支持python分布式计算框架Ray的示例分析

tangjin

作者

这篇文章将为大家详细讲解有关支持python分布式计算框架Ray的示例分析，小编觉得挺实用的，因此分享给大家做个参考，希望大家阅读完这篇文章后可以有所收获。

1、简介

Ray为构建分布式应用程序提供了一个简单、通用的API。Ray是一种分布式执行框架,便于大规模应用程序和利用先进的机器学习库。

Ray通过以下方式完成这项任务：

为构建和运行分布式应用程序提供简单的原语。

使最终用户能够并行化单个机器代码，而代码更改很少到零。

在核心Ray之上包含大量应用程序、库和工具，以支持复杂的应用程序。

2、安装

安装方式比较简单： pip install ray==1.4.1

[root@node2~]#pipinstall'ray[default]'
Lookinginindexes:https://mirrors.aliyun.com/pypi/simple/
Collectingray[default]
Downloadinghttps://mirrors.aliyun.com/pypi/packages/13/ec/f727ddd3fbcdc6102eace62c9d5dd9d9ad8112d40eeb7de8783676aca24d/ray-1.4.1-cp36-cp36m-manylinux2014_x86_64.whl(51.6MB)
|████████████████████████████████|51.6MB269kB/s
Collectingaiohttp
Downloadinghttps://mirrors.aliyun.com/pypi/packages/31/28/7b49246d7825c61e1a14306ab050e8eec48fcf5b009b19c8f6ea4e312467/aiohttp-3.7.4.post0-cp36-cp36m-manylinux2014_x86_64.whl(1.3MB)
|████████████████████████████████|1.3MB121.5MB/s
Requirementalreadysatisfied:dataclassesin/usr/local/lib/python3.6/site-packages(fromray[default])(0.8)
Collectingprotobuf>=3.15.3
Downloadinghttps://mirrors.aliyun.com/pypi/packages/53/4e/e2db88d0bb0bda6a879eea62fddbaf813719ce3770d458bc5580512d9c95/protobuf-3.17.3-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl(1.0MB)
|████████████████████████████████|1.0MB136.1MB/s
Collectingjsonschema
Downloadinghttps://mirrors.aliyun.com/pypi/packages/c5/8f/51e89ce52a085483359217bc72cdbf6e75ee595d5b1d4b5ade40c7e018b8/jsonschema-3.2.0-py2.py3-none-any.whl(56kB)
|████████████████████████████████|56kB7.1MB/s
Collectingaioredis
Downloadinghttps://mirrors.aliyun.com/pypi/packages/b0/64/1b1612d0a104f21f80eb4c6e1b6075f2e6aba8e228f46f229cfd3fdac859/aioredis-1.3.1-py3-none-any.whl(65kB)
|████████████████████████████████|65kB6.3MB/s
Collectingpydantic>=1.8
Downloadinghttps://mirrors.aliyun.com/pypi/packages/2b/7c/7d0b3f2d7959b7193018896db236ded165f9bca1bb75f46f4c32fa6f4f9d/pydantic-1.8.2-cp36-cp36m-manylinux2014_x86_64.whl(10.2MB)
|████████████████████████████████|10.2MB420kB/s
Collectingpyyaml
Downloadinghttps://mirrors.aliyun.com/pypi/packages/7a/5b/bc0b5ab38247bba158504a410112b6c03f153c652734ece1849749e5f518/PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl(640kB)
|████████████████████████████████|640kB19.1MB/s
Collectingnumpy>=1.16
Downloadinghttps://mirrors.aliyun.com/pypi/packages/14/32/d3fa649ad7ec0b82737b92fefd3c4dd376b0bb23730715124569f38f3a08/numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl(14.8MB)
|████████████████████████████████|14.8MB296kB/s
Collectinggpustat
Downloadinghttps://mirrors.aliyun.com/pypi/packages/b4/69/d8c849715171aeabd61af7da080fdc60948b5a396d2422f1f4672e43d008/gpustat-0.6.0.tar.gz(78kB)
|████████████████████████████████|78kB7.3MB/s
Collectingredis>=3.5.0
Downloadinghttps://mirrors.aliyun.com/pypi/packages/a7/7c/24fb0511df653cf1a5d938d8f5d19802a88cef255706fdda242ff97e91b7/redis-3.5.3-py2.py3-none-any.whl(72kB)
|████████████████████████████████|72kB968kB/s
Collectingaiohttp-cors
Downloadinghttps://mirrors.aliyun.com/pypi/packages/13/e7/e436a0c0eb5127d8b491a9b83ecd2391c6ff7dcd5548dfaec2080a2340fd/aiohttp_cors-0.7.0-py3-none-any.whl(27kB)
Collectingpy-spy>=0.2.0
Downloadinghttps://mirrors.aliyun.com/pypi/packages/9d/4d/1a9cbe9a0b543e6733cb38afe26451522a9ef8e4897b59e74cc76838f245/py_spy-0.3.7-py2.py3-none-manylinux1_x86_64.whl(3.1MB)
|████████████████████████████████|3.1MB131kB/s
Requirementalreadysatisfied:requestsin/usr/local/lib/python3.6/site-packages(fromray[default])(2.24.0)
Collectingcolorama
Downloadinghttps://mirrors.aliyun.com/pypi/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl(16kB)
Collectingfilelock
Downloadinghttps://mirrors.aliyun.com/pypi/packages/93/83/71a2ee6158bb9f39a90c0dea1637f81d5eef866e188e1971a1b1ab01a35a/filelock-3.0.12-py3-none-any.whl(7.6kB)
Collectingprometheus-client>=0.7.1
Downloadinghttps://mirrors.aliyun.com/pypi/packages/09/da/4e8471ff825769581593b5b84769d32f58e5373b59fccaf355d3529ad530/prometheus_client-0.11.0-py2.py3-none-any.whl(56kB)
|████████████████████████████████|56kB4.2MB/s
Collectingmsgpack<2.0.0,>=1.0.0
Downloadinghttps://mirrors.aliyun.com/pypi/packages/0c/0d/b1d9d32d03ce38ba5e2a37fbae850afd4530a14cc441e8335f1865a03705/msgpack-1.0.2-cp36-cp36m-manylinux1_x86_64.whl(272kB)
|████████████████████████████████|272kB127.6MB/s
Requirementalreadysatisfied:click>=7.0in/usr/local/lib/python3.6/site-packages(fromray[default])(7.1.2)
Collectingopencensus
Downloadinghttps://mirrors.aliyun.com/pypi/packages/18/59/12044123133d000f705383ad98579aeb0dd82d66b33a254a21b54bf0d6bb/opencensus-0.7.13-py2.py3-none-any.whl(127kB)
|████████████████████████████████|127kB128.9MB/s
Collectinggrpcio>=1.28.1
Downloadinghttps://mirrors.aliyun.com/pypi/packages/eb/32/332a2bb0cee0d4040331952c5ef3d040d119a15b3c622b60bdcbe768bded/grpcio-1.39.0-cp36-cp36m-manylinux2014_x86_64.whl(4.3MB)
|████████████████████████████████|4.3MB417kB/s
Collectingcolorful
Downloadinghttps://mirrors.aliyun.com/pypi/packages/b0/8e/e386e248266952d24d73ed734c2f5513f34d9557032618c8910e605dfaf6/colorful-0.5.4-py2.py3-none-any.whl(201kB)
|████████████████████████████████|201kB113.9MB/s
Requirementalreadysatisfied:six>=1.5.2in/usr/local/lib/python3.6/site-packages(fromgrpcio>=1.28.1->ray[default])(1.15.0)
Requirementalreadysatisfied:typing-extensions>=3.7.4.3in/usr/local/lib/python3.6/site-packages(frompydantic>=1.8->ray[default])(3.10.0.0)
Collectingmultidict<7.0,>=4.5
Downloadinghttps://mirrors.aliyun.com/pypi/packages/a1/35/b22524d6b9cacfb4c5eff413a069bbc17c6ea628e54da5c6c989998ced5f/multidict-5.1.0-cp36-cp36m-manylinux2014_x86_64.whl(141kB)
|████████████████████████████████|141kB120.6MB/s
Collectingidna-ssl>=1.0
Downloadinghttps://mirrors.aliyun.com/pypi/packages/46/03/07c4894aae38b0de52b52586b24bf189bb83e4ddabfe2e2c8f2419eec6f4/idna-ssl-1.1.0.tar.gz(3.4kB)
Collectingattrs>=17.3.0
Downloadinghttps://mirrors.aliyun.com/pypi/packages/20/a9/ba6f1cd1a1517ff022b35acd6a7e4246371dfab08b8e42b829b6d07913cc/attrs-21.2.0-py2.py3-none-any.whl(53kB)
|████████████████████████████████|53kB3.9MB/s
Requirementalreadysatisfied:chardet<5.0,>=2.0in/usr/local/lib/python3.6/site-packages(fromaiohttp->ray[default])(3.0.4)
Collectingyarl<2.0,>=1.0
Downloadinghttps://mirrors.aliyun.com/pypi/packages/da/08/52b26b44bce7b818b410aee37c5e424c9ea420c557bca97dc2adac29b151/yarl-1.6.3-cp36-cp36m-manylinux2014_x86_64.whl(293kB)
|████████████████████████████████|293kB130.8MB/s
Collectingasync-timeout<4.0,>=3.0
Downloadinghttps://mirrors.aliyun.com/pypi/packages/e1/1e/5a4441be21b0726c4464f3f23c8b19628372f606755a9d2e46c187e65ec4/async_timeout-3.0.1-py3-none-any.whl(8.2kB)
Requirementalreadysatisfied:idna>=2.0in/usr/local/lib/python3.6/site-packages(fromidna-ssl>=1.0->aiohttp->ray[default])(2.10)
Collectinghiredis
Downloadinghttps://mirrors.aliyun.com/pypi/packages/82/72/313fb6d30b7e413c8662447f9a0abe9105494ceb9266ecb08d442c14afbb/hiredis-2.0.0-cp36-cp36m-manylinux2010_x86_64.whl(84kB)
|████████████████████████████████|84kB4.8MB/s
Collectingnvidia-ml-py3>=7.352.0
Downloadinghttps://mirrors.aliyun.com/pypi/packages/6d/64/cce82bddb80c0b0f5c703bbdafa94bfb69a1c5ad7a79cff00b482468f0d3/nvidia-ml-py3-7.352.0.tar.gz(19kB)
Collectingpsutil
Downloadinghttps://mirrors.aliyun.com/pypi/packages/da/82/56cd16a4c5f53e3e5dd7b2c30d5c803e124f218ebb644ca9c30bc907eadd/psutil-5.8.0-cp36-cp36m-manylinux2010_x86_64.whl(291kB)
|████████████████████████████████|291kB131.6MB/s
Collectingblessings>=1.6
Downloadinghttps://mirrors.aliyun.com/pypi/packages/03/74/489f85a78247609c6b4f13733cbf3ba0d864b11aa565617b645d6fdf2a4a/blessings-1.7-py3-none-any.whl(18kB)
Requirementalreadysatisfied:setuptoolsin/usr/lib/python3.6/site-packages(fromjsonschema->ray[default])(39.2.0)
Collectingpyrsistent>=0.14.0
Downloadinghttps://mirrors.aliyun.com/pypi/packages/6c/19/1af501f6f388a40ede6d0185ba481bdb18ffc99deab0dd0d092b173bc0f4/pyrsistent-0.18.0-cp36-cp36m-manylinux1_x86_64.whl(117kB)
|████████████████████████████████|117kB138.2MB/s
Requirementalreadysatisfied:importlib-metadatain/usr/local/lib/python3.6/site-packages(fromjsonschema->ray[default])(4.3.0)
Requirementalreadysatisfied:zipp>=0.5in/usr/local/lib/python3.6/site-packages(fromimportlib-metadata->jsonschema->ray[default])(3.4.1)
Collectingopencensus-context==0.1.2
Downloadinghttps://mirrors.aliyun.com/pypi/packages/f1/33/990f1bd9e7ee770fc8d3c154fc24743a96f16a0e49e14e1b7540cc2fdd93/opencensus_context-0.1.2-py2.py3-none-any.whl(4.4kB)
Collectinggoogle-api-core<2.0.0,>=1.0.0
Downloadinghttps://mirrors.aliyun.com/pypi/packages/aa/51/629a31d4e7db2bbc0b0b789ac4034108c68647bea9cf5b54c8366df47b8e/google_api_core-1.31.0-py2.py3-none-any.whl(93kB)
|████████████████████████████████|93kB1.6MB/s
Collectingcontextvars
Downloadinghttps://mirrors.aliyun.com/pypi/packages/83/96/55b82d9f13763be9d672622e1b8106c85acb83edd7cc2fa5bc67cd9877e9/contextvars-2.4.tar.gz(9.6kB)
Collectinggoogle-auth<2.0dev,>=1.25.0
Downloadinghttps://mirrors.aliyun.com/pypi/packages/d7/13/35b1e0a63e160ecad04985e60339382e48e6020d22f8328a2ab47226c910/google_auth-1.33.1-py2.py3-none-any.whl(152kB)
|████████████████████████████████|152kB127.7MB/s
Collectingpackaging>=14.3
Downloadinghttps://mirrors.aliyun.com/pypi/packages/3c/77/e2362b676dc5008d81be423070dd9577fa03be5da2ba1105811900fda546/packaging-21.0-py3-none-any.whl(40kB)
|████████████████████████████████|40kB6.1MB/s
Collectinggoogleapis-common-protos<2.0dev,>=1.6.0
Downloadinghttps://mirrors.aliyun.com/pypi/packages/55/08/796a6bc0b550e2b7116041c953d3d5100016abea106131d71e5651826e7b/googleapis_common_protos-1.53.0-py2.py3-none-any.whl(198kB)
|████████████████████████████████|198kB136.3MB/s
Collectingsetuptools
Downloadinghttps://mirrors.aliyun.com/pypi/packages/bd/25/5bdf7f1adeebd4e3fa76b2e2f045ae53ee208e40a4231ad0f0c3007e4353/setuptools-57.4.0-py3-none-any.whl(819kB)
|████████████████████████████████|819kB127.0MB/s
Requirementalreadysatisfied:pytzin/usr/local/lib/python3.6/site-packages(fromgoogle-api-core<2.0.0,>=1.0.0->opencensus->ray[default])(2020.1)
Collectingcachetools<5.0,>=2.0.0
Downloadinghttps://mirrors.aliyun.com/pypi/packages/bf/28/c4f5796c67ad06bb91d98d543a5e01805c1ff065e08871f78e52d2a331ad/cachetools-4.2.2-py3-none-any.whl(11kB)
Collectingpyasn1-modules>=0.2.1
Downloadinghttps://mirrors.aliyun.com/pypi/packages/95/de/214830a981892a3e286c3794f41ae67a4495df1108c3da8a9f62159b9a9d/pyasn1_modules-0.2.8-py2.py3-none-any.whl(155kB)
|████████████████████████████████|155kB21.5MB/s
Collectingrsa<5,>=3.1.4
Downloadinghttps://mirrors.aliyun.com/pypi/packages/e9/93/0c0f002031f18b53af7a6166103c02b9c0667be528944137cc954ec921b3/rsa-4.7.2-py3-none-any.whl(34kB)
Collectingpyparsing>=2.0.2
Downloadinghttps://mirrors.aliyun.com/pypi/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl(67kB)
|████████████████████████████████|67kB7.2MB/s
Collectingpyasn1<0.5.0,>=0.4.6
Downloadinghttps://mirrors.aliyun.com/pypi/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl(77kB)
|████████████████████████████████|77kB7.0MB/s
Requirementalreadysatisfied:certifi>=2017.4.17in/usr/local/lib/python3.6/site-packages(fromrequests->ray[default])(2020.6.20)
Requirementalreadysatisfied:urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1in/usr/local/lib/python3.6/site-packages(fromrequests->ray[default])(1.25.10)
Collectingimmutables>=0.9
Downloadinghttps://mirrors.aliyun.com/pypi/packages/4a/52/e64a14a99c509cbdfe0405e9f076aef0331cb9548a3efa1d5bacd524978a/immutables-0.15-cp36-cp36m-manylinux1_x86_64.whl(100kB)
|████████████████████████████████|100kB18.2MB/s
Usinglegacy'setup.pyinstall'foridna-ssl,sincepackage'wheel'isnotinstalled.
Usinglegacy'setup.pyinstall'forgpustat,sincepackage'wheel'isnotinstalled.
Usinglegacy'setup.pyinstall'fornvidia-ml-py3,sincepackage'wheel'isnotinstalled.
Usinglegacy'setup.pyinstall'forcontextvars,sincepackage'wheel'isnotinstalled.
Installingcollectedpackages:pyasn1,setuptools,rsa,pyparsing,pyasn1-modules,protobuf,multidict,immutables,cachetools,yarl,packaging,idna-ssl,googleapis-common-protos,google-auth,contextvars,attrs,async-timeout,pyrsistent,psutil,opencensus-context,nvidia-ml-py3,hiredis,google-api-core,blessings,aiohttp,redis,pyyaml,pydantic,py-spy,prometheus-client,opencensus,numpy,msgpack,jsonschema,grpcio,gpustat,filelock,colorama,aioredis,aiohttp-cors,ray,colorful
Attemptinguninstall:setuptools
Foundexistinginstallation:setuptools39.2.0
Uninstallingsetuptools-39.2.0:
Successfullyuninstalledsetuptools-39.2.0
Runningsetup.pyinstallforidna-ssl...done
Runningsetup.pyinstallforcontextvars...done
Runningsetup.pyinstallfornvidia-ml-py3...done
Runningsetup.pyinstallforgpustat...done
Successfullyinstalledaiohttp-3.7.4.post0aiohttp-cors-0.7.0aioredis-1.3.1async-timeout-3.0.1attrs-21.2.0blessings-1.7cachetools-4.2.2colorama-0.4.4colorful-0.5.4contextvars-2.4filelock-3.0.12google-api-core-1.31.0google-auth-1.33.1googleapis-common-protos-1.53.0gpustat-0.6.0grpcio-1.39.0hiredis-2.0.0idna-ssl-1.1.0immutables-0.15jsonschema-3.2.0msgpack-1.0.2multidict-5.1.0numpy-1.19.5nvidia-ml-py3-7.352.0opencensus-0.7.13opencensus-context-0.1.2packaging-21.0prometheus-client-0.11.0protobuf-3.17.3psutil-5.8.0py-spy-0.3.7pyasn1-0.4.8pyasn1-modules-0.2.8pydantic-1.8.2pyparsing-2.4.7pyrsistent-0.18.0pyyaml-5.4.1ray-1.4.1redis-3.5.3rsa-4.7.2setuptools-57.4.0yarl-1.6.3

3、单机

下面是不使用分布式的代码示例：

#test_ray.py
importray
importcv2
importnumpy

importtime

#@ray.remote
deftest(img):
orb=cv2.AKAZE_create()
kb=orb.detect(img,None)
kp,des=orb.compute(img,kp)
returndes

#不使用注解的方式
test_remote=ray.remote(test)

#函数test加注解时使用方式
#test_remote=test.remote(test)

img=cv2.imread("/tmp/test.jpg")

start_time=time.time()

foriinrange(10):
test(img)

print(time.time()-start_time)

运行：python test_ray.py

4、集群

如果需要启动集群模式，则需要先启动服务，选择一台机器作为主服务器，然后按虾米那命令启动：

[root@node1~]#raystart--head--port=8888
LocalnodeIP:192.168.0.81
2021-07-2809:58:44,051INFOservices.py:1274--ViewtheRaydashboardathttp://127.0.0.1:8265

--------------------
Rayruntimestarted.
--------------------

Nextsteps
ToconnecttothisRayruntimefromanothernode,run
raystart--address='192.168.0.81:8888'--redis-password='5241590000000000'

Alternatively,usethefollowingPythoncode:
importray
ray.init(address='auto',_redis_password='5241590000000000')

Ifconnectionfails,checkyourfirewallsettingsandnetworkconfiguration.

ToterminatetheRayruntime,run
raystop

可以看到里面提示具体的用法。此时通过ray的web界面，端口8265访问，看到已有一台机器。

然后在另外两台机启动客户端节点，保证各个节点的ray版本一致,查看版本：

# pip freeze |grep ray

版本不一致的话，要升级为一致：pip install -U ray，此处选择的版本为1.4.1

从节点启动方式(密码可以在主节点启动时显示)：

raystart--address='192.168.0.81:8888'--redis-password='5241590000000000'

改写程序，用于分布式：

#test_ray.py
importray
importcv2
importnumpy

importtime

@ray.remote
deftest(img):
orb=cv2.AKAZE_create()
kb=orb.detect(img,None)
kp,des=orb.compute(img,kp)
returndes

#不使用注解的方式
#test_remote=ray.remote(test)

#函数test加注解时使用方式
test_remote=test.remote(test)

if__name__=="__main__":
ray.init(address='auto',_redis_password='5241590000000000')
start_time=time.time()
img=cv2.imread("/tmp/test.jpg")
futures=[test_remote.remote(img)foriinrange(100)]
ray.get(futures)
start_time=time.time()
print(time.time()-start_time)

在主节点运行程序，程序会在3台机器上启动。停止ray服务的方式：ray stop.

关于“支持python分布式计算框架Ray的示例分析”这篇文章就分享到这里了，希望以上内容可以对大家有一定的帮助，使各位可以学到更多知识，如果觉得文章不错，请把它分享出去让更多的人看到。

阅读全文

发布于 2021-07-29 21:58:22

Python