存在这样的情况似乎在DASK集群挂在重新启动时
模拟这个我写了这个愚蠢的代码:
import contextlib2
from distributed import Client, LocalCluster
for i in xrange(100):
print i
with contextlib2.ExitStack() as es:
cluster = LocalCluster(processes=True, n_workers=4)
client = Client(cluster)
es.callback(client.close)
es.callback(es.callback(client.close))
此代码永远不会完成环路我得到这个错误
raise_exc_info(self._exc_info)
File "//anaconda/lib/python2.7/site-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "//anaconda/lib/python2.7/site-packages/distributed/deploy/local.py", line 191, in _start
yield [self._start_worker(**self.worker_kwargs) for i in range(n_workers)]
File "//anaconda/lib/python2.7/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "//anaconda/lib/python2.7/site-packages/tornado/concurrent.py", line 269, in result
raise_exc_info(self._exc_info)
File "//anaconda/lib/python2.7/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "//anaconda/lib/python2.7/site-packages/tornado/concurrent.py", line 269, in result
raise_exc_info(self._exc_info)
File "//anaconda/lib/python2.7/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "//anaconda/lib/python2.7/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
即时通讯使用DASK分布式1.25.1和Python 2.7的Mac上运行
这是在一个DASK问题,而使用python 2.7 Linux上的唯一的方法来启动新的工作(多进程)是通过使用叉
叉反过来可能会导致死锁详见门票开放DASK https://github.com/dask/distributed/issues/2446