扭曲的反应堆在芹菜内部无法正常运行

Question

bounty到期在7天内。该问题的答案有资格获得+ 50声望赏金。Prashant正在寻找来自信誉良好的来源的答案。

系统/依赖项详细信息：

CPU --> 4
requirements --> celery==4.3.0, twisted==19.7.0 , python3.7 

下面是我的芹菜装置

from threading import Thread
from celery import Celery
from twisted.internet import threads, reactor, defer
from twisted.web.error import Error
from celery import signals

app =   Celery('tasks', broker='pyamqp://guest@localhost//')

@signals.worker_process_init.connect
def configure_infrastructure(**kwargs):
    Thread(target=reactor.run, name="reactor.run", args=(False,)).start()
    print('started new thread')

@signals.worker_process_shutdown.connect()
def shutdown_reactor(**kwargs):
    """
    This is invoked when the individual workers shut down. It just stops the twisted reactor
    @param kwargs:
    @return:
    """
    reactor.callFromThread(reactor.stop)
    print('REACTOR SHUTDOWN')

def getPage(inp):
    print(inp)
    return inp

def inThread():
    print('inside inthread method')
    try:
        result = threads.blockingCallFromThread(
            reactor, getPage, "http://twistedmatrix.com/")
    except Exception as exc:
        print(exc)
    else:
        print(result)
    #reactor.callFromThread(reactor.stop)


@app.task
def add(x, y):
    print('inside add method')
    inThread()
#    reactor.run()
    return x + y

正在运行的芹菜工人，如下：

celery -A run worker --loglevel=info

芹菜开始时的日志：

(2_env) ubuntu@gpy:~/app/env/src$ celery -A run worker --loglevel=info

[tasks]
  . run.add

[2020-04-09 07:25:29,357: WARNING/Worker-1] started new thread
[2020-04-09 07:25:29,362: WARNING/Worker-4] started new thread
[2020-04-09 07:25:29,362: WARNING/Worker-3] started new thread
[2020-04-09 07:25:29,364: WARNING/Worker-2] started new thread
[2020-04-09 07:25:29,367: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//

如下调用方法：

>>> run.add.delay(1,2)
<AsyncResult: d41680fd-7cc1-4e75-81be-6496bad0cc16>
>>> 

有时我可以看到它工作正常。

[2020-04-09 07:27:17,998: INFO/MainProcess] Received task: run.add[00934769-48c4-48b8-852c-8b746bdd5e03]
[2020-04-09 07:27:17,999: WARNING/Worker-4] inside add method
[2020-04-09 07:27:17,999: WARNING/Worker-4] inside inthread method
[2020-04-09 07:27:18,000: WARNING/Worker-4] http://twistedmatrix.com/
[2020-04-09 07:27:18,000: WARNING/Worker-4] http://twistedmatrix.com/
[2020-04-09 07:27:18,000: INFO/MainProcess] Task run.add[00934769-48c4-48b8-852c-8b746bdd5e03] succeeded in 0.00144551398989s: 3

有时我可以看到它无法调用getPage方法并像下面的日志一样被挂起

[2020-04-09 07:27:22,318: INFO/MainProcess] Received task: run.add[d41680fd-7cc1-4e75-81be-6496bad0cc16]
[2020-04-09 07:27:22,319: WARNING/Worker-2] inside add method
[2020-04-09 07:27:22,319: WARNING/Worker-2] inside inthread method

在reactor.run中使用Thread是否有问题？

Answer 1

您似乎已经采取了一些谨慎措施，可以在one线程中运行Twisted反应堆。但是，您将无法在多个线程中运行它，我想这是与Celery一起使用时发生的事情。它具有实例状态和全局状态，如果在多个线程中运行它将被踩踏。

相反，请尝试使用crochet协调从任意多个其他线程到在单个非主线程中运行的反应堆的调用。