在生产中多次重复 Celery 任务

Question

我是 Celery 新手，我创建了一个简单的应用程序，它连接到 Web 套接字服务器以接收任务并使用 Celery 安排它们。我的 Celery 队列根据消息类型显示任务（首先是文本消息，然后是单击其中一个任务时运行下一个任务的按钮）。 在本地，一切都按预期运行。但是，有些任务在生产中会重复多次，尤其是触发器（按钮）。在我的生产环境中，我为 Django 创建了一个 Web 服务，并创建了一个带有 Redis 数据库的 Celery 后台工作人员。

以下是我在生产中用来运行 Celery Worker 和 Beat 的命令：

# Start command (celery worker and beat)
celery -A bot.celery worker --beat --scheduler django --loglevel=info --concurrency 4
# Start command (Django)
daphne -b 0.0.0.0 bot.asgi:application

我的 Django 和 Celery 设置：

CELERY_BROKER_URL = "redis://localhost:6379/0" # localhost replaced with the internal Redis URL in production
CELERY_RESULT_BACKEND = "redis://localhost:6379/1" # localhost replaced with the internal Redis URL in production

TIME_ZONE = 'UTC'

CELERY_ENABLE_UTC = True
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'
ASGI_APPLICATION = 'bot.asgi.application'

部分工人输出：

Nov 1 09:00:15 AM  [2022-11-01 08:00:15,178: INFO/MainProcess] missed heartbeat from celery@srv-cdcl4202i3msb94icl70-5469f7b9d8-gzks7
Nov 1 09:05:33 AM  [2022-11-01 08:05:33,207: INFO/MainProcess] Task schedule_chat_message[574703ea-51d8-4acf-b9e9-202c37d0cbe5] received
Nov 1 09:05:33 AM  [2022-11-01 08:05:33,211: INFO/MainProcess] Task schedule_chat_message[ef2b2dd0-f297-4480-9577-da83a44bc244] received
Nov 1 09:05:33 AM  [2022-11-01 08:05:33,223: INFO/MainProcess] Task schedule_chat_message[a3619feb-9768-4e6b-b8be-966c335eae5f] received
Nov 1 09:05:34 AM  [2022-11-01 08:05:34,182: INFO/ForkPoolWorker-2] Task schedule_chat_message[a3619feb-9768-4e6b-b8be-966c335eae5f] succeeded in 0.9560789109673351s: None
Nov 1 09:05:34 AM  [2022-11-01 08:05:34,272: INFO/ForkPoolWorker-3] Task schedule_chat_message[574703ea-51d8-4acf-b9e9-202c37d0cbe5] succeeded in 1.0607709939940833s: None
Nov 1 09:05:34 AM  [2022-11-01 08:05:34,280: INFO/ForkPoolWorker-4] Task schedule_chat_message[ef2b2dd0-f297-4480-9577-da83a44bc244] succeeded in 1.0533532970002852s: None
Nov 1 09:05:34 AM  [2022-11-01 08:05:34,936: INFO/Beat] DatabaseScheduler: Schedule changed.
Nov 1 09:05:35 AM  [2022-11-01 08:05:35,274: INFO/Beat] Scheduler: Sending due task Scheduled to send to chat_1ea56ac at 2022-11-01 08:05:33.211860+00:00 (send_out_chat_message)
Nov 1 09:05:35 AM  [2022-11-01 08:05:35,289: INFO/MainProcess] Task send_out_chat_message[37854a6c-d20a-45b1-99ba-37b22d818af4] received
Nov 1 09:05:35 AM  [2022-11-01 08:05:35,562: INFO/ForkPoolWorker-3] Task send_out_chat_message[37854a6c-d20a-45b1-99ba-37b22d818af4] succeeded in 0.27148923301137984s: None
Nov 1 09:05:38 AM  [2022-11-01 08:05:38,090: INFO/Beat] Scheduler: Sending due task Scheduled to send to chat_1ea56ac at 2022-11-01 08:05:33.227478+00:00 (send_out_chat_message)
Nov 1 09:05:38 AM  [2022-11-01 08:05:38,096: INFO/MainProcess] Task send_out_chat_message[80ca730b-0ff4-43df-b3ec-2758b0b493df] received
Nov 1 09:05:38 AM  [2022-11-01 08:05:38,276: INFO/ForkPoolWorker-3] Task send_out_chat_message[80ca730b-0ff4-43df-b3ec-2758b0b493df] succeeded in 0.1784440579940565s: None
Nov 1 09:05:44 AM  [2022-11-01 08:05:44,359: INFO/Beat] Scheduler: Sending due task Scheduled to send to chat_1ea56ac at 2022-11-01 08:05:33.226634+00:00 (send_out_chat_message)
Nov 1 09:05:44 AM  [2022-11-01 08:05:44,365: INFO/MainProcess] Task send_out_chat_message[a1bcff89-78af-41b6-8e81-95aa5d3172d3] received
Nov 1 09:05:44 AM  [2022-11-01 08:05:44,561: INFO/ForkPoolWorker-3] Task send_out_chat_message[a1bcff89-78af-41b6-8e81-95aa5d3172d3] succeeded in 0.19544574100291356s: None
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,089: INFO/MainProcess] Task schedule_chat_message[a83a6683-1750-422f-8494-7226152249b4] received
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,099: INFO/MainProcess] Task schedule_chat_message[d983a9d0-2243-4a87-9844-5f2e986f7626] received
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,103: INFO/MainProcess] Task schedule_chat_message[f30d05ec-257f-4516-bfcb-5880b85d3e85] received
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,109: INFO/MainProcess] Task schedule_chat_message[f8e1a7a3-21a9-41f3-a8d9-9ceae4f0c10f] received
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,864: INFO/ForkPoolWorker-3] Task schedule_chat_message[a83a6683-1750-422f-8494-7226152249b4] succeeded in 0.773912389006s: None
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,869: INFO/ForkPoolWorker-4] Task schedule_chat_message[d983a9d0-2243-4a87-9844-5f2e986f7626] succeeded in 0.7689848589943722s: None
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,968: INFO/ForkPoolWorker-2] Task schedule_chat_message[f30d05ec-257f-4516-bfcb-5880b85d3e85] succeeded in 0.8630019359989092s: None

非常感谢您的帮助！

Answer 1

由于您使用 Redis 作为代理，因此它似乎与此 celery issue 有关。看起来使用 SQS 的开发人员可能会遇到类似的问题。

Answer 2

我遇到了同样的问题，我们设法解决了问题......

问题是我们在 django 中安排任务 (m/h/dm/my/d) * 13 * * 1-5。

我们这样做是为了在一周的每天下午 13 点运行任务，但在 django 中我们需要指定分钟！

将日程编辑为(m/h/dm/my/d) 00 13 * * 1-5后，问题解决了！

Answer 3

虽然这个话题已经很老了，但我想为那些仍然遇到同样问题的人，或者像我这样在多年的旧系统中突然遇到这个错误的人，对 txicos 的答案进行一些补充。

首先，如果您使用

ps -Af | grep celerybeat

检查当前的 celery 节拍过程，您可能会发现实际上有不止一个 celery 节拍过程在工作。所以任务倍增是很正常的，毕竟有多个节拍和工人在玩。

在 txicos 提供的 github 源代码中，它认为此错误可能与时区问题有关（UTC 为 false），或者可以通过使用visibility_timeout 变量来解决（celery docs）。可悲的是，这对我不起作用，唯一的选择就是更换经纪人。

您可以按照 rabbitmq celery docs 进行安装，这非常简单，并为我消除了错误。虽然我宁愿修复 Redis 服务器发生的错误，但这是一个足够的解决方案。