如何配置以SQS为后端的celery?

问题描述 投票:0回答:1

我正在尝试使用 celery 应用程序设置 SQS 代理,并在 django 项目中配置。 这是我的设置:

celery.py:

import os

from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "core.settings")

app = Celery("my-app")

app.config_from_object("django.conf:settings", namespace="CELERY")

app.autodiscover_tasks()

我的芹菜相关的django设置:

CELERY_BROKER_TRANSPORT_OPTIONS = {
    "region": "eu-west-1",
    "queue_name_prefix": f"celery-",
}
CELERY_BROKER_URL = "sqs://"
CELERY_TASK_DEFAULT_QUEUE = "default"

这个配置使用rabbitmq代理工作得很好,这让我认为整体配置是正确的,但是当我使用SQS作为代理时,消息被发送到SQS(“消息可用”计数器增加),但是一旦被我的接收到工作人员,他们会转到“飞行中的消息”并似乎永远呆在那里(至少几个小时)。 另外,我可以看到工作人员记录正在执行的操作,但由于某种原因从未实际执行该任务。

以下是一些芹菜工人日志:

 host;x-amz-date
 786cb490d758593ebf5a6e0c0b34cf025b3309bb0d777891344fd32bd01cb61b
 [2023-11-19 19:52:05,766: DEBUG/MainProcess] StringToSign:
 AWS4-HMAC-SHA256
 20231119T195205Z
 20231119/eu-west-1/sqs/aws4_request
 32a2b642b24ffcd3f42216826611c27799efb1cf756b26e7119d5553009f527c
 [2023-11-19 19:52:05,766: DEBUG/MainProcess] Signature:
 93bcf47703548603fa0787fbbd2ae8aa4b54460db91695f4f6c818b9141b620e
 [2023-11-19 19:52:10,959: DEBUG/MainProcess] Response headers: {}
 [2023-11-19 19:52:10,960: DEBUG/MainProcess] Response body:
 b'<?xml version="1.0"?><ReceiveMessageResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ReceiveMessageResult><Message><MessageId>c4e1b404-.....-c046dc4aeb60</MessageId><ReceiptHandle>AQEBy+4G0x9BF8su10zFWJQuyEXJOFSxF........cRPUdk/vjWzirMiBw97ZSG44M=</ReceiptHandle><MD5OfBody>483202aa2938b...da1b024b63</MD5OfBody><Body>eyJib2R5IjogIlcxdGRMQ0I3...CJyZXBseV90byI6ICJmOWZjY2FmDIyMTBkYTAyMiJ9fQ==</Body><Attribute><Name>ApproximateReceiveCount</Name><Value>1</Value></Attribute></Message></ReceiveMessageResult><ResponseMetadata><RequestId>28a76707-9a01...d3f3a42ad7</RequestId></ResponseMetadata></ReceiveMessageResponse>'
 [2023-11-19 19:52:10,960: DEBUG/MainProcess] Event choose-signer.sqs.ReceiveMessage: calling handler <function set_operation_specific_signer at 0xffff9b901750>
 [2023-11-19 19:52:10,960: DEBUG/MainProcess] Calculating signature using v4 auth.
 [2023-11-19 19:52:10,960: DEBUG/MainProcess] CanonicalRequest:
 POST
 /ACCOUNT_ID/celery-default


 host:sqs.eu-west-1.amazonaws.com
 x-amz-date:20231119T195210Z


 host;x-amz-date
 786cb490d758593ebf5a6e...d777891344fd32bd01cb61b
 [2023-11-19 19:52:10,961: DEBUG/MainProcess] StringToSign:
 AWS4-HMAC-SHA256
 20231119T195210Z
 20231119/eu-west-1/sqs/aws4_request
 a6b5c2bb4cb9eef89...4e16bc164d0a0d9b1a1
 [2023-11-19 19:52:10,961: DEBUG/MainProcess] Signature:
 81bd2621f3ce88b5a76...79dc18951c3bc0c8a
 [2023-11-19 19:52:21,000: DEBUG/MainProcess] Response headers: {}
 [2023-11-19 19:52:21,000: DEBUG/MainProcess] Response body:
 b'<?xml version="1.0"?><ReceiveMessageResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ReceiveMessageResult/><ResponseMetadata><RequestId>802bb8a8-...-e57c9a538959</RequestId></ResponseMetadata></ReceiveMessageResponse>'
 [2023-11-19 19:52:21,000: DEBUG/MainProcess] Event choose-signer.sqs.ReceiveMessage: calling handler <function set_operation_specific_signer at 0xffff9b901750>
 [2023-11-19 19:52:21,001: DEBUG/MainProcess] Calculating signature using v4 auth.
 [2023-11-19 19:52:21,001: DEBUG/MainProcess] CanonicalRequest:
 POST
 /ACCOUNT_ID/celery-default

我不认为这是权限问题,因为我可以使用执行角色在 AWS (fargate) 上重现该错误,也可以在本地使用管理员用户重现该错误,这两种情况下的结果相同。

请问您知道出了什么问题吗?

  • 尝试检查工人日志(如上所示)
  • 尝试检查AWS cloudtrail,但找不到任何有用的东西
  • 尝试切换到rabbitMQ代理,这解决了问题。
  • 在本地和AWS上尝试,以消除堆栈问题和权限问题
django amazon-web-services celery amazon-sqs django-celery
1个回答
0
投票

最后,更新芹菜(和昆布)解决了这个问题!

来源:github.com/celery/celery/pull/8646

© www.soinside.com 2019 - 2024. All rights reserved.