GCP 日志记录在 Celery 任务中不起作用

Question

我尝试在 celery 任务中使用 GCP 日志记录，但它不会将日志发送到 GCP Logs Explorer，它仅在运行 celery 应用程序的控制台上输出日志。类似的问题已被问到here，但尚未得到答复。这是我的代码：

from check_attributes import list_attributes
from celery import Celery
import json


# GCP Logging
import google.cloud.logging
client = google.cloud.logging.Client()
client.setup_logging()
import logging


# list_attributes(logging.getLogger())
logging.info('OUTSIDE Task')

app = Celery('celery_tasks', backend='redis://localhost//0', broker='redis://localhost//1')

@app.task
def dummyFunc(firstArg, secondArg):
    
    
    logging.info('INSIDE Task!!!')
    # list_attributes(logging.getLogger())
    
    result = {
                'sum' : firstArg + secondArg,
                'id': 15 
    }

用于运行 celery Worker 的命令：

celery -A celery_tasks worker --concurrency=1 --loglevel=info

logging.info('外部任务')

将消息发送到GCP日志，但是当任务执行时，而不是向上推送

“内部任务”

留言，

“外部任务”

消息再次推送。

如果我打印任务内部和外部记录器的属性，这就是我得到的。

任务之外：

dict : {'filters': [], 'name': 'root', 'level': 20, 'parent': None, 'propagate': True, 'handlers': [], 'disabled' : 假，'_cache': {}} 文档：
根记录器与任何其他记录器没有什么不同，除了 它必须有一个日志级别，并且只有一个实例 层次结构。

模块：日志记录 weakref：无 _cache : {} 已禁用 : 错误过滤器 : [] 处理程序 : [] 级别 : 20 管理器 : 名称：根父级： 无传播：真根：

任务内部：

[2023-08-18 16:51:07,465：警告/ForkPoolWorker-1] 模块 [2023-08-18 16:51:07,466：警告/ForkPoolWorker-1] [2023-08-18 16:51:07,466：警告/ForkPoolWorker-1]：[2023-08-18 16：51：07,466： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,466: 警告/ForkPoolWorker-1]日志记录[2023-08-18 16:51:07,466： 警告/ForkPoolWorker-1] weakref [2023-08-18 16:51:07,467： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,467: 警告/ForkPoolWorker-1]：[2023-08-18 16:51:07,467： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,467: 警告/ForkPoolWorker-1]无[2023-08-18 16:51:07,467： 警告/ForkPoolWorker-1] _cache [2023-08-18 16:51:07,467： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,467: 警告/ForkPoolWorker-1]：[2023-08-18 16:51:07,467： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,467: 警告/ForkPoolWorker-1] {20：真} [2023-08-18 16:51:07,467： 警告/ForkPoolWorker-1]已禁用[2023-08-18 16:51:07,467： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,467: 警告/ForkPoolWorker-1]：[2023-08-18 16:51:07,467： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1]错误[2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1]过滤器[2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1]：[2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1] [] [2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1]处理程序[2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1]：[2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1] [] [2023-08-18 16:51:07,468：警告/ForkPoolWorker-1]级别[2023-08-18 16:51:07,468：警告/ForkPoolWorker-1] [2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1]：[2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1] 20 [2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1]经理[2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1]：[2023-08-18 16:51:07,468： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,469: 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,469：警告/ForkPoolWorker-1]名称[2023-08-18 16:51:07,469：警告/ForkPoolWorker-1] [2023-08-18 16:51:07,469： 警告/ForkPoolWorker-1]：[2023-08-18 16:51:07,470： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,470: 警告/ForkPoolWorker-1] root [2023-08-18 16:51:07,470: 警告/ForkPoolWorker-1]父级[2023-08-18 16:51:07,470： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,470: 警告/ForkPoolWorker-1]：[2023-08-18 16:51:07,471： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,471： 警告/ForkPoolWorker-1]无[2023-08-18 16:51:07,471： 警告/ForkPoolWorker-1]传播[2023-08-18 16:51:07,471： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,472： 警告/ForkPoolWorker-1]：[2023-08-18 16:51:07,472： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,472： 警告/ForkPoolWorker-1] 正确 [2023-08-18 16:51:07,472： 警告/ForkPoolWorker-1] root [2023-08-18 16:51:07,472： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,472： 警告/ForkPoolWorker-1]：[2023-08-18 16:51:07,472： 警告/ForkPoolWorker-1] [2023-08-18 16:51:07,475： 警告/ForkPoolWorker-1]

Answer 1

我在使用 Google Cloud Logging 时也发现了同样的问题。

要修复它，应使用基于

custom class

的 CloudLoggingHandler 并将传输类设置为

SyncTransport

from google.cloud.logging_v2.handlers.transports import SyncTransport


class CustomCloudLoggingHandler(CloudLoggingHandler):

    def __init__(
            self,
            client,
            *,
            name=DEFAULT_LOGGER_NAME,
            transport=SyncTransport,
            resource=None,
            labels=None,
            stream=None,
    ):
        super(CloudLoggingHandler, self).__init__(stream)
        if not resource:
            # infer the correct monitored resource from the local environment
            resource = detect_resource(client.project)
        self.name = name
        self.client = client
        self.transport = transport(client, name, resource=resource)
        self.project_id = client.project
        self.resource = resource
        self.labels = labels
        # add extra keys to log record
        log_filter = CloudLoggingFilter(project=self.project_id, default_labels=labels)
        self.addFilter(log_filter)

确保为 celery 设置正确加载的新记录器类