Google Pub/Sub Big Query 订阅架构 Mismach

Question

我已成功为 Pub/Sub 主题创建 BigQuery 订阅。

使用的订阅模式选项：

• 使用表模式
• 写入元数据
• 删除未知字段

主题没有架构。我认为这不是必需的，但我可能是错的。

现在我正在向该主题发布消息，但是订阅显示错误“由于资源错误，订阅无法接收消息。”

订阅状态为：

table schema mismatch

。

我无法在日志中找到有关特定不匹配的任何内容。

[
    {
        "name": "subscription_name",
        "type": "STRING",
        "mode": "REQUIRED",
        "description": "Pub/Sub subscription name"
    },
    {
        "name": "message_id",
        "type": "STRING",
        "mode": "REQUIRED",
        "description": "Pub/Sub message identifier"
    },
    {
        "name": "publish_time",
        "type": "TIMESTAMP",
        "mode": "REQUIRED",
        "description": "Pub/Sub publishing time"
    },
    {
        "name": "attributes",
        "type": "JSON",
        "mode": "REQUIRED",
        "description": "Pub/Sub message attributes"
    },
    {
        "name": "id",
        "type": "STRING",
        "mode": "REQUIRED",
        "description": "Message identifier"
    },
    {
        "name": "vendor_id",
        "type": "STRING",
        "mode": "REQUIRED",
        "description": "Message vendor identifier"
    },
    {
        "name": "account",
        "type": "string",
        "mode": "REQUIRED",
        "description": "EmailEngine account identifier"
    },
    {
        "name": "path",
        "type": "STRING",
        "mode": "REQUIRED",
        "description": "Mailbox path"
    },
    {
        "name": "created_on",
        "type": "STRING",
        "mode": "REQUIRED",
        "description": "Creation date"
    },
    {
        "name": "from",
        "type": "STRING",
        "mode": "REQUIRED",
        "description": "Message sender"
    },
    {
        "name": "subject",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "Message subject"
    },
    {
        "name": "body",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "Message body"
    },
    {
        "name": "body_type",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "Message body type (plain, html or markdown)"
    },
    {
        "name": "to",
        "type": "STRING",
        "mode": "NULLABLE"
    },
    {
        "name": "cc",
        "type": "STRING",
        "mode": "NULLABLE"
    },
    {
        "name": "bcc",
        "type": "STRING",
        "mode": "NULLABLE"
    },
    {
        "name": "reply_to",
        "type": "STRING",
        "mode": "NULLABLE"
    },
    {
        "name": "in_reply_to",
        "type": "STRING",
        "mode": "NULLABLE"
    },
    {
        "name": "is_auto_reply",
        "type": "BOOLEAN",
        "mode": "NULLABLE"
    },
    {
        "name": "labels",
        "type": "STRING",
        "mode": "NULLABLE"
    },
    {
        "name": "flags",
        "type": "STRING",
        "mode": "NULLABLE"
    },
    {
        "name": "sender_role",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "Classified sender role"
    },
    {
        "name": "message_category",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "Classified message category"
    },
    {
        "name": "duration",
        "type": "INTEGER",
        "mode": "NULLABLE",
        "description": "Classification duration"
    },
    {
        "name": "prompt_tokens",
        "type": "INTEGER",
        "mode": "NULLABLE"
    },
    {
        "name": "completion_tokens",
        "type": "INTEGER",
        "mode": "NULLABLE"
    },
    {
        "name": "total_tokens",
        "type": "INTEGER",
        "mode": "NULLABLE",
        "description": "Total tokens used for the interaction with LLM"
    },
    {
        "name": "payload",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "Classification payload from LLM"
    }
]

发布订阅消息：

{
  "id": "<id.com>",
  "vendor_id": "AAAAAQAAAyY",
  "account": "naxj4rkbqadgqppi",
  "path": "INBOX",
  "created_on": "2024-02-08T15:15:53+00:00",
  "from": "John Doe <[email protected]>",
  "subject": "Subject",
  "body": "Message body",
  "body_type": "md",
  "to": "Mark Doe <[email protected]>",
  "cc": null,
  "bcc": null,
  "reply_to": null,
  "in_reply_to": null,
  "is_auto_reply": false,
  "labels": "",
  "flags": "",
  "sender_role": "internal",
  "message_category": null,
  "duration": 12,
  "prompt_tokens": 1637,
  "completion_tokens": 7,
  "total_tokens": 1644,
  "payload": "{"json": "payload"}"
}

问题是架构出了什么问题或者如何查找错误详细信息？

Answer 1

目前存在 use_table_schema 的

问题

，即使是一条错误消息也可能使订阅永久进入“架构不匹配”状态。这不是预期行为，修复程序将于 2024 年 2 月 16 日推出。