原因：com.github.shyiko.mysql.binlog.event.deserialization.MissingTableMapEventException：找不到表id：137的TableMapEventData

Question

我们有一个简单的 Spring Boot 3.1 应用程序，它包装了 Debezium。我们升级到最新版本2.4.0.Final。现在在服务运行几个小时后出现此错误：

2023-10-05 07:36:54 [pool-4-thread-1] INFO  io.debezium.embedded.EmbeddedEngine - Retriable exception thrown, connector will be restarted; errors.max.retries=-1
org.apache.kafka.connect.errors.RetriableException: An exception occurred in the change event producer. This connector will be restarted.
        at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:49)
        at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onEventDeserializationFailure(MySqlStreamingChangeEventSource.java:1299)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1101)
        at com.github.shyiko.mysql.binlog.BinaryLogClient._connect(BinaryLogClient.java:620)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:525)
        at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4{timestamp=1696426677000, eventType=EXT_DELETE_ROWS, serverId=73360770, headerLength=19, dataLength=8155, nextPosition=116155347, flags=0}
        at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1246)
        ... 6 common frames omitted
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1696426677000, eventType=EXT_DELETE_ROWS, serverId=73360770, headerLength=19, dataLength=8155, nextPosition=116155347, flags=0}
        at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:341)
        at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:244)
        at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:236)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1088)
        ... 4 common frames omitted
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.MissingTableMapEventException: No TableMapEventData has been found for table id:137. Usually that means that you have started reading binary log 'within the logical event group' (e.g. from WRITE_ROWS and not proceeding TABLE_MAP
        at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeRow(AbstractRowsEventDataDeserializer.java:109)
        at com.github.shyiko.mysql.binlog.event.deserialization.DeleteRowsEventDataDeserializer.deserializeRows(DeleteRowsEventDataDeserializer.java:64)
        at com.github.shyiko.mysql.binlog.event.deserialization.DeleteRowsEventDataDeserializer.deserialize(DeleteRowsEventDataDeserializer.java:56)
        at com.github.shyiko.mysql.binlog.event.deserialization.DeleteRowsEventDataDeserializer.deserialize(DeleteRowsEventDataDeserializer.java:32)
        at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:335)
        ... 7 common frames omitted

服务无法修复此错误。有什么想法吗？

Answer 1

这可能是因为连接器可能已重新启动并将 tablemapeventdata 提交到偏移主题，并且当连接器重新启动时，它将从 tablemapeventdata 数据已过去的偏移量开始。要修复此问题，如果您的架构支持至少一次传递语义，则使用较早的 binlog 文件（早于当前偏移量的 1 - 2 个文件）更新 kafka 偏移量主题并重新启动连接器。现在连接器可以感知表映射的上下文并处理事件。这对我们有用，没有数据丢失