在sql server 2019中设置cdc并注册debezium sql服务器连接器后运行kafka使用者时无法生成任何日志

问题描述 投票:0回答:2

[以前,我已经基于此link在SQL Server中设置CDC,并基于此link将Debezium SQL Server连接器注册到Kafka connect。我运行Kafka使用者脚本并在SQL Server中进行一些修改数据(插入,更新和删除),但是Kafka使用者不显示有关修改数据的任何JSON日志。

这是我的配置文件sqlserver-debezium.json

{
   "name":"sqlserver-debezium",
   "config":{
      "connector.class":"io.debezium.connector.sqlserver.SqlServerConnector",
      "database.user":"sa",
      "database.dbname":"cdc_sqlserver",
      "database.history.kafka.bootstrap.servers":"10.140.0.33:9092",
      "database.history.kafka.topic":"debezium_sqlserver",
      "database.server.name":"sqlserver-debezium",
      "database.port":"1433",
      "table.whitelist":"cdc_sqlserver_ipul.dbo.customers",
      "database.hostname":"10.140.15.214",
      "database.password":"test123",
      "name":"sqlserver-debezium",
      "include.query":"true",
      "snapshot.mode":"when_needed"
   }
}

我通过主机添加连接器,如下所示(与指南一样):

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d @sqlserver-debezium.json

运行脚本Kafka使用者:

kafka-console-consumer --bootstrap-server localhost:9092 --topic sqlserver-connector --from-beginning

将不胜感激。

sql-server apache-kafka apache-kafka-connect debezium
2个回答
0
投票

您能否再次检查table.whitelist设置?只能是dbo.customers


0
投票

此问题已解决。所以我将变量snapshot.mode的值更改为initial,删除sqlserver-debezium.json中的variabel include.query,重试注册连接器并在sql server数据库中进行一些修改(插入,更新,删除),然后显示json日志,如下所示

{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":true,"field":"first_name"
},{"type":"string","optional":true,"field":"last_name"},{"type":"string","optional":true,"field":"email"}],"optional":true,"name":"btpns_sqlserver_ipul.dbo.customer
s.Value","field":"before"},{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":true,"field":"first_name"},{"type":
"string","optional":true,"field":"last_name"},{"type":"string","optional":true,"field":"email"}],"optional":true,"name":"btpns_sqlserver_ipul.dbo.customers.Value","
field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"stri
ng","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"
parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":false,"f
ield":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"string","optional":true,"field":"change_lsn"},{"type":"string","optional":true,"field":"
commit_lsn"},{"type":"int64","optional":true,"field":"event_serial_no"}],"optional":false,"name":"io.debezium.connector.sqlserver.Source","field":"source"},{"type":
"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},
{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"
optional":false,"name":"btpns_sqlserver_ipul.dbo.customers.Envelope"},"payload":{"before":{"id":9,"first_name":"Andi","last_name":"Thompson","email":"[email protected]"
},"after":{"id":9,"first_name":"Deni","last_name":"Thompson","email":"[email protected]"},"source":{"version":"1.2.0-SNAPSHOT","connector":"sqlserver","name":"btpns-sql
server-ipul","ts_ms":1588740858363,"snapshot":"false","db":"btpns_cdc_sqlserver_ipul","schema":"dbo","table":"customers","change_lsn":"0000002d:00003da0:0002","comm
it_lsn":"0000002d:00003da0:0003","event_serial_no":2},"op":"u","ts_ms":1588740859225,"transaction":null}}
{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":true,"field":"first_name"
},{"type":"string","optional":true,"field":"last_name"},{"type":"string","optional":true,"field":"email"}],"optional":true,"name":"btpns_sqlserver_ipul.dbo.customer
s.Value","field":"before"},{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":true,"field":"first_name"},{"type":
"string","optional":true,"field":"last_name"},{"type":"string","optional":true,"field":"email"}],"optional":true,"name":"btpns_sqlserver_ipul.dbo.customers.Value","
field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"stri
ng","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"
parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":false,"f
ield":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"string","optional":true,"field":"change_lsn"},{"type":"string","optional":true,"field":"
commit_lsn"},{"type":"int64","optional":true,"field":"event_serial_no"}],"optional":false,"name":"io.debezium.connector.sqlserver.Source","field":"source"},{"type":
"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},
{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"
optional":false,"name":"btpns_sqlserver_ipul.dbo.customers.Envelope"},"payload":{"before":{"id":9,"first_name":"Deni","last_name":"Thompson","email":"[email protected]"
},"after":null,"source":{"version":"1.2.0-SNAPSHOT","connector":"sqlserver","name":"btpns-sqlserver-ipul","ts_ms":1588744428530,"snapshot":"false","db":"btpns_cdc_s
qlserver_ipul","schema":"dbo","table":"customers","change_lsn":"0000002d:00005400:0002","commit_lsn":"0000002d:00005400:0005","event_serial_no":1},"op":"d","ts_ms":
1588744430725,"transaction":null}}
© www.soinside.com 2019 - 2024. All rights reserved.