Kafka Mysql Connector插件。路径配置

Question

我正在尝试将mysql与kafka连接。我已经下载了debezium-debezium-connector-mysql

这是我的connect-standalone.properties：

# These are defaults. This file just demonstrates how to override some settings.
bootstrap.servers=localhost:9092

# The converters specify the format of data in Kafka and how to translate it into Connect data. Every Connect user will
# need to configure these based on the format they want their data in when loaded from or stored into Kafka
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
# Converter-specific settings can be passed in by prefixing the Converter's setting with the converter we want to apply
# it to
key.converter.schemas.enable=true
value.converter.schemas.enable=true

offset.storage.file.filename=/tmp/connect.offsets
# Flush much faster than normal, which is useful for testing/debugging
offset.flush.interval.ms=10000

# Set to a list of filesystem paths separated by commas (,) to enable class loading isolation for plugins
# (connectors, converters, transformations). The list should consist of top level directories that include 
# any combination of: 
# a) directories immediately containing jars with plugins and their dependencies
# b) uber-jars with plugins and their dependencies
# c) directories immediately containing the package directory structure of classes of plugins and their dependencies
# Note: symlinks will be followed to discover dependencies or plugins.
# Examples: 
# plugin.path=/usr/local/share/java,/usr/local/share/kafka/plugins,/opt/connectors,
plugin.path=/home/ec2-user/share/confluent-hub-components/debezium-debezium-connector-mysql/lib

我的test.config：

{
  "name": "mysql-source-demo-customers",
  "config": {
      "connector.class": "io.debezium.connector.mysql.MySqlConnector",
      "database.hostname": "localhost",
      "database.port": "3306",
      "database.user": "root",
      "database.password": "dsm1234",
      "database.server.id": "42",
      "database.server.name": "asgard",
      "table.whitelist": "demo.customers",
      "database.history.kafka.bootstrap.servers": "localhost:9092",
      "database.history.kafka.topic": "dbhistory.demo" ,
      "include.schema.changes": "true",
      "transforms": "unwrap,InsertTopic,InsertSourceDetails",
      "transforms.unwrap.type": "io.debezium.transforms.UnwrapFromEnvelope",
      "transforms.InsertTopic.type":"org.apache.kafka.connect.transforms.InsertField$Value",
      "transforms.InsertTopic.topic.field":"messagetopic",
      "transforms.InsertSourceDetails.type":"org.apache.kafka.connect.transforms.InsertField$Value",
      "transforms.InsertSourceDetails.static.field":"messagesource",
      "transforms.InsertSourceDetails.static.value":"Debezium CDC from MySQL on asgard"
  }
}

当我运行bin/connect-standalone.sh config/connect-standalone.properties test.config时，

我收到一个错误：

[[2019-11-19 09：10：45,146] INFO扫描插件类。这可能需要一点时间...（org.apache.kafka.connect.cli.ConnectStandalone：78）[2019-11-19 09：10：45,170] INFO正在从以下位置加载插件：/ home / ec2-user / share / confluent-hub-components / debezium-debezium-connector-mysql / lib / mysql-binlog-connector-java- 0.19.1.jar（org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader：222）[2019-11-19 09：10：45,291]信息注册的加载器：PluginClassLoader {pluginLocation = file：/ home / ec2-user / share / confluent-hub-components / debezium-debezium-connector-mysql / lib / mysql-binlog -connector-java-0.19.1.jar}（org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader：245）[2019-11-19 09：10：45,292]信息添加了插件'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy'（org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader：174）[2019-11-19 09：10：45,292]信息添加了插件'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy'（org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader：174）[2019-11-19 09：10：45,292]信息添加了插件'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy'（org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader：174）[2019-11-19 09：10：45,293] INFO正在从以下位置加载插件：/home/ec2-user/share/confluent-hub-components/debezium-debezium-connector-mysql/lib/debezium-connector-mysql-0.10。 0.Final.jar（org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader：222）[2019-11-19 09：10：45,324]错误由于错误而停止（org.apache.kafka.connect.cli.ConnectStandalone：130）java.lang.NoClassDefFoundError：io / debezium / util / IoUtil在io.debezium.connector.mysql.Module。（Module.java:19）在io.debezium.connector.mysql.MySqlConnector.version（MySqlConnector.java:47）在org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor（DelegatingClassLoader.java:350）在org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor（DelegatingClassLoader.java:355）在org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc（DelegatingClassLoader.java:331）在org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath（DelegatingClassLoader.java:311）在org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins（DelegatingClassLoader.java:244）处在org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin（DelegatingClassLoader.java:236）在org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader（DelegatingClassLoader.java:205）处在org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders（DelegatingClassLoader.java:182）在org.apache.kafka.connect.runtime.isolation.Plugins（Plugins.java:61）在org.apache.kafka.connect.cli.ConnectStandalone.main（ConnectStandalone.java:79）引起原因：java.lang.ClassNotFoundException：io.debezium.util.IoUtil在java.net.URLClassLoader.findClass（URLClassLoader.java:382）在java.lang.ClassLoader.loadClass（ClassLoader.java:418）在org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass（PluginClassLoader.java:104）在java.lang.ClassLoader.loadClass（ClassLoader.java:351）...还有12个]

[通过查看日志，io/debezium/util/IoUtil似乎是一个问题。（我用Google搜索和stackoverflowed了关于插件路径的信息。我在Stackoverflow中找到了一个类似的问题，并遵循了它，但仍然无法正常工作。）

我想念的是什么？

Answer 1

如果使用Confluent Hub客户端安装连接器，则无需编辑插件路径。它是递归扫描的，因此仅/home/ec2-user/share/confluent-hub-components应该可以工作。

旁注：我建议将插件存储在ec2-user主文件夹之外的其他位置，因为这样长时间运行的进程通常以其自己的受限用户帐户运行