Kafka spring boot应用程序生产商无法用Kafka Sink Connector和Avro格式反映出来

Question

我的目标是我有春天启动应用程序kafka生产者与Avro序列化属性，我除了推送到相应的主题，应该通过汇合的Sink连接器访问并插入到mysql / Oracle数据库表中的消息，我能够生成Avro序列化和弹簧启动消费者可以Avro反序列化，但我的Sink连接器不工作，我无法捕捉到什么样的有效载荷接收器连接器除了弹簧启动器生成器编码如何推送消息这样的方式，接收器连接器可以与那个性能

提前致谢 ：）

这是Spring启动应用程序中的application.yml

server: 
    port: 9000
    spring.kafka:
    bootstrap-servers: "localhost:9092"
    properties:
      schema.registry.url: "http://localhost:8081"
      specific.avro.reader: true
    producer:
      key-serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"
      value-serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"
    app:
      topic: event_pay2

这是从spring boot应用程序创建模式的有效负载

{
  "schema": {
    "type": "struct",
    "fields": [
      {
        "type": "string",
        "optional": false,
        "field": "userid"
      },
      {
        "type": "string",
        "optional": false,
        "field": "regionid"
      },
      {
        "type": "string",
        "optional": false,
        "field": "gender"
      }
    ],
    "optional": false,
    "name": "oracle.user"
  },
  "payload": {
    "userid": "User_1",
    "regionid": "Region_5",
    "gender": "FEMALE"
  }
}

Pom.hml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.0.5.RELEASE</version>
    <relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.kafka</groupId>
<artifactId>kafka-producer-example</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>kafka-producer-example</name>
<description>Demo project for Spring Boot</description>

 <repositories>
    <repository>
        <id>confluent</id>
        <url>http://packages.confluent.io/maven</url>
    </repository>
</repositories>

<properties>
    <java.version>1.8</java.version>
    <confluent.version>4.0.0</confluent.version>
</properties>

<dependencies>
<dependency>
        <groupId>com.google.code.gson</groupId>
        <artifactId>gson</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-devtools</artifactId>
        <scope>runtime</scope>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka-test</artifactId>
        <scope>test</scope>
    </dependency>
     <dependency>
        <groupId>io.confluent</groupId>
        <artifactId>kafka-avro-serializer</artifactId>
        <version>${confluent.version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro</artifactId>
  <version>1.8.2</version>
</dependency>
</dependencies>

<build>
 <pluginManagement>
    <plugins>
    <plugin>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro-maven-plugin</artifactId>
        <version>1.8.2</version>
        <executions>
            <execution>
                <phase>generate-sources</phase>
                <goals>
                    <goal>schema</goal>
                </goals>
                <configuration>
                    <sourceDirectory>${project.basedir}/src/main/resources/avro/</sourceDirectory>
                    <outputDirectory>${project.build.directory}/generated/avro</outputDirectory>
                </configuration>
            </execution>
        </executions>
    </plugin>


        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
    </plugins>
     </pluginManagement>
</build>

</project>

这是我的休息电话如何将消息推入kafka主题

@PostMapping("/publish/avrodata")
public String sendMessage(@RequestBody String request) {
    sender.send(request);
    return "Published successfully";
}

最后我的接收器

"name": "JDBC_Sink_EVENT_PAY",
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": "1",
"topics": "event_pay2",
"connection.url": "jdbc:mysql://localhost:3306/user",
"connection.user": "****",
"connection.password": "****",
"auto.create": "true",
"auto.evolve":"true",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://localhost:8081",
"key.converter.schemas.enable": "true",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://localhost:8081",
 "value.converter.schemas.enable": "true"

Answer 1

始终，始终在设置连接器之前调试您的主题。使用kafka-avro-console-consumer。如果这不起作用，那么Connect + AvroConverter可能也不会（根据我的经验），你可以减少问题空间。

---

如果我正确读取了您的代码，您发送了一个String，而不是Avro对象

public String sendMessage(@RequestBody String request) {
    sender.send(request);  // <--- here and ^^^^^^ here
    return "Published successfully";
}

相反，您需要将输入请求解析为作为/src/main/resources/avro架构数据的一部分创建的对象，而不是仅仅将传入请求作为字符串转发。

那个AVSC文件可能看起来像

{
  "type": "Record",
  "namespace": "oracle.user",
  "name": "User",
  "fields": [
      { "type": "string", "name: "user_id" },
      { "type": "string", "name": "regionid" },
      { "type" "string", "name" "gender" }
   ]
}

这将创建一个oracle.user.User对象，使你的KafkaTemplate需要像KafkaTemplate<String, oracle.user.User> sender