使用 Avro 架构和 JSON 数据创建 Avro GenericRecord

Question

我正在尝试使用以下架构和 JSON 数据生成 Avro GenericRecord。

Avro 架构

{
  "type": "record",
  "name": "Person",
  "fields": [
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "age",
      "type": "int"
    },
    {
      "name": "city",
      "type": "string"
    },
    {
      "name": "gender",
      "type": {
        "type": "enum",
        "name": "Gender",
        "symbols": ["MALE", "FEMALE"]
      }
    }
  ]
}

JSON 数据

{"name": "John", "age": 30, "city": "New York", "gender": "MALE"}

我用来生成 GenericRecord 的代码是

public class Main {

    public static void main(String[] args) throws IOException {
        Schema schema = readSchema();
        JsonNode data = readData();
        GenericRecord genericRecord = convertJsonToAvro(data, schema);
        System.out.println(genericRecord);
    }

    public static GenericRecord convertJsonToAvro(JsonNode jsonNode, Schema avroSchema) throws IOException {
        DatumReader<GenericRecord> reader = new GenericDatumReader<>(avroSchema);
        Decoder decoder = DecoderFactory.get().jsonDecoder(avroSchema, jsonNode.toString());
        return reader.read(null, decoder);
    }

    private static Schema readSchema() throws IOException {
        InputStream inputStream = Main.class.getClassLoader().getResourceAsStream("schemas/person.avsc");
        return new Schema.Parser().parse(inputStream);

    }

    private static JsonNode readData() throws IOException {
        InputStream inputStream = Main.class.getClassLoader().getResourceAsStream("sample_data/person.json");
        ObjectMapper objectMapper = new ObjectMapper();
        return objectMapper.readValue(inputStream, JsonNode.class);
    }

}

从上面的代码中，我可以成功生成一个GenericRecord。但是，当模式枚举字段更改为如下所示的可选字段时，它会抛出错误。

可选字段

{
    "name": "gender",
    "type": [
        "null",
        {
            "type": "enum",
            "name": "Gender",
            "symbols": ["MALE", "FEMALE"]
        }
    ],
    "default": null
}

错误

Exception in thread "main" org.apache.avro.AvroTypeException: Expected start-union. Got VALUE_STRING
at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:511)
at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:430)
at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:282)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:188)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:161)
at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:260)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:248)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:180)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:161)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:154)
at com.generic.Main.convertJsonToAvro(Main.java:27)
at com.generic.Main.main(Main.java:20)

FAILURE: Build failed with an exception.

如果有人可以解释这里发生的事情以及修复此错误的方法，我非常感激。我知道 allegro/json-avro-converter 但我想在没有它的情况下完成此操作。代码示例会非常有帮助。

请找到我在上述代码中使用的依赖项

implementation group: 'org.apache.avro', name: 'avro', version: '1.11.3'
implementation group: 'com.fasterxml.jackson.core', name: 'jackson-core', version: '2.16.0'

Answer 1

已更新

gender

中的枚举类型是

Gender

，当您将该属性视为对象时，有效负载应将其包含为子属性。

这是一个非空

gender

有效负载：

{"name": "John", "age": 30, "city": "New York", "gender": {"Gender": "MALE"}}

那么这是你的

GenericRecord

：

{"name": "John", "age": 30, "city": "New York", "gender": "MALE"}

这是一个空

gender

有效负载

{"name": "John", "age": 30, "city": "New York", "gender": null}