如何在Scala中创建和匹配模式

Question

嗨，我有一个如下所示的模式

|-- eventObject: struct (nullable = true)
|    |-- baseDivisionCode: string (nullable = true)
|    |-- countryCode: string (nullable = true)
|    |-- dcNumber: long (nullable = true)
|    |-- financialReportingGroup: string (nullable = true)
|    |-- itemList: array (nullable = true)
|    |    |-- element: struct (containsNull = true)
|    |    |    |-- availabletosellQty: long (nullable = true)
|    |    |    |-- distroAvailableQty: long (nullable = true)
|    |    |    |-- itemNumber: long (nullable = true)
|    |    |    |-- itemUPC: string (nullable = true)
|    |    |    |-- ossIndicator: string (nullable = true)
|    |    |    |-- turnAvailableQty: long (nullable = true)
|    |    |    |-- unitOfMeasurement: string (nullable = true)
|    |    |    |-- weightFormatType: string (nullable = true)
|    |    |    |-- whpkRatio: long (nullable = true)

要对此进行映射，我已经创建了以下架构类型

|-- eventObject: struct (nullable = true)
|    |-- baseDivisionCode: string (nullable = true)
|    |-- countryCode: string (nullable = true)
|    |-- dcNumber: integer (nullable = true)
|    |-- financialReportingGroup: string (nullable = true)
|    |-- itemList: struct (nullable = true)
|    |    |-- availabletosellQty: long (nullable = true)
|    |    |-- distroAvailableQty: long (nullable = true)
|    |    |-- itemNumber: long (nullable = true)
|    |    |-- itemUPC: string (nullable = true)
|    |    |-- ossIndicator: string (nullable = true)
|    |    |-- turnAvailableQty: long (nullable = true)
|    |    |-- unitOfMeasurement: string (nullable = true)
|    |    |-- weightFormatType: string (nullable = true)
|    |    |-- whpkRatio: long (nullable = true)

通过写这样的东西

 val testSchema = new StructType()
  .add("eventObject", new StructType()
    .add("baseDivisionCode", StringType)
    .add("countryCode",StringType)
    .add("dcNumber", IntegerType)
    .add("financialReportingGroup",StringType)

    .add("itemList",new StructType(
      Array(
        StructField("availabletosellQty",LongType),
        StructField("distroAvailableQty",LongType),
        StructField("itemNumber", LongType),
        StructField("itemUPC", StringType),
        StructField("ossIndicator",StringType),
        StructField("turnAvailableQty",LongType),
        StructField("unitOfMeasurement",StringType),
        StructField("weightFormatType",StringType),
        StructField("whpkRatio",LongType)))))

但是它与我收到的架构不匹配...在这方面我做错了什么？

当我尝试用一​​些数据填充时，我得到空值...

Answer 1

使用ArrayType指定数组类型：

val testSchema = new StructType()
    .add("eventObject", new StructType()
    .add("baseDivisionCode", StringType)
    .add("countryCode", StringType)
    .add("dcNumber", LongType)
    .add("financialReportingGroup", StringType)
    .add("itemList", new ArrayType(
      new StructType(
        Array(
          StructField("itemNumber", LongType),
          StructField("itemUPC", StringType),
          StructField("unitOfMeasurement", StringType),
          StructField("availabletosellQty", LongType),
          StructField("turnAvailableQty", LongType),
          StructField("distroAvailableQty", LongType),
          StructField("ossIndicator", StringType),
          StructField("weightFormatType", StringType))), containsNull = true)))