如何在 PySpark 中为嵌套 JSON 列创建架构？

Question

我有一个包含多列的镶木地板文件，其中有 2 列是 JSON/Struct，但它们的类型是字符串。可以存在任意数量的 array_elements。

{
  "addressline": [

    {
      "array_element": "F748DK’8U1P9’2ZLKXE"
    },
    {
      "array_element": "’O’P0BQ04M-"
    },
    {
      "array_element": "’fvrvrWEM-"
    }

  ],
  "telephone": [
    {
      "array_element": {
        "locationtype": "8.PLT",
        "countrycode": null,
        "phonenumber": "000000000",
        "phonetechtype": "1.PTT",
        "countryaccesscode": null,
        "phoneremark": null
      }
    }
  ]
}

如何创建一个模式来处理 PySpark 中的这些列？

Answer 1

将您提供的示例视为字符串，我创建了此数据框：

from pyspark.sql import functions as F, types as T
df = spark.createDataFrame([('{"addressline":[{"array_element":"F748DK’8U1P9’2ZLKXE"},{"array_element":"’O’P0BQ04M-"},{"array_element":"’fvrvrWEM-"}],"telephone":[{"array_element":{"locationtype":"8.PLT","countrycode":null,"phonenumber":"000000000","phonetechtype":"1.PTT","countryaccesscode":null,"phoneremark":null}}]}',)], ['c1'])

这是应用于此列的架构：

schema = T.StructType([
    T.StructField('addressline', T.ArrayType(T.StructType([
        T.StructField('array_element', T.StringType())
    ]))),
    T.StructField('telephone', T.ArrayType(T.StructType([
        T.StructField('array_element', T.StructType([
            T.StructField('locationtype', T.StringType()),
            T.StructField('countrycode', T.StringType()),
            T.StructField('phonenumber', T.StringType()),
            T.StructField('phonetechtype', T.StringType()),
            T.StructField('countryaccesscode', T.StringType()),
            T.StructField('phoneremark', T.StringType()),
        ]))
    ])))
])

为

from_json

函数提供架构的结果：

df = df.withColumn('c1', F.from_json('c1', schema))

df.show()
# +-------------------------------------------------------------------------------------------------------+
# |c1                                                                                                     |
# +-------------------------------------------------------------------------------------------------------+
# |{[{F748DK’8U1P9’2ZLKXE}, {’O’P0BQ04M-}, {’fvrvrWEM-}], [{{8.PLT, null, 000000000, 1.PTT, null, null}}]}|
# +-------------------------------------------------------------------------------------------------------+

df.printSchema()
# root
#  |-- c1: struct (nullable = true)
#  |    |-- addressline: array (nullable = true)
#  |    |    |-- element: struct (containsNull = true)
#  |    |    |    |-- array_element: string (nullable = true)
#  |    |-- telephone: array (nullable = true)
#  |    |    |-- element: struct (containsNull = true)
#  |    |    |    |-- array_element: struct (nullable = true)
#  |    |    |    |    |-- locationtype: string (nullable = true)
#  |    |    |    |    |-- countrycode: string (nullable = true)
#  |    |    |    |    |-- phonenumber: string (nullable = true)
#  |    |    |    |    |-- phonetechtype: string (nullable = true)
#  |    |    |    |    |-- countryaccesscode: string (nullable = true)
#  |    |    |    |    |-- phoneremark: string (nullable = true)