Databricks-使用PySpark从SQL列中爆炸JSON

Question

Databricks的新手。有一个我要从中创建数据框的SQL数据库表。列之一是JSON字符串。我需要将嵌套的JSON分解为多列。使用此post和此post可以使我到达现在的位置。

示例JSON：

{
  "Module": {
    "PCBA Serial Number": "G7456789",
    "Manufacturing Designator": "DISNEY",
    "Firmware Version": "0.0.0",
    "Hardware Revision": "46858",
    "Manufacturing Date": "10/17/2018 4:04:25 PM",
    "Test Result": "Fail",
    "Test Start Time": "10/22/2018 6:14:14 AM",
    "Test End Time": "10/22/2018 6:16:11 AM"
  }

到目前为止的代码：

    #define schema 
         schema = StructType(
          [
            StructField('Module',ArrayType(StructType(Seq
              StructField('PCBA Serial Number',StringType,True),
              StructField('Manufacturing Designator',StringType,True),
              StructField('Firmware Version',StringType,True),
              StructField('Hardware Revision',StringType,True),
              StructField('Test Result',StringType,True),
              StructField('Test Start Time',StringType,True),
              StructField('Test End Time',StringType,True))), True) ,True),
            StructField('Test Results',StringType(),True),
            StructField('HVM Code Errors',StringType(),True)
          ]

#use from_json to explode json by applying it to column
        df.withColumn("ActivityName", from_json("ActivityName", schema))\
            .select(col('ActivityName'))\
            .show()

错误：

    SyntaxError: invalid syntax
  File "<command-1632344621139040>", line 10
    StructField('PCBA Serial Number',StringType,True),
              ^
SyntaxError: invalid syntax

Answer 1

当您使用pyspark时，类型应为StringType()而不是StringType并删除Seq并替换为[]

schema = StructType([StructField('Module',ArrayType(StructType([
    StructField('PCBA Serial Number',StringType(),True),
    StructField('Manufacturing Designator',StringType(),True),
    StructField('Firmware Version',StringType(),True),
    StructField('Hardware Revision',StringType(),True),
    StructField('Test Result',StringType(),True),
    StructField('Test Start Time',StringType(),True),
    StructField('Test End Time',StringType(),True)])), True),
StructField('Test Results',StringType(),True),
StructField('HVM Code Errors',StringType(),True)])