将字符串作为嵌套JSON存储在列中到JSON文件-Pyspark

Question

我有一个pyspark数据框，看起来像这样

+------------------------------------+-------------------+-------------+--------------------------------+---------+
|member_uuid                         |Timestamp          |updated      |member_id                       |easy_id  |
+------------------------------------+-------------------+-------------+--------------------------------+---------+
|027130fe-584d-4d8e-9fb0-b87c984a0c20|2020-02-11 19:15:32|password_hash|ajuypjtnlzmk4na047cgav27jma6_STG|993269700|

我将上面的数据框转换为此，

 +---------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
|attribute|operation|params                                                                                                                                           |timestamp          |
+---------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
|profile  |UPDATE   |{"member_uuid":"027130fe-584d-4d8e-9fb0-b87c984a0c20","member_id":"ajuypjtnlzmk4na047cgav27jma6_STG","easy_id":993269700,"field":"password_hash"}|2020-02-11 19:15:32|

使用以下代码，

ll = ['member_uuid', 'member_id', 'easy_id', 'field']
df = df.withColumn('timestamp', col('Timestamp')).withColumn('attribute', lit('profile')).withColumn('operation', lit(col_name)) \
                    .withColumn('field', col('updated')).withColumn('params', F.to_json(struct([x for x in ll])))
    df = df.select('attribute', 'operation', 'params', 'timestamp')

我已将此数据框df转换为JSON后将其保存到文本文件中。我尝试使用以下代码执行相同的操作，

df_final.toJSON().coalesce(1).saveAsTextFile('file')

该文件包含，

{"attribute":"profile","operation":"UPDATE","params":"{\"member_uuid\":\"027130fe-584d-4d8e-9fb0-b87c984a0c20\",\"member_id\":\"ajuypjtnlzmk4na047cgav27jma6_STG\",\"easy_id\":993269700,\"field\":\"password_hash\"}","timestamp":"2020-02-11T19:15:32.000Z"}

我希望它以这种格式保存，

{"attribute":"profile","operation":"UPDATE","params":{"member_uuid":"027130fe-584d-4d8e-9fb0-b87c984a0c20","member_id":"ajuypjtnlzmk4na047cgav27jma6_STG","easy_id":993269700,"field":"password_hash"},"timestamp":"2020-02-11T19:15:32.000Z"}

to_json将params列中的值保存为字符串，有没有一种方法可以将json上下文保存在此处，以便将其保存为所需的输出？

Answer 1

一种简单的处理方法是对文件进行替换操作

sourceData=open('file').read().replace('"{','{').replace('}"','}').replace('\\','')
with open('file','w') as final:
    final.write(sourceData)

这可能不是您想要的，但会达到最终结果。