我有一个 Pandas 数据框,其中一列包含 JSON 数据(JSON 结构很简单:只有一层,没有嵌套数据):
ID,Date,attributes
9001,2020-07-01T00:00:06Z,"{"State":"FL","Source":"Android","Request":"0.001"}"
9002,2020-07-01T00:00:33Z,"{"State":"NY","Source":"Android","Request":"0.001"}"
9003,2020-07-01T00:07:19Z,"{"State":"FL","Source":"ios","Request":"0.001"}"
9004,2020-07-01T00:11:30Z,"{"State":"NY","Source":"windows","Request":"0.001"}"
9005,2020-07-01T00:15:23Z,"{"State":"FL","Source":"ios","Request":"0.001"}"
我想规范化 attributes 列中的 JSON 内容,以便 JSON 属性成为数据帧中的每一列。
ID,Date,attributes.State, attributes.Source, attributes.Request
9001,2020-07-01T00:00:06Z,FL,Android,0.001
9002,2020-07-01T00:00:33Z,NY,Android,0.001
9003,2020-07-01T00:07:19Z,FL,ios,0.001
9004,2020-07-01T00:11:30Z,NY,windows,0.001
9005,2020-07-01T00:15:23Z,FL,ios,0.001
我一直在尝试使用Pandas json_normalize,它需要字典。所以,我想我会将 attributes 列转换为字典,但它并没有完全按照预期的方式工作,因为字典具有以下形式:
df.attributes.to_dict()
{0: '{"State":"FL","Source":"Android","Request":"0.001"}',
1: '{"State":"NY","Source":"Android","Request":"0.001"}',
2: '{"State":"FL","Source":"ios","Request":"0.001"}',
3: '{"State":"NY","Source":"windows","Request":"0.001"}',
4: '{"State":"FL","Source":"ios","Request":"0.001"}'}
标准化采用键 (0, 1, 2, ...) 作为列名,而不是 JSON 键。
我感觉我已经很接近了,但我不太清楚如何准确地做到这一点。欢迎任何想法。
谢谢!
Normalize 期望作用于对象,而不是字符串。
import json
import pandas as pd
df_final = pd.json_normalize(df.attributes.apply(json.loads))
您不需要先转换为字典。
尝试:
import pandas as pd
pd.json_normalize(df[‘attributes’])
我找到了一个解决方案,但我对此并不太满意。我觉得效率很低。
import pandas as pd
import json
# Import full dataframe
df = pd.read_csv(r'D:/tmp/sample_simple.csv', parse_dates=['Date'])
# Create empty dataframe to hold the results of data conversion
df_attributes = pd.DataFrame()
# Loop through the data to fill the dataframe
for index in df.index:
row_json = json.loads(df.attributes[index])
normalized_row = pd.json_normalize(row_json)
# df_attributes = df_attributes.append(normalized_row) (deprecated method) use concat instead
df_attributes = pd.concat([df_attributes, normalized_row], ignore_index=True)
# Reset the index of the attributes dataframe
df_attributes = df_attributes.reset_index(drop=True)
# Drop the original attributes column
df = df.drop(columns=['attributes'])
# Join the results
df_final = df.join(df_attributes)
# Show results
print(df_final)
print(df_final.info())
这给了我预期的结果。然而,正如我所说,它存在一些低效率的地方。对于初学者来说,数据帧附加在 for 循环 中。根据文档,最佳实践是制作一个列表,然后追加,但我不知道如何在保持我想要的形状的同时做到这一点。我欢迎所有批评者和想法。
一行即可实现想要的输出:
df = pd.concat([df[['ID', 'Date']], pd.json_normalize(df['attributes'])], axis=1)