在Python中取消/规范化JSON

Question

我正在尝试在这里取消国会数据的嵌套：https://theunitedstates.io/congress-legislators/legislators-historical.json

示例结构：

    {
    "id": {
      "bioguide": "B000226",
      "govtrack": 401222,
      "icpsr": 507,
      "wikipedia": "Richard Bassett (politician)",
      "wikidata": "Q518823",
      "google_entity_id": "kg:/m/02pz46"
    },
    "name": {
      "first": "Richard",
      "last": "Bassett"
    },
    "bio": {
      "birthday": "1745-04-02",
      "gender": "M"
    },
    "terms": [
      {
        "type": "sen",
        "start": "1789-03-04",
        "end": "1793-03-03",
        "state": "DE",
        "class": 2,
        "party": "Anti-Administration"
      }
    ]
  }

如果我只使用json_normalize(data)，则这些“术语”并不会嵌套。

[如果我尝试专门打乱这些术语，例如json_normalize(data, 'terms', 'name')，那么我包括的其他内容（此处为名称）将保持dict格式，并以{u'last': u'Bassett', u'first': u'Richard'}作为行条目。

完整的当前代码，如果您想运行它：

import json
import urllib
import pandas as pd
from pandas.io.json import json_normalize

# load data
url = "https://theunitedstates.io/congress-legislators/legislators-historical.json"
json_url = urllib.urlopen(url)
data = json.loads(json_url.read())

# parse
congress_names = json_normalize(data, record_path='terms',meta='name')

Answer 1

我认为以下代码应该有效。可能有更好的归一化方法，但我不知道。

import requests
import pandas as pd
import re
import json
from pandas.io.json import json_normalize

url = ' https://theunitedstates.io/congress-legislators/legislators-historical.json'
resp = requests.get(url)
raw_dict = json.loads(resp.text)

df = pd.DataFrame()
for i in range(len(raw_dict)):    
     df1 = json_normalize(raw_dict[i], record_path = ['terms'], meta = ['name'])
     df1 = pd.concat([df1, df1['name'].apply(pd.Series)], axis=1)
     df = pd.concat([df,df1], axis=0, ignore_index =True, sort=True)

Answer 2

[将terms指定为rec_path时，需要将其余列的路径列表指定为meta。使用列表理解来建立这样的列表，如下所示

from pandas.io import json

l_meta = [[k, k1]  for k in data[0] if k != 'terms' for k1 in data[0][k]]
congress_names = json.json_normalize(data, 'terms', l_meta, errors='ignore')

Out[1105]:
  type       start         end state  class                party  district  \
0  sen  1789-03-04  1793-03-03    DE    2.0  Anti-Administration       NaN
1  rep  1789-03-04  1791-03-03    VA    NaN                  NaN       9.0

  id.bioguide id.govtrack id.icpsr                    id.wikipedia  \
0     B000226      401222      507    Richard Bassett (politician)
1     B000546      401521      786  Theodorick Bland (congressman)

  id.wikidata id.google_entity_id  name.first name.last bio.birthday  \
0     Q518823        kg:/m/02pz46     Richard   Bassett   1745-04-02
1    Q1749152        kg:/m/033mf4  Theodorick     Bland   1742-03-21

  bio.gender
0          M
1          M

注：为此测试，我仅从data中选择前2个元素/对象。我还假设第一个元素（data [0]）具有所有列。

---

方法2：

[normalize整体data作为主要congress_names。在该切片之后，仅将列terms和normalize列到新的df1并重新加入

congress_names = json.json_normalize(data)
df1 = json.json_normalize(congress_names.terms.str[0])
congress_names = congress_names.join(df1).drop('terms', axis=1)

Out[1130]:
  id.bioguide  id.govtrack  id.icpsr                    id.wikipedia  \
0     B000226       401222       507    Richard Bassett (politician)
1     B000546       401521       786  Theodorick Bland (congressman)

  id.wikidata id.google_entity_id  name.first name.last bio.birthday  \
0     Q518823        kg:/m/02pz46     Richard   Bassett   1745-04-02
1    Q1749152        kg:/m/033mf4  Theodorick     Bland   1742-03-21

  bio.gender  id.house_history type       start         end state  class  \
0          M               NaN  sen  1789-03-04  1793-03-03    DE    2.0
1          M            9479.0  rep  1789-03-04  1791-03-03    VA    NaN

                 party  district
0  Anti-Administration       NaN
1                  NaN       9.0