如何使用Python从Json文件的嵌套列表中提取对象？

Question

我收到了来自Lobbyview的json回复。我试图将其放在数据框中以仅访问某些变量，但没有成功。我如何只能以可导出到.dta的格式访问某些变量（例如id和委员会）？这是我尝试过的代码。

import requests, json
query = {"naics": "424430"}
results = requests.post('https://www.lobbyview.org/public/api/reports',
data = json.dumps(query))
print(results.json())

import pandas as pd
b = pd.DataFrame(results.json())

_id = data["_id"]
committee = data["_source"]["specific_issues"][0]["bills_by_algo"][0]["committees"]

对json的观察看起来像这样：

"_score": 4.421936, 
"_type": "object", 
"_id": "5EZUMbQp3hGKH8Uq2Vxuke", 
"_source": 
    {
    "issue_codes": ["CPT"], 
    "received": 1214320148, 
    "client_name": "INTELLECTUAL PROPERTY OWNERS ASSOCIATION", 
    "amount": 240000, 
    "client": 
        {
        "legal_name": "INTELLECTUAL PROPERTY OWNERS ASSOCIATION", 
        "name": "INTELLECTUAL PROPERTY OWNERS ASSOCIATION", 
        "naics": null, 
        "gvkey": null, 
        "ticker": "Unlisted", 
        "id": null, 
        "bvdid": "US131283992L"}, 
    "specific_issues": [
        {
        "text": "H.R. 34, H.R. 1908, H.R. 2336, H.R. 3093  S. 522, S. 681, S. 1145, S. 1745", 
        "bills_by_algo": [
            {
            "titles": ["To amend title 35, United States Code, to provide for patent reform.", "Patent Reform Act of 2007", "Patent Reform Act of 2007", "Patent Reform Act of 2007"], 
            "top_terms": ["Commerce", "Administrative fees"], 
            "sponsor": 
                {
                "firstname": "Howard", 
                "district": 28, 
                "title": "rep", 
                "id": 400025
                }, 
            "committees": ["House Judiciary"], 
            "introduced": 1176868800, 
            "type": "HR", "id": "110_HR1908"}, 
            {
            "titles": ["To amend title 35, United States Code, relating to the funding of the United States Patent and Trademark Office."], 
            "top_terms": ["Commerce", "Administrative fees"], 
            "sponsor": 
                {
                "firstname": "Howard", 
                "district": 28, 
                "title": "rep", 
                "id": 400025
                }, 
            "committees": ["House Judiciary"], 
            "introduced": 1179288000, 
            "type": "HR", 
            "id": "110_HR2336"
        }],

        "gov_entities": ["U.S. House of Representatives", "Patent and Trademark Office (USPTO)", "U.S. Senate", "UNDETERMINED", "U.S. Trade Representative (USTR)"], 
        "lobbyists": ["Valente, Thomas Silvio", "Wamsley, Herbert C"], 
        "year": 2007, 
        "issue": "CPT", 
        "id": "S4nijtRn9Q5NACAmbqFjvZ"}], 
    "year": 2007, 
    "is_latest_amendment": true,
     "type": "MID-YEAR AMENDMENT", 
    "id": "1466CDCD-BA3D-41CE-B7A1-F9566573611A", 
    "alternate_name": "INTELLECTUAL PROPERTY OWNERS ASSOCIATION"
    }, 
"_index": "collapsed"}```

Answer 1

由于您指定的数据非常深地嵌套在JSON响应中，因此您必须遍历该数据并将其临时保存到列表中。为了更好地理解响应数据，我建议您使用某种工具来查看JSON结构，例如online JSON-Viewer。并非JSON中的每个条目都包含必要的数据，因此我尝试通过try和except捕获错误。为了确保id和committees正确匹配，我选择将它们作为小词典添加到列表中。然后可以轻松地将此列表读入熊猫。保存为.dta要求您将committees列中的列表转换为字符串，相反，您可能还想另存为.csv，以获得更通用的格式。

import requests, json
import pandas as pd

query = {"naics": "424430"}
results = requests.post(
    "https://www.lobbyview.org/public/api/reports", data=json.dumps(query)
)


json_response = results.json()["result"]

resulting_data = []

# loop through the response
for data in json_response:
    # try to find entries with specific issues, bills_by_algo and committees
    try:
        # loop through the special issues
        for special_issue in data["specific_issues"]:
            # loop through the bills_by_algo's
            for x in special_issue["bills_by_algo"]:
                # append the id and committees in a dict
                resulting_data.append(({"id": x["id"], "committees": x["committees"]}))

    except KeyError as e:
        print(e, "not found in entry.")
        continue


# create a DataFrame
df = pd.DataFrame(resulting_data)
# export of list objects in the column is not supported by .dta, therefore we convert
# to strings with ";" as delimiter
df["committees"] = ["; ".join(map(str, l)) for l in df["committees"]]
print(df)
df.to_stata("result.dta")

结果在

             id                                         committees
0    110_HR1908                                    House Judiciary
1     110_S1145                                   Senate Judiciary
2    112_HR6083                                  House Agriculture
3     112_S3240        Senate Agriculture, Nutrition, and Forestry
4    113_HR2642                                  House Agriculture
..          ...                                                ...
406  113_HR2642                                  House Agriculture
407  114_HR1599  Senate Agriculture, Nutrition, and Forestry; H...
408  114_HR2029        House Appropriations; Senate Appropriations
409  112_HR6083                                  House Agriculture
410   112_S3240        Senate Agriculture, Nutrition, and Forestry

[411 rows x 2 columns]