使用pymysql将pandas中的json数据插入MySQL

Question

我有一个 csv 文件，其中的列是 json 转储列表或字典。

pictures

列

["https://images1.loopnet.com/i2/-zlV--gguqB1DTY1kXtod7Y11T6NjYHuEOsdziUbkIc/110/106-Caines-St-Seward-AK-Primary-Photo-1-Large.jpg", "https://images1.loopnet.com/i2/9oRXPv2oL67ts0OBDiQq--j6wl2NaP2_fVpRG4WmPWs/110/106-Caines-St-Seward-AK-Interior-Photo-2-Large.jpg", "https://images1.loopnet.com/i2/Whv9igp7CuqG14CkzA5y6oHo7NA0mltLBMttvdhTd1Y/110/106-Caines-St-Seward-AK-Interior-Photo-3-Large.jpg", "https://images1.loopnet.com/i2/WmAM3YCIRbr7NtaP322I98DjvqGY7MMf_p8fyKsZhVs/110/106-Caines-St-Seward-AK-Interior-Photo-4-Large.jpg", "https://images1.loopnet.com/i2/dB7SmPhSBDinpHJOM5oRxg6sLSF7Bp_Pa5SdyeCQpl0/110/106-Caines-St-Seward-AK-Interior-Photo-5-Large.jpg"]

的示例条目

financial_info

列示例

{"Gross Rental Income": {"annual": null, "annual_per_sf": null}, "Other Income": {"annual": null, "annual_per_sf": null}, "Vacancy Loss": {"annual": null, "annual_per_sf": null}, "Effective Gross Income": {"annual": null, "annual_per_sf": null}, "Net Operating Income": {"annual": 99999.0, "annual_per_sf": 9.99}}

我想将它从 csv 加载到 pandas 数据框并将其插入到 MySQL 数据库。我在 MySQL 中将这些列定义为 json 类型。

但是我明白

pymysql.err.ProgrammingError: (1064, 'You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \'["https://images1.loo...

我认为这是编码或引号的问题，但当我申请时它不起作用

import pandas as pd
from pymysql.converters import escape_string

data = pd.read_csv(filename, encoding="utf-8")
data["pictures"] = data["pictures"].apply(escape_string)

load_data_to_db(data)

Answer 1

在这段代码中，我们使用 pd.json.dumps 将 DataFrame 中的 JSON 数据转换为 JSON 格式的字符串。然后，当将数据插入 MySQL 数据库时，我们使用 %s 作为 JSON 格式字符串的占位符，以避免任何 SQL 注入问题。

代码：

import pandas as pd
import pymysql

data = pd.read_csv(filename, encoding="utf-8")

data["pictures"] = data["pictures"].apply(pd.json.dumps)
data["financial_info"] = data["financial_info"].apply(pd.json.dumps)

cursor = connection.cursor()

for index, row in data.iterrows():
    sql = "INSERT INTO your_table (pictures, financial_info) VALUES (%s, %s)"
    values = (row["pictures"], row["financial_info"])
    cursor.execute(sql, values)

从 JSON 格式字符串返回 JSON 数据的示例代码：

import json

json_str = '{"key": "value", "numbers": [1, 2, 3], "nested": {"foo": "bar"}}'
json_data = json.loads(json_str)
print(json_data)

print(json_data["key"])  # output: 'value'
print(json_data["numbers"])  # output: [1, 2, 3]