当使用pandas to_sql进入VARCHAR(MAX)SQL列时,数据截断。

问题描述 投票:3回答:1

问题:在将数据框导入到SQL表时,出现截断错误。

当把数据框导入到SQL表时,得到一个截断错误。

sqlalchemy.exc.DataError: (pyodbc.DataError) ('22001', '[22001] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]String or binary data would be truncated. (8152) (SQLExecDirectW); [22001] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]The statement has been terminated. (3621)')

问题列是 项目描述 (见下面的栏目表)。我已经引用了产生错误的那一行。我不完全确定这是问题的根源,但我认为该列最初是以VARCHAR(300)的形式创建的。引起问题的值是307个字符,其中9个被截断,如下图。这样就剩下298个了。原先的大小限制是300,2个为's',我想还剩下298吧?不确定,但截断的长度很奇怪。

导致该问题的代码。

dataframe.to_sql('tblImport', eng, schema='dbo', if_exists='append', index=False, dtype=dict_SQL_dtypes)

尝试过的解决方案:

  • 在SQL中设置TEXTSIZE
  • 将列类型改为VARCHAR NVARCHAR,将长度改为1000或MAX。
  • 将数据类型(dtype)设置为 sqlalchemy.types.VARCHAR (想法来自 此处)
  • 导入前,按ProjectDescription列的长度降序对数据框进行排序,将最长的值放在最上面一行。
  • 丢弃表格,将列重新创建为VARCHAR(MAX)作为数据类型,以防列的原始长度在某种程度上仍然限制了导入。
  • 直接通过SQL查询改变列类型

版本。

  • 操作系统:Windows 10
  • 蟒蛇。3.6
  • pandas: 0.24.1
  • sqlalchemy: 1.2.17
  • pyodbc:4.0.25
  • DB管理工具。MSSMS
  • SQL server 12

DB连接。

pyodbc.connect("DRIVER={{ODBC Driver 13 for SQL Server}};SERVER={0};DATABASE={1};UID={2};PWD={3}".format(serv, db, usr, pwd))

Engine:

create_engine('mssql+pyodbc://{}@sqlserver:{}@{}:1433/{}?driver=ODBC+Driver+13+for+SQL+Server'.format(usr, pwd, serv, db), echo=True)

ProjectDescription列 character_maximum_length from DB schema: -1

栏目。

Column Name          | Data Type      | Allow Nulls
ProjectUID           | varchar(60)    | Unchecked
Framework            | char(5)        | Checked
Partner              | char(3)        | Checked
SCUID                | char(9)        | Checked
ClientName           | varchar(255)   | Checked
PartnerProjectNumber | varchar(40)    | Checked
ProjectName          | varchar(255)   | Checked
ProjectDescription   | varchar(MAX)   | Checked

溯源。

  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\base.py", line 1216, in _execute_context
    cursor, statement, parameters, context
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\default.py", line 533, in do_executemany
    cursor.executemany(statement, parameters)
pyodbc.DataError: ('22001', '[22001] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]String or binary data would be truncated. (8152) (SQLExecDirectW); [22001] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]The statement has been terminated. (3621)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "run_for_tests.py", line 258, in <module>
    outmsg = run()
  File "run_for_tests.py", line 252, in run
    process_result = process_data(file_type, file_name, relative_path, dict_clients)
  File "run_for_tests.py", line 204, in process_data
    update_sql_tables(df_base)
  File "server_interaction.py", line 74, in update_sql_tables
    dataframe.to_sql('tblImport', eng, schema='dbo', if_exists='append', index=False, dtype=dict_SQL_dtypes)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\pandas\core\generic.py", line 2532, in to_sql
    dtype=dtype, method=method)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\pandas\io\sql.py", line 460, in to_sql
    chunksize=chunksize, dtype=dtype, method=method)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\pandas\io\sql.py", line 1174, in to_sql
    table.insert(chunksize, method=method)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\pandas\io\sql.py", line 686, in insert
    exec_insert(conn, keys, chunk_iter)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\pandas\io\sql.py", line 599, in _execute_insert
    conn.execute(self.table.insert(), data)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\base.py", line 980, in execute
    return meth(self, multiparams, params)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\sql\elements.py", line 273, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\base.py", line 1099, in _execute_clauseelement
    distilled_params,
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\base.py", line 1240, in _execute_context
    e, statement, parameters, cursor, context
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\base.py", line 1458, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\util\compat.py", line 296, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\util\compat.py", line 276, in reraise
    raise value.with_traceback(tb)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\base.py", line 1216, in _execute_context
    cursor, statement, parameters, context
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\default.py", line 533, in do_executemany
    cursor.executemany(statement, parameters)
  sqlalchemy.exc.DataError: (pyodbc.DataError) ('22001', '[22001] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]String or binary data would be truncated. (8152) (SQLExecDirectW); [22001] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]The statement has been terminated. (3621)') [SQL: 'INSERT INTO dbo.[tblImport] ([ProjectUID], [Framework], [Partner], [SCUID], [ClientName], [PartnerProjectNumber], [ProjectName], [ProjectDescription]) VALUES (?, ?, ?, ?, ?, ?, ?, ?)'] [parameters: (('xxx', 'xxxxx', 'xxx', 'xxxxxxx', 'ABC', 123, 'ABC', 'Provide competition facilities. Build this facility with a capacity of 9000 spectators  ... (9 characters truncated) ... m pool and diving facilities and athletes changing facilities to a competion standard and then, post games, convert it to a council operated facility'), ....)] (Background on this error at: http://sqlalche.me/e/9h9h)
python sql-server pandas sqlalchemy
1个回答
© www.soinside.com 2019 - 2024. All rights reserved.