pandas pyodbc unicode问题你\ 2013

Question

我使用mac与python 2.7和pyodbc来查询来自Microsoft Sql server的数据。有一个timestamp列，在我的数据帧中显示为datetime64 [ns]

该计划的结构 -

SQLCommand = (" SELECT Col1, Col2, Col3 from xyztable ") 
DF = pd.read_sql(SQLCommand,cnxn)

# extracting Day and month by converting to dt 
DF['TS']=DF['TS'].dt.strftime('%d%m%')
# Create labels from Categories (string type data column in SQL table), replacing each category
DF['Flag']= DF['CODE']
DF.dtypes


TS         datetime64[ns]
TIWOR            object
CODES           object
T-enc                int8
TS                    object
TS_HHMM               object
TS_DD                  int64
TS_DDMM                int64
Flag              object
dtype: object

# I am able to replace all categories but it fails at this step as u\2013 appears in the middle of string 
DF['Flag'].unique()
array([0, 1, nan, u'Dev \u2013 Env'], dtype=object) 

# All attempts to find and replace are not working, some records have 'nan' values and DF.dropna does not work.

试图解决

1. 已经提到了各种堆栈溢出文章，但它没有帮助。
2. 将Sql查询输出导出为CSV文件并加载到Dataframe中，仍然是同样的问题。 DF.to_csv('~/SQLoutput.csv', sep='\t', encoding='utf-8')
3. 导出sql查询输出到excel文件并加载到dataframe中，仍然是同样的问题DF=pd.read_excel('/Users/User1/SQLoutput.xlsx',sheet_name=0,encoding='utf-8')
4. 在代码开头添加了# -*- coding:utf-8 -*-，没有帮助
5. 验证pyodbc设置，没问题。 tsql -S sqlservername -U Username -P Password

Answer 1

虽然在熊猫中没有尝试过，但您可以使用（这只是一个记录的样本，尝试对整个列应用相同的）来解决unicode问题：

import unidecode
record = unidecode.unidecode_expect_nonascii(record)