处理文本时出现字符编码问题

Question

我正在尝试处理大型语料库，但在preprocess_string（）中返回如下所示的错误

Traceback (most recent call last): File "D:/Projects/docs_handler/data_preprocessing.py", line 60, in <module> for temp in batch(iterator,1000): File "D:/Projects/docs_handler/data_preprocessing.py", line 30, in batch for item in iterable: File "D:/Projects/docs_handler/data_preprocessing.py", line 23, in iter_tokenized_documents document = preprocess_string(open(os.path.join(root, file)).read().strip(),filters=CUSTOM_FILTERS) File "C:\Users\koradg\AppData\Local\Programs\Python\Python36\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 16144: character maps to <undefined>

---

Versions
Windows-10-10.0.17763-SP0
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit (AMD64)]
NumPy 1.17.0
SciPy 1.3.0
gensim 3.8.0
FAST_VERSION 0

---

def iter_tokenized_documents(input_directory):
    """Iterate over all documents, yielding a document (=list of utf8 tokens) at a time."""
    for root, dirs, files in os.walk(input_directory):
        for file in filter(lambda file: file.endswith('.txt'), files):
            document = preprocess_string(open(os.path.join(root, file)).read().strip(),filters=CUSTOM_FILTERS)
            if(len(document)):
                yield document

如何运行它而没有任何错误？

Answer 1

从您的问题来看，您似乎在编码错误。只需在打开文件中使用encoding ='utf-8'。如果该章程不重要，请在打开文件中使用此errors =“ ignore”。