python 3.5 nltk Stanford细分windows 10

Question

我按照官方NLTK wiki的指示设置了环境变量。我在第一个例子中遇到了以下错误。以下是代码：

from nltk.tokenize import StanfordSegmenter
datapath = "D:/Coding/stanford-segmenter/"
corporadict = datapath+"data/"
modelpath = datapath + "data/pku.gz"
dictpath = datapath + "data/dict-chris6.ser.gz"
segmenter = StanfordSegmenter(path_to_sihan_corpora_dict=corporadict,path_to_model=modelpath,path_to_dict=dictpath)
res = segmenter.segment(u"这是斯坦福中文分词器")

但Python给了我以下错误。 Traceback（最近一次调用最后一次）：

File "D:/Video data/data_processed/ugctext/test_stanford.py", line 19, in <module>
res = segmenter.segment(u"这是斯坦福中文分词器")
File "C:\Python35\lib\site-packages\nltk\tokenize\stanford_segmenter.py", line 164, in segment
return self.segment_sents([tokens])
File "C:\Python35\lib\site-packages\nltk\tokenize\stanford_segmenter.py", line 192, in segment_sents
stdout = self._execute(cmd)
File "C:\Python35\lib\site-packages\nltk\tokenize\stanford_segmenter.py", line 211, in _execute
stdout, _stderr = java(cmd, classpath=self._stanford_jar, stdout=PIPE, stderr=PIPE)
File "C:\Python35\lib\site-packages\nltk\internals.py", line 129, in java
p = subprocess.Popen(cmd, stdin=stdin, stdout=stdout, stderr=stderr)
File "C:\Python35\lib\subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "C:\Python35\lib\subprocess.py", line 1198, in _execute_child
args = list2cmdline(args)
File "C:\Python35\lib\subprocess.py", line 751, in list2cmdline
needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: argument of type 'NoneType' is not iterable

任何人都可以帮我解决这个问题吗？谢谢！

Answer 1

由于某种原因，list2cmdline(args)的subprocess.py正在返回[None]，并且没有得到妥善处理。我猜这是java()中stanford_segmenter.py调用的问题。

从here您可以看到代码已更新为2014年需要Java 8.如果您的Java版本低于此值，则可能是问题所在。

Answer 2

可能需要java_class参数。

例如：

segmenter = StanfordSegmenter(
    java_class='edu.stanford.nlp.ie.crf.CRFClassifier',
    path_to_sihan_corpora_dict=corporadict,
    path_to_model=modelpath,
    path_to_dict=dictpath
)