我在一个类中有一个方法需要在循环中做很多工作,我想把工作分散到我的所有核心上。
我写了下面的代码,如果我使用普通的map
,但是使用pool.map
会返回错误。
import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)
class OtherClass:
def run(sentence, graph):
return False
class SomeClass:
def __init__(self):
self.sentences = [["Some string"]]
self.graphs = ["string"]
def some_method(self):
other = OtherClass()
def single(params):
sentences, graph = params
return [other.run(sentence, graph) for sentence in sentences]
return list(pool.map(single, zip(self.sentences, self.graphs)))
SomeClass().some_method()
错误:
AttributeError:无法pickle本地对象'SomeClass.some_method..single'
为什么不能腌制single
?我甚至尝试将single
移动到全局模块范围(不在类中 - 使其独立于上下文):
import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)
class OtherClass:
def run(sentence, graph):
return False
def single(params):
other = OtherClass()
sentences, graph = params
return [other.run(sentence, graph) for sentence in sentences]
class SomeClass:
def __init__(self):
self.sentences = [["Some string"]]
self.graphs = ["string"]
def some_method(self):
return list(pool.map(single, zip(self.sentences, self.graphs)))
SomeClass().some_method()
我收到以下错误:
AttributeError:无法从'... / test.py'获取模块'main'上的属性'single'
在定义函数和类之前,您正在启动池,这样子进程就无法继承任何代码。移动池开始到底部并用if __name__ == '__main__':
保护它
import multiprocessing
class OtherClass:
def run(self, sentence, graph):
return False
def single(params):
other = OtherClass()
sentences, graph = params
return [other.run(sentence, graph) for sentence in sentences]
class SomeClass:
def __init__(self):
self.sentences = [["Some string"]]
self.graphs = ["string"]
def some_method(self):
return list(pool.map(single, zip(self.sentences, self.graphs)))
if __name__ == '__main__': # <- prevent RuntimeError for 'spawn'
# and 'forkserver' start_methods
with multiprocessing.Pool(multiprocessing.cpu_count() - 1) as pool:
print(SomeClass().some_method())