在for循环中使用Multiprocessing.Pool的意外行为

Question

这是我的代码：

import multiprocessing as mp
import numpy as np

def foo(p):
    global i
    return p*i

global lower, upper
lower = 1
upper = 4

for i in range(lower, upper):
    if __name__ == '__main__':
        dataset = np.linspace(1, 100, 100)
        agents = mp.cpu_count() - 1
        chunksize = 5
        pool = mp.Pool(processes=agents)
        result = pool.map(foo, dataset, chunksize)
        print result
        print i
        pool.close()
        pool.join()

控制台打印出阵列[3,6,9，...，300]三次，每个阵列打印输出之间的整数1,2,3。所以我正确地在低位和高位（不包括）之间进行迭代，但我希望它首先打印出数组[1,2,3，...，100]，然后是[2,4,6，...， 200]，最后[3,6,9，...，300]。我不明白为什么它只将i的最终值传递给foo然后映射三次。

Answer 1

当您运行新进程时，它就是它所看到的：

import multiprocessing as mp
import numpy as np

def foo(p):
    global i
    return p*i

global lower, upper
lower = 1
upper = 4

for i in range(lower, upper):
    if __name__ == '__main__':
        # This part is not run, as
        # in a different process,
        # __name__ is set to '__mp_main__'
# i is now `upper - 1`, call `foo(p)` with the provided `p`

在执行之后，它被告知要运行foo（它必须再次运行整个脚本以找出foo是什么，只是因为它的酸洗效果如何）

因此，在它运行之后，i将是upper - 1，它将永远返回p * 3。

你想让i成为给foo或一些多处理特定内存共享对象的参数，如descibed here

Answer 2

让我本地和使用functools.partial可以解决您的问题：

import multiprocessing as mp
import numpy as np
import functools

def foo(p,i):
    return p*i

global lower, upper
lower = 1
upper = 4

for i in range(lower, upper):
    if __name__ == '__main__':
        dataset = np.linspace(1, 100, 100)
        agents = mp.cpu_count() - 1
        chunksize = 5
        pool = mp.Pool(processes=agents)
        foo2 = functools.partial(foo, i)
        result = pool.map(foo2, dataset, chunksize)
        print(result)
        print(i)
        pool.close()
        pool.join()