我在下面[抽象]了多处理脚本,我正在尝试:
1)在两个进程之间分配工作负载(在列表中为每个变量添加1并将新变量附加到新列表)
2)将两个进程的输出加入到新的全局列表中以供将来处理。
关于如何获取两个进程的输出,并将该输出一起加入全局列表的任何想法?在执行之后我想要最终得到的是:
new_id_list = [2,4,6,8,10,3,5,7,9,11]来自worker1的#new_id_list +来自worker2的new_id_list
#python2
from multiprocessing import Process, Queue
from time import sleep
#records to process
id_list = [1,2,3,4,5,6,7,8,9,10]
#new output id list
new_id_list = []
queue = Queue()
def mp_worker(queue):
while queue.qsize() >0 :
record = queue.get()
new_id_list.append(record+1)
sleep(.1)
print(new_id_list)
###how would I go about passing this new_id_list as the global variable
print("worker closed")
def mp_handler():
# Spawn two processes, assigning the method to be executed
# and the input arguments (the queue)
processes = [Process(target=mp_worker, args=(queue,)) for _ in range(2)]
for process in processes:
process.start()
print('Process started')
for process in processes:
process.join()
if __name__ == '__main__':
for id in id_list:
queue.put(id)
mp_handler()
我假设您遇到的问题是两个进程无法共享new_id_list
。
您需要做的是创建另一个Queue
,它将代表结果队列并将其传递给两个进程。在流程内部根据需要附加到队列,并在两个流程执行结束时(在process.join()
之后),您只需从列表中的队列中提取所有内容。
找到关于解决方案的article here。
下面的工作代码。基本上:
1)我们使用multiprocessing.Manager()
2)使用Manager生成列表
3)将列表传递给每个worker,然后让每个worker将输出附加回列表。
from multiprocessing import Process, Queue
from time import sleep
import multiprocessing
#records to process
id_list = [1,2,3,4,5,6,7,8,9,10]
#new output id list
new_id_list = []
queue = Queue()
def mp_worker(d,queue):
while queue.qsize() >0 :
record = queue.get()
new_id_list.append(record+1)
d.append(record+1)
sleep(.1)
print(new_id_list)
print("worker closed")
def mp_handler():
# Spawn two processes, assigning the method to be executed
# and the input arguments (the queue)
processes = [Process(target=mp_worker, args=(d,queue,)) for _ in range(2)]
for process in processes:
process.start()
print('Process started')
for process in processes:
process.join()
if __name__ == '__main__':
mgr = multiprocessing.Manager()
d = mgr.list()
for id in id_list:
queue.put(id)
mp_handler()
print(d) #