加入多处理工作者的输出 - python2

问题描述 投票:0回答:2

我在下面[抽象]了多处理脚本,我正在尝试:

1)在两个进程之间分配工作负载(在列表中为每个变量添加1并将新变量附加到新列表)

2)将两个进程的输出加入到新的全局列表中以供将来处理。

关于如何获取两个进程的输出,并将该输出一起加入全局列表的任何想法?在执行之后我想要最终得到的是:

new_id_list = [2,4,6,8,10,3,5,7,9,11]来自worker1的#new_id_list +来自worker2的new_id_list

#python2
from multiprocessing import Process, Queue
from time import sleep

#records to process
id_list = [1,2,3,4,5,6,7,8,9,10]

#new output id list
new_id_list = []

queue = Queue()

def mp_worker(queue):

    while queue.qsize() >0 :
        record = queue.get()
        new_id_list.append(record+1)
        sleep(.1)
    print(new_id_list)
    ###how would I go about passing this new_id_list as the global variable
    print("worker closed")

def mp_handler():

    # Spawn two processes, assigning the method to be executed 
    # and the input arguments (the queue)
    processes = [Process(target=mp_worker, args=(queue,)) for _ in range(2)]

    for process in processes:
        process.start()
        print('Process started')

    for process in processes:
        process.join()



if __name__ == '__main__':

    for id in id_list:
        queue.put(id)
    mp_handler()
python multiprocessing
2个回答
1
投票

我假设您遇到的问题是两个进程无法共享new_id_list

您需要做的是创建另一个Queue,它将代表结果队列并将其传递给两个进程。在流程内部根据需要附加到队列,并在两个流程执行结束时(在process.join()之后),您只需从列表中的队列中提取所有内容。


1
投票

找到关于解决方案的article here

下面的工作代码。基本上:

1)我们使用multiprocessing.Manager()

2)使用Manager生成列表

3)将列表传递给每个worker,然后让每个worker将输出附加回列表。

from multiprocessing import Process, Queue
from time import sleep
import multiprocessing

#records to process
id_list = [1,2,3,4,5,6,7,8,9,10]

#new output id list
new_id_list = []

queue = Queue()

def mp_worker(d,queue):

    while queue.qsize() >0 :
        record = queue.get()
        new_id_list.append(record+1)
        d.append(record+1)
        sleep(.1)
    print(new_id_list)  
    print("worker closed")

def mp_handler():

    # Spawn two processes, assigning the method to be executed 
    # and the input arguments (the queue)
    processes = [Process(target=mp_worker, args=(d,queue,)) for _ in range(2)]

    for process in processes:
        process.start()
        print('Process started')

    for process in processes:
        process.join()



if __name__ == '__main__':
    mgr = multiprocessing.Manager()
    d = mgr.list()
    for id in id_list:
        queue.put(id)
    mp_handler()
    print(d)  #
© www.soinside.com 2019 - 2024. All rights reserved.