在不同的python进程之间共享大对象的最快方法

问题描述 投票:0回答:1

假定我有3个不同的Process,它们在永远的循环中执行不同的逻辑。我想并行运行所有它们,而每个Process可以访问shared_object,这是一类的重要对象。因此,我尝试将multiprocessing与管理器一起使用,以将其归档,如下所示:

import multiprocessing
import inspect
from multiprocessing.managers import BaseManager, NamespaceProxy
import time
import random

class SharedObject():
    def __init__(self):
        self.a = 1

    def show_a(self):
        print(self.a)


class ProcessManager(BaseManager):
    pass


class ProxyBase(NamespaceProxy):
    _exposed_ = ('__getattribute__', '__setattr__', '__delattr__')


class ManagerProxy(ProxyBase):
    pass


def register_proxy(name, cls, proxy):
    for attr in dir(cls):
        if callable(getattr(cls, attr)) and not attr.startswith("__"):
            proxy._exposed_ += (attr,)
            setattr(proxy, attr,
                    lambda s: object.__getattribute__(s, '_callmethod')(attr))
    ProcessManager.register(name, cls, proxy)


register_proxy('shared_object', SharedObject, ManagerProxy)

process_manager = ProcessManager()
process_manager.start()
shared_object = process_manager.shared_object()

def process_1():
    while True:
        print('Process 1 see {}'.format(shared_object.a))
        shared_object.a = 1
        time.sleep(1)

def process_2():
    while True:
        print('Process 2 see {}'.format(shared_object.a))
        shared_object.a = 2
        time.sleep(1)

def process_3():
    while True:
        print('Process 3 see {}'.format(shared_object.a))
        shared_object.a = 3
        if random.randint(0,1) == 1:
            shared_object.show_a()
        time.sleep(1)

first_process = multiprocessing.Process(name="First process", target=process_1)
first_process.start()

second_process = multiprocessing.Process(name="Second process", target=process_2)
second_process.start()

third_process = multiprocessing.Process(name="Third process", target=process_3)
third_process.start()

shared_object.show_a()

while True:
    time.sleep(10)

它有效,但是对我来说太慢了,因为我必须绕过大的numpy数组。还有其他方法可以使其更快(实时速度)吗?非常感谢

python process multiprocessing
1个回答
0
投票

看来这是由multiprocessing.shared_memory解决的问题,但是至少需要重新构建代码:

  • 分配正确的尺寸
  • 将共享库的名称传递给进程
  • 并且记得最后将其关闭
© www.soinside.com 2019 - 2024. All rights reserved.