生产者/消费者 - Queue.通过列表获取

Question

我正在尝试从数据库获取数据以输入机器学习模型。但我的Producer只是将数据一行一行地放入DB中的队列，Consumer也只能一行一行地获取。我的机器学习模型需要 2000 行来处理。

我希望生产者和消费者每次放置和获取一定数量的行，而不仅仅是一行一行

这是我的Python代码

def read_data():
    
    torque_data_queryset = Torque.get_torque_data()
    return list(torque_data_queryset)


def producer(queue):
    print('Producer: Running')
    data = read_data()
    for item in data:
       
        time.sleep(1)
        
        queue.put(item)
        print(f'> Producer added {item}')
    
    queue.put(None)
    print('Producer: Done')

# consumer task
def consumer(queue):
    print('Consumer: Running')
    while True:
        
        item = queue.get()
        print(item)
        
        if item is None:
            break
        
        time.sleep(1)
        
        print(f'> Consumer got {item}')
    print('Consumer: Done')

def thread_function():
    
    queue = Queue()
    
    consumer_thread = Thread(target=consumer, args=(queue,))
    consumer_thread.start()
    
    producer_thread = Thread(target=producer, args=(queue,))
    producer_thread.start()
    
    producer_thread.join()
    consumer_thread.join()
    
def thread_result(request):
    
    thread_function()
    return render(request, 'async_processing_result.html', {'message': 'Threads completed successfully!'})

Answer 1

您可以将对象列表put()放入queue.Queue中。消费者将立即get()整个列表。

例如：

import threading
import queue


def consumer(q: queue.Queue):
    while (value := q.get()) is not None:
        print(*value, sep="\n")


def producer(q: queue.Queue):
    # create a list of dictionaries
    data = [{"int": i, "chr": chr(i + ord("a"))} for i in range(10)]
    q.put(data)
    q.put(None)


def main():
    q = queue.Queue()
    threads = []
    for func in producer, consumer:
        (t := threading.Thread(target=func, args=(q,))).start()
        threads.append(t)
    for t in threads:
        t.join()


if __name__ == "__main__":
    main()

输出：

{'int': 0, 'chr': 'a'}
{'int': 1, 'chr': 'b'}
{'int': 2, 'chr': 'c'}
{'int': 3, 'chr': 'd'}
{'int': 4, 'chr': 'e'}
{'int': 5, 'chr': 'f'}
{'int': 6, 'chr': 'g'}
{'int': 7, 'chr': 'h'}
{'int': 8, 'chr': 'i'}
{'int': 9, 'chr': 'j'}