使用concurrent.futures.ThreadPoolExecutor处理列表字典的每个元素

Question

假设我有一本字典，比如

dict = {A:[1,2,3], B:[4,5,6], C:[7,8,9], ......}

我想逐个处理特定键列表中的每个元素，但可以使用并发.futures.ThreadPoolExecutor 并行处理各个键

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as ex:
  ex.submit(process_element, contents of A)
  ex.submit(process_element, contents of B)
  ex.submit(process_element, contents of C)
  .
  .
  .

所以输出应该是

result of process_element A[0]
result of process_element B[0]
result of process_element C[0]

.
.
.

result of process_element A[1]

但不一定要按这个顺序

上述方法有什么问题吗？

Answer 1

看起来您传递的不是列表，而是一个名为“A”的变量。我不确定当您运行代码时为什么没有收到错误。你想要的就在下面

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as ex:
  ex.submit(process_element, dict['A'])
  ex.submit(process_element, dict['B'])
  ex.submit(process_element, dict['C'])

要逐一处理每个元素，您可以简单地使用列表构造：

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as ex:
  procsA = [ex.submit(process_element, elem) for elem in dict['A']
  ...
  ...

Answer 2

对于任何在这里了解如何使用映射将列表字典传递给线程池执行器的实际主题的人，您只需将

items()

方法应用于字典，该方法将创建一个可以分发到线程池的元组列表。线程功能：

import concurrent.futures

def process_data(key_value_pair):
    key, data_list = key_value_pair
    # Process the data associated with the key
    print(f"Processing data for key '{key}': {data_list}")

data_dict = {
    'key1': [1, 2, 3],
    'key2': [4, 5, 6],
    'key3': [7, 8, 9]
}

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    # Execute the partial function concurrently for each key-value pair
    result = executor.map(process_data, data_dict.items())