尝试编码：链除外

Question

我有一个并发的.futures，通常与chunksize = 1一起使用。但是，偶尔我遇到需要更大块大小的大型数据集。目前，我已使用以下代码解决了此问题：

for i in datasets:
    try:
        with concurrent.futures.ProcessPoolExecutor() as executor:
            results=tuple(executor.map(do_something, parameters, chunksize=1)
    except concurrent.futures.process.BrokenProcessPool:
        try:
            with concurrent.futures.ProcessPoolExecutor() as executor:
                results=tuple(executor.map(do_something, parameters, chunksize=2)
        except concurrent.futures.process.BrokenProcessPool:
            try:
                with concurrent.futures.ProcessPoolExecutor() as executor:
                    results=tuple(executor.map(do_something, parameters, chunksize=4)
            etc. etc. etc....
            except concurrent.futures.process.BrokenProcessPool:
                   print('code failed')

这很好，但显然确实很丑陋。无论如何，我可以更简单地做到这一点？

Answer 1

您可以使用for循环遍历块大小，如果成功，则可以使用break循环，或者输入else块以输出错误消息：

for i in datasets:
    for chunksize in 1, 2, 4:
        try:
            with concurrent.futures.ProcessPoolExecutor() as executor:
                results=tuple(executor.map(do_something, parameters, chunksize=chunksize)
            break
        except concurrent.futures.process.BrokenProcessPool:
            pass
    else:
        print('code failed')

Answer 2

您应该使用另一个循环来完成此操作。每次执行的代码都相同，但块大小不同。这是使用for循环的示例

for i in datasets:

    for chunksize in [1,2,4]:
      try:
          with concurrent.futures.ProcessPoolExecutor() as executor:
              results=tuple(executor.map(do_something, parameters, chunksize=chunksize)
          break
      except concurrent.futures.process.BrokenProcessPool:
          if chunksize == 4:
            raise SomeErrorForMaxRetries
          else:
            continue