context.futures中的超时。上下文管理器中的ThreadPoolExecutor无法正常工作

Question

当我在上下文管理器中使用超时时，有人可以帮我解释为什么超时无法正常工作吗？

它在没有使用上下文管理器的情况下正常工作，它会在5s后引发TimeoutException但是使用上下文管理器它不会在5秒后引发异常。

实现没有上下文管理器

from concurrent import futures
MAX_WORKERS = 20

def download_one(c):
    import time
    time.sleep(100)


def download_many():
    executor = futures.ThreadPoolExecutor(MAX_WORKERS)
    res = executor.map(download_one, [1,2,3,4],timeout=5)
    print(list(res))

    return len(list(res))


download_many()

上下文管理器的实现

from concurrent import futures
MAX_WORKERS = 20

def download_one(c):
    import time
    time.sleep(100)


def download_many():
    with futures.ThreadPoolExecutor(MAX_WORKERS) as executor:
        res = executor.map(download_one, [1,2,3,4],timeout=5)
        print(list(res))

    return len(list(res))

download_many()

Answer 1

在两种情况下，超时都能正常工作。调用concurrent.futures._base.TimeoutError后，list(res)会上升5秒（=指定的超时值）。但是，当使用上下文管理器（with语句）时，将调用上下文管理器的__exit__方法。在这种情况下，它将等待所有线程完成后再离开上下文并（重新）引发原始错误。

这可以通过在正确的位置捕获和记录异常来证明：

import concurrent
import logging
import time
from concurrent import futures

MAX_WORKERS = 20


def download_one(c):
    logging.info('download_one(%s)' % str(c))
    time.sleep(10)  # Note: reduced to 10 seconds


def sub(executor):
    try:
        futures = executor.map(download_one, [1,2,3,4], timeout=5)
    except concurrent.futures._base.TimeoutError:
        logging.info('map timed out!')  # this is never logged
        raise

    try:
        results = list(futures)
    except concurrent.futures._base.TimeoutError:
        logging.info('list timed out!')  # here it happens!
        raise

    logging.info(results)
    logging.info('sub done')
    return len(result)


def download_many1():  # without context manager
    logging.info('download_many1')
    executor = futures.ThreadPoolExecutor(MAX_WORKERS)
    return sub(executor)


def download_many2():  # with context manager
    logging.info('download_many2')
    with futures.ThreadPoolExecutor(MAX_WORKERS) as executor:
        return sub(executor)


logging.basicConfig(level=logging.DEBUG, format='%(asctime)-15s   %(message)s')

logging.info('start 1')
try:
    download_many1()
except concurrent.futures._base.TimeoutError:
    logging.info('timeout!')
finally:
    logging.info('1 finished\n')

logging.info('start 2')
try:
    download_many2()
except concurrent.futures._base.TimeoutError:
    logging.info('timeout!')
finally:
    logging.info('2 finished\n')

这输出：

2019-04-27 21:17:20,578   start 1
2019-04-27 21:17:20,578   download_many1
2019-04-27 21:17:20,578   download_one(1)
2019-04-27 21:17:20,578   download_one(2)
2019-04-27 21:17:20,578   download_one(3)
2019-04-27 21:17:20,578   download_one(4)
2019-04-27 21:17:25,593   list timed out!   # actual timeout after 5 seconds
2019-04-27 21:17:25,593   timeout!          # the timeout you see at the same time
2019-04-27 21:17:25,593   1 finished

2019-04-27 21:17:25,593   start 2
2019-04-27 21:17:25,593   download_many2
2019-04-27 21:17:25,593   download_one(1)
2019-04-27 21:17:25,593   download_one(2)
2019-04-27 21:17:25,593   download_one(3)
2019-04-27 21:17:25,593   download_one(4)
2019-04-27 21:17:30,610   list timed out!   # actual timeout after 5 seconds
2019-04-27 21:17:35,608   timeout!          # the timeout you see 5 seconds later!!
2019-04-27 21:17:35,608   2 finished