AWS lambda，scrapy和捕获异常

Question

我正在将scrapy作为AWS lambda函数运行。在我的函数内部，我需要一个计时器来查看它是否运行超过1分钟，如果是这样，我需要运行一些逻辑。这是我的代码：

def handler():
    x = 60
    watchdog = Watchdog(x)
    try:
        runner = CrawlerRunner()
        runner.crawl(MySpider1)
        runner.crawl(MySpider2)
        d = runner.join()
        d.addBoth(lambda _: reactor.stop())
        reactor.run()
    except Watchdog:
        print('Timeout error: process takes longer than %s seconds.' % x)
        # some other logic here
    watchdog.stop()

我从this answer获得的看门狗定时器类。问题是代码永远不会命中except Watchdog阻塞，而是抛出异常：

Exception in thread Thread-1:
 Traceback (most recent call last):
   File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
     self.run()
   File "/usr/lib/python3.6/threading.py", line 1182, in run
     self.function(*self.args, **self.kwargs)
   File "./functions/python/my_scrapy/index.py", line 174, in defaultHandler
     raise self
 functions.python.my_scrapy.index.Watchdog: 1

我需要在函数中捕获异常。我该怎么做呢？ PS：我对Python很新。

Answer 1

好吧，这个问题让我有点疯狂，这就是为什么这不起作用：

Watchdog对象的作用是创建另一个引发异常但未处理的线程（异常仅在主进程中处理）。幸运的是，扭曲有一些简洁的功能。

您可以在另一个线程中运行reactor：

import time
from threading import Thread
from twisted.internet import reactor

runner = CrawlerRunner()
runner.crawl(MySpider1)
runner.crawl(MySpider2)
d = runner.join()
d.addBoth(lambda _: reactor.stop())
Thread(target=reactor.run, args=(False,)).start()  # reactor will run in a different thread so it doesn't lock the script here

time.sleep(60)  # Lock script here

# Now check if it's still scraping
if reactor.running:
    # do something
else:
    # do something else

我正在使用python 3.7.0

Answer 2

Twisted具有调度原语。例如，该程序运行大约60秒：

from twisted.internet import reactor
reactor.callLater(60, reactor.stop)
reactor.run()