Boto3 和 Pandas 在 AWS Lambda 上导致错误

Question

我正在使用带有 boto3 和 pandas==1.5.3 的 Python 3.9。 在我的requirements.txt中，我安装了pandas。

我的代码很简单，还没完成，但是我需要Pandas和s3客户端，当我本地运行时，它可以工作：

import faulthandler
import pandas as pd
import boto3

faulthandler.enable()


def lambda_handler(event, context):
    s3_client = boto3.client('s3')

    # Not implemented yet

    return {"status_code": 200}

在以下图像上运行：

FROM public.ecr.aws/lambda/python:3.9

COPY src/ ${LAMBDA_TASK_ROOT}

COPY requirements/requirements.txt  .

RUN  pip3 install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"

CMD [ "lambda_function.lambda_handler" ]

但是在 AWS 上运行 Lambda 函数时，出现错误：

Error: Runtime exited with error: signal: segmentation fault

File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 1173 in create_module
File "<frozen importlib._bootstrap>", line 565 in module_from_spec
File "<frozen importlib._bootstrap>", line 666 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/var/lang/lib/python3.9/multiprocessing/shared_memory.py", line 23 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 850 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1058 in _handle_fromlist
File "/var/lang/lib/python3.9/multiprocessing/managers.py", line 36 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 850 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/var/runtime/s3transfer/compat.py", line 94 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 850 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/var/runtime/s3transfer/__init__.py", line 143 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 850 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 972 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/var/runtime/boto3/s3/transfer.py", line 131 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 850 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/var/runtime/boto3/s3/inject.py", line 18 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 850 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/var/runtime/boto3/utils.py", line 56 in import_module
File "/var/runtime/boto3/utils.py", line 65 in _handler
File "/var/runtime/botocore/hooks.py", line 239 in _emit
File "/var/runtime/botocore/hooks.py", line 256 in emit
File "/var/runtime/botocore/hooks.py", line 412 in emit
File "/var/runtime/botocore/client.py", line 201 in _create_client_class
File "/var/runtime/botocore/client.py", line 147 in create_client
File "/var/runtime/botocore/session.py", line 997 in create_client
File "/var/runtime/boto3/session.py", line 299 in client
File "/var/runtime/boto3/__init__.py", line 92 in client
File "/var/task/lambda_function.py", line 21 in lambda_handler
File "/var/runtime/awslambdaric/bootstrap.py", line 188 in handle_event_request
File "/var/runtime/awslambdaric/bootstrap.py", line 499 in run
File "/var/runtime/awslambdaric/__main__.py", line 21 in main
File "/var/runtime/bootstrap.py", line 57 in main
File "/var/runtime/bootstrap.py", line 60 in <module>

当我没有 pandas 导入时，代码可以工作。 我尝试过更改Python、Pandas和boto3版本。

我为 lambda 配置了最大内存： Lambda 配置

lambda 策略拥有整个 s3 的权限。

Answer 1

我终于发现了。在我的需求文件中，我有pandas==1.5.3，我也有pandera==0.8.1，我不知道为什么，但是当我将pandera更新到0.19.2时，它可以工作。