从 python 代码在 aws lambda 中下载 nltk 数据

Question

我正在尝试从 python 代码下载 AWS lambda 中的 nltk 数据。但它说

{
  "errorMessage": "[Errno 30] Read-only file system: 'layers'",
  "errorType": "OSError",
  "requestId": "",
  "stackTrace": [
    "  File \"/var/lang/lib/python3.9/importlib/__init__.py\", line 127, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n",
    "  File \"<frozen importlib._bootstrap>\", line 1030, in _gcd_import\n",
    "  File \"<frozen importlib._bootstrap>\", line 1007, in _find_and_load\n",
    "  File \"<frozen importlib._bootstrap>\", line 986, in _find_and_load_unlocked\n",
    "  File \"<frozen importlib._bootstrap>\", line 680, in _load_unlocked\n",
    "  File \"<frozen importlib._bootstrap_external>\", line 850, in exec_module\n",
    "  File \"<frozen importlib._bootstrap>\", line 228, in _call_with_frames_removed\n",
    "  File \"/opt/python/lib/python3.9/site-packages/datadog_lambda/handler.py\", line 30, in <module>\n    handler_module = import_module(modified_mod_name)\n",
    "  File \"/var/lang/lib/python3.9/importlib/__init__.py\", line 127, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n",
    "  File \"<frozen importlib._bootstrap>\", line 1030, in _gcd_import\n",
    "  File \"<frozen importlib._bootstrap>\", line 1007, in _find_and_load\n",
    "  File \"<frozen importlib._bootstrap>\", line 972, in _find_and_load_unlocked\n",
    "  File \"<frozen importlib._bootstrap>\", line 228, in _call_with_frames_removed\n",
    "  File \"<frozen importlib._bootstrap>\", line 1030, in _gcd_import\n",
    "  File \"<frozen importlib._bootstrap>\", line 1007, in _find_and_load\n",
    "  File \"<frozen importlib._bootstrap>\", line 986, in _find_and_load_unlocked\n",
    "  File \"<frozen importlib._bootstrap>\", line 680, in _load_unlocked\n",
    "  File \"<frozen importlib._bootstrap_external>\", line 850, in exec_module\n",
    "  File \"<frozen importlib._bootstrap>\", line 228, in _call_with_frames_removed\n",
    "  File \"/var/task/entrypoints/__init__.py\", line 14, in <module>\n    from utils.container import state_machine\n",
    "  File \"/var/task/utils/container.py\", line 10, in <module>\n    from service_layer.models.nltk_downloader import is_nltk_data_download\n",
    "  File \"/var/task/service_layer/models/nltk_downloader.py\", line 12, in <module>\n    nltk.download(\"punkt\", download_dir=environ[\"LAMBDA_NLTK_DATA\"])\n",
    "  File \"/var/task/nltk/downloader.py\", line 777, in download\n    for msg in self.incr_download(info_or_id, download_dir, force):\n",
    "  File \"/var/task/nltk/downloader.py\", line 642, in incr_download\n    yield from self._download_package(info, download_dir, force)\n",
    "  File \"/var/task/nltk/downloader.py\", line 699, in _download_package\n    os.makedirs(download_dir)\n",
    "  File \"/var/lang/lib/python3.9/os.py\", line 215, in makedirs\n    makedirs(head, exist_ok=exist_ok)\n",
    "  File \"/var/lang/lib/python3.9/os.py\", line 225, in makedirs\n    mkdir(name, mode)\n"
  ]
}

这是我的代码

        nltk.download("punkt", download_dir=download_dir)
        nltk.download("wordnet", download_dir=download_dir)
        nltk.download("omw-1.4", download_dir=download_dir)

我尝试了不同的 download_dir 选项。但没有任何效果。例如，

download_dir = /layers/nltk_data
download_dir = nltk_data
etc

有什么想法吗？

Answer 1

/tmp

中有可用于 lambda 的临时存储。对于更永久的存储，您可以考虑写入 S3 或 EFS。

Answer 2

您只能写入文件夹

/tmp

。默认为 512Mb，但您可以在常规配置中将其增加到 10Gb。 但是存储大文件不是一个好主意，因为每次冷启动时，都需要重新下载。

您可以尝试使用文件系统，为 lambda 挂载 efs