无法在数据流中安装诗歌

问题描述 投票：0回答：1

运行任何诗歌可执行文件时出现此错误

Traceback (most recent call last):
  File "/root/.local/bin/poetry", line 5, in <module>
    from poetry.console.application import main
  File "/root/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/console/application.py", line 11, in <module>
    from cleo.application import Application as BaseApplication
ModuleNotFoundError: No module named 'cleo'

我的容器就是用这个逻辑构建的。

FROM gcr.io/dataflow-templates-base/python38-template-launcher-base:flex_templates_base_image_release_20230508_RC00
ARG DIR=/dataflow/template
ARG dataflow_file_path
ARG PROJECT_ID
# environment to pull the right containers
ARG ENV
ARG TOKEN
ENV COMPOSER_$ENV=1
ENV HAMSTERDAM_ROOT=/dataflow/template/

# copying over necessary files
RUN mkdir -p ${DIR}
WORKDIR ${DIR}
COPY transform/dataflow/${dataflow_file_path}.py beam.py
COPY deploy/dataflow/poetry.lock .
COPY deploy/dataflow/pyproject.toml .

# env var in order to use hamsterdam lib, for more info, see:
# https://cloud.google.com/dataflow/docs/guides/templates/configuring-flex-templates#set_required_dockerfile_environment_variables
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${DIR}/beam.py"
ENV FLEX_TEMPLATE_PYTHON_EXTRA_PACKAGES=""
ENV FLEX_TEMPLATE_PYTHON_PY_OPTIONS=""
ENV PIP_NO_DEPS=True

# install poetry
RUN curl -sSL https://install.python-poetry.org | python -
ENV PATH "/root/.local/bin/:${PATH}"
RUN poetry --version

我已经尝试卸载它和建议来自：

这些并不是真正适用的，因为它们是 

poetry
非 docker 环境中的可执行文件，但不确定还能做什么。我有一个从 
apache/beam_python3.8_sdk:2.45.0
构建的数据流 SDK，它具有相同的逻辑并且正在运行。

我忽略了最后一个

RUN
命令并构建了容器，这些是一些检查的输出

❯ docker run --rm --entrypoint /bin/bash dataflow -c 'which poetry'
/root/.local/bin/poetry
❯ docker run --rm --entrypoint /bin/bash dataflow -c 'poetry'
Traceback (most recent call last):
  File "/root/.local/bin/poetry", line 5, in <module>
    from poetry.console.application import main
  File "/root/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/console/application.py", line 11, in <module>
    from cleo.application import Application as BaseApplication
ModuleNotFoundError: No module named 'cleo'
❯ docker run --rm --entrypoint /bin/bash dataflow -c 'which python'
/usr/local/bin/python

我的假设是 

poetry
可执行文件正在导入其他库未安装到的 virtualenv。

更新：

我走进了一个兔子洞，做了这个

RUN pip install --no-cache-dir poetry cleo rapidfuzz importlib_metadata zipp crashtest

并且运行 

poetry --version
有效但是 
poetry config virtualenvs.create false
或任何其他命令将抛出此错误

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/poetry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/poetry/console/application.py", line 409, in main
    exit_code: int = Application().run()
  File "/usr/local/lib/python3.8/site-packages/cleo/application.py", line 338, in run
    self.render_error(e, io)
  File "/usr/local/lib/python3.8/site-packages/poetry/console/application.py", line 180, in render_error
    self.set_solution_provider_repository(self._get_solution_provider_repository())
  File "/usr/local/lib/python3.8/site-packages/poetry/console/application.py", line 398, in _get_solution_provider_repository
    from poetry.mixology.solutions.providers.python_requirement_solution_provider import (  # noqa: E501
  File "/usr/local/lib/python3.8/site-packages/poetry/mixology/__init__.py", line 5, in <module>
    from poetry.mixology.version_solver import VersionSolver
  File "/usr/local/lib/python3.8/site-packages/poetry/mixology/version_solver.py", line 8, in <module>
    from poetry.core.packages.dependency import Dependency
ModuleNotFoundError: No module named 'poetry.core'
python google-cloud-dataflow python-poetry
1个回答
0
投票

我发现了问题。在构建 Dataflow Flex 模板时，它似乎是 

Dockerfile
的顺序

这会起作用

# installing poetry binaries
RUN curl -sSL https://install.python-poetry.org | python3 -
ENV PATH "/root/.local/bin/:${PATH}"

# copying over necessary files
RUN mkdir -p ${DIR}
WORKDIR ${DIR}
COPY transform/dataflow/${dataflow_file_path}.py beam.py
COPY deploy/dataflow/poetry.lock .
COPY deploy/dataflow/pyproject.toml .

# env var in order to use custom lib, for more info, see:
# https://cloud.google.com/dataflow/docs/guides/templates/configuring-flex-templates#set_required_dockerfile_environment_variables
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${DIR}/beam.py"
ENV FLEX_TEMPLATE_PYTHON_EXTRA_PACKAGES=""
ENV FLEX_TEMPLATE_PYTHON_PY_OPTIONS=""
ENV PIP_NO_DEPS=True

但这不会

# copying over necessary files
RUN mkdir -p ${DIR}
WORKDIR ${DIR}
COPY transform/dataflow/${dataflow_file_path}.py beam.py
COPY deploy/dataflow/poetry.lock .
COPY deploy/dataflow/pyproject.toml .

# env var in order to use custom lib, for more info, see:
# https://cloud.google.com/dataflow/docs/guides/templates/configuring-flex-templates#set_required_dockerfile_environment_variables
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${DIR}/beam.py"
ENV FLEX_TEMPLATE_PYTHON_EXTRA_PACKAGES=""
ENV FLEX_TEMPLATE_PYTHON_PY_OPTIONS=""
ENV PIP_NO_DEPS=True


# installing poetry binaries
RUN curl -sSL https://install.python-poetry.org | python3 -
ENV PATH "/root/.local/bin/:${PATH}"

看得更深一点，可能是环境变量在起作用，尽管我不太确切如何。这是出错时的环境变量

这是工作图的

env

❯ docker run --rm --entrypoint /bin/bash dataflow -c 'env'
_=/usr/bin/env
GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=/etc/ssl/certs/ca-certificates.crt
PATH=/root/.local/bin/:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/gcloud/google-cloud-sdk/bin
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/1a96dc5acd0303c4700e02655aefd3bc68c78958/public/get-pip.py
PYTHON_GET_PIP_SHA256=d1d09b0f9e745610657a528689ba3ea44a73bd19c60f4c954271b790c71c2653
LD_LIBRARY_PATH=/usr/local/lib
PYTHON_PIP_VERSION=22.0.4
SHLVL=0
GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568
LANG=C.UTF-8
HOME=/root
PYTHON_SETUPTOOLS_VERSION=57.5.0
PWD=/
CLOUDSDK_CORE_DISABLE_PROMPTS=yes
PYTHON_VERSION=3.8.16
HOSTNAME=acbe81dff820

这是出错时的

env
输出

❯ docker run --rm --entrypoint /bin/bash dataflow -c 'env'
_=/usr/bin/env
PIP_NO_DEPS=True
GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=/etc/ssl/certs/ca-certificates.crt
PATH=/root/.local/bin/:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/gcloud/google-cloud-sdk/bin
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/1a96dc5acd0303c4700e02655aefd3bc68c78958/public/get-pip.py
PYTHON_GET_PIP_SHA256=d1d09b0f9e745610657a528689ba3ea44a73bd19c60f4c954271b790c71c2653
LD_LIBRARY_PATH=/usr/local/lib
PYTHON_PIP_VERSION=22.0.4
SHLVL=0
FLEX_TEMPLATE_PYTHON_EXTRA_PACKAGES=
FLEX_TEMPLATE_PYTHON_PY_OPTIONS=
FLEX_TEMPLATE_PYTHON_PY_FILE=/dataflow/template/beam.py
GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568
LANG=C.UTF-8
HOME=/root
PYTHON_SETUPTOOLS_VERSION=57.5.0
PWD=/dataflow/template
CLOUDSDK_CORE_DISABLE_PROMPTS=yes
PYTHON_VERSION=3.8.16
HOSTNAME=6556fc251ce4
最新问题
© www.soinside.com 2019 - 2024. All rights reserved.