Docker 构建 PyFlink 容器时遇到的问题

Question

有人在 Mac m1 机器上构建 PyFlink Docker 容器时遇到同样的问题吗？ 在 apache-flink 和 apache-flink-libraries 的 pip3 安装过程中，构建在步骤 4/6 失败。获取有关 Include 文件夹的所有时间错误应该位于“/opt/java/openjdk/include”，但不存在。 Contaner 中似乎没有正确安装 JDK，这是不真实的。

升级/安装 JDK 没有帮助。是容器本身的问题吗？ 我不喜欢使用 apache/flink docker 镜像存储库，因为只有 amd64 镜像，而且它是赞助的 OSS。

Dockerfile：

FROM flink:1.17.1

# Install JDK and essential libraries
RUN apt-get update -y && \
    apt-get install -y openjdk-11-jdk build-essential libssl-dev zlib1g-dev libbz2-dev libffi-dev lzma liblzma-dev bash

# Install Python 3.10.13
RUN set -ex; \
    wget https://www.python.org/ftp/python/3.10.13/Python-3.10.13.tgz && \
    tar -xvf Python-3.10.13.tgz && \
    cd Python-3.10.13 && \
    ./configure --without-tests --enable-shared && \
    make -j6 && \
    make install && \
    ldconfig /usr/local/lib && \
    cd .. && rm -f Python-3.10.13.tgz && rm -rf Python-3.10.13 && \
    ln -s /usr/local/bin/python3 /usr/local/bin/python

RUN pip3 install --upgrade pip

# Install Apache Flink
COPY apache-flink*.tar.gz /
RUN pip3 install --no-cache-dir /apache-flink-libraries*.tar.gz -v
RUN pip3 install --no-cache-dir /apache-flink*.tar.gz -v

# Download connector libraries
RUN wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-json/1.17.1/flink-json-1.17.1.jar; \
    wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka/1.17.1/flink-sql-connector-kafka-1.17.1.jar; \
    wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-elasticsearch7/3.0.1-1.17/flink-sql-connector-elasticsearch7-3.0.1-1.17.jar;

WORKDIR /opt/flink

错误日志如下：

18.98   WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
19.03   Installing build dependencies: finished with status 'done'
19.03   Getting requirements to build wheel: started
19.03   Running command Getting requirements to build wheel
19.10   Include folder should be at '/opt/java/openjdk/include' but doesn't exist. Please check you've installed the JDK properly.
19.12   error: subprocess-exited-with-error
19.12   
19.12   × Getting requirements to build wheel did not run successfully.
19.12   │ exit code: 255
19.12   ╰─> See above for output.
19.12   
19.12   note: This error originates from a subprocess, and is likely not a problem with pip.
19.12   full command: /usr/local/bin/python3.10 /usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpwujq3uju
19.12   cwd: /tmp/pip-install-71b1sm0q/pemja_2bee4b6bba71423b9a43f21b71487474
19.12   Getting requirements to build wheel: finished with status 'error'
19.12 error: subprocess-exited-with-error
19.12 
19.12 × Getting requirements to build wheel did not run successfully.
19.12 │ exit code: 255
19.12 ╰─> See above for output.
19.12 
19.12 note: This error originates from a subprocess, and is likely not a problem with pip.
------
dockerfile:24
--------------------
  22 |     COPY apache-flink*.tar.gz /
  23 |     RUN pip3 install --no-cache-dir /apache-flink-libraries*.tar.gz -v
  24 | >>> RUN pip3 install --no-cache-dir /apache-flink*.tar.gz -v
  25 |     
  26 |     # Download connector libraries
--------------------
ERROR: failed to solve: process "/bin/sh -c pip3 install --no-cache-dir /apache-flink*.tar.gz -v" did not complete successfully: exit code: 1

Answer 1

我在相同的情况下遇到了相同的问题（

17.1

，M1）。我认为这是一个 ARM 问题，因为我能够在 x86 机器上成功构建完全相同的 Dockerfile（如下）。

我在尝试构建示例 pyflink Dockerfile 时也复制了同样的问题。

我的 Dockerfile

FROM flink:1.17

ENV \
  PYTHON_VER=3.10.12-1~22.04.2 \
  PIP_VER=22.0.2+dfsg-1ubuntu0.3 \
  JDK_VER=11.0.20.1+1-0ubuntu1~22.04

SHELL ["/bin/bash", "-ceuxo", "pipefail"]

RUN apt-get update -y && \
  apt-get install -y --no-install-recommends \
    python3.10=${PYTHON_VER} \
    python3-pip=${PIP_VER} \
  && \
  apt-get clean && \
  rm -rf /var/lib/apt/lists/*

RUN pip3 install "apache-flink>=1.17.1,<1.18" && \
  pip cache purge

USER flink
RUN mkdir /opt/flink/usrlib
COPY python_demo.py /opt/flink/usrlib/python_demo.py