气流 PYTHONPATH

Question

我发现了错误：

ModuleNotFoundError: No module named 'scripts.marketing_functions'

项目结构示例。

scripts/
├──__init__.py
├── marketing_functions.py
└── marketing/
    └── some_script.py  # Here my marketing_functions.py is used
dags/
    └── my_dag.py # Dag with bashOperator. it uses some_script.py with marketing_function.py import    

1. 我将脚本放入 /opt/bitnami/airflow/scripts 中。
2. 我将 dags 放入 /opt/bitnami/airflow/dags 中。
3. 我添加到 PYTHONPATH /opt/bitnami/airflow/scripts。我使用命令“airflow info”看到正确的 PYTHONPATH：

Paths info
airflow_home    | /opt/bitnami/airflow                                                                                                                                                                                 
system_path     | /opt/bitnami/common/bin:/opt/bitnami/python/bin:/opt/bitnami/postgresql/bin:/opt/bitnami/airflow/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin                               
python_path     | /opt/bitnami/python/bin:/opt/bitnami/airflow/scripts:/opt/bitnami/python/lib/python39.zip:/opt/bitnami/python/lib/python3.9:/opt/bitnami/python/lib/python3.9/lib-dynload:/opt/bitnami/python/lib/pyt
                | hon3.9/site-packages:/opt/bitnami/python/lib/python3.9/site-packages/setuptools-63.4.3-py3.9.egg:/opt/bitnami/python/lib/python3.9/site-packages/pip-21.3.1-py3.9.egg:/opt/bitnami/airflow/dags:/opt/
                | bitnami/airflow/config:/opt/bitnami/airflow/plugins                                                                                                                                                  
airflow_on_path | True 

Dag 代码示例：

import datetime

from airflow import DAG
from airflow.operators.bash import BashOperator

with DAG(
        dag_id="marketing_some_script",
        start_date=datetime.datetime(2023, 4, 16),
        schedule_interval="30 2 * * *",
) as dag:
    some_script = "/opt/bitnami/airflow/scripts/marketing/some_script.py"

    task_some_script = BashOperator(
        task_id="task_some_report",
        bash_command=f"python {some_script}"
    )
task_some_script

some_script.py 文件头

from datetime import datetime

import gspread
import pandas as pd
from airflow.providers.postgres.hooks.postgres import PostgresHook
from oauth2client.service_account import ServiceAccountCredentials

from scripts.marketing_functions import *

请。帮我找到解决方案。

UPD。即使我只使用 python 控制台我也有同样的问题：

I have no name!@0a8ff614c4b8:/$ python /opt/bitnami/airflow/scripts/marketing/some_script.py
Traceback (most recent call last):
  File "/opt/bitnami/airflow/scripts/marketing/export_marketing_daily_report.py", line 8, in <module>
    from scripts.marketing_functions import *
ModuleNotFoundError: No module named 'scripts.marketing_functions'

但是

I have no name!@0a8ff614c4b8:/$ echo $PYTHONPATH
/opt/bitnami/airflow/scripts

Answer 1

@mkrieger 帮助我发表了评论： “/opt/bitnami/airflow/scripts 内没有脚本。PYTHONPATH 应包含 /opt/bitnami/airflow 才能查找脚本”