我正在尝试使用 Azure blob 配置远程日志记录。
Airflow version: 1.10.2
Python: 3.6.5
Ubuntu: 18.04
以下是我所做的步骤:
错误如下:
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 382, in resolve
found = getattr(found, frag)
AttributeError: module 'airflow.utils.log' has no attribute 'wasb_task_handler'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 384, in resolve
self.importer(used)
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/log/wasb_task_handler.py", line 23, in <module>
from airflow.contrib.hooks.wasb_hook import WasbHook
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/contrib/hooks/wasb_hook.py", line 22, in <module>
from airflow.hooks.base_hook import BaseHook
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 28, in <module>
from airflow.models import Connection
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/models.py", line 86, in <module>
from airflow.utils.dag_processing import list_py_file_paths
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/dag_processing.py", line 49, in <module>
from airflow.settings import logging_class_path
ImportError: cannot import name 'logging_class_path'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 558, in configure
handler = self.configure_handler(handlers[name])
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 708, in configure_handler
klass = self.resolve(cname)
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 391, in resolve
raise v
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 384, in resolve
self.importer(used)
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/log/wasb_task_handler.py", line 23, in <module>
from airflow.contrib.hooks.wasb_hook import WasbHook
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/contrib/hooks/wasb_hook.py", line 22, in <module>
from airflow.hooks.base_hook import BaseHook
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 28, in <module>
from airflow.models import Connection
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/models.py", line 86, in <module>
from airflow.utils.dag_processing import list_py_file_paths
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/dag_processing.py", line 49, in <module>
from airflow.settings import logging_class_path
ValueError: Cannot resolve 'airflow.utils.log.wasb_task_handler.WasbTaskHandler': cannot import name 'logging_class_path'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/gsingh/venv/bin/airflow", line 21, in <module>
from airflow import configuration
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/__init__.py", line 36, in <module>
from airflow import settings, configuration as conf
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/settings.py", line 262, in <module>
logging_class_path = configure_logging()
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/logging_config.py", line 73, in configure_logging
raise e
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/logging_config.py", line 68, in configure_logging
dictConfig(logging_config)
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 795, in dictConfig
dictConfigClass(config).configure()
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 566, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'processor': Cannot resolve 'airflow.utils.log.wasb_task_handler.WasbTaskHandler': cannot import name 'logging_class_path'
我不确定我缺少哪个配置。有人遇到同样的问题吗
我遇到了同样的错误,但是如果我向上滚动,我可以看到在 ValueError 之前抛出了另一个异常。这是一个 PermissionError。
PermissionError: [Errno 13] Permission denied: '/usr/local/airflow/logs/scheduler'
我收到该错误的原因是因为我在运行 Airflow docker 容器之前没有创建最初的 3 个文件夹(dags、logs、plugins)。所以docker似乎已经自动创建了,但是权限是错误的。
修复步骤:
docker-compose down --volumes --remove-orphans
mkdir logs dags plugins
docker-compose up airflow-init
docker-compose up
sudo chown 50000:0 dags logs plugins
就我而言。
我尝试使用所有这些容器(依赖于这 3 个卷转发)运行官方
docker-compose.yml
,或者简单地将 airflow standalone
包装到单个容器中以进行调试。事实证明,卷是使用 root 所有权而不是气流创建的。
您需要安装azure包。
pip install 'apache-airflow[azure_blob_storage,azure_data_lake,azure_cosmos,azure_container_instances]
根据 updating.md
现在应该安装
pip install apache-airflow[azure]
但这对我不起作用。
对于 Linux 以外的操作系统,您可能会收到未设置 AIRFLOW_UID 的警告,但您可以安全地忽略它。您还可以在与 docker-compose.yaml 相同的文件夹中手动创建一个包含以下内容的 .env 文件,以消除警告:
AIRFLOW_UID=50000