我根据this将我的Airflow设置配置为使用systemd运行。它好几天,但它已经抛出了一些我无法弄清楚如何解决的错误。运行sudo systemctl start airflow-webserver.service
并没有真正做任何事情,但运行airflow webserver
工作(但是,我们的目的需要使用systemd)。
要了解错误是什么,我运行sudo systemctl status airflow-webserver.service
,它会给出以下状态和错误:
Feb 20 18:54:43 ip-172-31-25-17.ec2.internal airflow[19660]: [2019-02-20 18:54:43,774] {models.py:258} INFO - Filling up the DagBag from /home/ec2-user/airflow/dags
Feb 20 18:54:43 ip-172-31-25-17.ec2.internal airflow[19660]: /home/ec2-user/airflow/dags/statcan_1410009501.py:33: SyntaxWarning: name 'pg_hook' is assigned to before global declaration
Feb 20 18:54:43 ip-172-31-25-17.ec2.internal airflow[19660]: global pg_hook
Feb 20 18:54:43 ip-172-31-25-17.ec2.internal airflow[19660]: /usr/lib/python2.7/site-packages/airflow/utils/helpers.py:346: DeprecationWarning: Importing 'PythonOperator' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operat...irely in Airflow 2.0.
Feb 20 18:54:43 ip-172-31-25-17.ec2.internal airflow[19660]: DeprecationWarning)
Feb 20 18:54:43 ip-172-31-25-17.ec2.internal airflow[19660]: /usr/lib/python2.7/site-packages/airflow/utils/helpers.py:346: DeprecationWarning: Importing 'BashOperator' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator...irely in Airflow 2.0.
Feb 20 18:54:43 ip-172-31-25-17.ec2.internal airflow[19660]: DeprecationWarning)
Feb 20 18:54:44 ip-172-31-25-17.ec2.internal airflow[19660]: [2019-02-20 18:54:44,528] {settings.py:174} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
Feb 20 18:54:45 ip-172-31-25-17.ec2.internal airflow[19660]: [2019-02-20 18:54:45 +0000] [19733] [INFO] Starting gunicorn 19.9.0
Feb 20 18:54:45 ip-172-31-25-17.ec2.internal airflow[19660]: Error: /run/airflow doesn't exist. Can't create pidfile.
调度程序似乎运行正常,运行systemctl status airflow-scheduler.service
和journalctl -f
后验证。
这是以下systemd文件的设置:
/US人/礼拜/system的/system/airflow-webserver.service
[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
[Service]
EnvironmentFile=/etc/sysconfig/airflow
User=ec2-user
Type=simple
ExecStart=/bin/airflow scheduler
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target
/etc/tmp files.的/airflow.conf
D /run/airflow 0755 airflow airflow
的/ etc / SYSCONFIG /气流
AIRFLOW_CONFIG= $AIRFLOW_HOME/airflow.cfg
AIRFLOW_HOME= /home/ec2-user/airflow
在此错误之前,我将我的气流安装从root移动到主目录。不确定它是否会影响我的设置,但如果相关则将其放在这里。
任何人都可以提供任何解释错误以及如何解决它?我尽力将systemd配置为尽可能接近指令,但也许我错过了什么?
编辑2:
对不起,我粘贴了错误的代码。所以这是我的airflow-webserver.service的代码
[Unit]
Description=Airflow webserver daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
[Service]
EnvironmentFile=/etc/sysconfig/airflow
User=ec2-user
Type=simple
ExecStart=/bin/airflow webserver --pid /run/airflow/webserver.pid
Restart=on-failure
RestartSec=5s
PrivateTmp=true
[Install]
WantedBy=multi-user.target
我也遇到了这个问题,并且能够通过在[Service]
单元文件中的airflow-webserver.service
下提供运行时目录参数来解决此问题:
[Service]
RuntimeDirectory=airflow
RuntimeDirectoryMode=0775
我无法弄清楚如何单独使用/etc/tmpfiles.d/airflow.conf
。
看起来您正在运行调度程序而不是Web服务器:
ExecStart=/bin/airflow scheduler
您可能想要执行以下操作:
ExecStart=/bin/airflow webserver -p 8080 --pid /run/airflow/webserver.pid
也许您只是复制粘贴错误的文件,在这种情况下共享正确的文件(airflow-webserver.service),以便我们可以帮助您解决此问题。
配置文件/etc/tmpfiles.d/airflow.conf
在启动时由systemd-tmpfiles-setup
服务使用。因此,服务器重启应该创建/ run / airflow目录。根据https://github.com/systemd/systemd/issues/8684,无法重新启动此服务。
正如上面的链接所示,在将airflow.conf
复制到/etc/tmpfiles.d/
之后,只需运行sudo systemd-tmpfiles --create
并创建/run/airflow
。