所有 Airflow 作业在本地失败错误：“任务已退出，返回代码 Negsignal.SIGABRT”

Question

我遇到一个问题，无论我运行什么 Airflow 作业，它都会失败并出现错误：

[2024-04-03, 15:26:57 PDT] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: local_to_s3_dag.local_to_s3_task manual__2024-04-03T22:26:53.513679+00:00 [queued]>
[2024-04-03, 15:26:57 PDT] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: local_to_s3_dag.local_to_s3_task manual__2024-04-03T22:26:53.513679+00:00 [queued]>
[2024-04-03, 15:26:57 PDT] {taskinstance.py:1308} INFO - Starting attempt 1 of 1
[2024-04-03, 15:26:57 PDT] {taskinstance.py:1327} INFO - Executing <Task(PythonOperator): local_to_s3_task> on 2024-04-03 22:26:53.513679+00:00
[2024-04-03, 15:26:57 PDT] {standard_task_runner.py:57} INFO - Started process 9751 to run task
[2024-04-03, 15:26:57 PDT] {standard_task_runner.py:84} INFO - Running: ['airflow', 'tasks', 'run', 'local_to_s3_dag', 'local_to_s3_task', 'manual__2024-04-03T22:26:53.513679+00:00', '--job-id', '3', '--raw', '--subdir', 'DAGS_FOLDER/local_to_s3_dag.py', '--cfg-path', '/var/folders/z7/50r70c7n2klfyy8qsv4fs97c0000gn/T/tmpbtxeqy6d']
[2024-04-03, 15:26:57 PDT] {standard_task_runner.py:85} INFO - Job 3: Subtask local_to_s3_task
[2024-04-03, 15:26:57 PDT] {task_command.py:410} INFO - Running <TaskInstance: local_to_s3_dag.local_to_s3_task manual__2024-04-03T22:26:53.513679+00:00 [running]> on host nicholass-macbook-pro-2.local
[2024-04-03, 15:26:57 PDT] {taskinstance.py:1545} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='NAME' AIRFLOW_CTX_DAG_ID='local_to_s3_dag' AIRFLOW_CTX_TASK_ID='local_to_s3_task' AIRFLOW_CTX_EXECUTION_DATE='2024-04-03T22:26:53.513679+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-04-03T22:26:53.513679+00:00'
[2024-04-03, 15:26:57 PDT] {logging_mixin.py:149} INFO - Got all filenames
[2024-04-03, 15:26:57 PDT] {local_task_job_runner.py:225} INFO - Task exited with return code Negsignal.SIGABRT
[2024-04-03, 15:26:57 PDT] {taskinstance.py:2653} INFO - 0 downstream tasks scheduled from follow-on schedule check

起初我认为这是我正在做的工作所特有的，因为我读到错误代码意味着任务出了问题，但现在即使我使用一个简单的任务进行测试，例如将数据从本地计算机移动到 S3 存储桶，作业将会失败。这是我在下面测试的 dag：

from datetime import datetime
from airflow import DAG
from pytz import timezone
from airflow.operators.python import PythonOperator
import boto3
# from airflow.providers.amazon.aws.transfers.local_to_s3 import LocalToS3Operator

def upload_file_to_s3(filename, bucket_name, s3_key):
    s3 = boto3.client('s3')
    s3.upload_file(filename, bucket_name, s3_key)
    print(f"File {filename} uploaded to S3 bucket {bucket_name} with key {s3_key}")

def upload_file_to_s3_task(**kwargs):
    filename = kwargs.get('filename')
    bucket_name = kwargs.get('bucket_name')
    s3_key = kwargs.get('s3_key')
    print('Got all filenames')
    upload_file_to_s3(filename, bucket_name, s3_key)

pst = timezone('US/Pacific')

default_args = {
    'owner': 'Name',
    'start_date': datetime(2024, 1, 29, 8, 53, tzinfo=pst),
    'end_date': datetime(2025, 1, 31, 8, 54, tzinfo=pst)
}

with DAG(dag_id="local_to_s3_dag",
         default_args=default_args,
         description="Takes local CSV and moves to S3 bucket",
         schedule="@weekly") as dag:
             
    upload_task = PythonOperator(
    task_id='local_to_s3_task',
    python_callable=upload_file_to_s3_task,
    op_kwargs={
        'filename': '/Users/myuser/Desktop/filepath', 
        'bucket_name': 'bucket',
        's3_key': 'data.csv'  
    }
    )


    upload_task

我已经排查了几个小时的问题，但不知道问题是什么，有人可以帮助我吗？

我尝试卸载并重新安装airflow，更新所有相关的库，并使用不同的DAGS来查看它们是否有效。

Answer 1

我也遇到了类似的问题，但运气不佳。其他线程表明这可能是与空间相关的问题，但这似乎也不是问题。很想听听别人的想法