Airflow 无法使用 gcsfuse 挂载 google 云存储桶

问题描述 投票：0回答：1

我想将 Google Cloud Bucket 安装到我的气流环境中，以便我可以在该 GCS Bucket 上读取和写入文件。我正在使用 Cloud Composer 2（composer-2.1.14-airflow-2.5.1 图像）

在气流中，我创建了一个 DAG 来运行以下 bash 脚本

#!/bin/bash

BUCKET="my-bucket"
MOUNT_DIR="/home/airflow/gcs/data/my-bucket"

#Creating $MOUNT_DIR directory & granting it permissions
mkdir -p $MOUNT_DIR
sudo chmod g+w $MOUNT_DIR

# Mounting GCS Bucket
gcsfuse --foreground --debug_fuse --debug_fs --debug_gcs --debug_http -o nonempty $BUCKET $MOUNT_DIR

以下是 Airflow 的日志：

[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - Start gcsfuse/0.42.3 (Go version go1.19.5) for app "" using mount point: /home/airflow/gcs/data/my-bucket
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - Opening GCS connection...
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - Creating a mount at "/home/airflow/gcs/data/my-bucket"
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - Creating a new server...
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - Set up root directory for bucket my-bucket
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - gcs: Req              0x0: <- ListObjects("")
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - gcs: Req              0x0: -> ListObjects("") (131.395831ms): OK
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - Mounting file system "my-bucket"...
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - fuse_debug: Beginning the mounting kickoff process
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - fuse_debug: Parsing fuse file descriptor
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - fuse_debug: Preparing for direct mounting
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - fuse_debug: Directmount failed. Trying fallback.
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - fuse_debug: Creating a socket pair
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - fuse_debug: Creating files to wrap the sockets
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - fuse_debug: Starting fusermount/os mount
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - /usr/bin/fusermount: fuse device not found, try 'modprobe fuse' first
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - Error while mounting gcsfuse: mountWithConn: Mount: mount: running /usr/bin/fusermount: exit status 1
[2023-09-20, 11:46:39 PDT] {subprocess.py:93} INFO - mountWithArgs: mountWithConn: Mount: mount: running /usr/bin/fusermount: exit status 1

我已经通过运行以下命令验证了 Airflow 可以访问存储桶，并且我看到了存储桶中的文件列表：

gsutil ls gs://$BUCKET

我什至尝试运行以下命令，但仍然遇到与上面相同的错误：

sudo mount -t gcsfuse -o rw,user $BUCKET $MOUNT_DIR

我已经参考了以下内容和其他一些页面，但我仍然无法安装它：

更新： 我将composer环境更新为composer-2.4.2-airflow-2.5.3，但仍然看到以下错误：

[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"INFO","severity":"INFO","message":"Start gcsfuse/1.0.1 (Go version go1.20.5) for app \"\" using mount point:/home/airflow/gcs/data/my-bucket\n","timestampSeconds":1695254138,"timestampNanos":83062812}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"INFO","severity":"INFO","message":"Opening GCS connection...\n","timestampSeconds":1695254138,"timestampNanos":83799366}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a mount at \"/home/airflow/gcs/data/datavant/my-bucket\"\n","timestampSeconds":1695254138,"timestampNanos":87562370}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a new server...\n","timestampSeconds":1695254138,"timestampNanos":87589651}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"INFO","severity":"INFO","message":"Set up root directory for bucket my-bucket\n","timestampSeconds":1695254138,"timestampNanos":87599362}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req              0x0: \u003c- ListObjects(\"\")\n","timestampSeconds":1695254138,"timestampNanos":87612220}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req              0x0: -\u003e ListObjects(\"\") (106.665835ms): OK\n","timestampSeconds":1695254138,"timestampNanos":194287578}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"INFO","severity":"INFO","message":"Mounting file system \"my-bucket\"...\n","timestampSeconds":1695254138,"timestampNanos":194342795}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Beginning the mounting kickoff process\n","timestampSeconds":1695254138,"timestampNanos":194916407}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Parsing fuse file descriptor\n","timestampSeconds":1695254138,"timestampNanos":194977401}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Preparing for direct mounting\n","timestampSeconds":1695254138,"timestampNanos":194984093}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Directmount failed. Trying fallback.\n","timestampSeconds":1695254138,"timestampNanos":195003380}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Creating a socket pair\n","timestampSeconds":1695254138,"timestampNanos":195238613}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Creating files to wrap the sockets\n","timestampSeconds":1695254138,"timestampNanos":195260643}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Starting fusermount/os mount\n","timestampSeconds":1695254138,"timestampNanos":195270306}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - /usr/bin/fusermount: fuse device not found, try 'modprobe fuse' first
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - {"name":"root","levelname":"INFO","severity":"INFO","message":"Error while mounting gcsfuse: mountWithConn: Mount: mount: running /usr/bin/fusermount: exit status 1\n","timestampSeconds":1695254138,"timestampNanos":198067902}
[2023-09-20, 16:55:38 PDT] {subprocess.py:93} INFO - mountWithArgs: mountWithConn: Mount: mount: running /usr/bin/fusermount: exit status 1
google-cloud-storage google-cloud-composer airflow-2.x gcsfuse
1个回答
0
投票

看来问题不是来自 gcsfuse 方面，而是保险丝安装的问题。您可以尝试这个解决方案吗https://forum.odroid.com/viewtopic.php?p=314535&sid=5decaed4623a9aa6c71619ac677d3bf2#p314535

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.