我在将静态 Ceph 卷挂载到 K8s 时遇到这个问题。
MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph │
│ 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/test1-pv/gl │
│ obalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-1586083215,mds_namespace=myfs,_netdev] stderr: mount error 13 = Permission denied
但我不确定这样做是否正确。请指出正确的方向。这是我的用例：我想设置一个共享文件系统，可以从所有命名空间中的所有 Pod 访问该文件系统。并发写入操作不是一个大问题，因为大多数 Pod 将从这个共享位置读取，例如 Python 包等
重复使用同一个 PVC 是不可能的，因为它是一个命名空间对象。
我所做的是在 SubVolumeGroup 下的 Ceph 中创建一个静态卷，并为每个命名空间创建一个 pv-pvc 对，并期望它将访问 Ceph 卷中的相同文件。
这是我安装到 Pod 的卷：
ubuntu@host1:~$ ls -l /mnt/ceph/volumes/sharedvg/sharedvolume/
total 0
drwxrwxrwx 2 root root 0 Mar 9 11:22 8a370586-60e6-4ec7-9d5b-c8c7ce7786c6
ubuntu@host1:~$ ls -l /mnt/ceph/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6/
total 0
ubuntu@host1:~$
这里是 PV 和 PVC yaml 文件。我在 rook-ceph 命名空间的秘密“rook-csi-cephfs-provisioner”中复制了 adminID 和 adminKey 值。
apiVersion: v1
kind: Secret
metadata:
name: rook-csi-cephfs-static-provisioner
type: Opaque
data:
userID: "XXX"
userKey: "XXX"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: test1-pv
namespace: default
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 128Gi
csi:
driver: rook-ceph.cephfs.csi.ceph.com
nodeStageSecretRef:
name: rook-csi-cephfs-static-provisioner
namespace: default
volumeAttributes:
clusterID: rook-ceph
fsName: "myfs"
staticVolume: "true"
rootPath: /volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6
volumeHandle: test1-pv
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
claimRef:
name: test-pvc-1
namespace: default
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-pvc-1
namespace: default
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 128Gi
storageClassName: ""
volumeMode: Filesystem
volumeName: test1-pv
这是busybox部署yaml文件：
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-deployment
namespace: default
labels:
app: test
spec:
replicas: 1
strategy:
type: RollingUpdate
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
containers:
- name: test
image: busybox
imagePullPolicy: IfNotPresent
volumeMounts:
- name: test1
mountPath: /test1
command: ['sh', '-c', 'echo Container 1 is Running ; sleep 3600']
volumes:
- name: test1
persistentVolumeClaim:
claimName: test-pvc-1
这是 Pod 的日志：
Warning FailedMount 29m kubelet MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal desc = an error (exit status 32) │
│ occurred while running mount args: [-t ceph 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6 /var/lib/kube │
│ let/plugins/kubernetes.io/csi/pv/test1-pv/globalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-973267258,mds_namespace=myfs,_netdev] stderr: mount error 13 = │
│ Permission denied │
│ Warning FailedMount 27m kubelet MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal desc = an error (exit status 32) │
│ occurred while running mount args: [-t ceph 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6 /var/lib/kube │
│ let/plugins/kubernetes.io/csi/pv/test1-pv/globalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-2348945139,mds_namespace=myfs,_netdev] stderr: mount error 13 │
│ = Permission denied │
│ Warning FailedMount 25m kubelet MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal desc = an error (exit status 32) │
│ occurred while running mount args: [-t ceph 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6 /var/lib/kube │
│ let/plugins/kubernetes.io/csi/pv/test1-pv/globalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-3861388178,mds_namespace=myfs,_netdev] stderr: mount error 13 │
│ = Permission denied │
│ Warning FailedMount 23m kubelet MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal desc = an error (exit status 32) │
│ occurred while running mount args: [-t ceph 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6 /var/lib/kube │
│ let/plugins/kubernetes.io/csi/pv/test1-pv/globalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-4165129570,mds_namespace=myfs,_netdev] stderr: mount error 13 │
│ = Permission denied │
│ Warning FailedMount 7m14s (x10 over 34m) kubelet Unable to attach or mount volumes: unmounted volumes=[test1], unattached volumes=[test1 kube-api-access-fwr79]: tim │
│ ed out waiting for the condition │
│ Warning FailedMount 3m3s (x13 over 21m) kubelet (combined from similar events): MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal d │
│ esc = an error (exit status 32) occurred while running mount args: [-t ceph 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7 │
│ -9d5b-c8c7ce7786c6 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/test1-pv/globalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-2075406143,mds_namespace=myfs, │
│ _netdev] stderr: mount error 13 = Permission denied │
│ Warning FailedMount 27s (x2 over 29m) kubelet Unable to attach or mount volumes: unmounted volumes=[test1], unattached volumes=[kube-api-access-fwr79 test1]: tim │
│ ed out waiting for the condition
Pod 处于“ContainerCreating”状态。
有什么建议吗？谢谢
我最近遇到了完全相同的问题。
我要强调的错误消息中的主要关键点是
MountVolume.MountDevice failed for volume ... exit status 32 ...
<your_shared_volume_path> ... mount error 13 ... Permission denied
搜索这些基本关键字和错误代码使我找到了解释“权限被拒绝”错误的帖子。
首先，只关注mount命令的错误我发现了这个post的第一条评论的亮点：
凭据文件的权限
不幸的是，在这种情况下，当无法访问处于“ContainerCreating”状态的 pod 外壳时，无法检索更多日志。
其次，同时搜索这两个错误代码使我想到了解决方案提及
观察到的文件共享秘密不正确
我认为凭据更改不会这么晚，所以放在一起检查存储这些凭据的 Kubernetes 秘密。
更新 Kubernetes 秘密中的这些存储访问凭证解决了这个问题，使 pod 在 nextrestart 后进入“运行”状态。