使用 azurefile csi 在 AKS 上部署 Bitbucket 时，本地主页需要 POSIX

Question

我正在按照 Atlassian 安装指南将 Bitbucket 部署到 AKS 集群：安装 Atlassian DC Helm Charts 并使用 azurefile-csi 为本地家庭动态创建 PV 和 PVC。

部署 Helm Chart 后，带有 Bitbucket 的 pod 会启动，但由于以下错误而从未达到就绪状态：

2024-01-08 17:52:21,922 ERROR [spring-startup]  c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started


org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'meshHealthcheck' defined in URL [jar:file:/opt/atlassian/bitbucket/app/WEB-INF/lib/bitbucket-git-mesh-8.9.8.jar!/com/atlassian/stash/internal/scm/git/mesh/MeshHealthcheck.class]:

 Unsatisfied dependency expressed through constructor parameter 0; 

 nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'meshClient' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: 

 Unsatisfied dependency expressed through method 'meshClient' parameter 2; 

 nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'routePropagatingClientInterceptor' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: 

 Unsatisfied dependency expressed through method 'routePropagatingClientInterceptor' parameter 0; 

 nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'meshRouter' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: 

 Unsatisfied dependency expressed through method 'meshRouter' parameter 5; 

 nested exception is org.springframework.beans.factory.BeanCreationException:

 Error creating bean with name 'sidecarManager': 

 Invocation of init method failed; 

 nested exception is java.nio.file.FileSystemException: 

 /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem:

 Operation not permitted

 ...

 at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160)

 ... 5 common frames omitted

Caused by: java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted

 at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)

 at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)

 at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)

 at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:254)

 at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:276)

 at java.base/java.nio.file.Files.setPosixFilePermissions(Files.java:2080)

 at com.atlassian.bitbucket.mesh.util.KeyUtils.write(KeyUtils.java:382)

 at com.atlassian.bitbucket.mesh.util.KeyUtils.writePublicKey(KeyUtils.java:260)

 at com.atlassian.stash.internal.scm.git.mesh.MeshLauncher.start(MeshLauncher.java:121)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.startSidecar(DefaultSidecarManager.java:272)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.internalStart(DefaultSidecarManager.java:223)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.start(DefaultSidecarManager.java:178)

 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

 ... 5 common frames omitted

检查堆栈跟踪时，我意识到错误：

Caused by: java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted

执行时抛出：

at java.base/java.nio.file.Files.setPosixFilePermissions(Files.java:2080)
at com.atlassian.bitbucket.mesh.util.KeyUtils.write(KeyUtils.java:382)

 at com.atlassian.bitbucket.mesh.util.KeyUtils.writePublicKey(KeyUtils.java:260)

 at com.atlassian.stash.internal.scm.git.mesh.MeshLauncher.start(MeshLauncher.java:121)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.startSidecar(DefaultSidecarManager.java:272)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.internalStart(DefaultSidecarManager.java:223)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.start(DefaultSidecarManager.java:178)

对于本地主页，我使用动态创建 azure 文件中的内置存储类 azurefile-csi 动态创建 pv 和 pvc。

对于本地家庭，我在 values.yaml

上传递以下值

localHome:

# Dynamic provisioning of local-home using the K8s Storage Classes
#
# https://kubernetes.io/docs/concepts/storage/persistent-volumes/#dynamic
# https://atlassian.github.io/data-center-helm-charts/examples/storage/aws/LOCAL_STORAGE/
#
persistentVolumeClaim:

# -- If 'true', then a 'PersistentVolume' and 'PersistentVolumeClaim' will be dynamically
# created for each pod based on the 'StorageClassName' supplied below.
#
create: true

# -- Specify the name of the 'StorageClass' that should be used for the local-home
# volume claim.
#
storageClassName: "azurefile-csi"

# -- Specifies the standard K8s resource requests and/or limits for the local-home
# volume claims.
#
resources:
requests:
storage: 1Gi

我也尝试提出索赔：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-local-home
namespace: bitbucket
spec:
accessModes:
- ReadWriteMany
storageClassName: "azurefile-csi" 
resources:
requests:
storage: 1Gi

并将其传递为：

localHome:

# Dynamic provisioning of local-home using the K8s Storage Classes
#
# https://kubernetes.io/docs/concepts/storage/persistent-volumes/#dynamic
# https://atlassian.github.io/data-center-helm-charts/examples/storage/aws/LOCAL_STORAGE/
#
persistentVolumeClaim:

# -- If 'true', then a 'PersistentVolume' and 'PersistentVolumeClaim' will be dynamically
# created for each pod based on the 'StorageClassName' supplied below.
#
create: false

...

# -- Static provisioning of local-home using K8s PVs and PVCs
#
# NOTE: Due to the ephemeral nature of pods this approach to provisioning volumes for
# pods is not recommended. Dynamic provisioning described above is the prescribed
# approach.
#
# When 'persistentVolumeClaim.create' is 'false', then this value can be used to define
# a standard K8s volume that will be used for the local-home volume(s). If not defined,
# then an 'emptyDir' volume is utilised. Having provisioned a 'PersistentVolume', specify
# the bound 'persistentVolumeClaim.claimName' for the 'customVolume' object.
# https://kubernetes.io/docs/concepts/storage/persistent-volumes/#static
#
customVolume:
 persistentVolumeClaim:
 claimName: "pvc-local-home"
 

两者都导致了上面所示的相同错误。

Bitbucket 尝试在 Azure 文件中设置 POSIX 时出现问题，因为 Azure 文件 CSI 与 POSIX 不兼容，因此必须使用 NFS 文件共享。

使用本地主目录的emptyDir运行helm安装时，不会发生此问题。

我在安装指南的第 5 节配置持久存储中找不到任何提及 POSIX 的内容。

我是否做错了什么，或者需要符合 POSIX 的卷才能在 Kubernetes 上正确运行 Bitbucket，而文档中没有提到这一点？

Answer 1

Azure 文件缺乏完整的 POSIX 合规性，您可以改用 Azure 磁盘或设置 NFS 服务器。 Azure 磁盘更符合 POSIX 标准，但只能安装到单个节点，这使得它们适合单节点设置，同时设置 NFS 服务器允许多节点访问，同时保持 POSIX 合规性。 按照guide，请修改您从 helm get values bitbucket -n 获得的values.yaml 中的持久性部分。以及 Azure 磁盘详细信息

persistence:
  localHome:
    enabled: true
    storageClassName: "your AzureDiskstorageclass"
    size: "10Gi"
  sharedHome:
    enabled: true
    storageClassName: "your AzureDiskstorageclass"
    size: "20Gi"

之后，部署bitbucket。

helm install bitbucket atlassian-data-center/bitbucket --namespace default --values values.yaml 

kubectl get pods

您还可以尝试设置 NFS 并相应更新 bitbucket 的 value.yaml 文件引用 NFS

localHome:
  customVolume:
    persistentVolumeClaim:
      claimName: bitbucket-nfs-pvc

然后使用重新部署 bitbucket

helm upgrade <release-name> atlassian-data-center/bitbucket --namespace <namespace> --values values.yaml

参考文档： Bitbucket 设置指南 Bitbucket 的 NFS