K8S无法将AWS EBS作为pod的持久卷安装

Question

Question

请告知无法在pod中安装AWS EBS卷的错误原因。

journalctl -b -f -u kubelet

1480 kubelet.go:1625] Unable to mount volumes for pod "nginx_default(ddc938ee-edda-11e7-ae06-06bb783bb15c)": timeout expired waiting for volumes to attach/mount for pod "default"/"nginx". list of unattached/unmounted volumes=[ebs]; skipping pod
1480 pod_workers.go:186] Error syncing pod ddc938ee-edda-11e7-ae06-06bb783bb15c ("nginx_default(ddc938ee-edda-11e7-ae06-06bb783bb15c)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"nginx". list of unattached/unmounted volumes=[ebs]
1480 reconciler.go:217] operationExecutor.VerifyControllerAttachedVolume started for volume "pv-ebs" (UniqueName: "kubernetes.io/aws-ebs/vol-0d275986ce24f4304") pod "nginx" (UID: "ddc938ee-edda-11e7-ae06-06bb783bb15c")
1480 nestedpendingoperations.go:263] Operation for "\"kubernetes.io/aws-ebs/vol-0d275986ce24f4304\"" failed. No retries permitted until 2017-12-31 03:34:03.644604131 +0000 UTC m=+6842.543441523 (durationBeforeRetry 2m2s). Error: "Volume not attached according to node status for volume \"pv-ebs\" (UniqueName: \"kubernetes.io/aws-ebs/vol-0d275986ce24f4304\") pod \"nginx\" (UID: \"ddc938ee-edda-11e7-ae06-06bb783bb15c\") "

Steps

1. 在AWS中使用kubeadm（没有EBS卷装，pod工作）部署K8S 1.9（us-west-1和AZ是us-west-1b）。
2. 根据Kubernetes - Cloud Providers和kubelets failing to start when using 'aws' as cloud provider配置IAM角色。
3. 根据Easily Replace or Attach an IAM Role to an Existing EC2 Instance by Using the EC2 Console将IAM角色分配给EC2实例。
4. 在清单中部署PV / PVC / POD。

来自kubectl的状态：

kubectl得到

NAME      READY     STATUS              RESTARTS   AGE       IP        NODE
nginx     0/1       ContainerCreating   0          29m       <none>    ip-172-31-1-43.us-west-1.compute.internal

NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM             STORAGECLASS   REASON    AGE
pv/pv-ebs   5Gi        RWO            Recycle          Bound     default/pvc-ebs                            33m

NAME          STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc/pvc-ebs   Bound     pv-ebs    5Gi        RWO                           33m

kubectl描述pod nginx

Events:
  Type     Reason                 Age                From                                                Message
  ----     ------                 ----               ----                                                -------
  Normal   Scheduled              27m                default-scheduler                                   Successfully assigned nginx to ip-172-31-1-43.us-west-1.compute.internal
  Normal   SuccessfulMountVolume  27m                kubelet, ip-172-31-1-43.us-west-1.compute.internal  MountVolume.SetUp succeeded for volume "default-token-dt698"
  Warning  FailedMount            6s (x12 over 25m)  kubelet, ip-172-31-1-43.us-west-1.compute.internal  Unable to mount volumes for pod "nginx_default(ddc938ee-edda-11e7-ae06-06bb783bb15c)": timeout expired waiting for volumes to attach/mount for pod "default"/"nginx".                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Warning  FailedMount            6s (x12 over 25m)  kubelet, ip-172-31-1-43.us-west-1.compute.internal  Unable to mount volumes for pod "nginx_default(ddc938ee-edda-11e7-ae06-06bb783bb15c)": timeout expired waiting for volumes to attach/mount for pod "default"/"nginx".

表现

---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-ebs
  labels:
    type: amazonEBS
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  awsElasticBlockStore:
    volumeID: vol-0d275986ce24f4304
    fsType: ext4
  persistentVolumeReclaimPolicy: Recycle
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-ebs
  labels:
    type: amazonEBS
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
kind: Pod
apiVersion: v1
metadata:
  name: nginx
spec:
  containers:
    - name: myfrontend
      image: nginx
      volumeMounts:
      - mountPath: "/var/www/html"
        name: ebs
  volumes:
    - name: ebs
      persistentVolumeClaim:
        claimName: pvc-ebs

现在政策

环境

$ kubectl version -o json
{
  "clientVersion": {
    "major": "1",
    "minor": "9",
    "gitVersion": "v1.9.0",
    "gitCommit": "925c127ec6b946659ad0fd596fa959be43f0cc05",
    "gitTreeState": "clean",
    "buildDate": "2017-12-15T21:07:38Z",
    "goVersion": "go1.9.2",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "serverVersion": {
    "major": "1",
    "minor": "9",
    "gitVersion": "v1.9.0",
    "gitCommit": "925c127ec6b946659ad0fd596fa959be43f0cc05",
    "gitTreeState": "clean",
    "buildDate": "2017-12-15T20:55:30Z",
    "goVersion": "go1.9.2",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}

$ cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core) 

EC2

EBS

Answer 1

解

找到了说明如何配置AWS云提供程序的文档。

• K8S AWS Cloud Provider Notes

脚步

1. 使用KubernetesCluster = $ {kubernetes cluster name}标记EC2实例和SG。如果用kubeadm创建，它就像Ability to configure user and cluster name in AdminKubeConfigFile中的kubernetes一样
2. 运行kubeadm init --config kubeadm.yaml。

kubeadm.yaml (Ansible template)

kind: MasterConfiguration
apiVersion: kubeadm.k8s.io/v1alpha1
api:
  advertiseAddress: {{ K8S_ADVERTISE_ADDRESS }}
networking:
  podSubnet:        {{ K8S_SERVICE_ADDRESSES }}
cloudProvider:      {{ K8S_CLOUD_PROVIDER }}

结果

$ journalctl -b -f CONTAINER_ID=$(docker ps | grep k8s_kube-controller-manager | awk '{ print $1 }')

Jan 02 04:48:28 ip-172-31-4-117.us-west-1.compute.internal dockerd-current[8063]: I0102 04:48:28.752141
1 reconciler.go:287] attacherDetacher.AttachVolume started for volume "kuard-pv" (UniqueName: "kubernetes.io/aws-ebs/vol-0d275986ce24f4304") from node "ip-172-3
Jan 02 04:48:39 ip-172-31-4-117.us-west-1.compute.internal dockerd-current[8063]: I0102 04:48:39.309178
1 operation_generator.go:308] AttachVolume.Attach succeeded for volume "kuard-pv" (UniqueName: "kubernetes.io/aws-ebs/vol-0d275986ce24f4304") from node "ip-172-

$ kubectl describe pod kuard
...
Volumes:
  kuard-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  kuard-pvc
    ReadOnly:   false

$ kubectl describe pv kuard-pv
Name:            kuard-pv
Labels:          failure-domain.beta.kubernetes.io/region=us-west-1
                 failure-domain.beta.kubernetes.io/zone=us-west-1b
                 type=amazonEBS
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"type":"amazonEBS"},"name":"kuard-pv","namespace":""},"spec":{"acce...
                 pv.kubernetes.io/bound-by-controller=yes
StorageClass:    
Status:          Bound
Claim:           default/kuard-pvc
Reclaim Policy:  Retain
Access Modes:    RWO
Capacity:        5Gi
Message:         
Source:
    Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
    VolumeID:   vol-0d275986ce24f4304
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
Events:         <none>

$ kubectl version -o json
{
  "clientVersion": {
    "major": "1",
    "minor": "9",
    "gitVersion": "v1.9.0",
    "gitCommit": "925c127ec6b946659ad0fd596fa959be43f0cc05",
    "gitTreeState": "clean",
    "buildDate": "2017-12-15T21:07:38Z",
    "goVersion": "go1.9.2",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "serverVersion": {
    "major": "1",
    "minor": "9",
    "gitVersion": "v1.9.0",
    "gitCommit": "925c127ec6b946659ad0fd596fa959be43f0cc05",
    "gitTreeState": "clean",
    "buildDate": "2017-12-15T20:55:30Z",
    "goVersion": "go1.9.2",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}