containerd 无法从 nexus 中拉取图像

Question

我们已将 kops kubernetes 集群从 v1.19.2 (docker://19.3.15) 升级到 v1.24.11 (containerd://1.6.18)，但升级后我们无法从我们的 nexus 存储库中提取 docker 镜像。

报错：

Failed to pull image "myregistry.com/infra-tools/busybox:latest": rpc error: code = NotFound desc = failed to pull and unpack image "myregistry.com/infra-tools/busybox:latest": failed to resolve reference "myregistry.com/infra-tools/busybox:latest": myregistry.com/infra-tools/busybox:latest: not found

图像存在，我可以使用 docker 拉/推。

来自工作节点：

ctr --debug images pull myregistry.com/infra-tools/busybox:latest
DEBU[0000] fetching                                      image="myregistry.com/infra-tools/busybox:latest"
DEBU[0000] resolving                                     host=myregistry.com
DEBU[0000] do request                                    host=myregistry.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.6 request.method=HEAD url="https://myregistry.com/v2/infra-tools/busybox/manifests/latest"
DEBU[0000] fetch response received                       host=myregistry.com response.header.cache-control="post-check=0, pre-check=0" response.header.connection=keep-alive response.header.content-length=2031 response.header.content-security-policy="sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation" response.header.content-type=text/html response.header.date="Fri, 10 Mar 2023 10:54:29 GMT" response.header.expires=0 response.header.pragma=no-cache response.header.server=nginx response.header.x-content-type-options=nosniff response.status="404 Not Found" url="https://myregistry.com/v2/infra-tools/busybox/manifests/latest"
INFO[0000] trying next host - response was http.StatusNotFound  host=myregistry.com
ctr: failed to resolve reference "myregistry.com/infra-tools/busybox:latest": myregistry.com/infra-tools/busybox:latest: not found

有人面临同样的问题吗？

Answer 1

这个问题出现在nexus 3.2.0之前的版本，升级到新版本后问题消失

原因：
containerd 在拉取镜像时提交 HTTP HEAD 请求，但旧的 nexus 版本返回 404 响应 HEAD 请求。

重现问题模拟请求：

curl -v -X HEAD -H "Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" "https://<docker-registry>/v2/<image-name>/manifests/<image-tag>"