如何使用 Gitlab CI 在两个阶段的工作期间保持 docker 镜像构建？

问题描述 投票：0回答：2

我在 EC2 上使用 Gitlab runner 来连接 ECS 上的 

build
、
test
和 
deploy
docker 镜像。

我使用“推/拉”逻辑开始我的 CI 工作流程：我在第一阶段构建所有 docker 映像并将它们推送到我的 gitlab 存储库，然后在 

test
阶段拉它们。

我认为，通过将 

build
阶段构建的图像保持在 
build
和 
test
阶段之间，我可以大大缩短工作流程时间。

我的

gitlab-ci.yml
看起来像这样：

stages:
  - build
  - test
  - deploy

build_backend:
  stage: build
  image: docker
  services:
    - docker:dind
  before_script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
  script:
    - docker build -t backend:$CI_COMMIT_BRANCH ./backend
  only:
    refs:
      - develop
      - master

build_generator:
  stage: build
  image: docker
  services:
    - docker:dind
  before_script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
  script:
    - docker build -t generator:$CI_COMMIT_BRANCH ./generator
  only:
    refs:
      - develop
      - master

build_frontend:
  stage: build
  image: docker
  services:
    - docker:dind
  before_script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
  script:
    - docker build -t frontend:$CI_COMMIT_BRANCH ./frontend
  only:
    refs:
      - develop
      - master

build_scraping:
  stage: build
  image: docker
  services:
    - docker:dind
  before_script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
  script:
    - docker build -t scraping:$CI_COMMIT_BRANCH ./scraping
  only:
    refs:
      - develop
      - master


test_backend:
  stage: test
  needs: ["build_backend"]
  image: docker
  services:
    - docker:dind
  before_script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
    - DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
    - mkdir -p $DOCKER_CONFIG/cli-plugins
    - apk add curl
    - curl -SL https://github.com/docker/compose/releases/download/v2.3.2/docker-compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose
    - chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose
  script:
    - docker compose -f docker-compose-ci.yml up -d backend
    - docker exec backend pip3 install --no-cache-dir --upgrade -r requirements-test.txt
    - docker exec db sh mongo_init.sh
    - docker exec backend pytest test --junitxml=report.xml -p no:cacheprovider
  artifacts:
    when: always
    reports:
      junit: backend/report.xml
  only:
    refs:
      - develop
      - master

test_generator:
  stage: test
  needs: ["build_generator"]
  image: docker
  services:
    - docker:dind
  before_script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
    - DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
    - mkdir -p $DOCKER_CONFIG/cli-plugins
    - apk add curl
    - curl -SL https://github.com/docker/compose/releases/download/v2.3.2/docker-compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose
    - chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose
  script:
    - docker compose -f docker-compose-ci.yml up -d generator
    - docker exec generator pip3 install --no-cache-dir --upgrade -r requirements-test.txt
    - docker exec generator pip3 install --no-cache-dir --upgrade -r requirements.txt
    - docker exec db sh mongo_init.sh
    - docker exec generator pytest test --junitxml=report.xml -p no:cacheprovider
  artifacts:
    when: always
    reports:
      junit: generator/report.xml
  only:
    refs:
      - develop
      - master
   
[...]


gitlab-runner/config.toml
：

concurrent = 5
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "Docker Runner"
  url = "https://gitlab.com/"
  token = ""
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:19.03.12"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/certs/client", "/cache"]
    shm_size = 0


docker-compose-ci.yml
：

services:
  backend:
    container_name: backend
    image: backend:$CI_COMMIT_BRANCH
    build:
      context: backend
    volumes:
      - ./backend:/app
    networks:
      default:
    ports:
      - 8000:8000
      - 587:587
      - 443:443
    environment:
      - ENVIRONMENT=development
    depends_on:
      - db

  generator:
    container_name: generator
    image: generator:$CI_COMMIT_BRANCH
    build:
      context: generator
    volumes:
      - ./generator:/var/task
    networks:
      default:
    ports:
      - 9000:8080
    environment:
      - ENVIRONMENT=development
    depends_on:
      - db

  db:
    container_name: db
    image: mongo
    volumes:
      - ./mongo_init.sh:/mongo_init.sh:ro
    networks:
      default:
    environment:
      MONGO_INITDB_DATABASE: DB
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: admin
    ports:
      - 27017:27017

  frontend:
    container_name: frontend
    image: frontend:$CI_COMMIT_BRANCH
    build:
      context: frontend
    volumes:
      - ./frontend:/app
    networks:
      default:
    ports:
      - 8080:8080
    depends_on:
      - backend

networks:
  default:
    driver: bridge

当我在我的 

context:
中评论 
docker-compose-ci.yml
时，Docker 找不到我的镜像，而且它确实没有在作业之间保留。

CI 期间 

build
-> 
test
-> 
deploy
的最佳 Docker 方法是什么？ 我应该压缩我的 docker 映像并使用工件在阶段之间共享它们吗？这似乎不是最有效的方法。

我有点不知道应该使用哪种方法来使用 Docker 在 Gitlab CI 中执行这样的常见工作流程。

docker docker-compose gitlab-ci gitlab-ci-runner
2个回答
8
投票

最好的方法是将镜像推送到注册表，并在其他需要的阶段拉取它。您似乎缺少推/拉逻辑。

您还想确保在 docker 构建中利用 docker 缓存。您可能需要在撰写文件中指定 

cache_from:
键。

例如：

build:
  stage: build
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    # pull latest image to leverage cached layers
    - docker pull $CI_REGISTRY_IMAGE:latest || true

    # build and push the image to be used in subsequent stages
    - docker build --cache-from $CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA  # push the image

test:
  stage: test
  needs: [build]
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    # pull the image that was built in the previous stage
    - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker-compose up # or docker run or whatever

编辑：

在带有 Docker buildkit/buildx 的现代 Docker 版本中，您可以使用 buildkit 内联缓存，而不是提前拉取镜像。这需要将更大的图像推送到您的存储库，但会使缓存拉取速度更快，因为 docker 可以在拉取它们之前判断哪些层对于缓存有效。

export DOCKER_BUILDKIT=1

docker build --build-arg BUILDKIT_INLINE_CACHE=1 \
             --cache-from "$CI_REGISTRY_IMAGE:latest"
             --cache-from "$CI_REGISTRY_IMAGE:$CI_REF_NAME" \
             --tag "$CI_REGISTRY_IMAGE:$CI_REF_NAME" \
             --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" \ 
             .

docker push "$CI_REGISTRY_IMAGE:$CI_REF_NAME"
docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
2
投票

尝试将“Docker Root Dir”安装为由 runners 团队共享的持久/nfs 卷。

Docker 镜像存储在“Docker Root Dir”路径中。您可以通过运行找到您的 docker root：

# docker info
...
 Storage Driver: overlay2
 Docker Root Dir: /var/lib/docker
...

一般根据操作系统的默认路径是

Ubuntu: /var/lib/docker/
Fedora: /var/lib/docker/
Debian: /var/lib/docker/
Windows: C:\ProgramData\DockerDesktop
MacOS: ~/Library/Containers/com.docker.docker/Data/vms/0/

一旦正确安装到所有代理，您将能够访问所有本地 docker 镜像。

参考资料：

https://docs.gitlab.com/runner

https://blog.nestybox.com/2020/10/21/gitlab-dind.html

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.