Docker - 库初始化失败 - 无法分配文件描述符表 - 内存不足

Question

我一直尝试在 Docker 容器上运行 Zookeeper 和 Kafka。 我的终端中出现了很多错误

[error occurred during error reporting , id 0xb]

和

[Too many errors, abort]

。然后

library initialization failed - unable to allocate file descriptor table - out of memory

。

我使用以下 docker-compose.yml 文件

version: "3.8"
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.5.4
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  kafka:
    image: confluentinc/cp-kafka:5.5.4
    # If you want to expose these ports outside your dev PC,
    # remove the "127.0.0.1:" prefix
    ports:
      - 127.0.0.1:9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Answer 1

我通过覆盖 docker.service 中的命令参数

ExecStart

解决了这个问题

sudo systemctl edit docker

然后输入

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --default-ulimit nofile=65536:65536 -H fd://

保存并运行

sudo systemctl daemon-reload

和

sudo systemctl restart docker

。

Answer 2

尝试像这样向zookeeper添加ulimits配置

---
version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ulimits:
      nofile:
        soft: 65536
        hard: 65536
    ports:
      - "2181:2181"

Answer 3

我认为你的kafka经纪人需要依赖于你的zookeper。 这对我有用：

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-server:6.2.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'

Answer 4

version: "3.9"
networks:
  nrixmw:
    driver: bridge
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    ulimits:
      nofile:
        soft: 65536
        hard: 65536
    hostname: zookeeper
    ports:
      - 2181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: '2181'
      ZOOKEEPER_TICK_TIME: '2000'
    networks:
      - nrixmw
  broker:
    image: confluentinc/cp-kafka:5.4.2
    ulimits:
      nofile:
        soft: 65536
        hard: 65536
    hostname: broker
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
      - 9092:9092
      - 9101:9101
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      # KAFKA_ADVERTISED_HOST_NAME: 'broker'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENERS: 'PLAINTEXT://:29092,PLAINTEXT_HOST://:9092'
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
      KAFKA_DELETE_TOPIC_ENABLE: 'true'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
    networks:
      - nrixmw
    mem_limit: 6g
    restart: always
  schema-registry:
    image: confluentinc/cp-schema-registry:5.5.1
    ulimits:
      nofile:
        soft: 65536
        hard: 65536
    depends_on:
      - zookeeper
      - broker
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: 'zookeeper:2181'
    networks:
      - nrixmw

这就是我的做法，希望这能完全解决您的错误。 我的系统配置 - Linux sourav-tuf 5.15.114-2-MANJARO #1 SMP PREEMPT Sun Jun 4 10:32:43 UTC 2023 x86_64 GNU/Linux (24g ram)

我有一台薄荷机，我不需要所有这些 ulimit，并且它仍在工作，这可能是 arch os 问题。不确定。

Answer 5

我确实尝试了上面提出的大部分解决方案，但没有成功。最后，我通过在 docker run 命令中添加

--ulimit nofile=8096:8096

来启动 Jenkins 容器。

例如：

docker run -d --ulimit nofile=8096:8096 -p 8089:8080 -v ~/jenkins_home/jenkins:/var/jenkins_home --name jenkins jenkins:2.60.1

现在容器已成功运行。