无法使用docker让hadoop HDFS数据持久化

问题描述 投票:0回答:1

我有一个使用此 docker-compos.yaml 文件创建的名称节点和数据节点

version: "3"
services:
   namenode:
      image: apache/hadoop:3
      hostname: 192.168.105.139
      command: ["hdfs", "namenode"]
      # ports:
      #   - 8020:8020
      #   - 9000:9000
      #   - 9870:9870
      network_mode: host
      env_file:
        - ./config
      environment:
          ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
      volumes:
       - ./hadoop/name:/tmp/hadoop-hadoop/dfs/name
   datanode:
      image: apache/hadoop:3
      command: ["hdfs", "datanode"]
      hostname: 192.168.105.139
      # ports:
      #   - 9864:9864
      #   - 9866:9866
      network_mode: host
      env_file:
        - ./config
      volumes:
       - ./hadoop/data:/tmp/hadoop-hadoop/dfs/data
      depends_on:
       - namenode

还有这个conf文件

HADOOP_HOME=/opt/hadoop
# CORE-SITE.XML_hadoop.tmp.dir=/opt/hadoop/data/
CORE-SITE.XML_fs.defaultFS=hdfs://192.168.105.139:9000
CORE-SITE.XML_hadoop.http.staticuser.user=hadoop
CORE-SITE.XML_hadoop_http_cross-origin_allowed-origins=*
CORE-SITE.XML_hadoop_http_cross-origin_allowed-methods=GET,POST,HEAD,DELETE,OPTIONS
CORE-SITE.XML_hadoop_http_cross-origin_allowed-headers=X-Requested-With,Content-Type,Accept,Origin
CORE-SITE.XML_hadoop_http_cross-origin_max-age=1800
CORE-SITE.XML_hadoop.http.cross-origin.enabled=true
# HDFS-SITE.XML_dfs.namenode.support.allow.format=false
# HDFS-SITE.XML_dfs.replication=1
# HDFS-SITE.XML_dfs.namenode.name.dir.restore=true
MAPRED-SITE.XML_mapreduce.framework.name=yarn
MAPRED-SITE.XML_yarn.app.mapreduce.am.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
MAPRED-SITE.XML_mapreduce.map.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
MAPRED-SITE.XML_mapreduce.reduce.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
YARN-SITE.XML_yarn.resourcemanager.hostname=resourcemanager
YARN-SITE.XML_yarn.nodemanager.pmem-check-enabled=false
YARN-SITE.XML_yarn.nodemanager.delete.debug-delay-sec=600
YARN-SITE.XML_yarn.nodemanager.vmem-check-enabled=false
YARN-SITE.XML_yarn.nodemanager.aux-services=mapreduce_shuffle
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.maximum-applications=10000
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.maximum-am-resource-percent=0.1
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.queues=default
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.capacity=100
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.user-limit-factor=1
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.maximum-capacity=100
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.state=RUNNING
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.acl_submit_applications=*
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.acl_administer_queue=*
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.node-locality-delay=40
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.queue-mappings=
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.queue-mappings-override.enable=false

我希望我的数据能够持久,因此我如上所述将数据安装到主机上。启动时,将创建包含 CLUSTERID 的版本文件。当我“关闭”集群并重新运行它时,我在数据节点上遇到了这个错误

java.io.IOException: Incompatible clusterIDs in /tmp/hadoop-hadoop/dfs/data: namenode clusterID = CID-846f2da4-7fad-40c7-891b-97ac6653031a; datanode clusterID = CID-c9ba5304-0c5b-4564-83fe-aaf6c2e3e019

docker hadoop docker-compose hdfs namenode
1个回答
0
投票

namenode容器将自我格式化并在每次启动时生成一个唯一的ID。您必须覆盖入口点脚本才能防止这种情况发生。

否则,请考虑使用 Hadoop Ozone 或 MinIO 映像而不是 HDFS 来实现与 Hadoop 兼容的持久性。

© www.soinside.com 2019 - 2024. All rights reserved.