我有一个使用此 docker-compos.yaml 文件创建的名称节点和数据节点
version: "3"
services:
namenode:
image: apache/hadoop:3
hostname: 192.168.105.139
command: ["hdfs", "namenode"]
# ports:
# - 8020:8020
# - 9000:9000
# - 9870:9870
network_mode: host
env_file:
- ./config
environment:
ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
volumes:
- ./hadoop/name:/tmp/hadoop-hadoop/dfs/name
datanode:
image: apache/hadoop:3
command: ["hdfs", "datanode"]
hostname: 192.168.105.139
# ports:
# - 9864:9864
# - 9866:9866
network_mode: host
env_file:
- ./config
volumes:
- ./hadoop/data:/tmp/hadoop-hadoop/dfs/data
depends_on:
- namenode
还有这个conf文件
HADOOP_HOME=/opt/hadoop
# CORE-SITE.XML_hadoop.tmp.dir=/opt/hadoop/data/
CORE-SITE.XML_fs.defaultFS=hdfs://192.168.105.139:9000
CORE-SITE.XML_hadoop.http.staticuser.user=hadoop
CORE-SITE.XML_hadoop_http_cross-origin_allowed-origins=*
CORE-SITE.XML_hadoop_http_cross-origin_allowed-methods=GET,POST,HEAD,DELETE,OPTIONS
CORE-SITE.XML_hadoop_http_cross-origin_allowed-headers=X-Requested-With,Content-Type,Accept,Origin
CORE-SITE.XML_hadoop_http_cross-origin_max-age=1800
CORE-SITE.XML_hadoop.http.cross-origin.enabled=true
# HDFS-SITE.XML_dfs.namenode.support.allow.format=false
# HDFS-SITE.XML_dfs.replication=1
# HDFS-SITE.XML_dfs.namenode.name.dir.restore=true
MAPRED-SITE.XML_mapreduce.framework.name=yarn
MAPRED-SITE.XML_yarn.app.mapreduce.am.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
MAPRED-SITE.XML_mapreduce.map.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
MAPRED-SITE.XML_mapreduce.reduce.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
YARN-SITE.XML_yarn.resourcemanager.hostname=resourcemanager
YARN-SITE.XML_yarn.nodemanager.pmem-check-enabled=false
YARN-SITE.XML_yarn.nodemanager.delete.debug-delay-sec=600
YARN-SITE.XML_yarn.nodemanager.vmem-check-enabled=false
YARN-SITE.XML_yarn.nodemanager.aux-services=mapreduce_shuffle
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.maximum-applications=10000
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.maximum-am-resource-percent=0.1
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.queues=default
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.capacity=100
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.user-limit-factor=1
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.maximum-capacity=100
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.state=RUNNING
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.acl_submit_applications=*
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.acl_administer_queue=*
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.node-locality-delay=40
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.queue-mappings=
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.queue-mappings-override.enable=false
我希望我的数据能够持久,因此我如上所述将数据安装到主机上。启动时,将创建包含 CLUSTERID 的版本文件。当我“关闭”集群并重新运行它时,我在数据节点上遇到了这个错误
java.io.IOException: Incompatible clusterIDs in /tmp/hadoop-hadoop/dfs/data: namenode clusterID = CID-846f2da4-7fad-40c7-891b-97ac6653031a; datanode clusterID = CID-c9ba5304-0c5b-4564-83fe-aaf6c2e3e019
。
namenode容器将自我格式化并在每次启动时生成一个唯一的ID。您必须覆盖入口点脚本才能防止这种情况发生。
否则,请考虑使用 Hadoop Ozone 或 MinIO 映像而不是 HDFS 来实现与 Hadoop 兼容的持久性。