Haddop 集群有 3 个活动节点，但资源管理器中只有 1 个活动节点

Question

我使用来自 docker hub 的 apache/hadoop 镜像在 docker 上运行 hadoop。

我只是使用默认配置运行并添加 2 个模式数据节点

这是我的 docker-compose.yaml

version: "2"
services:
   namenode:
      image: apache/hadoop:3.3.6
      hostname: namenode0
      command: ["hdfs", "namenode"]
      ports:
        - 9870:9870
      env_file:
        - ./config
      environment:
          ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
   datanode-0:
      image: apache/hadoop:3.3.6
      command: ["hdfs", "datanode"]
      env_file:
        - ./config
   datanode-1:
      image: apache/hadoop:3.3.6
      command: ["hdfs", "datanode"]
      env_file:
        - ./config
   datanode-2:
      image: apache/hadoop:3.3.6
      command: ["hdfs", "datanode"]
      env_file:
        - ./config
   resourcemanager:
      image: apache/hadoop:3.3.6
      hostname: resourcemanager
      command: ["yarn", "resourcemanager"]
      ports:
         - 8088:8088
      env_file:
        - ./config
      volumes:
        - ./test.sh:/opt/test.sh
   nodemanager:
      image: apache/hadoop:3.3.6
      command: ["yarn", "nodemanager"]
      env_file:
        - ./config

但是在 namenode UI 中有 3 个活动节点

如果运行这个

$ hdfs dfsadmin -report

Configured Capacity: 763068678144 (710.66 GB)
Present Capacity: 405373390848 (377.53 GB)
DFS Remaining: 405373366272 (377.53 GB)
DFS Used: 24576 (24 KB)
DFS Used%: 0.00%
Replicated Blocks:
        Under replicated blocks: 0
        Blocks with corrupt replicas: 0
        Missing blocks: 0
        Missing blocks (with replication factor 1): 0
        Low redundancy blocks with highest priority to recover: 0
        Pending deletion blocks: 0
Erasure Coded Block Groups: 
        Low redundancy block groups: 0
        Block groups with corrupt internal blocks: 0
        Missing block groups: 0
        Low redundancy blocks with highest priority to recover: 0
        Pending deletion blocks: 0

-------------------------------------------------
Live datanodes (3):

Name: 172.25.0.4:9866 (hadoop_datanode-0_1.hadoop_default)
Hostname: 7eb5f69ae955
Decommission Status : Normal
Configured Capacity: 254356226048 (236.89 GB)
DFS Used: 8192 (8 KB)
Non DFS Used: 115378343936 (107.45 GB)
DFS Remaining: 135124455424 (125.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 53.12%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Sat Oct 07 15:43:14 UTC 2023
Last Block Report: Sat Oct 07 15:21:42 UTC 2023
Num of Blocks: 0


Name: 172.25.0.5:9866 (hadoop_datanode-1_1.hadoop_default)
Hostname: 29d464fd39ac
Decommission Status : Normal
Configured Capacity: 254356226048 (236.89 GB)
DFS Used: 8192 (8 KB)
Non DFS Used: 115378343936 (107.45 GB)
DFS Remaining: 135124455424 (125.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 53.12%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Sat Oct 07 15:43:13 UTC 2023
Last Block Report: Sat Oct 07 15:21:37 UTC 2023
Num of Blocks: 0


Name: 172.25.0.7:9866 (hadoop_datanode-2_1.hadoop_default)
Hostname: dccb08fe06b8
Decommission Status : Normal
Configured Capacity: 254356226048 (236.89 GB)
DFS Used: 8192 (8 KB)
Non DFS Used: 115378343936 (107.45 GB)
DFS Remaining: 135124455424 (125.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 53.12%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Sat Oct 07 15:43:12 UTC 2023
Last Block Report: Sat Oct 07 15:21:42 UTC 2023
Num of Blocks: 0

但是yarn和资源管理器中只有1个节点处于活动状态

$ yarn node -list

2023-10-07 15:47:05 INFO  DefaultNoHARMFailoverProxyProvider:64 - Connecting to ResourceManager at resourcemanager/172.25.0.6:8032
Total Nodes:1
         Node-Id             Node-State Node-Http-Address       Number-of-Running-Containers
c8ac8dd71a0d:41135              RUNNING c8ac8dd71a0d:8042                                  0

如何将另一个节点添加到活动节点？

Answer 1

数据节点和节点管理器是两个不同的东西。添加更多 Nodemanager 来更新 YARN