大数据之Hadoop集群中Yarn常用命令

Yarn状态的查询，除了可以在hadoop103:8088页面查看以外，还可以通过命令操作。常见的命令操作如下所示。

需求：执行WordCount案例，并通过Yarn命令查看任务运行情况。原文：sw-code

# 启动集群

[hadoop@hadoop102 bin]$ myhadoop.sh start

# 运行wordcount案例

[hadoop@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output

yar application查看任务

1、列出所有Application：

[hadoop@hadoop102 hadoop-3.1.3]$ yarn application -list

2022-06-06 16:51:19,896 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032

Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0

                Application-Id      Application-Name        Application-Type          User           Queue                   State              Final-State             Progress                        Tracking-URL

2、根据Application状态过滤：yarn application -list -appStates [ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED]

[hadoop@hadoop102 hadoop-3.1.3]$ yarn application -list -appStates FINISHED

2022-06-06 16:53:49,886 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032

Total number of applications (application-types: [], states: [FINISHED] and tags: []):2

                Application-Id      Application-Name        Application-Type          User           Queue                   State              Final-State             Progress                        Tracking-URL

application_1654496324557_0001            word count               MAPREDUCE        hadoop         default                FINISHED                   FAILED                 100% http://hadoop102:19888/jobhistory/job/job_1654496324557_0001

application_1654496324557_0002            word count               MAPREDUCE        hadoop         default                FINISHED                   FAILED                 100% http://hadoop102:19888/jobhistory/job/job_1654496324557_0002

2、Kill掉Application

[hadoop@hadoop102 hadoop-3.1.3]$ yarn application -kill application_1654496324557_0001

2022-06-06 16:57:34,082 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032

Application application_1654496324557_0001 has already finished

yarn logs查看日志

1、查询Application日志：yarn logs -applicationId <ApplicationId>

[hadoop@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1654496324557_0001

2、查询Container日志：yarn logs -applicationId <ApplicationId> -containerId <ContainerId>

[hadoop@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1654496324557_0001 -containerId container_1654496324557_0001_01_000001

yarn applicationattempt查看尝试的任务

1、列出所有Application尝试的列表：yarn applicationattempt -list <ApplicationId>

[hadoop@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -list application_1654496324557_0001

2022-06-06 17:05:10,914 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032

Total number of application attempts :1

         ApplicationAttempt-Id                 State                        AM-Container-Id                            Tracking-URL

appattempt_1654496324557_0001_000001                FINISHED    container_1654496324557_0001_01_000001  http://hadoop103:8088/proxy/application_1654496324557_0001/

2、打印ApplicationAttempt状态：yarn applicationattempt -status <ApplicationAttemptId>

[hadoop@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -status appattempt_1654496324557_0001_000001

2022-06-06 17:08:56,570 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032

Application Attempt Report :

        ApplicationAttempt-Id : appattempt_1654496324557_0001_000001

        State : FINISHED

        AMContainer : container_1654496324557_0001_01_000001

        Tracking-URL : http://hadoop103:8088/proxy/application_1654496324557_0001/

        RPC Port : 32771

        AM Host : hadoop103

        Diagnostics : Task failed task_1654496324557_0001_m_000000

Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

yarn container查看容器

1、列出所有Container：yarn container -list <ApplicationAttemptId>

[hadoop@hadoop103 ~]$ yarn container -list appattempt_1654496324557_0004_000001

2022-06-06 17:14:40,313 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032

Total number of containers :2

                  Container-Id            Start Time             Finish Time                   State                    Host       Node Http Address                                 LOG-URL

container_1654496324557_0004_01_000001  星期一 六月 06 17:13:55 +0800 2022                       N/A                 RUNNING         hadoop102:46450     http://hadoop102:8042   http://hadoop102:8042/node/containerlogs/container_1654496324557_0004_01_000001/hadoop

container_1654496324557_0004_01_000004  星期一 六月 06 17:14:38 +0800 2022                       N/A                 RUNNING         hadoop103:44058     http://hadoop103:8042   http://hadoop103:8042/node/containerlogs/container_1654496324557_0004_01_000004/hadoop

2、打印Container状态：yarn container -status <ContainerId>

[hadoop@hadoop103 ~]$ yarn container -status container_1654496324557_0006_01_000001

2022-06-06 17:20:51,357 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032

Container Report :

        Container-Id : container_1654496324557_0006_01_000001

        Start-Time : 1654507199188

        Finish-Time : 0

        State : RUNNING

        Execution-Type : GUARANTEED

        LOG-URL : http://hadoop103:8042/node/containerlogs/container_1654496324557_0006_01_000001/hadoop

        Host : hadoop103:44058

        NodeHttpAddress : http://hadoop103:8042

        Diagnostics : null

注：只有任务跑的途中才能看到container的状态

yarn node查看节点状态

[hadoop@hadoop103 ~]$ yarn node -list -all

2022-06-06 17:25:20,336 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032

Total Nodes:3

         Node-Id             Node-State Node-Http-Address       Number-of-Running-Containers

 hadoop104:38945                RUNNING    hadoop104:8042                                  0

 hadoop103:44058                RUNNING    hadoop103:8042                                  0

 hadoop102:46450                RUNNING    hadoop102:8042                                  0

yarn rmadmin更新配置

加载队列配置：yarn rmadmin -refreshQueues

[hadoop@hadoop103 ~]$ yarn rmadmin -refreshQueues

2022-06-06 17:27:10,830 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8033

yarn queue查看队列

打印队列信息：yarn queue -status <QueueName>

[hadoop@hadoop103 ~]$ yarn queue -status default

2022-06-06 17:28:54,282 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032

Queue Information :

Queue Name : default

        State : RUNNING

        Capacity : 100.0%

        Current Capacity : .0%

        Maximum Capacity : 100.0%

        Default Node Label expression : <DEFAULT_PARTITION>

        Accessible Node Labels : *

        Preemption : disabled

        Intra-queue Preemption : disabled