Understanding different ID’s that are generated during the Map Reduce Application.
In Hadoop 2, Map Reduce jobs are executed using the YARN(Yet Another Resource Negotiator). Let us understand the different id’s that are created while executing a mapreduce application.
Application Id:
When a MR job is submitted by the client, the resource manager will first create the application ID. Application ID is composed of the time that the resource manager is started and an incrementing counter maintained by the RM to uniquely identify the application.
application_1622829088382_0005
In the above example, 1622829088382 refers to the start time format of resource manager (not the application) and 0005 indicates that it is the fifth application that is executed using this RM. The counter for Application ID starts with 1 and it is of four digits. If the counter exceeds digits (10000) it will not be reset, instead will continue the same.
Job Id:
Once after the application Id is generated, RM launches the application master which takes care running the MR job. Job Id is similar to that of the application ID and it is created by the replacing the application prefix from application ID with a job prefix.
job_1622829088382_0005
Task Id:
In a MR job, multiple tasks will be executed and each task can be uniquely identified by using Task ID. It is composed of RM start time, RM counter, type of task (map/reduce) and task counter.
task_1622829088382_0005_m_000001
Here the m represents the map task and 000001 refers to the second map task of the job with ID job_1622829088382_0005. The counter for task will start from 0 (unlike application counter that starts with 1).
Attempt_id:
An application or a task might be executed more than once, due to failures or speculative execution. In order to identify the unique instance we use the attempt id.
appattempt_1622829088382_0005_000002
The above example is for a second attempt of an application with ID application__1622829088382_0005.
attempt_1622829088382_0005_m_000001_1
The above example is for a second attempt of a amp task with ID task_1622829088382_0005_m_000001.
All these unique helps us in understanding how a MR job runs in a distributed fashion.
Hope you like this article. Happy Learning!!
