I have 4 nodes(1 master 3 slave) and running a job. I have a problem and cannot resolve. Can you help me? (I see Container killed on request. Exit code is 143 [2022-07-28 06:16:54.444]Container exited with a non-zero exit code 143. but I couldn't find a solution)
2022-07-28 06:16:54,428 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1658988882858_0001_m_000082_0 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
2022-07-28 06:16:54,428 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2022-07-28 06:16:54,428 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1658988882858_0001_m_000087_0 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
2022-07-28 06:16:54,428 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2022-07-28 06:16:54,432 WARN [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://01namenode:9000/OUT/_temporary/1/_temporary/attempt_1658988882858_0001_m_000082_0
2022-07-28 06:16:54,432 WARN [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://01namenode:9000/OUT/_temporary/1/_temporary/attempt_1658988882858_0001_m_000087_0
2022-07-28 06:16:54,432 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1658988882858_0001_m_000082_0 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED
2022-07-28 06:16:54,432 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1658988882858_0001_m_000087_0 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED
2022-07-28 06:16:54,432 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1658988882858_0001_m_000082 Task Transitioned from KILL_WAIT to KILLED
2022-07-28 06:16:54,432 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1658988882858_0001_m_000087 Task Transitioned from KILL_WAIT to KILLED
2022-07-28 06:16:54,433 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1658988882858_0001Job Transitioned from FAIL_WAIT to FAIL_ABORT
2022-07-28 06:16:54,434 INFO [CommitterEvent Processor #4] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT
2022-07-28 06:16:54,442 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1658988882858_0001Job Transitioned from FAIL_ABORT to FAILED
2022-07-28 06:16:54,443 INFO [Thread-235] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly, recording last MRAppMaster retry
2022-07-28 06:16:54,443 INFO [Thread-235] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
2022-07-28 06:16:54,443 INFO [Thread-235] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified that shouldUnregistered is: true
2022-07-28 06:16:54,443 INFO [Thread-235] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2022-07-28 06:16:54,443 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
2022-07-28 06:16:54,443 INFO [Thread-235] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
2022-07-28 06:16:54,443 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 16
2022-07-28 06:16:54,445 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2022-07-28 06:16:54,448 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2022-07-28 06:16:54,448 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2022-07-28 06:16:54,449 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2022-07-28 06:16:54,449 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2022-07-28 06:16:54,451 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2022-07-28 06:16:54,452 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2022-07-28 06:16:54,452 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2022-07-28 06:16:54,453 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2022-07-28 06:16:54,453 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event MAP_ATTEMPT_KILLED
2022-07-28 06:16:54,454 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2022-07-28 06:16:54,455 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event MAP_ATTEMPT_KILLED
2022-07-28 06:16:54,455 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event MAP_ATTEMPT_KILLED
2022-07-28 06:16:54,456 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2022-07-28 06:16:54,456 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2022-07-28 06:16:54,457 INFO [Thread-235] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_FAILED
2022-07-28 06:16:55,171 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:22 AssignedReds:0 CompletedMaps:25 CompletedReds:0 ContAlloc:141 ContRel:0 HostLocal:56 RackLocal:0
2022-07-28 06:16:55,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1658988882858_0001: ask=5 release= 0 newContainers=17 finishedContainers=22 resourcelimit=<memory:4096, vCores:6> knownNMs=3
2022-07-28 06:16:55,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1658988882858_0001_01_000139
2022-07-28 06:16:55,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1658988882858_0001_01_000140
2022-07-28 06:16:55,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1658988882858_0001_01_000135
2022-07-28 06:16:55,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1658988882858_0001_01_000141
2022-07-28 06:16:55,185 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658988882858_0001_m_000023_3: [2022-07-28 06:16:54.283]Container killed by the ApplicationMaster.
[2022-07-28 06:16:54.292]Container killed on request. Exit code is 143
[2022-07-28 06:16:54.293]Container exited with a non-zero exit code 143.
2022-07-28 06:16:55,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1658988882858_0001_01_000134
2022-07-28 06:16:55,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1658988882858_0001_01_000136
2022-07-28 06:16:55,185 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658988882858_0001_m_000024_3: [2022-07-28 06:16:54.374]Container killed by the ApplicationMaster.
[2022-07-28 06:16:54.383]Container killed on request. Exit code is 143
[2022-07-28 06:16:54.384]Container exited with a non-zero exit code 143.
2022-07-28 06:16:55,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1658988882858_0001_01_000137
2022-07-28 06:16:55,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1658988882858_0001_01_000138
2022-07-28 06:16:55,185 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658988882858_0001_m_000069_2: [2022-07-28 06:16:54.385]Container killed by the ApplicationMaster.
[2022-07-28 06:16:54.392]Container killed on request. Exit code is 143
[2022-07-28 06:16:54.393]Container exited with a non-zero exit code 143.
2022-07-28 06:16:55,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1658988882858_0001_01_000124
2022-07-28 06:16:55,185 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658988882858_0001_m_000027_3: [2022-07-28 06:16:54.396]Container killed by the ApplicationMaster.
[2022-07-28 06:16:54.407]Container killed on request. Exit code is 143
[2022-07-28 06:16:54.408]Container exited with a non-zero exit code 143.
2022-07-28 06:16:55,185 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658988882858_0001_m_000067_2: [2022-07-28 06:16:54.401]Container killed by the ApplicationMaster.
[2022-07-28 06:16:54.412]Container killed on request. Exit code is 143
[2022-07-28 06:16:54.412]Container exited with a non-zero exit code 143.
2022-07-28 06:16:55,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1658988882858_0001_01_000120
2022-07-28 06:16:55,185 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658988882858_0001_m_000080_0: [2022-07-28 06:16:54.401]Container killed by the ApplicationMaster.
[2022-07-28 06:16:54.414]Container killed on request. Exit code is 143
[2022-07-28 06:16:54.421]Container exited with a non-zero exit code 143.
2022-07-28 06:16:55,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1658988882858_0001_01_000121
2022-07-28 06:16:55,185 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658988882858_0001_m_000082_0: [2022-07-28 06:16:54.422]Container killed by the ApplicationMaster.
[2022-07-28 06:16:54.432]Container killed on request. Exit code is 143
[2022-07-28 06:16:54.434]Container exited with a non-zero exit code 143.
2022-07-28 06:16:55,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1658988882858_0001_01_000122
2022-07-28 06:16:55,185 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658988882858_0001_m_000087_0: [2022-07-28 06:16:54.423]Container killed by the ApplicationMaster.
[2022-07-28 06:16:54.432]Container killed on request. Exit code is 143
[2022-07-28 06:16:54.434]Container exited with a non-zero exit code 143.
2022-07-28 06:16:55,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1658988882858_0001_01_000125
2022-07-28 06:16:55,185 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658988882858_0001_m_000000_3: [2022-07-28 06:16:54.215]Container killed by the ApplicationMaster.
[2022-07-28 06:16:54.310]Container killed on request. Exit code is 143
[2022-07-28 06:16:54.311]Container exited with a non-zero exit code 143.
2022-07-28 06:16:55,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1658988882858_0001_01_000123
2022-07-28 06:16:55,185 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658988882858_0001_m_000016_3: [2022-07-28 06:16:54.231]Container killed by the ApplicationMaster.
[2022-07-28 06:16:54.348]Container killed on request. Exit code is 143
[2022-07-28 06:16:54.384]Container exited with a non-zero exit code 143. 