mapreduce job on yarn exited with exitCode: -1000 beacuse of resource changed on src filesystem

Question

    Application application_1552978163044_0016 failed 5 times due to AM Container for appattempt_1552978163044_0016_000005 exited with exitCode: -1000

Diagnostics:

java.io.IOException: Resource abfs://[email protected]/hdp/apps/2.6.5.3006-29/mapreduce/mapreduce.tar.gz changed on src filesystem (expected 1552949440000, was 1552978240000 Failing this attempt. Failing the application.

Your description is too simple. If possible, please post more details about your issue at here, which include the necessary code, the environment version or others? — Peter Pan

Peter Pan Peter Pan · Accepted Answer · 2019-03-25T04:29:31

Just based on the exception information, it seems to be caused by Azure Storage could not keep the original timestamp of the copied file. I searched a workaround that recommended to change the source code of yarn-common to disable the code block of timestamp check when copy file to avoid the exception throws to make the MR job continous to work.

Here is the source code in the latest version of yarn-common which check the timestamp for copied file and throws the exception.

/** #L255
   * Localize files.
   * @param destination destination directory
   * @throws IOException cannot read or write file
   * @throws YarnException subcommand returned an error
   */
  private void verifyAndCopy(Path destination)
      throws IOException, YarnException {
    final Path sCopy;
    try {
      sCopy = resource.getResource().toPath();
    } catch (URISyntaxException e) {
      throw new IOException("Invalid resource", e);
    }
    FileSystem sourceFs = sCopy.getFileSystem(conf);
    FileStatus sStat = sourceFs.getFileStatus(sCopy);
    if (sStat.getModificationTime() != resource.getTimestamp()) {
      throw new IOException("Resource " + sCopy +
          " changed on src filesystem (expected " + resource.getTimestamp() +
          ", was " + sStat.getModificationTime());
    }
    if (resource.getVisibility() == LocalResourceVisibility.PUBLIC) {
      if (!isPublic(sourceFs, sCopy, sStat, statCache)) {
        throw new IOException("Resource " + sCopy +
            " is not publicly accessible and as such cannot be part of the" +
            " public cache.");
      }
    }

    downloadAndUnpack(sCopy, destination);
  }

mapreduce job on yarn exited with exitCode: -1000 beacuse of resource changed on src filesystem

1 Answers