flink job submission org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find Flink job

Question

Getting the following flink job submission error,

@centos1 flink-1.10.0]$ ./bin/flink run -m 10.0.2.4:8081 ./examples/batch/WordCount.jar --input file:///storage/flink-1.10.0/test.txt --output file:///storage/flink-1.10.0/wordcount_out
Job has been submitted with JobID 33d489aee848401e08c425b053c854f9

------------------------------------------------------------
 The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: org.apache.flink.runtime.rest.util.RestClientException: [org.apache.flink.runtime.rest.handler.RestHandlerException: org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find Flink job (33d489aee848401e08c425b053c854f9)

....

    Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find Flink job (33d489aee848401e08c425b053c854f9)
Caused by: org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find Flink job (33d489aee848401e08c425b053c854f9)
        at org.apache.flink.runtime.dispatcher.Dispatcher.getJobMasterGatewayFuture(Dispatcher.java:776)
        at org.apache.flink.runtime.dispatcher.Dispatcher.requestJobStatus(Dispatcher.java:505)
        ... 27 more
]

logs from the taskmanger nodes: saying the file not found.. Is the correct way of pointing files in a flink cluster setup.

2020-03-19 13:15:29,843 ERROR org.apache.flink.runtime.operators.BatchTask                  - Error in task code:  CHAIN DataSource (at main(WordCount.java:69) (org.apache.flink.api.java.io.TextInputFormat)) -> FlatMap (FlatMap at main(WordCount.java:84)) -> Combine (SUM(1), at main(WordCount.java:87) (1/2)
java.io.IOException: Error opening the Input Split file:/storage/flink-1.10.0/test.txt [0,19]: /storage/flink-1.10.0/test.txt (No such file or directory)
        at org.apache.flink.api.common.io.FileInputFormat.open(FileInputFormat.java:824)
        at org.apache.flink.api.common.io.DelimitedInputFormat.open(DelimitedInputFormat.java:470)

how to troubleshoot the above error, what to check , very less clues in the flink logs

Is this file file:/storage/flink-1.10.0/test.txt accessible from the TaskManager? — Till Rohrmann
Assuming it's a valid Flink job you already uploaded at some point, and that the file is exactly where you specified it in the shell command, I'd again suggest to look if all instances of your flink cluster have sufficient disk space. — kopaka
@TillRohrmann file:/storage/flink-1.10.0/test.txt is only accessible in Jobmanager node. — vinsent paramanantham
Then the problem is that the TaskManagers try to read a file which is not accessible. Please store the file somewhere where it is accessible from all TaskManager nodes. — Till Rohrmann

Anish Sarangi Anish Sarangi · Accepted Answer · 2020-04-23T20:30:18

The reason why is happening is because you are submitting a job across a distributed cluster and the location you have specified is perhaps only accessible by Job manager or machine from where you have submitted your job. However, actual program and Job execution takes place in Task Manager. Better approach for this would be by specifying a location which is accessible by all the nodes, may be HDFS or NFS.

flink job submission org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find Flink job

1 Answers