Apache Flink Rest-Client Jar-Upload not working

Question

I am struggling to automatically deploy new Flink jobs within our CI/CD workflows by using the Flink rest-api (which may be found here in the flink Github repository).

Documentation only says that that jar upload may be achieved by using /jars/upload, but not how exactly a valid rest request has to be build (which Headers, which Body type, which Authorization, which Method and so on).

So I took a look at the Flink dashboard code of flink/flink-runtime-web project on Github and searched for the implementation they used to upload a jar and - Yippie! Its implemented by calling the rest-api I am trying to use (using POST as method). After that I tried to figure out with Postman which is correct way to send requests using different Content-Type headers and Body types, but none of them worked for me now.

I would have filed a ticket directly to the flink project, but could not find any reference to their ticket system.

So the basic Question here is:

• How do I have to call the rest endpoint /jars/upload to successfully upload a file?

Answer 1

I've run into the same issue and solved it by looking at the network request in chrome when uploading a jar with the web UI.

The request must

• Use multipart upload
• The field name must be jarfile
• The multi part content must include the file Content-Type as well (otherwise you'll get a 500 from Flink complaining about the missing header)

Here is a python script using requests that does the upload

upload = requests.post(                                                                                               
    base_url + "/jars/upload",                                                                                        
    files={
        "jarfile": (
            os.path.basename(path), 
            open(path, "rb"), 
            "application/x-java-archive"
         )
    }
)       

Answer 2

Those who are more partial to a command line can use curl:

curl -X POST -H "Expect:" -F "jarfile=@path/to/flink-job.jar" http://hostname:8081/jars/upload

Answer 3

For those who want a java solution can use:

CloseableHttpClient client = HttpClients.createDefault();
        HttpPost httpPost = new HttpPost("YOUR API URL");
        MultipartEntityBuilder builder = MultipartEntityBuilder.create();
        builder.addBinaryBody("jarfile", new File("jarr.jar"));
        HttpEntity multipart = builder.build();
        httpPost.setEntity(multipart);

        try{
            CloseableHttpResponse response = client.execute(httpPost);
            System.out.println(response);
            BufferedReader rd = new BufferedReader(new InputStreamReader(
                    response.getEntity().getContent()));

            String line = "";
            while ((line = rd.readLine()) != null) {
                System.out.println(line);
                }
        }
        catch (Exception e){
            e.printStackTrace();
        }