Accessing Flink Classloader before Stream Start

Question

In my project I would like to access the Flink User Classloader before the stream is executed. I have been instantiating my own Classloader to deserialize classes (doing my best to avoid issues related to multiple classloaders) prior to stream execution.

However the further I am progressing the more issues I am having to write (bad) code to avoid this issue.

This could be solved if I can access the Flink User Classloader and use that, however I don't see a mechanism to do so outside of "RichFunctions" (https://ci.apache.org/projects/flink/flink-docs-stable/api/java/org/apache/flink/api/common/functions/RichFunction.html) which require the stream to be running.

Any guidance here would be appreciated

Where do you want to access the user classloader? In the main method, in your user defined functions? — Robert Metzger

yunfan yunfan · Accepted Answer · 2018-10-17T09:09:31

You can use yourself class loader in flink.

Building the graph yourself and submit the class loader with the client.

The code is like follows:

final StandaloneClusterClient client;
try {
    client = new StandaloneClusterClient(configuration);
} catch (final Exception e) {
    throw new RuntimeException("Could not establish a connection to the job manager", e);
}

try {
    ClassLoader classLoader = JobWithJars.buildUserCodeClassLoader(
            Collections.<URL>singletonList(uploadedJarUrl),
            Collections.<URL>emptyList(),
            this.getClass().getClassLoader());
    client.runDetached(jobGraph, classLoader);
} catch (final ProgramInvocationException e) {
    throw new RuntimeException("Cannot execute job due to ProgramInvocationException", e);
}

But I still wonder why you want yourself classloader, it may can be implement by other way.

Accessing Flink Classloader before Stream Start

1 Answers