Problem with using collect() function of DataSet in apache flink

Question

I am trying to calculate the AdamicAdar index of relations in a Social Media following graph. I set up my Edges, Vertices, Dataset and Graph, using apache flink-gelly lirbrarie. Here is my code:



    import org.apache.flink.api.java.ExecutionEnvironment;
    import org.apache.flink.api.java.operators.DataSource;
    import org.apache.flink.api.java.tuple.Tuple2;
    import org.apache.flink.graph.Graph;
    import org.apache.flink.graph.library.similarity.AdamicAdar;
    import org.apache.flink.types.NullValue;
    import org.apache.flink.types.StringValue;
    
    import java.util.List;
    
    public class MyMain {
    
        public static void main(String[] args) {
    
            ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
            DataSource> edgeDataSet = env.
                    readCsvFile(String.valueOf(MyMain.class.getResource("dataset/edges.csv"))).
                    types(StringValue.class, StringValue.class);
            Graph graph = Graph.fromTuple2DataSet(edgeDataSet, env);
    
            List list = null;
            try {
                list = graph.run(new AdamicAdar()).collect();
            } catch (Exception e) {
                e.printStackTrace();
            }
    
            System.out.println(list.get(0));
        }
    }

and here is the error I get:



    Exception in thread "main" java.lang.NoSuchMethodError: org.apache.flink.configuration.ConfigUtils.decodeListFromConfig(Lorg/apache/flink/configuration/ReadableConfig;Lorg/apache/flink/configuration/ConfigOption;Lorg/apache/flink/util/function/FunctionWithException;)Ljava/util/List;
        at org.apache.flink.client.cli.ExecutionConfigAccessor.getJars(ExecutionConfigAccessor.java:75)
        at org.apache.flink.client.deployment.executors.PipelineExecutorUtils.getJobGraph(PipelineExecutorUtils.java:61)
        at org.apache.flink.client.deployment.executors.LocalExecutor.getJobGraph(LocalExecutor.java:98)
        at org.apache.flink.client.deployment.executors.LocalExecutor.execute(LocalExecutor.java:79)
        at org.apache.flink.api.java.ExecutionEnvironment.executeAsync(ExecutionEnvironment.java:962)
        at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:878)
        at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:862)
        at org.apache.flink.api.java.DataSet.collect(DataSet.java:413)
        at MyMain.main(MyMain.java:23)
    
    Process finished with exit code 1

Also this is a part of edges.csv file I use:

in which 5 316 means vertex number five is connected to vertex number 216 and this defines an edge.

And here is my pom.xml file pom.xml

Could you please provide your pom.xml/gradle.build file ? This is defintely a library conflict issue. Also, could you please specify the way you run the application ? Thank you — Mikalai Lushchytski
Yes you are right. But I don't know what version I have imported wrong. I edited the question and added my pom.xml file. I just press the run button of the intellij ide to run the application. @MikalaiLushchytski — amin rahman
First of all, I would suggest trying to get the same scala veersion across libraries so please replace flink-clients_2.12 with flink-clients_2.11. — Mikalai Lushchytski
Also, not sure why flink-shaded is required, but it seems that flink-shaded-asm-7 should be of version 7.1-11.0 — Mikalai Lushchytski
I changes the versions and I still get the same error. @MikalaiLushchytski — amin rahman

Bryan Kyle Bryan Kyle · Accepted Answer · 2020-09-11T12:45:10

A NoSuchMethodError usually means there’s a version mismatch in your dependencies. Looking at your pom.xml you’ve got what appears to be a mixture of different versions of your Flink related libraries.

As an aside, I also noticed that your .csv is not comma separated. You may want to simplify the program and verify that edgeDataset is actually a 2-Tuple and not a single value.

Problem with using collect() function of DataSet in apache flink

1 Answers