Flink standalone cluster

4
votes

I am trying to setup a simple standalone flink cluster. I followed these instructions. I have two scenarios:

1) 1 master and 2 slaves which are in the company network, which according to the IT services allows me to access any port (I doubt that).

2) 1 master and 1 slave which are both in my home network.

In both scenarios I am able to access the Flink Web UI after running the start-cluster.sh script in my master/job manager. However, I cannot see task managers. In scenario 2 I was able to make it work (see task managers) only if I run the start-cluster.sh in both master and slave (which is not something described in the tutorials), with only the master having password-less (still using the passphrase) access to slaves (see my questions below to find out why I mention this). I still cannot see the task managers in the scenario 1 even if I run start-cluster.sh on all the nodes (master and slaves).

Base on these I have 4 questions:

Q1) Do I need to run the start-cluster.sh script on each machine (i.e., masters and slaves)?

Q2) Documentation suggests the need of password-less ssh. I followed the tutorials for this and I was able to do that but it still asks me for a passphrase when I start the cluster to connect to each slave. Is this how a password-less connection should look like or I should try something else?

Q3) Do I need my slaves to be able to ssh password-less the master, or only the master to connect password-less to slaves?

Q4) If I need to ask IT services to open flink ports for me, could you please verify that I will only need them to open 8081 (for Web UI) and 6123 which I define (default value) in the parameter jobmanager.rpc.port: in my flink-conf.yaml file?

Configuration for scenario 1: 3 Ubuntu Linux machines running Open JDK 8

Configuration for scenario 2: 2 Mac OS X 10 machines running Oracle JDK 8

In both cases I use Apache Flink 1.7.1

1
cluster-computingapache-flink

1 Answers

2
votes

I managed to make the master see the task managers. The problem was with the Java installation. Although I was pretty sure that I had installed openjdk 8 to all the machines, setting the PATH and the JAVA_HOME variable to all of them pointing to the same JDK, it seems that the problem was solved by just adding the env.java.home parameter in the flink-conf.yaml to explicitly point to my JDK path (e.g., /usr/lib/jvm/java-8-openjdk-amd64)

Turns out that I am now able to answer some of my questions:

Q1) No, start-cluster.sh should only be executed at the master.

Q2) I switched to an ssh connection without a passphrase (just regenerated the key and pressed enter when asked for the passphrase leaving it empty). Though I do not know if this is needed.

Q3) No, only master should be able to have password-less access to slaves.

Q4) Master opens and listens to ports 8081 and 6123. It also opens some more ports (meaybe one for each slave?). If you do sudo lsof -i -P -n you will find which other ports are opened as they change each time.