Elasticsearch to BigQuery pipeline deployment fails on cloud data fusion instance

Question

I am deploying a data fusion pipeline which takes data from an index at elasticsearch and load that data to bigQuery table. Pipeline simply consist of elasticsearch plugin connector to BigQuery connector. When I run the pipeline it generates the following error. Source of the error is

io.cdap.cdap.internal.app.runtime.distributed.remote.RemoteExecutionTwillRunnerService#548-runtime-startup-1

Actual error is

java.io.IOException: com.jcraft.jsch.JSchException: java.net.ConnectException: Connection timed out (Connection timed out)
    at io.cdap.cdap.common.ssh.DefaultSSHSession.<init>(DefaultSSHSession.java:88) ~[na:na]
    at io.cdap.cdap.internal.app.runtime.distributed.remote.RemoteExecutionTwillPreparer.lambda$start$0(RemoteExecutionTwillPreparer.java:436) ~[na:na]
    at io.cdap.cdap.internal.app.runtime.distributed.remote.RemoteExecutionTwillRunnerService$ControllerFactory.lambda$create$0(RemoteExecutionTwillRunnerService.java:524) ~[na:na]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_232]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_232]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_232]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_232]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_232]
Caused by: com.jcraft.jsch.JSchException: java.net.ConnectException: Connection timed out (Connection timed out)
    at com.jcraft.jsch.Util.createSocket(Util.java:349) ~[com.jcraft.jsch-0.1.54.jar:na]
    at com.jcraft.jsch.Session.connect(Session.java:215) ~[com.jcraft.jsch-0.1.54.jar:na]
    at com.jcraft.jsch.Session.connect(Session.java:183) ~[com.jcraft.jsch-0.1.54.jar:na]
    at io.cdap.cdap.common.ssh.DefaultSSHSession.<init>(DefaultSSHSession.java:85) ~[na:na]
    ... 7 common frames omitted
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_232]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_232]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_232]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_232]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_232]
    at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_232]
    at java.net.Socket.connect(Socket.java:556) ~[na:1.8.0_232]
    at java.net.Socket.<init>(Socket.java:452) ~[na:1.8.0_232]
    at java.net.Socket.<init>(Socket.java:229) ~[na:1.8.0_232]
    at com.jcraft.jsch.Util.createSocket(Util.java:343) ~[com.jcraft.jsch-0.1.54.jar:na]
    ... 10 common frames omitted

How can I resolve this issue ?

data fusion is a GCP tool where you create pipelines without explicitly coding anything. You can create pipelines by just drag and drop relevant plugins... I used two plugins 1) Elasticsearch source plugin 2) BigQuery Sink connector — Muhammad Aqeel

Ali Anwar Ali Anwar · Accepted Answer · 2019-12-01T19:21:41

Please see the networking requirements as documented in https://cloud.google.com/data-fusion/docs/how-to/create-instance.

Specifically:

New projects start with a default network. The default network is pre-populated with a firewall rule, default-allow-ssh, that allows incoming connections on TCP port 22 from any source to any instance in the network. If such a rule doesn't exist in the network used by your Cloud Data Fusion instance, you need to create such a rule.

Elasticsearch to BigQuery pipeline deployment fails on cloud data fusion instance

1 Answers