Run Erlang Observer with ssh port forwarding

3
votes

I have a remote Erlang node and local developer's PC. I want to start local node [email protected], start observer, call c:nl/1 and other debug actions. I wrote this:

   #!/bin/sh
   export ERL_EPMD_PORT=43690
   PORT=`ssh -l user target -p 5022 "/usr/bin/epmd -names" | awk '$2 == "target-node" {print $5}'`
   pkill -f ssh.*-fxN.*target
   ssh -fxNL 43690:`hostname`:4369 target -p 5022 -l user
   ssh -fxNL $PORT:`hostname`:$PORT target -p 5022 -l user
   ssh -fxNR 9001:`hostname`:9001 target -p 5022 -l user
   erl -name [email protected] -setcookie ABCDEFFJKGSK \
   -kernel inet_dist_listen_min 9001 inet_dist_listen_max 9001 \
   -eval "net_adm:ping('nodename@target')."
   pkill -f ssh.*-fxN.*target

But when I run this script I get message like:

bind: Address already in use Bad local forwarding specification ':debian:' Erlang/OTP 17 [erts-6.1] [source] [64-bit] [smp:4:4] [async-threads:10] [kernel-poll:false]

Eshell V6.1 (abort with ^G) ([email protected])1>

How can I run observer on local machine and connect them to the remote node?

4
ssherlangportforwardingssh-tunnel

4 Answers

4
votes

There are at least three things:

  • not closing TCP connections properly
  • node names
  • tunneling wrong

If you close process, that holds TCP connection with kill it can go to TIME_WAIT state for a couple of minutes: read more

In the ssh -L port:host:hostport, you have 2 components:

  • first one is port and this says, use this port on my local machine
  • host:hostport mean: "connect to this host and port from remote machine", you can imagine, that you are already on the remote machine and you want to connect to the third one

You are using hostname inside, which is evaluated on your local machine and returns debian and this doesn't make sense, because probably debian doesn't mean anything on your remote machine. Try using:

ssh -fxNR 9001:localhost:9001 target -p 5022 -l user

This should work as expected without the error about specification.

Erlang distributed uses node names to locate the machines. On your local node you are calling:

net_adm:ping('nodename@target').

which tries to contact port 4369 directly on target without port forwarding.

Erlang distributed was designed to be used in local trusted networks. It is really hard to set it up with port forwarding. You can start the debug node on the remote machine and connect to your system - this should be easy. If the remote system has X server, you can even forward the observer window (ssh -Y).

3
votes

First find the remote port: 

   $ ssh remote-host "epmd -names"
    epmd: up and running on port 4369 with data:
    name some_node at port 58769

Note the running on port for epmd itself and the port of the node you're interested in debugging. Reconnect to the remote host with these ports forwarded:

$ ssh -L 4369:localhost:4369 -L 58769:localhost:58769 remote-host

On your machine, start a hidden Erlang node running the observer app:

$ erl -name [email protected] -setcookie if-server-has-one -hidden -run observer

N.B.: If the server is using -sname, you need to use -sname as well, because Erlang™. You'll also need to change your system's hostname to match the target host, for the same reason.

In observer, go to Nodes - Connect Node and type [email protected].

You can also get a remote shell:

$ erl -name [email protected] -setcookie if-server-has-one -remsh [email protected]
1
votes

I tried to automate the same ssh port forwarding approach and came to this project: https://github.com/virtan/ccl

Add sys.config with {ccl, [ {slaves, [ {"ef1", "ef1.somehost.net", use} ]} ]} and start ccl application to get tunnel to ef1.somehost.net established and remote node started and connected.

0
votes

@tkowal's answer best explains the problem here but I want to share how I deal with this problem. I often do like to forward ports listed by epmd -names to localhost so I can easily use my local GUI tools to examine the state of things on the remote nodes. I wrote a shell script automatically forward the ports for me so I don't have to remember a bunch of commands. It's easy to use:

$ ./epmd_port_forwarder --server=remote-erlang-server.net
epmd: Cannot connect to local epmd
Killed local epmd
epmd: up and running on port 4369 with data:
name remote_erlang_node at port 46259
Forwarding ports
+ ssh -N -L 4369:localhost:4369 -L 46259:localhost:46259 remote-erlang-server.net

After you start the script you can then use epmd -names to list the remote nodes that are being forwarded.