SSH Tunnel through Ubuntu bastion to EC2 instance in private subnet

Question

According this AWS doc: Scenario 2: VPC with Public and Private Subnets (NAT) I have my own VPC with two subnets: private and public. In public subnet I have deployed an Ubuntu 16.04 Instance with assigned EIP. It also has next security group inbound rules:

Type   Protocol Port Range Source            Description
SSH    TCP      22         xx.xx.xx.xx/32    Home IP

and outbound accordingly:

Type   Protocol Port Range Source            Description
SSH    TCP      22         sg-xxprivatexx    Security group ID for instance in private subnet

Looks nice, I can ssh it externally from my home. No problem.

In private subnet I have deployed another one Ubuntu 16.04 machine with next security group (inbound rules):

Type   Protocol Port Range Source            Description
HTTP   TCP      80         sg-xxpublicxxx    Security Group ID for bastion instance in public subnet
SSH    TCP      22         sg-xxpublicxxx    -

and no outbound rules (actually it has 80, 443 outbound ports opened, but its not an interesting part as I guess). And I still can reach this virtual machine using ssh from my bastion.

Right now I just want to make only one simple thing - run ssh port forwarding so I can run localhost:8080 on my home PC browser and see the webpage I published on my private instance. If I understand it correctly from here and here (and from here as well) I have to run something like:

 ssh -N -v -L 8080:10.0.1.112:80 [email protected]

Which as I guess basically means: just forward a traffic from private subnet instance with IP 10.0.1.112:80 to my localhost:8080 through my bastion VM with username ubuntu hosted on EIP 3.121.46.99.

Debug ends with lines:

debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:ZyVHgnF8z5vE5gfNr1S2JDfjhdydZVTNevPRgJZ+sRA /home/matterai/.ssh/key.pem
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/matterai/.ssh/id_rsa
debug1: Trying private key: /home/matterai/.ssh/id_dsa
debug1: Trying private key: /home/matterai/.ssh/id_ecdsa
debug1: Trying private key: /home/matterai/.ssh/id_ed25519
debug1: No more authentication methods to try.
[email protected]: Permission denied (publickey).

I am playing around it few days and I still can't get what am I doing wrong. Its so strange: I can ssh -A (to allow forwarding) to my bastion, I can ssh to my private instance from bastion. But I cant establish SSH tunnel to see my webpage (in the future it will be mongodb) without an error. Need some advice or point to the right direction, please! Thank you.

UPD#1

Ok then. If I make manual forwarding using my local machine and my bastion, I get an expected result. Basically it means run this command on bastion:

ubuntu@bastion: ssh -v -N -L 5000:localhost:8000 [email protected]

After that runs command on local/home machine:

matterai@homepc: ssh -v -N -L 5000:localhost:5000 [email protected]

When I make a request to localhost:5000 on my local machine, I can see the result page. May I and how if it's possible to combine this two commands? (spoiler: yes, it's possible: see the answer!)

Answer 1

Ok, it's easy. Hope my answer will help somebody.

1. You need to use ssh -J option to connect through your bastion virtual machine:

 -J [user@]host[:port]
         Connect to the target host by first making a ssh connection to
         the jump host and then establishing a TCP forwarding to the ulti‐
         mate destination from there.  Multiple jump hops may be specified
         separated by comma characters.  This is a shortcut to specify a
         ProxyJump configuration directive.

2. Then you need to forward traffic from your destination virtual machine port (:8000) where the app (or database) started to your localhost port (:5001) using ssh -L:

 -L [bind_address:]port:host:hostport
 -L [bind_address:]port:remote_socket
 -L local_socket:host:hostport
 -L local_socket:remote_socket
         Specifies that connections to the given TCP port or Unix socket
         on the local (client) host are to be forwarded to the given host
         and port, or Unix socket, on the remote side.  This works by
         allocating a socket to listen to either a TCP port on the local
         side, optionally bound to the specified bind_address, or to a
         Unix socket.  Whenever a connection is made to the local port or
         socket, the connection is forwarded over the secure channel, and
         a connection is made to either host port hostport, or the Unix
         socket remote_socket, from the remote machine.

        Port forwardings can also be specified in the configuration file.
         Only the superuser can forward privileged ports.  IPv6 addresses
         can be specified by enclosing the address in square brackets.

        By default, the local port is bound in accordance with the
         GatewayPorts setting.  However, an explicit bind_address may be
         used to bind the connection to a specific address.  The
         bind_address of “localhost” indicates that the listening port be
         bound for local use only, while an empty address or ‘*’ indicates
         that the port should be available from all interfaces.

3. Full ssh command will look like:

matterai@homepc: ssh -v -N -A -J [email protected] -L 5001:localhost:8000 [email protected]

UPD: Also you can simplify a bit your command. In ~/.ssh/config you can add your jumphost (bastion) and your final destination VM IP:

Host bastion
        HostName 3.121.46.99
        User ubuntu
        Port 22
        IdentityFile ~/.ssh/secret.pem
        ForwardAgent yes

Host server
        HostName 10.0.1.112
        User ubuntu
        Port 22
        IdentityFile ~/.ssh/secret.pem
        ProxyJump bastion

Now, you can run command:

ssh -v -N -A -J bastion -L 80:localhost:8000 server

Looks much better. Also you can just simply connect via ssh using ssh server.

Answer 2

You seem to have things correctly configured, but the error is saying that it can't find a private key to use for the connection.

To test the port forwarding, start by using the ssh command that logs into your public instance.

Then, take that exact command, and simply add: -L 8080:10.0.1.112:80

If it works for 'normal' ssh, then it will work with port forwarding too.

By the way, in general you should never need to modify the outbound rules of a security group. The default settings permit all outbound traffic. This 'trusts' the apps running on the instance and allows them to communicate outwards to anywhere. You would only need to restrict such rules where you wish to enforce a high-security environment.