QR Code contains TinyURL of this article.Bulletproof, Firewall-busting SSH Back to Your Computer

Background

I need to remotely connect to my computers from time to time. For example, I will sometimes SSH1 in to my home MacBook from the office, so that I can check on the progress of a long-running job, or take a peek at the results of another. When I log in over SSH my connection is securely encrypted, fast and dumps me right into a terminal shell where, with a single su command, I have complete control of the machine. Perfect.

Conditions

The ability to SSH in to a computer is dependent on three things:

  1. The host computer (for example: my MacBook at home) must have an SSH daemon listening for connection attempts;
  2. The host must be directly addressable from the Internet (for example: with a static IP address);
  3. There must be a network route from the client (for example: my workstation in the office) to the host.

With macOS, satisfying the first condition is as simple as enabling “Remote Login” from the “Sharing” preference pane in “System Preferences.”2  However, depending on your network configuration, the second and third conditions can be more difficult to satisfy. Consider the following illustration of how my MacBook connects to the Internet:3

Illustration of home computer with router/firewall connection to the Internet.

My MacBook connects to the Internet through an ADSL router. My service provider assigns a dynamic IP address to my ADSL router when it connects. A DHCP service on the router assigns IP addresses to each and every device on my network. The wonders of Network Address Translation (NAT) ensure that all devices on the internal network can play nicely with the larger Internet. All of which means that my MacBook can’t satisfy the second condition — it is not directly addressable on the Internet.

In addition, the router has an integrated firewall. The firewall has no open ports, so anything that connects through it to the Internet is inaccessible from the Internet.4  As the arrows in the illustration above indicate, the direction of requests are all outbound only. Of course, data can travel from the Internet to my MacBook, but only in response to requests initiated from it. Thus I also fail to satisfy the third condition — there is no route to the host.

The Wrong Way

I could resolve the dynamic IP address issue if I employ a service such as entryDNS or no-ip to map a memorable domain-name to the moving target that is my IP address. Then I could set up port forwarding on my router to direct SSH requests to my MacBook. One obvious problem with this is that it introduces a dependency on that specific router… not the bulletproof solution the title of this article promises.

Furthermore, for this to work, I should update the dynamic DNS provider any time my router acquires a new IP address. The most reliable way to do this is directly from the router itself,5 but again, that restricts me to a specific router. Thus, I would have to update the dynamic DNS service from my MacBook. However, I use the Internet almost exclusively over a VPN and that makes it difficult to determine the router’s external IP address from the MacBook. All way too complicated and messy.

The Right Way

But what if we could take the dynamic IP allocation, network address translation and firewall out of the equation? Well we can. If we employ some SSH black magic. What we need is something called a “reverse SSH tunnel.”

Consider the following illustration:

Illustration of reverse SSH Tunnel.

So what’s happening here? My MacBook makes a SSH request to the “middleman” server (I’ll explain what this is later in this article) and says, in effect, “create a persistent SSH connection from yourself back to me.” This is the reverse SSH tunnel.

Now, from my client terminal (for example: my office computer) I can SSH to my account on the middleman (which is a directly addressable computer on the Internet) and, once connected I can then connect directly to my MacBook over that same reverse SSH tunnel. As the MacBook was the initiator of the connection, NAT and firewall negotiation has already taken place, so they are no longer a barrier.

I initiate the connection on the MacBook with the following command:

ssh -f -N -R 10022:localhost:22 me@middleman.example.com

Let’s break this down:

  • -f instructs the SSH process to go into the background when it runs, so as to not tie up the shell process;
  • -N instructs the SSH process to not execute a remote command (ie: just set up the connection);
  • -R instructs the SSH process to listen for connections on the remote machine (ie: the middleman server);
  • 10022:localhost:22 instructs the SSH process to listen on port 10022 on the remote machine and forward all requests made to that port to port 22 on localhost (ie: the SSH daemon on my MacBook);
  • me@middleman.example.com - the username for my account on — and the address of — the middleman server.

We can confirm that the tunnel is up with the following command:

lsof -i -n | egrep '\<ssh\>'

Which results in something like this:

ssh  7173 username  3u  IPv4  0x8f69abd5bc8ca8ad  0t0  TCP xxx.xxx.xxx.xxx:52320->xxx.xxx.xxx.xxx:ssh (ESTABLISHED)

Now I can connect to my MacBook by issuing the following command on the client terminal:

ssh -t me@middleman.example.com 'ssh localhost -p 10022'

Which SSH translates as follows: execute the command ssh localhost -p 10022 on the remote host middleman.example.com after logging in to the user account me.

Okay so far? Good. If you have followed along and have actually set up a tunnel and made a connection to the target computer, then you should now exit the tunnelled connection and kill the corresponding ssh process(es).

Intermission

“Hang on a minute,” I hear you cry, “what’s this middleman server? Where does that come from?”

The middleman is any Internet connected and Internet addressable host to which you have shell access and on which you have permission to create SSH tunnels. You might already have access to such a server. If you don’t, then you can pay for a such an account from a provider like the SDF (with whom I have an account). Just ensure that the access you buy allows for SSH tunnelling (with the SDF, for example, you need a “MetaARPA” account).

Once you have suitable middleman, you should set up public key authentication between your target computer (the machine you want to SSH in to) and the middleman — because it won’t be bulletproof if you have to manually enter a password each time you want to establish the reverse SSH tunnel.

So where are we at? We can now SSH into our chosen target, bypassing the constraints of dynamic IP addressing, NAT and firewall. Let’s take it to the next level.

Testing a bulletproof vest in Washington, D.C. September 1923.

Making it Bulletproof

One problem with the tunnel, as it stands, is that it’s fragile. If something interrupts Internet connectivity, or either the MacBook or the middleman goes down, the tunnel will collapse. Then we’re right back where we started, with no ready remote access to the MacBook.

So the next step is to add some fault-tolerance. We’ll start by swapping out our ssh command with a call to the aptly named autossh.

autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic. The idea is from rstunnel (Reliable SSH Tunnel), but implemented in C.”

Unfortunately, autossh is not natively installed with macOS, but it is available as a Homebrew package.6  Let’s install it:

brew install autossh

Once installation is complete, test autossh with the following command:

/usr/local/bin/autossh -M 0 -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=2 -N -R 10022:localhost:22 me@middleman.example.com

Note:

Homebrew may have installed autossh into a different path on your system than it does on mine. Use the command which autossh to get its path and replace the string /usr/local/bin/autossh with it. Also, remember to replace the string me@middleman.example.com with the appropriate one for your middleman server.

The additional parameters on the autossh command, compared with the regular ssh one are:

  • -M 0 - an autossh option to specify a monitoring port. We’re not using this, hence the 0;
  • -o ExitOnForwardFailure=yes - specifies whether ssh should terminate the connection if it cannot set up all requested dynamic, tunnel, local, and remote port forwardings;
  • -o ServerAliveInterval=30 - the number of seconds that the client will wait before sending a null packet to the server (to keep the connection alive);
  • -o ServerAliveCountMax=2 - describes the limit of how long autossh will allow the server to be unresponsive. Acts as a multiplier to the ServerAliveInterval: thus, if the server does not respond within (ServerAliveInterval × ServerAliveCountMax) seconds, then autossh will tear down the connection then restart it.

See if the tunnel is up:

lsof -i -n | egrep '\<ssh\>'

Confirm that we can login to the target:

ssh -t me@middleman.example.com 'ssh localhost -p 10022'

Still okay? Great. You should now exit this tunnelled connection and kill the corresponding ssh process(es).

Persistence

If everything is roses at this point, then we have just one final task: to ensure that the system initiates our tunnel following a cold-boot or restart.

On macOS it’s ridiculously easy to do this, with launchd.7  First, in a terminal shell, enter the following command (macOS may prompt you for your password):

sudo nano /usr/local/bin/autotunnel

Paste the following into the nano editor:

#!/bin/bash

# wait a while: allow the system time to establish network connectivity
sleep 60

# then, instantiate the reverse SSH tunnel
/usr/local/bin/autossh -M 0 -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=2 -N -R 10022:localhost:22 me@middleman.example.com

Again, you should replace the strings /usr/local/bin/autossh and me@middleman.example.com with the ones appropriate for your tunnel. You should also confirm that the first line is the correct path to bash.

Hit control+O, then return to save, followed by control+X to exit.

Make the script executable:

sudo chmod a+x /usr/local/bin/autotunnel

Run it:

/usr/local/bin/autotunnel

See if the tunnel is up:

lsof -i -n | egrep '\<ssh\>'

Confirm that we can login to the target:

ssh -t me@middleman.example.com 'ssh localhost -p 10022'

You should now exit the tunnelled connection and kill the corresponding ssh process(es).

I know, there’s a lot of repetition here. It’s always best to test thoroughly though. Don’t worry, we’re nearly there.

Now we’ll create a LaunchAgent for our autotunnel script:

sudo nano /Library/LaunchAgents/com.bulletproof-ssh-tunnel.plist

Then paste the following into the nano editor:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>com.bulletproof-ssh-tunnel.plist</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/bin/autotunnel</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
  </dict>
</plist>

Hit control+O, then return to save, followed by control+X to exit.

Finally, enter the following command:

launchctl bootstrap gui/`stat -f %u` /Library/LaunchAgents/com.bulletproof-ssh-tunnel.plist

The new LaunchAgent will now start up and instantiate the reverse SSH tunnel. If the tunnel collapses, launchd will recreate it. launchd will also create the tunnel automatically following a cold-start or reboot.

In fact, the reverse SSH tunnel is now so resilient that — if you issue a kill command — the tunnel will collapse, only for launchd to immediately create a new one.

Therefore, if you want to take the tunnel down, you should enter the following command(s):

sudo launchctl stop com.bulletproof-ssh-tunnel.plist; launchctl bootout gui/`stat -f %u` /Library/LaunchAgents/com.bulletproof-ssh-tunnel.plist

Remember, you can always check on the presence or otherwise of the tunnel with:

lsof -i -n | egrep '\<ssh\>'

Finishing up

It’s worth setting up a handful of aliases in the .bashrc or .zshrc file on the target computer, for example:

alias showTunnels="lsof -i -n | egrep '\<ssh\>'"

alias tunnelUp="launchctl bootstrap gui/`stat -f %u` /Library/LaunchAgents/com.bulletproof-ssh-tunnel.plist"

alias tunnelDown="sudo launchctl stop com.bulletproof-ssh-tunnel.plist; launchctl bootout gui/`stat -f %u` /Library/LaunchAgents/com.bulletproof-ssh-tunnel.plist"

It’s also handy to have a corresponding alias on the client machine(s). My MacBook has the rather cool machine name, Sparta, so I have a fantastically named alias on my office computer:

alias gotoSparta="ssh -t me@middleman.example.com 'ssh localhost -p 10022'"

Bonus Feature

I hope you never need to benefit from this bonus feature, but here it is anyway… if your computer is ever stolen — and you have a bulletproof, reverse SSH connection to it — then you might get to fool around with the thief… Just sayin’.

  1. Secure Shell↩︎

  2. Setting up a SSH daemon will require a different procedure for non-macOS users. I leave this as an exercise for the reader. ¯\_(ツ)_/¯ ↩︎

  3. Mine is a typical example of a domestic broadband network configuration. ↩︎

  4. At least, that’s the theory. ↩︎

  5. Most routers support dynamic DNS in their firmware, YMMV↩︎

  6. If you’re a macOS user and you don’t have Homebrew installed, you should do so immediately. Homebrew rocks. 😃 ↩︎

  7. On other operating systems the procedure for setting up a persistent, start-at-boot program will differ. You’ll have to Google it. ↩︎