Unlimited tunnels to localhost for little money

Jan 13, 2020
7 min read

You might have enjoyed our previous post How to mock OAuth 2.0 in Go. We talked about getting rid of calls to GitHub during development. We were able to mock OAuth and use hard coded data for the GUI. This is all great but sometimes we need the real data. Especially when building new features you might not know about the structure of the new data. In this case we must make valid requests to GitHub. We also need a way to receive webhooks on events.

During development the application runs on our local machine. This machine is not available from the internet and GitHub cannot access it. That means GitHub is not able to send a request to our server on an installation or any other event. If you want to make your machine accessible from the outside several solutions already exist. Here is a comprehensive list of services we found online. There are certainly more that we haven't mentioned.

Why not use an existing solution?

All those options have several pros and cons. They might be easy to get started but in the long run we had multiple issues. After having built our own solution they are all gone and we can focus on building our application. Here are our three main problems.

1. Random URLs

One major issue is the random URL. When using a free service you're assigned a random URL like a1b2c3.service.com and the next time you'll get a URL like d4e5f6.service.com. However you need a fixed URL to use webhooks and callbacks. They are hardcoded in the GitHub Application settings page.

GitHub settings

2. Single point of failure

Another issue is the dependency. In the past we used serveo a lot. However they started having issues and are currently down. Another service didn't update their TLS certificates and we couldn't use https. We need https for secure cookies. During development and in production we want to use the same business logic. We don't want to use insecure cookies during development. We liked the idea of simply using SSH reverse proxies. Some services make use of them but others force you to download third party tools and install them globally.

3. Price

Last but not least those services are quite expensive. They are around the same price you'll end up with when you follow our solution but you are more flexible and you'll get a server. When you use this server for additional services you can save a lot of money.

Requirements

  1. First of all you need a server or virtual machine in the internet. We're using a small Digital Ocean (referral link, you get $100 credit, we get $25) droplet for 5 $/month.
  2. You also need a domain. We will create several subdomains. Each subdomain is used by a different tunnel.
  3. Next is a valid https certificate from Let's Encrypt. This is free.
  4. On our server we need a web server and will use nginx for this demo. You can use any server you're familiar with.
  5. Last but not least we need ssh on our developer machine.

Create the subdomain

The first thing you've got to do is to create a new subdomain. We're using AWS Route 53 as our DNS. The following steps should be similar for your own DNS. Create two new record sets. One A for IPv4 and one AAAA record set for IPv6. Enter your subdomain and point them to the IP addresses of our droplet. A request to dev.foo.com now reaches our server.

AWS Route 53 settings

The next step is about handling this request on our server.

Server setup

On our server we need a web server. The web server is responsible for a few things. First of all it is responsible for TLS termination. It listens on port 443 and handles https requests. Thus we need valid certificates. We're going to use Let's Encrypt here. Afterwards the web server has to forward the request to another port on localhost. Later on we will use this port for our SSH tunnel endpoint. This is the high level overview so let's get started.

First of all SSH into your server, install nginx and follow the instructions to install certbot.

$ sudo apt update
$ sudo apt install software-properties-common
$ sudo add-apt-repository universe
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt update
$ sudo apt install nginx certbot python-certbot-nginx

Make sure nginx is running.

$ systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-12-16 19:59:32 UTC; 3 weeks 1 days ago
     Docs: man:nginx(8)
 Main PID: 17560 (nginx)
    Tasks: 2 (limit: 1137)
   Memory: 5.6M
   CGroup: /system.slice/nginx.service
           ├─17560 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           └─20069 nginx: worker process

Then run certbot --nginx to create valid certificates via Let's Encrypt. The --nginx flags tells certbot to update our nginx configuration.

$ sudo certbot --nginx

Use an editor to open the default nginx configuration.

$ vim /etc/nginx/sites-enabled/default

Within this configuration change the server_name to your URL. Use proxy_pass to forward all incoming requests to 127.0.0.1:8080. The port 8080 is the endpoint for our SSH tunnel. More on this later in the post. The rest of the config should have been automatically added by certbot.

server {
    server_name dev.foo.com;

	location / {
		proxy_pass http://127.0.0.1:8080;
	}

    listen [::]:443 ssl ipv6only=on;
    listen 443 ssl;

    ssl_certificate /etc/letsencrypt/live/dev.foo.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/dev.foo.com/privkey.pem;

    include /etc/letsencrypt/options-ssl-nginx.conf;

    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
}

server {
    if ($host = dev.foo.com) {
        return 301 https://$host$request_uri;
    }

	listen 80;
	listen [::]:80 ;
    server_name dev.foo.com;
    return 404;
}

You must tweak your SSH configuration a little. Open the sshd_config file.

$ sudo vim /etc/ssh/sshd_config

Change the GatewayPorts option from no to yes. The default value is no. By setting it to yes we allow anyone to connect to the forwarded ports. As our server is on the public internet, anyone could connect to the port. Just be aware of it.

...
GatewayPorts yes
...

Restart the SSH daemon and the nginx webserver to ensure all settings are up-to-date.

$ sudo systemctl restart sshd
$ sudo systemctl reload nginx

We're done on the server. Close the SSH connection. The rest of the work needs to be done on the client.

Start the tunnel

Start your application on your development machine. Then start the reverse SSH tunnel and connect to the server. It assumes your application is running on port 8081. We're using the same port 8080 again that we've used in our nginx configuration. Now everything that's coming in on this port will be forwarded to our local machine.

ssh -N -R 8080:localhost:8081 root@dev.foo.com

Conclusion

The final setup is rock solid. We just have to renew our certificates from time to time. You could also automate this process and have even less work. Add more subdomains and more nginx server configurations to different ports. You'll get unlimited tunnels to your development machines for very little money. Our droplet costs us 5 $/month and our domain 1 $/month. In total we pay 6 $/month or 72 $/year. We like this setup a lot and are very happy we invested the time to build it. In the end it's all about developer experience. If the developer experience is great you will get great products.