Blog | Tristan Kernan

“That some of us should venture to embark on a synthesis of facts and theories, albeit with second-hand and incomplete knowledge of some of them – and at the risk of making fools of ourselves” (Erwin Schrödinger)

Inter-Server Networking

I've been hard at work expanding Simple Routing, adding additional server infrastructure. I am self hosting on bare metal across Hetzner and OVH, so cloud networking solutions don't quite fit. I'm also aiming for simple mental models to avoid getting bogged down in a complex hosting infrastructure. To that end, I decided on Tailscale for secure networking between servers.

architecture-beta group internet(internet)[Public Internet] service clients(internet)[HTTP Clients] in internet service s1pub(server)[server1 public] in internet group tailnet(cloud)[Tailnet] service s1(server)[server1 tailnet] in tailnet service server2(server)[server2] in tailnet service laptop(disk)[Dev Laptop] in tailnet clients:R --> L:s1pub s1pub:B -- T:s1 s1:R -- L:server2 laptop:T -- B:s1 laptop:R -- B:server2

I had an existing tailnet set up for my homelab; thankfully, tailscale supports access control policies for limiting access between tagged hosts. In my case, my laptop is able to access all servers, but simple routing servers can only access each other, and homelab is otherwise unable to connect to other hosts. Sample code:

JSON
{
    "grants": [
        // your devices ↔ your devices
        {"src": ["autogroup:member"], "dst": ["autogroup:self"], "ip": ["*"]},
        // you → servers
        {"src": ["autogroup:member"], "dst": ["tag:sr-server"], "ip": ["*"]},
        // you → homelab
        {"src": ["autogroup:member"], "dst": ["tag:homelab"], "ip": ["*"]},
        // servers ↔ each other
        {"src": ["tag:sr-server"], "dst": ["tag:sr-server"], "ip": ["*"]},
    ]
}

Installing tailscale on each host with a tag means that the sessions won't expire. The remaining piece of the puzzle for me was networking across docker networks: each host uses docker compose or swarm, so services are by default unreachable from the host without port forwarding. Typical forwarding with <host>:<container> will bind the port to 0.0.0.0, or all network devices, which has the unfortunate side effect of bypassing the ufw firewall and exposing the service to the open internet.

The solution that I settled on is to bind service ports to 127.0.0.1 only, as in 127.0.0.1:19100:9100. This avoids exposing the service to the public internet. Then, I set up a tailscale forwarding service via:

Bash
sudo tailscale serve --bg --tcp=19100 tcp://localhost:19100

which will in effect route the service over the tailscale network. It's also possible to bind the service ports to the tailnet network device or ip address itself, but I decided against that to avoid a race condition on restart: docker will fail to start the containers if tailscale hasn't started yet.

Note that in the above command, I am using tcp over http. This is to avoid using magicdns (tailscale's friendly url service) on production servers, favoring plain ip addresses (which are stable for a node's lifetime). Hosts connect to each other via the tailscale ip, as in http://100.102.xxx.yyy:19100.

I think this is a solid solution at my current scale, as it's generally set and forget. The one improvement I'd want is to be able to configure tailscale services via code rather than cli, as ideally new server setup can be an automated script.