One tunnel per proxy: what proxied copies cost, and the cheapest safe design¶

Design note behind the shared tunnel of proxied copies. Date: 2026-11-09.

The problem¶

A proxied copy runs an earner on one box while its traffic leaves from another public IP. Until this change every copy was two containers: the earner and a tunnel of its own, the earner living in the tunnel's network namespace. A Tailscale tunnel runs its own tailscaled and joins the tailnet as its own ephemeral device.

Copies through the same proxy already share its exit IP, so a tunnel each buys them nothing. But the copies are what a user adds, and memory grew with them.

What we measured¶

A box running seven copies through one Tailscale exit node (Docker on btrfs):

The tunnels cost more than the earners they carried, and nearly all of a tunnel's cost is fixed. On btrfs the page cache is not shared between containers: every tailscaled mapping is private, so each container pays for its own copy of the binary. On overlay2 those pages would be shared.

The exit box served one tailnet peer per copy (7 of its 28 peers were these tunnels), and each tunnel was a device in the tailnet admin console.

The lab (scripts/proxy_shared_tunnel_lab.sh, a local headscale tailnet, Docker on overlayfs, an idle stand-in earner) drives the worker's own code. Sum of the proxy's tunnels' cgroup memory.current, 25 seconds after the last deploy:

The lab tunnel is smaller than a production one: overlayfs shares the binaries' page cache between containers, the test tailnet has two nodes, and the stand-in earner moves almost nothing. The shape is the point: the tunnel side stops growing with the copies.

The options¶

All memory figures are the tunnel side only; earners cost the same in every option.

Not ranked:

• Host policy routing with no container at all. Tailscale cannot choose an exit node per source address (an upstream request for it was declined, and a node cannot use an exit node while advertising one). It would need our own tunnel plus ip rule edits on the host, which only works on rootful Linux and can break the host's own networking. Option C gets the same memory inside a namespace, without touching the host's routes.
• A shared tun2proxy for SOCKS5 and HTTP proxies. That is option A for those kinds, and comes with it.
• Routing at the routers. Needs capable routers at both ends; nothing we can ship.
• tsnet, or tailscaled in userspace mode with a SOCKS listener. Both cost more memory, not less, and userspace mode dials from the box itself when the exit node's route is missing (why the tunnel runs in kernel mode).

The decision¶

1. Option A now. One tunnel per (box, proxy), named cashpilot-tunnel-p<id>, tailnet host name cashpilot-p<id>. It saves almost all of the memory with no new requirement on anyone's host, and nothing new to set up.
2. Option C later, as an advanced mode, once a lab has measured it, offered only where the worker finds a rootful Linux engine with the wireguard module. Option A stays the fallback everywhere.

How option A works¶

• Who belongs to which tunnel. An earner's tunnel is the container its NetworkMode names, never a name lookup. That one rule makes legacy pairs (one tunnel per copy, from older workers), a half-finished move and the shared tunnel all read the same way in the heartbeat.
• Adding copy N+1. If the proxy's tunnel runs, is ready, and was started on the same settings (a digest kept in its root-only secret volume, never a label) and the same image, only the earner is created. No other copy is touched.
• Copies are not recreated for a change. Several earners keep a device id in their writable layer (docker diff shows, for example, /.config/traffmonetizer/cli_device_ids.json, /etc/earnfm/deviceid, /mnt/app/InstallationId.ini, /root/.packetstream/data), and proxied copies have no volumes. A recreated copy is a new device to its provider, so the tunnel changes in place wherever it can:
• an edited proxy while the tunnel runs: the new settings first prove an exit in a throwaway Test tunnel, then are written into the running tunnel, which restarts. Its copies keep their NetworkMode (the container id does not change) and are restarted into the new namespace. If the tunnel then fails on settings the test passed, the old settings go back;
• a stopped tunnel is started in place with the settings rewritten;
• a new tunnel image (each worker release) does not replace a tunnel copies live in. A deploy takes it only when no other copy is in the tunnel.
• Replacing the tunnel is left for when there is no other way: Docker cannot start the container at all, or the image changes on a deploy of the only copy. A new tunnel starts under a staged name and proves its exit, then each copy is recreated inside it from docker inspect (image, environment, command, labels, capabilities, limits), because a container's namespace is fixed when it is created. Each move creates the clone under <name>-next, renames the copy aside to <name>-moving, gives the clone the name, starts it, and only then removes the old container; a failure puts it back. A worker that dies midway leaves both, and recover_moves (worker start, and the heartbeat) finishes or undoes the move. A staged tunnel copies live in is finished, never swept.
• The settings volume is a Mount, not volumes=[path]. docker-py turns the latter into a Binds entry with no source. On Docker 29 such a container comes back from a daemon restart as get: no such volume and cannot be started again; the Mount form (HostConfig.Mounts, type volume, no source) comes back with its files.
• Stops are written down. With a tunnel per copy, a stopped earner beside a ready tunnel could only mean a reboot or a crash, so the heartbeat started it. With a shared tunnel that is also what a stopped copy looks like while its siblings run. So a stop writes the earner's name to /data/.proxied_held before anything is stopped (and refuses if it cannot), a start or a removal clears it, and the heartbeat never starts a held copy. The tunnel stops when no copy that should run is left in it, and is removed with its secret volume when no copy is left at all.
• Traffic and the cap. Every copy reports its tunnel's counters and id. The cap already keys each baseline by worker:tunnel id, so N entries with one id count once. Per-copy traffic is gone; per-proxy-per-box traffic is what the UI shows. That loss is accepted.
• Migration. At worker start, per proxy, the settings are read back from a running, ready legacy tunnel's secret files (nothing is asked of the UI), the shared tunnel is started and proves its exit, and each legacy copy is recreated inside it before its old tunnel is removed. A legacy pair that was stopped moves held, so it stays stopped. Any failure leaves the pairs not moved yet as they are: each is valid and fails closed. A proxy with no running tunnel moves on its next start. Running it again does nothing.

The fail-closed argument¶

1. The firewall loads before anything else. Before tailscaled or tun2proxy starts, cp-tunnel loads table inet cashpilot with an output policy of drop. Egress on the real interface is allowed only for tailscaled's own fwmark, or for the proxy's address and port.
2. Earners cannot forge that mark: they get no NET_ADMIN, NET_RAW or SYS_ADMIN. Sharing changes nothing here. N earners in one namespace are N unprivileged processes behind the same table.
3. Exit down, tailscaled crashed, re-authentication failed: no route through tailscale0, so the drop policy applies. Kernel mode means no userspace dialer.
4. The tunnel container dies or restarts: the earners stay in the old namespace, which keeps only lo. The heartbeat restarts them into the new one only after /run/ready is written, which happens only after the firewall is loaded and a probe saw an exit that is not the box's own.
5. Reboot or Docker restart: earners have restart policy no, so Docker never starts one before the tunnel. Only the heartbeat gate does.
6. The LAN and tailnet fence (private and tailnet ranges dropped on the way into the tunnel) applies to the whole namespace, so to every copy in it.
7. New: copies in one namespace see each other's sockets in /proc/net and can reach each other's 127.0.0.1 listeners. They still cannot sniff traffic or change the firewall. One tailscaled failure cuts every copy on that proxy on that box; it fails closed, and those copies already shared the exit.

How it is tested¶

• Unit tests (tests/test_proxy_shared_tunnel.py) run the real module against an in-memory Docker that tracks labels, status, NetworkMode, the secret volume and the ready file: reuse, replacement, a tunnel that cannot start, stop and start of one copy among several, held copies and the heartbeat, a reboot, the cap counting one tunnel once, remove_all keeping a tunnel another service uses, and migration (idempotent, untouched on failure, stopped pairs stay stopped).
• scripts/proxy_tunnel_mutants.py applies twelve mutants, one per promise above, and requires every one to turn those tests red.
• scripts/proxy_shared_tunnel_lab.sh runs against a real Docker daemon and a local headscale tailnet. With two earners in one tunnel it checks egress through the exit only, no interruption of one copy by another's stop, no tailnet access, no DNS name on the tunnel's interface, no interface during a tunnel restart, and no escape or packet with the exit node stopped, tailscaled frozen or killed, or no exit node set. Three negative controls must turn those checks red: the firewall deleted, a copy created on the bridge instead of inside the tunnel, and a tunnel image whose output policy is accept. Then it drives the worker's own code with 1 and 7 copies, on this tree and on the base branch, counts the tunnels, measures cgroup memory, and migrates copies the base branch deployed.

Follow-ups¶

• Recover in place. Supervise tailscaled and tun2proxy inside the tunnel and restart the process instead of exiting, removing /run/ready during the gap. The namespace and the firewall survive, so the copies need no restart: a gap of seconds instead of a heartbeat.
• Shrink the tunnel. Measure GOMEMLIMIT and GOGC for tailscaled (start at 64 to 96 MiB), build one multicall binary so the 34 MB CLI is not paged in at start, and set a memory limit above the startup peak.
• A copy that wants its own tunnel, for owners who do not want two providers' software in one namespace.
• CashPilot Desktop still runs one tunnel per copy and gets the same change separately.
• Kernel WireGuard (option C). A one-shot privileged helper creates the interface and moves it into a --network none holder; on the exit side, a one-shot helper sets up the interface, forwarding and NAT, and exits. The same leak suite and negative controls as option A.
• Your tailnet policy. Grant the tunnels' tag only autogroup:internet through the exit node, and nothing into the tag (before you add one).