nsjail with netns

Nsjail and netns

I’ve been a long time user of nsjail which is a pretty handy tool to create linux namespaces. My common usecase for nsjail is to put services into their own, well, jail - by only giving them access to specific folders in the filesystem.

As an example: The webserver which served you this page runs with the following configuration:

mode: ONCE
hostname: "nsjail-hostname"

clone_newnet: false
time_limit: 0

rlimit_cpu_type: INF
rlimit_nofile: 4096
rlimit_fsize: 320960

uidmap {
        inside_id: "1099"
        outside_id: "psa-www"
}
gidmap {
        inside_id: "1099"
        outside_id: "psa-www"
}
mount {
	src: "/tank/websites"
	dst: "/web"
	is_bind: true
}
mount {
        src: "/usr/local/bin/psa-www"
        dst: "/psa-www"
        is_bind: true
}
# standard paths.
mount {
        src: "/dev/urandom"
        dst: "/dev/urandom"
        is_bind: true
}

And this works fine: The process is neatly isolated and can only access resources it has to - it doesn’t even see any other processes given that it runs in its own pid namespace.

However, one thing that always bothered me is that the process still has full network access, so how can we limit that? Nsjail does support this with macvlan, but this never really worked for me in practice (requiring your hosting provider to allow you to show up with unexpected mac addresses), so i started to look around for options.

Option 1: Filter based on UID

nftables allows matching by uid which is simple and works, but always felt a bit meh.

Example:

chain output {
	type filter hook output priority 0; policy accept;
	meta skuid psa-www counter jump psawww-filter
	counter
}

chain psawww-filter {
	udp dport 123 counter accept
	counter drop
}

Option 2: Create a new network namespace

This is a more involved option but feels much cleaner: We use ip to create a new network namespace - and then launch nsjail in it.

Creating a new network namespace requires a little bit of work which i automated in a tool called setupns.pl.

Example:

$ git clone https://codeberg.org/adrian-blx/setup-ns.git && cd setup-ns
$ su -
# ./setup-ns.pl -t -r -n foobar -4 10 -6 3

This will create a new namespace called foobar with 2 new interfaces:

• foobar-out with IP 10.97.10.1/24 and fd00:97::3:1/112
• foobar-in (inside the new network ns) with IP 10.97.10.2/24 and fd00:97::3:2/112

After that, you can launch an nsjail inside the new namespace:

# ip netns exec foobar nsjail  -C ./nsjail.conf  -- /bin/foo ...

Note that setup-ns.pl will setup a default route for both IPv4 and IPv6 since we launched it with the -r option. In order for routing to work, you’ll need to configure masquerading and forwarding in nftables.

jail_ifs = { foobar-out }
wan_if   = { enp4s0 }
# ....

table inet filter {
	chain forward {
		type filter hook forward priority 0; policy drop;
			iifname $jail_ifs  oif     $wan_if   counter accept;
			iif     $wan_if    oifname $jail_ifs counter accept;
	}
}
# Postrouting masquerade
table inet nat {
	chain postrouting {
		type nat hook postrouting priority 100;
		ip  saddr 10.97.0.0/16 iifname $jail_ifs oif $wan_if counter masquerade
		ip6 saddr fd00:97::/96 iifname $jail_ifs oif $wan_if counter masquerade
		counter
	}
}

and of course, forwarding also needs to be enabled:

# sysctl -w net.ipv6.conf.all.forwarding=1
# sysctl -w net.ipv4.conf.all.forwarding=1

That’s all: With this setup, you’ll get a completely new network namespace with a dedicated interface / route which you can easily filter on.