Fully isolate WFAs
Fully isolate WFAs. Right now they run as root, in an isolated netns. I spent a lot of time looking at the ways to fix this, but they are surprisingly hard. Basically, we want the WFA to run with CAP_NET_RAW, so it can speak the capability protocol over the virtual iface in its netns, but we want it to run as nobody, and ideally have only a single
rlimit'd dir it can r/w into. Giving CAP_NET_RAW to an untrusted child process with reduced privs wasn't possible on Linux until ambient capabilities came in kernel 4.3 . (The default Cloudlab images are at 4.1, so nothing comes easy for us, but although I've already written a privilege deescalating exec() for us before I knew about all this). Anyway, you had to use filesystem caps and set the CAP_NET_RAW bit on the binary you wanted to execute. We can't set CAP_NET_RAW for python, obviously! And caps never transfer across the exec() boundary (until ambient caps in 4.3). Moreover, in python 2, there is no recvfrom(), so you cannot get a previously-opened raw socket over a unix socket from some other process. There is a python
rawsock module that works around this by a binary module hack where there's a binary that opens a raw sock, which is CAP_NET_RAW in the filesystem, and it gets run by a python program. Then, another binary module loaded by the python script that invoked the first binary connects to the first binary over a unix socket (or something), and recvfrom()s the raw socket. Then that sockfd is turned into a python file from its file descriptor number... and then the python script can send raw packets on it. This is not super ideal for us... but we could do our own version that is more restrictive for our case.
Then, we have to restrict filesystem access. We could actually build an lxc container for each WFA that has its own FS, or has read-only unionfs mount to the root fs (but that might be hard, last time I looked at this, unionfs was still outside the mainline kernel, although Ubuntu always built it).
It should be straightforward to put the WFA into a separate cgroup to rlimit it.