Race condition in vnode code can lead to vnode destruction
Our vnode setup is structured such that when we start a vnode via vnodesetup
, it then fires off an instance of mkvnode.pl
to do the actual work. Both of these processes then hang around, mkvnode
in case the vnode dies on it own, and vnodesetup
as the point of contact for signals from outside. In the latter case, vnodesetup
will propogate a signal to mkvnode
to shutdown or teardown the vnode.
mkvnode
interprets HUP to mean "destroy all evidence of the vnode", USR1 to mean "halt the vnode and retain the disk", and USR2 to mean "reboot".
vnodesetup
interprets TERM or HUP to mean "halt", USR1 to mean "destroy", USR2 to mean "reboot".
In other words, the meanings of HUP and USR1 are reversed. One can already see where hilarity might ensue...
When running on Xen 4.9 (aka Ubuntu 18 dom0), I have seen a case where, when shutting down the physical host, systemd
will send a TERM to all processes and then someone (systemd
again?) sends a HUP to all processes. If mkvnode
happens to get the HUP from whereever before it gets the USR1 trickled down from vnodesetup
, it will destroy the vnode. If it gets the USR1 first, it blocks all other signals and thus never gets the HUP and the vnode is just shut down.
The specific case showed (in syslog):
Aug 30 09:51:35 vhost3 systemd[1]: session-2746.scope: Killing process 42486 (vnodesetup) with signal SIGTERM.
Aug 30 09:51:35 vhost3 systemd[1]: session-2746.scope: Killing process 42527 (mkvnode.pl) with signal SIGTERM.
and in the vnodesetup/mkvnode log we see:
vnodesetup (42486) caught a SIGTERM!
TIMESTAMP: 08/30/2019 09:51:35.234850 vnodesetup shutting down ...
mkvnode (42527) caught a SIGHUP!
vnodesetup (42486) caught a SIGHUP!
TIMESTAMP: 08/30/2019 09:51:35.235559 vnodesetup shutting down ...
Container is being killed
Killing the vnode ...
Note the SIGHUP to mkvnode
and the fact that vnodesetup
gets both a TERM and a HUP before starting to do anything.