Isolated ixgbe driver works on a single threaded LCD, where packets sent by
multiple userspace threads are handled. LCD takes care of sending these packets
(xmit) and cleaning up the descriptors once they are sent out (ixgbe_poll).
When multiple threads are sending packets, a single threaded LCD is not able
to clean up the used descriptors (poll), whereas in a native driver multiple
SOFTIRQ threads can concurrently work on cleaning up the descriptors.
This prevents us from sending packets from more than 2 threads.
Microkernel creates 'N' LCD threads instead of one (1 parent thread, N-1
child threads). All threads enter Vt-X domain with the same EPT, so that they
can access the same set of memory as the parent LCD.
The LCD parent thread or the first thread that gets created does most of the
work, from setting up the memory to initializing the device. The control
plane is taken care of by LCD parent thread.
During the setup phase, each LCD child thread creates a communication channel to
communicate with KLCD.
We could split the work across LCD threads in any fashion.
Parent thread sends packets and child thread(s) clean up the descriptors.
Similar to a single-threaded LCD, all threads are just a clone of the
parent thread. (They do both tx and cleanup).
The first thread that enters the isolation domain initializes LCD and the
remaining threads wait for completion of init. Once initialization is done,
all other threads proceed to create async channels for communicating with KLCD.
Spinlocks: All memory allocation routines need to have locks (kmalloc and
friends). The locks are already in place but just elided by macros from
mutexes: Mutexes should be converted to spin locks for LCDs as mutexes
require scheduler routines to put a process to sleep and wakeup.
We need two LCD threads to be entering VT-x domain. Ideally, they should have
two different VMCS. Microkernel creates two VMCS structures and enters VT-x
using them on two separate CPUs
When operating on the same LCD structure, all hypercalls should be protected
by a lock in the microkernel. Normally, the control plane of the driver would be
taken care by the parent thread, we should protect the LCD struct in
microkernel as we don't trust the isolated code.