|
|
# SMP support for LCDs
|
|
|
----------------------
|
|
|
|
|
|
### Need for SMP
|
|
|
|
|
|
* Isolated ixgbe driver works on a single threaded LCD, where packets sent by
|
|
|
multiple user space threads are handled. LCD takes care of sending these packets
|
|
|
(`xmit`) and cleaning up the descriptors once they are sent out (`ixgbe_poll`).
|
|
|
|
|
|
* When multiple threads are sending packets, a single threaded LCD is not able
|
|
|
to cleanup the used descriptors (poll), whereas in a native driver multiple
|
|
|
SOFTIRQ threads can concurrently work on cleaning up the descriptors.
|
|
|
|
|
|
* This prevents us from sending packets from more than 2 threads.
|
|
|
|
|
|
|
|
|
### Design
|
|
|
* Microkernel creates 'N' LCD threads instead of one (1 parent thread, N-1
|
|
|
child threads). All threads enter Vt-X domain with the same EPT, so that they
|
|
|
can access the same set of memory as the parent LCD.
|
|
|
|
|
|
* The LCD parent thread or the first thread that gets created does most of the
|
|
|
work, from setting up the memory to initializing the device. The control
|
|
|
plane is taken care by LCD parent thread.
|
|
|
|
|
|
* During the setup phase, each LCD child thread creates a communication channel to
|
|
|
communicate with KLCD.
|
|
|
|
|
|
* We could split the work across LCD threads in any fashion.
|
|
|
- Parent thread sends packets and child thread(s) cleanup the descriptors.
|
|
|
- Similar to a single-threaded LCD, all threads are just a clone of the
|
|
|
parent thread. (They do both tx and cleanup).
|
|
|
|
|
|
### LCD initialization
|
|
|
|
|
|
* The first thread that enters the isolation domain initializes LCD and the
|
|
|
remaining threads wait for completion of init. Once initialization is done,
|
|
|
all other threads proceed to create async channels for communicating with KLCD.
|
|
|
|
|
|
### Locks
|
|
|
|
|
|
* **Spinlocks**: All memory allocation routines need to have locks (kmalloc and
|
|
|
friends). The locks are already in-place but just elided by macros from
|
|
|
`post_hook.h`
|
|
|
|
|
|
* **mutexes**: Mutexes should be converted to spin locks for LCDs as mutexes
|
|
|
require scheduler routines to put a process to sleep and wakeup.
|
|
|
|
|
|
### EPTs
|
|
|
|
|
|
* We need two LCD threads to be entering VT-x domain. Ideally they should have
|
|
|
two different VMCS. Microkernel creates two VMCS structures and enter VT-x
|
|
|
using them on two separate CPUs
|
|
|
|
|
|
### Hypercalls
|
|
|
|
|
|
* When operating on the same LCD structure, all hypercalls should be protected
|
|
|
by a lock in the microkernel. Normally, control plane of the driver would be
|
|
|
taken care by the parent thread, we should protect the LCD struct in
|
|
|
microkernel as we don't trust the isolated code. |