Commit ded1cd32 authored by Charles Jacobsen's avatar Charles Jacobsen Committed by Vikram Narayanan

Support for multiple threads in lcd (limited). Major clean up of code.

In line with more recent design discussions, we now have (limited)
support for running multiple threads inside an lcd.

Each thread will have its own hardware vm, but share a guest
physical address space and cspace.

It's limited for now because threads cannot handle interrupts/exceptions
internally in the lcd. This will require a per-thread TSS (much like
Linux's per-core TSS/interrupt stack). I removed the gdt/idt/tss for
now (Cf. with Dune, they don't use gdt/idt/tss) and will tackle that later
after finishing more important stuff.

I have only tested the code for running one hardware vm inside an lcd. Some
code is missing proper locking for the future when we have multiple threads
inside an lcd. I'm leaving this for now.

The microkernel uses a simple bitmap for guest physical page allocation.

Removed blob loading - code is set up for running modules exclusively.

See the headers and Documentation/lcd-domains for more info.

I put a flag at the top of files that are not currently in use, and will
probably be deleted/incorporated later.
parent b9e42c88
This code is in virt/lcd-domains and include/lcd-domains. It is the
arch-independent layer of the LCD microkernel.
The two main objects are struct lcd and struct lcd_thread, defined in
struct lcd contains the guest physical address space (in
underlying lcd_arch), and a list of lcd_thread's. It will soon contain
the lcd's cspace when that is incorporated.
struct lcd_thread corresponds with a host kernel thread that is running
inside the hardware virtual machine. It contains a pointer to the thread's
utcb (for easy microkernel access), and a pointer to the underlying
lcd_arch_thread (the hardware vm).
Why have one kernel thread / hardware vm for each lcd_thread? Answer: To keep
the microkernel simple. The microkernel could add an additional layer of
virtualization on top of the hardware vm, so that we didn't have so many
hardware vm's floating around. But it would be complicated and we would then
have to write scheduler code in the microkernel.
A struct lcd is created by providing a module name. The module will be loaded
inside the lcd, and an initial lcd_thread will be created (stored in the
struct lcd's init_thread field) that, when started, will execute the module's
init code.
See also the comments in the header lcd-domains.h (above) and the test cases
in virt/lcd-domains/lcd-tests.c.
Aside from building and installing the kernel code, you will need to do one
extra step, explained in detail below.
We don't want tricky logic for locating modules, so we want to re-use the
request_module facility in the kernel. But this relies on the user space
modprobe tools. So, we did the following:
-- we modified the module loading code in the kernel so that a caller
can safely load a module that is destined for an lcd in the host
(modules destined for an lcd *will not* have their init code executed
when installed in the host, nor their exit code executed when they
are uninstalled from the host)
-- we added an ioctl interface that user code can use to
load a module destined for an lcd; it uses the patched module loading
-- we created a patched modprobe that uses this interface
-- we patched request_module to allow kernel code to load a module
destined for an lcd, using the patched modprobe
So, when you call lcd_create, the kernel loads the module using the patched
This means you need to have the patched modprobe properly installed!
Step 1
Build and install the kernel and all modules. In the root directory of the
kernel source,
[ 1 ] make menuconfig
-- go into Virtualization (2) and select Lightweight Capability
Domains and Intel Support for LCDs
-- it is recommended you build them as modules, for debugging
[ 2 ] exit and save the configuration
[ 3 ] make
-- use make -j 8 if e.g. you have 8 cores, will go faster
[ 4 ] sudo make modules_install install
-- order is important!
-- this should automatically update the grub boot menu
Step 2 - Patched Modprobe Setup
The patched version is inside tools/module-init-tools.
To build and install, enter the module-init-tools directory,
and do the following:
[ 1 ] aclocal -I m4 && automake --add-missing --copy && autoconf
[ 2 ] ./configure --prefix=/ --program-prefx=lcd-
[ 3 ] make
[ 4 ] (sudo) make install
This will install the patched /sbin/lcd-modprobe and /sbin/lcd-insmod,
as well as the other init tools that were left untouched. The
request_module will use lcd-modprobe to load the module.
The man pages won't install on emulab (since /share is read only).
You can specify a different man dir via configure if you wish.
[Note: The only changes to init tools are in modprobe.c and insmod.c; only
the changes in modprobe.c are of interest (lcd-insmod is not currently
used/needed). Instead of doing the Linux init_module system call,
lcd-modprobe does an ioctl call to the LCD driver (hence, the LCD driver
must be loaded), with the bytes of the module, its size, and command
line options.]
Step 3 - Reboot and install
After rebooting the machine, select the new kernel to boot it.
After booting, if you built the lcd system as modules, do:
[ 1 ] insmod ${MODULE_PATH}/arch/x86/lcd-domains/lcd-domains-arch.ko
[ 2 ] insmod ${MODULE_PATH}/virt/lcd-domains/lcd-domains.ko
where ${MODULE_PATH} is something like /lib/modules/3.10.14/kernel.
This will install the lcd system.
You can now create an lcd using the example below.
Here is an example of how to start up an lcd with a module named foo.ko. foo.ko
should already be compiled and installed in the system's module load path.
struct lcd *lcd;
struct lcd_thread *lcd_thread;
int ret;
* Create the lcd
ret = lcd_create("foo.ko", &lcd);
* Start the lcd's init thread (will run foo.ko's init routine)
ret = lcd_thread_start(lcd->init_thread);
/* (...wait for a while, maybe sleep...) */
* Kill the init thread
ret = lcd_thread_kill(lcd->init_thread);
* Tear down the LCD
This one is a real zinger.
A good chunk of the current arch-independent code is for setting up the
boot guest virtual address space for an lcd. We assume that the lcd will take
over managing this, so we've kept allocation logic dirt simple.
Note that the microkernel is protected from what the lcd does to its guest
virtual address space. The microkernel manages the lcd's guest physical
address space, and the host pages the lcd has access to, so it can safely
write to memory without causing a page fault.
......@@ -3,13 +3,95 @@
This code is in arch/x86/lcd-domains/ and
The two main objects are struct lcd_arch and struct lcd_arch_thread, defined in
struct lcd_arch contains the extended page table (EPT) guest physical address
space information, and a list of the contained lcd_arch_thread's.
struct lcd_arch_thread corresponds with an Intel VT-x virtual machine, and
contains all state needed to manage it. Each lcd_arch_thread uses the EPT of
its parent lcd_arch.
See the comments in the header lcd-domains-arch.h (above), and the test
cases in arch/x86/lcd-domains/lcd-domains-arch-tests.c.
Here is an example how to set up one lcd_arch with one lcd_arch_thread, and
run it (without the error checking).
struct lcd_arch *lcd_arch;
struct lcd_arch_thread *lcd_arch_thread;
* Create the lcd_arch
lcd_arch = lcd_arch_create();
/* (...Allocate and map pages in the LCD...) */
* Create and add a thread
lcd_arch_thread = lcd_arch_add_thread(lcd_arch);
/* (...Set up the thread's program counter, etc...) */
* Tear down the thread
* Tear down the LCD
One of the hardest, error-prone, confusing parts of this is we need to keep
track of four different kinds of addresses:
(1) host physical
(2) host virtual
(3) guest physical
(4) guest virtual
We have created four different data types (much like those used for
page tables -- pte_t, pmd_t, ...):
(1) host physical hpa_t
(2) host virtual hva_t
(3) guest physical gpa_t
(4) guest virtual gva_t
and routines for converting between them. Some conversions are not supported
directly, and must be done with more work -- e.g., guest physical to host
virtual requires an EPT. But some, like host physical to host virtual, can
be done using host macros. See the arch header.
Host physical addresses need to be stored in the EPT.
Guest physical addresses need to be stored in the guest virtual page tables.
>>>>>>> Support for multiple threads in lcd (limited). Major clean up of code.
TODO: Some of the arch code is not thread safe.
Locks are used on an lcd_arch's ept and its list of lcd_arch_threads.
We use mutexes for now, so some functions are not safe to call from
......@@ -21,16 +103,22 @@ This makes debugging easier since we can interrupt locks (no deadlocks).
We probably don't need a gdt/tss/idt right now. I'm removing that.
We will need GDT/TSS/IDT in the future when we provide full support for running
multiple threads inside an LCD, and if those threads are handling some
exceptions/interrupts (like guest virtual page faults).
When an LCD thread is spawned, the creator should provide 4 pages of memory:
In the future, we could provide a secure interface for an lcd thread to
change its gdt/tss/idt info in its vmcs. The lcd thread could allocate a gdt,
etc. inside, and use the interface to update the gdtr, tr, idtr as needed.
The lcd thread could also tell the microkernel not to vmexit for certain
external interrupts.
-- 1 for thread's stack/utcb
-- 1 for its gdt
-- 1 for its tss
-- 1 for its idt
If we do tss, each thread will probably need its own - since the tss contains
a stack pointer that varies with the threads.
The IDT/GDT could be shared by threads, but it might be complicated, and the
microkernel would have to make certain assumptions or enforce certain polices
to protect itself (e.g., if it needed to modify the IDT/GDT when setting up
a new thread). Having a TSS per thread makes some sense - a thread here is more
like a logical core - and this is what Linux does.
I'm putting the gdt / tss init code here for reference since it was quite
complicated to do (some of the macros won't be relevant anymore):
This diff is collapsed.
* Anton Burtsev, University of Utah, 2014
* elfnote.h
* Author: Anton Burtsev <>
* Copyright: University of Utah
#include <linux/slab.h>
#include <lcd/cap.h>
#include <linux/elf.h>
#include <stdio.h>
#include <unistd.h>
// Sync IPC specific routines
#include <linux/types.h>
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment