GitLab

This code is ugly, but it's working.

Tested with basic module, and appears to be working
properly. I will soon incorporate the patched
modprobe into the kernel tree, and then this code
will be usable by everyone.

The ipc code is still unimplemented. The only
hypercall handled is yield. Also note that other
exit conditions (e.g. external interrupt) have not
been fully tested.

Overview:
-- kernel code calls lcd_create_as_module with
   the module's name
-- lcd_create_as_module loads the module using
   request_lcd_module (request_lcd_module calls
   the patched modprobe to load the module, and
   the patched modprobe calls back into the lcd
   driver via the ioctrl interface to load the
   module)
-- lcd_create_as_module then finds the loaded
   module, spawns a kernel thread and passes off
   the module to it
-- the kernel thread initializes the lcd and
   maps the module inside it, then suspends itself
-- lcd_run_as_module wakes up the kernel thread
   and tells it to run
-- lcd_delete_as_module stops the kernel thread
   and deletes the module from the host kernel

File-by-file details:

arch/x86/include/asm/lcd-domains-arch.h
arch/x86/lcd-domains/lcd-domains-arch-tests.c
arch/x86/lcd-domains/lcd-domains-arch.c
-- lcd was not running in 64-bit mode, and my
   checks had one subtle bug
-- fixed %cr3 load to properly load vmcs first
-- fixed set program counter to use guest virtual
   rather than guest physical address

include/linux/sched.h
-- added struct lcd to task_struct

include/linux/init_task.h
-- lcd pointer set to null when task_struct is
   initialized

include/linux/module.h
kernel/module.c
-- made init_module and delete_module system calls
   callable from kernel code
-- available in module.h via do_sys_init_module and
   do_sys_delete_module
-- simply moved the majority of the guts of the
   system calls into a non-system call, exported
   routine
-- take an extra flag, for_lcd; when set, the init
   code skips over running (and deallocating) the
   module's init code, and the delete code skips
   over running the module exit
-- system calls from user code set for_lcd = 0; this
   ensures existing code still works

include/linux/kmod.h
kernel/kmod.c
kernel/sysctl.c
-- changed __request_module to __do_request_module; takes
   one extra argument, for_lcd
-- __request_module   ==>  __do_request_module with for_lcd = 0
-- request_lcd_module ==>  __do_request_module with for_lcd = 1
-- call_modprobe conditionally uses lcd_modprobe_path, the path
   to a patched modprobe accessible via sysfs

include/lcd-domains/lcd-domains.h
-- added lcd status enum; see source code doc
-- three routines for creating/running/destroying
   lcd's that use modules; see source code doc

include/uapi/linux/lcd-domains.h
-- added interface defns for patched modprobe to call into
   lcd driver for module init; lcd driver loads
   module (via slightly refactored module.c code) on behalf
   of modprobe

virt/lcd-domains/lcd-domains.c
-- implementation of routines for modules inside lcd's
-- implementation of module init / delete for lcd's
   (uses patched module.c code)

virt/lcd-domains/Kconfig
virt/lcd-domains/Makefile
virt/lcd-domains/lcd-module-load-test.c
virt/lcd-domains/lcd-tests.c
-- added test module for lcd module code
-- test runs automatically when lcd module is inserted

In the process of debugging, major corrections and
debug checks implemented (this is a big squash):
-- coded up entry checks documented in Intel SDM
   V3 chp 26 (this should help later if settings
   are changed, make it less risky to experiment
   and give more confidence)
-- fixed host tr base addr bug (the worst bug! caused
   system to hang and then crash, since the tss
   was erroneous...)
-- fixed vmx_entry to properly set host rsp before
   entry
-- setting host sysenter and idt info
-- fixed cs ar bytes bug
-- fixed gdt limit bug
-- fixed tr type bug
-- extra settings added to cr0 and cr4, but may
   not be needed ... (debug fix attempt)
-- lstar mstar autoload, may not be needed ...
   (debug fix attempt)

  -- LCD export an API to its domains that is accessible via
  capability invocations (well, syscalls if you like)

  -- Kernel runs a thread that implement this API, e.g. serves
  capability invocations

  -- I've split capability allocation and cspaces, this makes much more
     sense -- complex domains can implement custom allocation policies,
     simple domains go with a static set of caps
  -- This is a list-based (zone like) allocator

Fixed (another) nasty casting bug in the
code that initializes the ept pointer. I
was using the old way of casting, and overlooking
that I was storing a host virtual pointer
inside a host physical pointer. This caused
invept to fail (invept checks if the eptp is
valid).

I also fixed invept and invvpid to print an
error message (like vmload and vmclear), rather
than generate an invalid opcode exception. It's
easier to debug and understand the problem.

In the future, we could more carefully parse the
error returned.

(Sorry for the barrage of pushes, but I want
to make sure I push changes in case I crash
the machine...)

Constructors and casting inlines are in arch-dep
code header.

Updated tests.

Originally, destroying the ept structures also
freed the mapped host physical memory. This is
leading to bad double free errors. Instead, I
will report a potential memory leak if an ept
pte is present.

Users of the ept should unmap any host phys
memory before destroying the ept (will shift
over ept unmap range from arch-indep to
arch-dep next).

Updated arch test04 to use vmx_free_ept, now
that it doesn't try to free mapped host phys
mem.

Added pgd_pfn to asm/pgtables.h.

Need to load vmcs in order to modify it.

Can't set stack pointer until vmcs is loaded.

Pushing in case of crash ...

This was tricky. Unfortunately, since guest physical
addresses are wrapped in the paging entries, we can't
use some of the linux macros / routines (e.g.,
pud_offset). This would allow us to benefit from pud-
and pmd-folding, making the code more portable. Perhaps
we'll think of something later...

The code will only work on a linux with 4 paging levels
(pgd, pud, pmd, page table). lcd_mm_gva_walk is the
main routine that traverses the hierarchy. Each subroutine
translates a paging structures guest physical to host
virtual address, and gets the correct entry.

A simple watermark is used (rather than a bitmap) for
allocating guest physical memory.

Changed to use macro accessor in lcd_arch. I think
this is less confusing and less bug prone. So,
to access lcd's utcb, do:

struct lcd_utcb *utcb;
utcb = LCD_ARCH_GET_UTCB(vcpu);

(Code before relied on free ept to free memory. But
this could be confusing and hard to debug in the future...)

Just clears the ept entry.

arch/x86/lcd-domains/...
-- Fixed program counter and stack pointer initialization
-- Added guest virtual paging pointer modifier routine

include/lcd-domains/lcd-domains.h
-- Added the minimum amount of data for guest virtual paging
   setup

virt/lcd-domains/lcd-domains.c
-- Added general routines for lcd create / destroy.
-- Moved mm code to this file (to avoid future name clashes and
   be more self contained ... is this why kernel source files are
   so big?)
-- Added general routine for initializing an lcd's gva.
-- Re-factored blob initialization to use mm routines. Documented
   memory layout in the source code.

Problem: I'm relying heavily on the ept tear down to free any
memory mapped in there. But this is getting ugly ... Next step:
Write quick routines to properly unmap and explicitly free memory.

User code calls ioctl with LCD_RUN_BLOB ioctl number and
lcd_blob_info (containing userspace address of blob and
blob order) -- defined in public include/linux/lcd-domains.h.

The blob must be N pages, and N must be a power of 2 (for
easy driver code). blob_order = log2(N). The blob consists
of machine instructions that are loaded in the lcd and
executed. The machine instructions cannot access any memory,
including the stack (for now, until gv paging is in place).

-- Added lcd_arch_set_pc for setting the lcd's program counter.
-- Added driver code in lcd-domains.c for handling the ioctl
   request, loading the blob from user space into a fresh lcd,
   and running the lcd (in a loop).

Some data structures in struct lcd_arch are mapped in
the ept (so that the lcd has access to them). When the
ept is freed, these data structures are freed.

All tests passing now for lcd_arch_create.

Switch from macros to inline functions for type safety
and sanity.

EPT stress test04 now passes (aside from free ept).

Successful build and insmod on emulab, but errors when trying
to create lcd (perhaps errors mapping in ept). Regression tests
should help and save debugging time in future.

Updated Kbuild, Kconfig, Makefiles.

(forgot to git add utcb header)

Syscall (hypercall?) codes are in include/lcd-domains/syscall.h.

Added syscall code accessor macro to lcd-domains-arch.h.

Fixed capability register accessor macros to use %rbx.

Simple name changes and slightly different access pattern
for ipc registers. Updated headers and code. Removed ipc.h
header for now.