1. 26 Oct, 2016 3 commits
    • Charlie Jacobsen's avatar
      Adds support for passing short strings back and forth. · 92e45ef3
      Charlie Jacobsen authored and Vikram Narayanan's avatar Vikram Narayanan committed
      Simple illustrative example in test-mods/string_example. Ping-pongs
      a string 'hello' back and forth a few times.
      
      The string/data needs to be contained inside a single page
      for now. This should hopefully handle most of our immediate
      use cases.
      
      Idea: The sender grants the receiver access to the page that
      contains the data. The receiver maps it in their address space. This
      is clearly too slow for the main data path, and it's a bit hackie.
      But it works for strings that are passed in control plane interactions
      (rather than trying to cram the darn thing inside the message buffer).
      
      Wiki to be updated soon with more details.
      92e45ef3
    • Charlie Jacobsen's avatar
      Major overhaul of build process. · 8198c2fb
      Charlie Jacobsen authored and Vikram Narayanan's avatar Vikram Narayanan committed
      Full kernel build no longer required. Yay! This should
      cut down on dev time a lot.
      
      I moved all of the LCD source into $(kernel-src)/lcd-domains/,
      so it's all in one spot. There is now a top-level makefile in
      there that triggers building liblcd, the microkernel, and the
      examples. This is built as an *external* build now, even
      though the directory is in the kernel source. The build now takes
      under a minute to do everything LCD related.
      
      This should also make verification easier in the future (e.g.
      building with clang) if we aren't ensnared in the kernel
      source.
      
      Of course, to use the microkernel and examples, you have to
      build the patched kernel and install it. But now when you
      make a few lines of changes in e.g. an example, you don't have
      to trigger a top-level kernel build to rebuild it. Running
      the full kernel build takes on average about 3 - 4 minutes
      (some files are generated everytime, linking is done, and so
      on), and can take upwards of 30 minutes for a full build if you
      re-config'd.
      
      Which brings me to my other change: no more config for LCDs
      in menuconfig. If we create menu entries for every example
      and so on, we end up changing the config too often, and this
      triggers full kernel rebuilds == waste of time. We can use
      macros by setting them via compiler flags (e.g., -DSOME_FLAG).
      Furthermore, it wasn't making sense to me to do conditional
      compilation for LCD support (we always want to compile for that).
      Yes, changes aren't clearly delineated with macros, but you can
      see changes made by just doing 'git diff v3.10.14 some-file-or-dir'.
      
      The wiki has been fully updated with instructions for building,
      and other relevant parts (updated paths to files).
      
      I also took the opportunity to clean up some old stuff lying around
      that is dead (like lcdguest). I incorporated all of the documentation
      in Documentation/lcd-domains into the wiki so it's all in one
      spot now (including some helpful debug tips).
      8198c2fb
    • Charlie Jacobsen's avatar
      Adds full functioning example with glue code. · 424ac36d
      Charlie Jacobsen authored and Vikram Narayanan's avatar Vikram Narayanan committed
      Example is in virt/lcd-domains/test-mods/glue-example. It consists
      of a fake minix and fake vfs. The original "unmodified" code is
      in minix/original/main.c, vfs/original/main.c, and include/vfs.h.
      The glue code (written by hand) is in minix/glue and vfs/glue. The
      IDL is in idl/.
      
      You can build the fake minix and vfs for isolation or as regular
      modules. To build for isolation, run `make menuconfig' and under
      Test Modules --> Example Exercising IDL and Glue.
      
      To run minix and vfs in isolation, install the "boot" module,
      lcd-test-mod-glue-example-boot.ko. (Its code is under isol-boot/main.c.)
      
      The example will print status messages to the kernel logs so you
      can see a trace of the interaction.
      
      A few hacks were necessary to fully exercise all of the code. For example,
      the fake vfs is the one that invokes new_file and rm_file, in the middle
      of its dispatch loop.
      
      The code for the trampolines in vfs/glue/vfs_caller.{c,lds} is probably
      the ugliest part. That required a little bit of trickery and low-level
      hacking.
      424ac36d
  2. 25 Oct, 2016 37 commits
    • Charlie Jacobsen's avatar
      Adds support for creating a kLCD from another kLCD. · e49732d2
      Charlie Jacobsen authored and Vikram Narayanan's avatar Vikram Narayanan committed
      This is a boring but important commit that we need for
      our glue code example (that is still in progress).
      
      The main motiviation is: One kLCD (like a booting kLCD) may
      want to create other threads to run kernel modules, and it
      may want to grant them capabilities to resources. (Before
      this, this wasn't possible - all kLCDs were isolated from
      each other.) Another alternative (possibly better) for the
      future is to allow threads to share cspaces. But we're
      probably not there yet.
      
      Adds the following to the kLIBLCD interface:
      
      -- klcd_create_klcd - essentially creates another kernel
         thread and initializes the lcd-specific stuff like its
         cspace; the creator can then use lcd_cap_grant to grant
         this thread capabilities
      
      -- klcd_create_module_klcd - creates a klcd (using
         klcd_create_klcd) and loads a kernel module for the klcd
         to run
      
      -- klcd_destroy_module_klcd - destroys klcd (the kernel
         thread, etc.) and unloads the module
      
      Adds test and an example group of modules that show how to use
      this.
      e49732d2
    • Charlie Jacobsen's avatar
      Breaks apart microkernel into smaller pieces, and refactors regression tests. · 40bfcbdb
      Charlie Jacobsen authored and Vikram Narayanan's avatar Vikram Narayanan committed
      Trying to make the code base saner for others to read through.
      
      Tests are no longer ran every time you insert the microkernel. They are built
      as a separate kernel module (in virt/lcd-domains/tests/lcd-tests.ko).
      
      Ran tests again, and some of the LCD functional tests. All appears OK.
      40bfcbdb
    • Charlie Jacobsen's avatar
      Basic 'data store' code added to libkernel. · ef040dc7
      Charlie Jacobsen authored and Vikram Narayanan's avatar Vikram Narayanan committed
      Just the capability code adapted for use inside the libkernel.
      
      Basic tests passing. Coalesced a lot of the liblcd tests into
      one spot (in liblcd/lcd/test.c).
      ef040dc7
    • Charlie Jacobsen's avatar
      Linux slab allocator and page allocator inside LCDs. · 35ed57d7
      Charlie Jacobsen authored and Vikram Narayanan's avatar Vikram Narayanan committed
      Single threaded, no locks, no fancy NUMA/percpu.
      
      Passing some simple examples. Added a memory management
      example module, in test-mods/mm, that exercises a lot
      of this new code.
      
      1:
      
      I moved in and adapted our existing guest virtual
      paging code from kliblcd.c. I'm using statically
      allocated bitmaps and arrays for tracking allocations
      in the guest virtual and physical address spaces.
      Using identity mapping for ease. (I decided not to
      use Linux's page allocator since it's too intertwined
      with the boot process - percpu variables, freeing
      init mem, boot allocator, all kinds of complexity ...)
      It might not be too hard to reimplement the buddy
      allocator algorithms, since I had to include a
      statically allocated array of struct pages anyway.
      
      I've set aside about 16 MBs for dynamic page allocations,
      but this can be changed using macros. You can allocate
      1, 2, 4, 8, etc. pages at a time. (The slab allocator
      requires this.)
      
      2:
      
      I finally broke down and set up boot info pages - 4
      boot pages right now, can be adjusted with a macro.
      Whoever boots the lcd needs to pack in information about the
      lcd's address space, initial cspace, and so on. 4 pages
      is enough to pack in information for larger modules
      like the mm example.
      
      3:
      
      I moved liblcd to a separate directory, and hacked the
      build system so that we can build liblcd as a static
      library and link it with example modules.
      
      liblcd/ contains lcd/, which has code for interacting
      with the microkernel and my simple page allocator.
      
      The Linux slab allocator is inside mm/, and some
      needed dependencies are in lib/. I made very few
      changes to the source code itself, but used some
      preprocessor/compiler hacks to make everything work. See
      Documentation/lcd-domains/liblcd.txt. I elided all of
      the locking and made it single core, single NUMA node.
      It's possible we'll see some bigs in the future, in code I haven't
      excerised yet (will probably manifest themselves as
      page faults).
      
      Ideally, we should have a separate tree for liblcd
      and building modules. That way we can avoid some of these
      hacks (maybe not all).
      
      4:
      
      Updated a lot of the documentation in
      Documentation/lcd-domains/.
      35ed57d7
    • Charlie Jacobsen's avatar
      Two simple IPC tests are passing. Still getting mysterious hang. · c667fea5
      Charlie Jacobsen authored and Vikram Narayanan's avatar Vikram Narayanan committed
      
      
      Working on moving Linux's mm into lcd's.
      
      I gave up trying to debug the hang. Confirmed the pages for the lcd's
      vm are the ones I expect. Turned on red zones. All tests are passing.
      Hang happens after insmod/rmmod of the lcd module about 10 - 20 times, it
      varies. Sometimes one core just silently dies / doesn't even respond to
      an NMI. Sometimes the ethernet driver complains (this could be an
      unrelated bug that was fixed upstream).
      
      Few things in this commit:
      
        1
      =====
      
      Updated documentation in Documentation/lcd-domains/.
      
        2
      =====
      
      Baby version of lib kernel, inside arch/x86/lcd-domains/liblcd.c.
      Unfortunately due to the recursive make, this needs to be textually
      included inside the modules destined for lcd's, for now.
      
        3
      =====
      
      Added new test modules and modified directory structure and
      build system. See documentation in Documentation/lcd-domains.
      
        4
      =====
      
      A few tweaks to the nmi handler to print a backtrace. May remove that in
      the future, as it's probably not safe to do inside an nmi handler (but if
      we're in that error state, we might be desperate to know what's happening ...).
      
        5
      =====
      
      Changed interrupt handling in arch-dependent code. The KVM code we were using
      is probably wrong for 64-bit - it doesn't properly switch stacks, etc., which
      is super important for 64-bit and may be impossible to emulate in
      software. I think this could be stale code inside KVM, but not sure. Dune
      doesn't use it. KVM doesn't ack external interrutps on vm exit, so I think
      this interrupt emulation code is always skipped (at least for non-nested
      VMs).
      
      Instead, we're not ack'ing interrupts on exit, and letting the native code
      do the right thing, like Dune.
      
      I was thinking this might be the source of the bad hang (stack
      overflow, e.g.), but not true.
      
      Conflicts:
      	include/linux/sched.h
      	kernel/watchdog.c
      	virt/lcd-domains/lcd-cspace-tests2.c
      Resolved-by: Vikram Narayanan's avatarVikram Narayanan <vikram186@gmail.com>
      c667fea5
    • Charlie Jacobsen's avatar
      Except IPC, kliblcd fully tested. Everything is working. · 664628a4
      Charlie Jacobsen authored and Vikram Narayanan's avatar Vikram Narayanan committed
      Documentation in Documentation/lcd-domains/...
      
      Loading, mapping, and running a module is working correctly, using
      all of the capability code that interposes on each operation (mapping,
      freeing pages, etc.).
      
      cptr allocation and indexing into cspaces is working correctly.
      
      IPC testing and debugging is coming next.
      664628a4
    • Charlie Jacobsen's avatar
      Muktesh's capabilities fully incorporated. Capsicum-style enter/exit. · 6ee9a51f
      Charlie Jacobsen authored and Vikram Narayanan's avatar Vikram Narayanan committed
      
      
      Builds, but not fully tested. Good tests for capability subsystem, some tests
      for kliblcd.
      
      Non-isolated kernel threads can "enter" the lcd system by doing
      klcd_enter / klcd_exit. They can create other lcd's, set them up, etc. They
      use the same interface that regular lcd's will use, so such code could be
      moved to an lcd, as we had planned. Will document this in Documentation folder
      tomorrow ( == today ).
      
      Capability system does checks now when a capability is deleted/revoked: for
      example, if it's for a page, the microkernel checks if the page is mapped, and
      unmaps it. If the last capability goes away, the page is freed. Documentation
      is in Documentation/lcd-domains/cap.txt.
      
      IPC code is in place, but not tested yet (pray for me).
      
      Debug is taking some time. Sometimes requires a power cycle which adds an
      extra 5 - 10 minutes. Build is slow the first time after reboot. Give me a user
      level program and I'll debug it in 30 seconds! argc
      
      Main arch-independent files:
      
          include/lcd-domains/kliblcd.h, types.h
      
             This is what non-isolated kernel code should include to use the
             kliblcd interface to the microkernel.
      
          virt/lcd-domains/main.c, kliblcd.c, cap.c, ipc.c, internal.h
      
             The microkernel, broken up into pieces.
      
          virt/lcd-domains/tests/
      
             The tests, in progress.
      
      Some old files are still hanging around in virt/lcd-domains and will be
      incorprated/cleaned up soon.
      
      I couldn't squash over the merge from the decomposition branch, so there's a
      bunch of junk commits coming over. (I should've just copied Muktesh's files.)
      
      Conflicts:
      	drivers/Kconfig
      	drivers/lcd-cspace/test.h
      	include/lcd-domains/cap.h
      	include/lcd-prototype/lcd.h
      	include/lcd/console.h
      	include/lcd/elfnote.h
      	include/linux/init_task.h
      	include/linux/module.h
      	include/linux/sched.h
      	virt/lcd-domains/cap.c
      	virt/lcd-domains/ipc.c
      	virt/lcd-domains/lcd-cspace-tests2.c
      Resolved-by: Vikram Narayanan's avatarVikram Narayanan <vikram186@gmail.com>
      6ee9a51f
    • Charles Jacobsen's avatar
      Support for multiple threads in lcd (limited). Major clean up of code. · ded1cd32
      Charles Jacobsen authored and Vikram Narayanan's avatar Vikram Narayanan committed
      In line with more recent design discussions, we now have (limited)
      support for running multiple threads inside an lcd.
      
      Each thread will have its own hardware vm, but share a guest
      physical address space and cspace.
      
      It's limited for now because threads cannot handle interrupts/exceptions
      internally in the lcd. This will require a per-thread TSS (much like
      Linux's per-core TSS/interrupt stack). I removed the gdt/idt/tss for
      now (Cf. with Dune, they don't use gdt/idt/tss) and will tackle that later
      after finishing more important stuff.
      
      I have only tested the code for running one hardware vm inside an lcd. Some
      code is missing proper locking for the future when we have multiple threads
      inside an lcd. I'm leaving this for now.
      
      The microkernel uses a simple bitmap for guest physical page allocation.
      
      Removed blob loading - code is set up for running modules exclusively.
      
      See the headers and Documentation/lcd-domains for more info.
      
      I put a flag at the top of files that are not currently in use, and will
      probably be deleted/incorporated later.
      ded1cd32
    • root's avatar
      Working lookup, insert, and delete · a5d6ea85
      root authored and Vikram Narayanan's avatar Vikram Narayanan committed
      a5d6ea85
    • Anton Burtsev's avatar
      Meaningful approach for creating and connecting LCDs · e7a70c32
      Anton Burtsev authored and Vikram Narayanan's avatar Vikram Narayanan committed
      e7a70c32
    • Anton Burtsev's avatar
      Client-server example... getting really wild (checkpoint) · 84067c4d
      Anton Burtsev authored and Vikram Narayanan's avatar Vikram Narayanan committed
      84067c4d
    • Anton Burtsev's avatar
      LCD API client and server code · fc2675ff
      Anton Burtsev authored and Vikram Narayanan's avatar Vikram Narayanan committed
        -- LCD export an API to its domains that is accessible via
        capability invocations (well, syscalls if you like)
      
        -- Kernel runs a thread that implement this API, e.g. serves
        capability invocations
      fc2675ff
    • Anton Burtsev's avatar
      Kicked out Muktesh's code, wrote my own stub · 6085bcd5
      Anton Burtsev authored and Vikram Narayanan's avatar Vikram Narayanan committed
      -- Mine is primitive but works, Muktesh will have to come
         back and rewrite
      -- Cspace is an array, only insert and lookup are supported
      6085bcd5
    • Anton Burtsev's avatar
      0de25ed1
    • Anton Burtsev's avatar
      Fix undefined symbol · d1f239ac
      Anton Burtsev authored and Vikram Narayanan's avatar Vikram Narayanan committed
      d1f239ac
    • Anton Burtsev's avatar
      Re-implemented ipc send and receive functions · 6404e62e
      Anton Burtsev authored and Vikram Narayanan's avatar Vikram Narayanan committed
      6404e62e
    • Anton Burtsev's avatar
      Cleaned up (sort of) capability code · 1d9c6dfb
      Anton Burtsev authored and Vikram Narayanan's avatar Vikram Narayanan committed
      1d9c6dfb
    • Anton Burtsev's avatar
      Rename capability type (*_t is a good idea sometimes) · 42684499
      Anton Burtsev authored and Vikram Narayanan's avatar Vikram Narayanan committed
      42684499
    • Charlie Jacobsen's avatar
      Simple renaming lcd => lcd-domains. · f34be85a
      Charlie Jacobsen authored and Vikram Narayanan's avatar Vikram Narayanan committed
      Updated Kbuild, Kconfig, Makefiles.
      f34be85a
    • Anton Burtsev's avatar
      Move things around, solve module dependencies problems · d242a850
      Anton Burtsev authored and Vikram Narayanan's avatar Vikram Narayanan committed
      d242a850
    • Anton Burtsev's avatar
      Remove init_module, cleanup_module from cap.c · ea3949b7
      Anton Burtsev authored and Vikram Narayanan's avatar Vikram Narayanan committed
      ea3949b7
    • Charlie Jacobsen's avatar
      FIXED INDENTATION TO USE TABS INSTEAD OF SPACES. · 4b722aaf
      Charlie Jacobsen authored and Vikram Narayanan's avatar Vikram Narayanan committed
      Per Linux coding style spec. Updated files in
      arch/x86/lcd, drivers/lcd, include/lcd.
      4b722aaf
    • Anton Burtsev's avatar
      Move things around · 0ec257ef
      Anton Burtsev authored and Vikram Narayanan's avatar Vikram Narayanan committed
      Not ideal. Some problems:
      
        - The arch dependent and independent parts are not completely
          separated.
      
        - The module builds in arch/x86/lcd and this is a bit strange
      
        - I can't build with LCD buildin, but build works if I build it
          as a module
      
        - There is still a bunch of old warrnings from Weibin's code
      0ec257ef
    • Charles Jacobsen's avatar
      MOVED ARCH-INDEPENDENT CODE TO VIRT/LCD. · 07f42b10
      Charles Jacobsen authored and Vikram Narayanan's avatar Vikram Narayanan committed
      Capability code in xcap/ moved to virt/lcd.
      
      Arch-independent code in arch/x86/lcd moved
      to virt/lcd (including tools and guest directories).
      07f42b10
    • Muktesh Khole's avatar
      Add check for grant permissions in lcd_cap_grant_capability · 5c7887d9
      Muktesh Khole authored and Vikram Narayanan's avatar Vikram Narayanan committed
      5c7887d9
    • Muktesh Khole's avatar
      implementation of lcd_cap_mint_capability · ca725b16
      Muktesh Khole authored and Vikram Narayanan's avatar Vikram Narayanan committed
      ca725b16
    • Muktesh Khole's avatar
      Complete Re-Design of LCD Capability module · a62128fc
      Muktesh Khole authored and Vikram Narayanan's avatar Vikram Narayanan committed
      a62128fc
    • Muktesh Khole's avatar
      LCD Capability module Re-Design Phase3 · 238d0181
      Muktesh Khole authored and Vikram Narayanan's avatar Vikram Narayanan committed
      238d0181
    • Muktesh Khole's avatar
      LCD Capability Module Re-Design Phase2 · 1a1f21d7
      Muktesh Khole authored and Vikram Narayanan's avatar Vikram Narayanan committed
      1a1f21d7
    • Muktesh Khole's avatar
      Capability Module Re-Design Phase1 · f33d65d3
      Muktesh Khole authored and Vikram Narayanan's avatar Vikram Narayanan committed
      f33d65d3
    • Muktesh Khole's avatar
      Fix bug with lcd_cap_delete locking mechanism Part1 · 3c7979bf
      Muktesh Khole authored and Vikram Narayanan's avatar Vikram Narayanan committed
      3c7979bf
    • Muktesh Khole's avatar
      Documenting external interfaces · 904364c9
      Muktesh Khole authored and Vikram Narayanan's avatar Vikram Narayanan committed
      904364c9
    • Muktesh Khole's avatar
      Moving lcd_lookup_capability to external interface · 06a58b9b
      Muktesh Khole authored and Vikram Narayanan's avatar Vikram Narayanan committed
      06a58b9b
    • Muktesh Khole's avatar
      Renaming capability.c and capability.h to lcd_cap.c and lcd_cap.h to avoid... · 9a4c14f4
      Muktesh Khole authored and Vikram Narayanan's avatar Vikram Narayanan committed
      Renaming capability.c and capability.h to lcd_cap.c and lcd_cap.h to avoid conflict /linux/capability.h files
      9a4c14f4
    • Muktesh Khole's avatar
      Adding cspace initialization to INIT_TASK macro · d6dfb02c
      Muktesh Khole authored and Vikram Narayanan's avatar Vikram Narayanan committed
      
      
      Conflicts:
      	include/linux/init_task.h
      	include/linux/sched.h
      Resolved-by: Vikram Narayanan's avatarVikram Narayanan <vikram186@gmail.com>
      d6dfb02c
    • Muktesh Khole's avatar
      Complete implementation of lcd_cap_revoke · a06b7d57
      Muktesh Khole authored and Vikram Narayanan's avatar Vikram Narayanan committed
      a06b7d57
    • Muktesh Khole's avatar
      Complete implementation of lcd_cap_delete · 382b4a14
      Muktesh Khole authored and Vikram Narayanan's avatar Vikram Narayanan committed
      382b4a14