1. 26 Oct, 2016 40 commits
    • Charles Jacobsen's avatar
      test-v2: ipc1 example works. · 9252bb21
      Charles Jacobsen authored
      Missing lcd_get_utcb function; added it.
      
      Another cptr_cache ABI issue - need to compile isolated code with
      padding flag; works now.
      
      Couple small bugs in microkernel (return value not initialized for
      certain error case).
      9252bb21
    • Charlie Jacobsen's avatar
      test-v2: ipc1 example set up. · d184c6d4
      Charlie Jacobsen authored
      d184c6d4
    • Charlie Jacobsen's avatar
      test-v2: load example set up. · 301a3b7e
      Charlie Jacobsen authored
      301a3b7e
    • Charles Jacobsen's avatar
      test-v2: RAM map tests pass. · c543053a
      Charles Jacobsen authored
      Couple duplicate memory interval tree inserts/deletes were leading
      to some use-after-frees/page faults. Cleaned that up. Added some
      resource tree debug code along the way.
      
      Also, caught something subtle (noted in code). I didn't consider
      the following scenario: Heap tries to allocate pages; the allocator
      notices it needs to bring in more fresh pages, and notifies the
      heap (via a callback); the heap allocs the fresh pages (from the
      microkernel), maps them, and inserts those pages into the memory interval
      tree; the memory interval tree kmallocs a tree node; kmalloc
      calls back into the heap to grow a slab cache.
      
      That last bit could be a potential problem (recursive call back
      into the heap before we finish the original call). Lucky for me,
      I designed the heap/allocator so that (1) the pages from the first
      call are already marked as in use (not on a free list); (2) the
      fresh pages are mapped first *before* inserting the corresponding
      cptr into the memory interval tree.
      
      The Linux kernel deals with these same recursive issues (they
      resolve them using special GFP_ flags so that you don't get
      recursion). In my case, the recursion is risky, but works.
      c543053a
    • Charlie Jacobsen's avatar
      test-v2: RAM map (kmap) test. · 73de8d7f
      Charlie Jacobsen authored
      73de8d7f
    • Charles Jacobsen's avatar
      test-v2: kmalloc tests pass, big page alloc test passes. · 42ad48cc
      Charles Jacobsen authored
      Caught a bug in heap allocator - wasn't setting an out
      parameter (that design pattern I use - out params - will be the
      death of me).
      
      Updated max page allocation order to match the host. The microkernel
      uses the host page allocator, so that is the only limiting
      factor now (so e.g., on x86_64, you can now allocate up to
      2^10 = 1024 pages = 4 MBs inside the LCD, and this is the maximum
      you could allocate in non-isolated code too). Before, the allocation
      size was limited to about 2^6 = 64 pages = 256 KBs.
      42ad48cc
    • Charlie Jacobsen's avatar
      test-v2: kmalloc and kmem cache tests. · 2d246902
      Charlie Jacobsen authored
      2d246902
    • Charles Jacobsen's avatar
      test-v2: Page alloc tests pass. · 69f9f5fe
      Charles Jacobsen authored
      Woo-hoo! Lucky.
      69f9f5fe
    • Charlie Jacobsen's avatar
      test-v2: Higher-level page alloc tests. · 5cc91d4d
      Charlie Jacobsen authored
      5cc91d4d
    • Charles Jacobsen's avatar
    • Charlie Jacobsen's avatar
    • Charles Jacobsen's avatar
      test-v2: LCD boots without apparent issue. · a708dcb8
      Charles Jacobsen authored
      Buddy allocator, interval trees, etc. seem to all be
      operational. THC runtime initializes. Tear down appears
      graceful. Will need to stress test ...
      a708dcb8
    • Charles Jacobsen's avatar
      test-v2: More allocator bugs - translate page order to block order. · 569f0180
      Charles Jacobsen authored
      LCD boots most of the way, getting assertion failures in THC code.
      569f0180
    • Charles Jacobsen's avatar
      test-v2: Fix more allocator bugs. · 94bbac3f
      Charles Jacobsen authored
      I needed to re-think the free lists initialization.
      Also bug in demand paging.
      
      Heap and kmalloc now initialize, but I still crash afterward.
      94bbac3f
    • Charles Jacobsen's avatar
      test-v2: Fix some allocator bugs, fix cptr cache bug. · 3710d903
      Charles Jacobsen authored
      I was using PAGE_SIZE instead of PAGE_SHIFT in some places.
      
      Also, non-isolated code sets up the cptr cache for isolated
      code; but the size of the mutex is different for each environment
      (16 bytes versus 8 bytes). So, I hacked libcap to take care of
      the difference (using padding).
      3710d903
    • Charles Jacobsen's avatar
      test-v2: Fix liblcd syscalls, gv page tables, this_module linkage. · 666f9dae
      Charles Jacobsen authored
      Couple interesting issues here:
      
         1 - Syscalls were breaking without "asm volatile". I tried
             just plain "asm" to be more compiler-friendly (I guess that
             is more friendly), but the compiler optimized away entire
             vmcall's, haha.
         2 - Guest virtual page table set up was way off for the pud (wasn't
             pointing to the pmd's, and some of the surrounding address
             arithmetic was bad). It's still bad (too specific for x86_64
             perhaps), but that's acceptable for now.
         3 - This was the most interesting one. I didn't even think that
             when we build liblcd (the .a file), the kernel doesn't think
             we're building a kernel module, so THIS_MODULE expands to
             NULL. In order to fix this, I manually set the MODULE macro
             (via a -D cc flag) so that the module.h macros expand as
             expected (we will always link liblcd.a with a kernel module,
             so e.g. struct module __this_module will always be defined).
      666f9dae
    • Charlie Jacobsen's avatar
      test-v2: Use 2MB pages instead of 1GB pages for guest virtual. · d08c3ad1
      Charlie Jacobsen authored
      This only costs us 64 KBs versus 8 KBs, which isn't much on a modern
      arch.
      d08c3ad1
    • Charles Jacobsen's avatar
      ac26533c
    • Charles Jacobsen's avatar
      test-v2: Add rb tree code to liblcd, fix unresolved mutex symbols. · 87fbf2ff
      Charles Jacobsen authored
      There is a limitation with my preprocessor hacks: If a header
      invokes a function inside another static inline function, I
      can't modify the call site (that is the key) using a macro
      undef/re-define. Instead, I need to just define the function
      in another source file.
      
      Another limitation (yet to be encountered) is a problematic
      macro function invoked in a header. I can't re-define the
      macro function before the call site using the post hooks. Will
      cross that bridge when I get there.
      87fbf2ff
    • Charles Jacobsen's avatar
      test-v2: LCD module create functioning. · 38d3c5d0
      Charles Jacobsen authored
      Host can successfully load LCD, using new interval tree/memory
      object logic. (Still get EPT fault while booting LCD.)
      
      Some fixes/changes:
      
          1 - Need to use page-level granularity for memory objects.
              Unlike alloc_pages, vmalloc is page-granularity (rather
              than power-of-2 page granularity).
          2 - The kernel module region base virtual address needs to
              match the host's upper 2GB address, because of how address
              calculations are done internally and because we use the
              host module loader with no linkage changes (it's possible,
              but not worth it right now). Add assertion to make sure
              this is the case.
          3 - Needed to make LCD the owner of its cspace (set owner
              pointer).
      38d3c5d0
    • Sarah Spall's avatar
      ffc17a50
    • Charles Jacobsen's avatar
      test-v2: Fix cap types bug and sync create endpoint bug. · 6390c4c5
      Charles Jacobsen authored
      Need to ensure types array and microkernel types enum order
      match.
      
      Wasn't setting out parameter for create sync endpoint.
      
      Allocate two cptr's to reserve call/reply endpoints for
      LCD (in module create).
      6390c4c5
    • Charles Jacobsen's avatar
      test-v2: Fix cptr cache configuration bug. · cd6e20a1
      Charles Jacobsen authored
      The cptr cache was too big. Shrunk it a bit to the size
      we were using before.
      cd6e20a1
    • Charles Jacobsen's avatar
      test-v2: Simple example builds. · e34ca44e
      Charles Jacobsen authored
      A few things -
      
         1 - Needed to add export symbols for kliblcd.
      
         2 - Auto generate path to liblcd.a and paste the path in each Kbuild
             example file
      
         3 - Set up CC flags for examples, non-isolated and isolated
      
         4 - Need to point to kliblcd's Module.symvers so that kernel doesn't
             complain when we build non-isolated .ko's.
      e34ca44e
    • Charlie Jacobsen's avatar
      test-v2: Set up initial basic load test. · 0dad2249
      Charlie Jacobsen authored
      Manually pulling in Anton's upstream change for the test mods dir
      macro.
      0dad2249
    • Charlie Jacobsen's avatar
      test-v2: Refactor test modules/examples build for external build. · 3511bec6
      Charlie Jacobsen authored
      I re-create the test_mods directory hierarchy in the build dir, and
      bring over the Kbuilds. I also auto-generate the 'src =' line in
      the Kbuilds for the external build hack.
      
      Throughout the top-level Makefile, I use an additional hack. Let me
      explain. We use recursive make to build dependencies (like versions
      of libcap), because only the recursive make knows how to check that
      the dependencies are up to date. But this means we have to make the
      dependency targets phonies, so they always run. This in turn means
      that those targets that depend on *them* will always run. To avoid
      this, I use 'check for presence only' dependencies for these types
      of dependencies that would trigger a full rebuild. The trick though
      is to specify the final build output file as the dependency, rather
      than the phony target, and to not create a rule for the final build
      output file. (This way, if the file does not exist, the top-level
      Makefile will just fail and say it doesn't know how to build the
      file. It won't know that one of the phony targets does that.)
      
      Anyway, test_mods is one example. It depends on liblcd.a being built,
      but in order to prevent triggering a recursive make on liblcd.a, we
      just use a 'check for presence only' type of dependency. It's up
      to the person building to ensure they rebuild top-level targets
      as necessary. (It doesn't seem easy to chain the whole build
      together, with proper dependency management, since we're using
      recursive make for dependency builds and kernel builds.)
      
      build_all.sh updated to trigger build of all targets in the correct
      order.
      3511bec6
    • Charles Jacobsen's avatar
      build-refactor-2: Microkernel, kliblcd, and liblcd all build. · 862baeb8
      Charles Jacobsen authored
      In one giant build circus.
      
      Test mods build not re-configured yet. Going to do some
      runtime testing and debugging of the new code before I fix
      those up.
      862baeb8
    • Charlie Jacobsen's avatar
      build-refactor-2: Add ar script to combine libcap.a into liblcd. · bd5bde51
      Charlie Jacobsen authored
      This relies on GNU ar scripts (not supported by other versions
      of ar, fyi).
      bd5bde51
    • Charles Jacobsen's avatar
      build-refactor-2: libcap builds for isolated environment. · ff630892
      Charles Jacobsen authored
      Still need to mash it together with liblcd. Going to use a
      GNU ar script for that.
      ff630892
    • Charlie Jacobsen's avatar
    • Charles Jacobsen's avatar
      4ca91d0c
    • Charles Jacobsen's avatar
      build-refactor-2: Set up liblcd build, and libcap build for liblcd. · 2c945e8c
      Charles Jacobsen authored
      The libcap build won't work quite yet, or won't be set up properly
      for liblcd rather. I need to tweak the libcap kernel build and/or
      add a new platform.
      2c945e8c
    • Charles Jacobsen's avatar
      build-refactor-2: Fix build errors. Microkernel and kliblcd build. · 088970e6
      Charles Jacobsen authored
      Now that microkernel libcap build is separated from microkernel build
      itself, when we trigger a top-level microkernel build, it doesn't
      rebuild the whole thing every time. Kind of nice.
      088970e6
    • Charlie Jacobsen's avatar
      build-refactor-2: Separate libcap build from microkernel build. · 0a3b25ab
      Charlie Jacobsen authored
      I tried to do both in one shot, but it didn't seem possible to
      do it the way I wanted it. (If you combine them, there
      didn't seem an easy way to conditionally install libcap. This
      would be nice because then the microkernel wouldn't be fully
      rebuilt every time.)
      0a3b25ab
    • Charlie Jacobsen's avatar
      liblcd-v2: Add bootup code for liblcd. · 5fa0c548
      Charlie Jacobsen authored
      5fa0c548
    • Charlie Jacobsen's avatar
      liblcd-v2: Add slab dependencies. · 1b64ac34
      Charlie Jacobsen authored
      1b64ac34
    • Charlie Jacobsen's avatar
      0d822f1c
    • Charlie Jacobsen's avatar
      a94f0e92
    • Charlie Jacobsen's avatar
      liblcd-v2: Fix generalized page alloc metadata alloc/free. · 2979d763
      Charlie Jacobsen authored
      It seemed dumb to have the generalized page allocator code
      repeatedly call out to the user to get memory for metadata.
      For one thing, it was buggy. For another, in many use cases
      we can alloc the metadata in one shot.
      
      RAM map allocator partially started.
      2979d763
    • Charlie Jacobsen's avatar
      liblcd-v2: Make room for RAM map region (separate from heap). · b9ce0034
      Charlie Jacobsen authored
      This is kind of like an mmap region, but only for RAM. (The guest
      virtual page tables will be configured for write back).
      
      Use case: You get a RAM capability from some other guy, and you
      want to map it into your physical/virtual address space. This is
      reasonable to figure out manually for big RAM chunks that are
      mapped one/two times. But for little tedious ones (string sharing),
      it's helpful to have an allocator to assist and track free parts
      of guest physical.
      
      The RAM map region uses very course-grained physical address space
      allocation (minimum 64 MBs). This leads to a lot of internal
      fragmentation, but it should be tolerable since the region is big.
      It also means we have a smaller bit of metadata for tracking the
      region.
      b9ce0034