1. 21 May, 2015 1 commit
  2. 15 May, 2015 1 commit
    • Leigh Stoller's avatar
      Directory based image paths. · 3a21f39e
      Leigh Stoller authored
      Soon, we will have images with both full images and deltas, for the same
      image version. To make this possible, the image path will now be a
      directory instead of a file, and all of the versions (ndz,sig,sha1,delta)
      files will reside in the directory.
      
      A new config variable IMAGEDIRECTORIES turns this on, there is also a check
      for the ImageDiretories feature. This is applied only when a brand new
      image is created; a clone version of the image inherits the path it started
      with. Yes, you can have a mix of directory based and file based image
      descriptors.
      
      When it is time to convert all images over, there is a script called
      imagetodir that will go through all image descriptors, create the
      directory, move/rename all the files, and update the descriptors.
      Ultimately, we will not support file based image paths.
      
      I also added versioning to the image metadata descriptors so that going
      forward, old clients can handle a descriptor from a new server.
      3a21f39e
  3. 09 Feb, 2015 1 commit
  4. 28 Jul, 2014 1 commit
  5. 11 Jul, 2014 1 commit
  6. 09 Jul, 2014 1 commit
  7. 01 Jul, 2014 1 commit
  8. 16 Dec, 2013 1 commit
  9. 13 Aug, 2013 1 commit
  10. 06 Aug, 2013 1 commit
  11. 30 Jul, 2013 1 commit
  12. 11 Jul, 2013 1 commit
  13. 01 Jul, 2013 1 commit
  14. 20 Jun, 2013 1 commit
    • Leigh Stoller's avatar
      A big set of changes to how we create XEN guest disks. · fbc26aea
      Leigh Stoller authored
      Prior to this commit, XEN guests disks were single partition, no MBR,
      the bits dumped into the lvm. This makes a snapshot of a XEN node,
      look completely different then a physical disk image, especially if
      users want more disk space (mkextrafs) inside the guest, and then want
      to take a snapshot of that, and then run it on a physical node (which
      was not possible).
      
      With these changes, guests now use the same MBR layout as our version
      two MBR, which makes them interchangeable with physical disk images.
      In fact, the goal is to be able to switch back and forth as needed,
      based on physical resource availability.
      fbc26aea
  15. 31 May, 2013 1 commit
    • Leigh Stoller's avatar
      Do not default XEN guest images to "packages". Lets make the default · 3b352486
      Leigh Stoller authored
      a single slice image, since we can now pull the kernel (ramdisk) out
      from the guest filesystem (using pygrub for linux, or just mounting
      BSD filesystems). This is a lot faster and easier to deal with. I
      added an option to the newimage page so that people can set this, but
      in general we need a better way to guess that we need it. Always set
      for EC2 images.
      3b352486
  16. 14 May, 2013 2 commits
    • Leigh Stoller's avatar
      Add prototype EC2 image import plumbing. · 980aa180
      Leigh Stoller authored
      To create a new descriptor that will be an import from EC2 (and thus
      run under XEN), add ?ec2=1 to newimage_ez.php3. Eventually will link
      it in someplace. The form will create a XEN based VM, but instead of
      node to snapshot from, provide user@host for the EC2 instance.
      
      On the image snapshot page, instead of node use user@host for the EC2
      instance.
      
      The backend script (create_image) will call over to ops and invoke
      Srikanth's code. I have called that script ec2import-image.pl. See
      create_image for how arguments are passed to the script.
      980aa180
    • Leigh Stoller's avatar
      f0131807
  17. 26 Nov, 2012 1 commit
  18. 10 Oct, 2012 1 commit
  19. 28 Sep, 2012 1 commit
  20. 24 Sep, 2012 2 commits
    • Leigh Stoller's avatar
      Minor bug fix. · f2f41448
      Leigh Stoller authored
      f2f41448
    • Eric Eide's avatar
      Replace license symbols with {{{ }}}-enclosed license blocks. · 6df609a9
      Eric Eide authored
      This commit is intended to makes the license status of Emulab and
      ProtoGENI source files more clear.  It replaces license symbols like
      "EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
      blocks that contain actual license statements.
      
      This change was driven by the fact that today, most people acquire and
      track Emulab and ProtoGENI sources via git.
      
      Before the Emulab source code was kept in git, the Flux Research Group
      at the University of Utah would roll distributions by making tar
      files.  As part of that process, the Flux Group would replace the
      license symbols in the source files with actual license statements.
      
      When the Flux Group moved to git, people outside of the group started
      to see the source files with the "unexpanded" symbols.  This meant
      that people acquired source files without actual license statements in
      them.  All the relevant files had Utah *copyright* statements in them,
      but without the expanded *license* statements, the licensing status of
      the source files was unclear.
      
      This commit is intended to clear up that confusion.
      
      Most Utah-copyrighted files in the Emulab source tree are distributed
      under the terms of the Affero GNU General Public License, version 3
      (AGPLv3).
      
      Most Utah-copyrighted files related to ProtoGENI are distributed under
      the terms of the GENI Public License, which is a BSD-like open-source
      license.
      
      Some Utah-copyrighted files in the Emulab source tree are distributed
      under the terms of the GNU Lesser General Public License, version 2.1
      (LGPL).
      6df609a9
  21. 27 Aug, 2012 1 commit
    • Leigh Stoller's avatar
      Add support for creating snapshot images of OpenVZ containers via the · fdd08bb4
      Leigh Stoller authored
      web interface.
      
      Most of this commit is about making it easier to create the
      descriptors (of any kind) The newimageid_ez page now takes an optional
      node_id on which to base the new descriptor; most of the fields are
      initialized.
      
      You can also provide optional imageid and osinfo arguments, which
      allows for slightly less info to be initialized.
      
      Then on the ShowNode page, the link to create an image will take you
      to the snapshot image page if the user has write access to the image,
      otherwise to the newimageid_iz page to create and snapshot a new
      image descriptor.
      fdd08bb4
  22. 11 Jul, 2012 1 commit
    • Leigh Stoller's avatar
      Cleanup in the web interface to prevent XSS attacks. · 6cf701f9
      Leigh Stoller authored
      We had a couple of different problems actually.
      
      * We allow users to insert html into many DB fields (say, a project or
        experiment description).
      
      * We did not sanitize that output when displaying back.
      
      * We did not sanitize initial page arguments that were reflected in the
        output (say, in a form).
      
      Since no one has the time to analyze every line of code, I took a couple of
      shortcuts. The first is that I changed the regex table to not allow any <>
      chars to go from the user into the DB. Brutal, but in fact there are only a
      couple of places where a user legitimately needs them. For example, a
      startup command that includes redirection. I handle those as special
      cases. As more come up, we can fix them.
      
      I did a quick pass through all of the forms, and made sure that we run
      htmlspecialchars on everything including initial form args. This was not
      too bad cause of the way all of the forms are structured, with a
      "formfields" array.
      
      I also removed a bunch of obsolete code and added an update script to
      actually remove them from the www directory.
      
      Lastly, I purged some XMLRPC code I did a long time ago in the Begin
      Experiment path. Less complexity, easier to grok and fix.
      
      	modified:   sql/database-fill.sql
      	modified:   sql/dbfill-update.sql
      6cf701f9
  23. 22 Dec, 2011 1 commit
  24. 30 Aug, 2011 1 commit
  25. 18 Jul, 2008 1 commit
  26. 19 Nov, 2007 1 commit
  27. 09 Nov, 2007 1 commit
  28. 19 Oct, 2007 1 commit
    • Russ Fish's avatar
      Move newimageid_ez page form logic to a backend Perl script. · 9a586d0b
      Russ Fish authored
           www/newimageid_ez.php3 - The reworked PHP page.  Calls Image::NewImageId with ez=1.
           www/newimageid.php3 - Call Image::NewImageId with ez=0.
           www/imageid_defs.php - Re-use the Image::NewImageId class method, adding an 'ez' arg.
           backend/{newimageid_ez,GNUmakefile}.in configure configure.in - New backend script.
               After checks, calls OSinfo->Create and Image->Create with the same XML args array.
           db/Image.pm.in - Re-use the Image->Create method, adding an imageid over-ride arg.
           db/OSinfo.pm.in - Filter out extraneous db slot args from XML in the Create method.
           db/libdb.pm.in - Add TB_{OS,IMAGE}ID_* constants from dbdefs.php3 .
           sql/database-fill.sql - Add OS entries to the table_regex 'images' pattern set.
      9a586d0b
  29. 10 Sep, 2007 1 commit
    • Leigh Stoller's avatar
      Rework the newosid web page as an example of how I want all of our current · 77540494
      Leigh Stoller authored
      form processing to be done.
      
      The gist is that I have moved all of the data checking and DB work to
      the backend into a new script called utils/newosid. This script does
      all the field checking that used to be done in php. It takes a simple
      XML file as input and returns a set of strings to format as errors (if
      there are any).
      
      The overall goal to make a big push to move this code out of PHP and
      perl.  A nice side effect is that many operations that are current
      only available via the web interface will also become available
      command line (and also XMLRPC with a little moew work).
      77540494
  30. 27 Aug, 2007 1 commit
  31. 24 Aug, 2007 1 commit
    • Mike Hibler's avatar
      First crack at limiting the node types assigned to new images. · fff92262
      Mike Hibler authored
      Currently we declare the valid types for an image to be those that are of
      class "pc" and for which we have at least one node in the testbed.
      
      With this change, when an image is created as a snapshot from an existing
      node, we look at the partition info for the existing image to further
      restrict the types that the derived image will work on.
      
      This will prevent images derived from old images like RHL90 or FBSD410
      from getting newer node types such as pc745d as a valid type.
      
      One problem with the current solution is that, we don't know til the form
      is submitted once, what node you are snapshot'ing, so the first time a
      user submits the form, it is almost certain to come back with an error.
      
      Further, they have no choice but to uncheck the offending types.  If part
      of their customization was to add support for a new node type...too bad,
      ya' can't add it!  There are numerous ways to fix this, all of which exceed
      my Friday afternoon mad hacking skilz.
      fff92262
  32. 21 Aug, 2007 1 commit
    • Leigh Stoller's avatar
      Another round of widearea node hacking for CMU. These changes add · 99346dc0
      Leigh Stoller authored
      widearea reloading support.
      
      * New slot in the images table to store an access key which remote
        sites must provide in order to download an image (via https).
      
      * tmcd returns a different kind of ADDRESS field from doloadinfo.
        Instead of the multicast stuff, return a URL that points to boss'
        web server. The URL is of the form:
      
         https://www.myemulab.net/spewimage.php?imageid=10013&access_key=abcdef
      
        which as you can see is fully specified; the client does not need
        to know anything else.
      
      * New webpage and backend scripts appropriately called "spewimage"
        which also includes support for the http HEAD request (from wget) to
        avoid downloading images that are already on the node. I just
        learned about this HEAD request stuff today ... but otherwise these
        operate as expected, spewing the image if the access key is provided.
      
      * Changes to rc.frisbee to deal with remote loading. In addition to
        URL support, I also added support for simple paths, the intent being
        that we will probably distribute images offline (say, at night) so
        that when a node reboots it doesn't actually have to wait 60 minutes
        for an image to download. I have not added any server side support
        for this yet though. Maybe later this week.
      
      * Other bits and pieces and fixes to make this work.
      99346dc0
  33. 09 Aug, 2007 1 commit
  34. 17 Apr, 2007 1 commit
  35. 13 Mar, 2007 1 commit
  36. 06 Mar, 2007 1 commit
  37. 13 Feb, 2007 1 commit
  38. 12 Feb, 2007 1 commit
    • Leigh Stoller's avatar
      * Replace the argument processing code in all pages. Currently we rely on · 48acc8e3
      Leigh Stoller authored
        register_globals=1 to turn POST/GET/COOKIES arguments in local variables.
        This is known to be a terrible security risk, and we keep saying we are
        going to fix it, and now I am. In order to accomplish this on a
        transitional basis (since I don't want the entire web interface to stop
        working while I debug it), and because the code just needs the cleanup, I
        am doing it like this: Each page will sport new declarations at the top:
      
      	RequiredPageArguments("experiment", PAGEARG_EXPERIMENT,
                                    "template",   PAGEARG_TEMPLATE,
                                    "instance",   PAGEARG_INSTANCE,
                                    "metadata",   PAGEARG_METADATA,
                                    "osinfo",     PAGEARG_OSINFO,
                                    "image",      PAGEARG_IMAGE,
                                    "project",    PAGEARG_PROJECT,
                                    "group",      PAGEARG_GROUP,
                                    "user",       PAGEARG_USER,
      			      "node",       PAGEARG_NODE,
      			      "yesno",      PAGEARG_BOOLEAN,
      			      "message",    PAGEARG_STRING,
      			      "age",        PAGEARG_INTEGER,
                                    "cost",       PAGEARG_NUMERIC,
                                    "formfields", PAGEARG_ARRAY,
                                    "unknown",    PAGEARG_ANYTHING);
      
      	OptionalPageArguments("canceled", PAGEARG_BOOLEAN);
      
        The first token in each pair is the name of the global variable to
        set, and the second token is the type. So, for "experiment" we look at
        the URL for a pid/eid or exptidx, etc, sanity check them (safe for a
        DB query), and then try to find that experiment in the DB. If it maps
        to an experiment, set global variable $experiment to the object. Since
        its a required argument, produce an error if not supplied. Similar
        treatment for optional arguments, with the obvious difference.
      
        The goal is to have ALL argument processing in one place, consistent,
        and correct. I've found numerous places where we leak unchecked
        arguments into queries. It also cuts out a lot of duplicated code.
      
      * To make the above easier to deal with, I've been replacing lots of
        hardcoded URLS in the code of the form:
      
      	foo.php3?pid=$pid&eid=$eid ...
      
        with
      
              CreateURL("foo", $experiment)
      
        which creates and returns the neccessary url string, by looking at
        the type of its arguments (experiment, template, instance, etc.)
      
        Eventually plan to replace them all so that URL handling throughout
        the code is all defined in one place (all the new URL code is in
        url_defs.php).
      
      * I have cranked up error reporting to tell me anytime a variable is
        used before it is initialized, plus a bunch of other stuff that PHP
        deems improper. Think of it like -Wall ... and boy we get a lot of
        warnings.  A very large percentage of the diffs are to fix all these
        warnings.
      
        The warnings are currently going to /usr/testbed/log/php-errors.log,
        and I'll be adding a script to capture them each night and mail them
        to tbops. This file also gets errors (this will be a change for
        developers; rather then seeing errors and warnings dumped in the
        middle of web pages, they will go to this file instead).
      
      * Major refactoring of the code. More objects (nodes, images, osids).
        Moving tons of queries into the objects in the hopes of someday
        getting to a point where we can split the web interface onto a
        different server.  Lots of general cleanup.
      48acc8e3