GitLab

the tb-set-node-os command with a second optional argument; if that is
present, the first arg is the child OS and the second is the parent OS.
We add some new features in ptopgen (OS-parentOSname-childOSname) based
off a new table that maps which child OSes can run on which parents, and
the right desires get added to match.  We setup the reloads in os_setup
along with the parents.  Also needed a new opmode, RELOAD-PCVM, to handle
all this.

For now, users only have to specify that their images can run on pcvms, a
special hack for which type the images can run on.  This makes sense in
general since there is no point conditionalizing childOS loading on
hardware type at the moment, but rather on parentOS.  Hopefully this stuff
wiill mostly work on shared nodes too, although we'll have to be more
aggressive on the client side garbage collecting old frisbee'd images for
long-lived shared hosts.

I only made these changes in libvtop, so assi...

* Add external resource identifiers to the reserved table so we can
  figure out the external node that is mapped onto the local (proxy)
  node.

* Change limit on fixed_iface to 128 chars to accomodate urns.

between experiments and projects using VMs (jail, vservers, openvz,
etc).

* Add new slot to virt_nodes called "sharing_mode". Only admins can set
  this, via the NS file, to create an experiment consisting of nodes
  that are shared. A text string, in case we want to support different
  kinds of sharing modes.

* Add new slot to reserved called "sharing_mode". This is copied from
  the virt_nodes table as above. Also, virtnodes that are currently
  allocated to a shared node, get this slot set, so that the rest of
  the system knows.

* Add a toklen to the "erole" slot of the reserved table, called
  "sharedhost"; erole is returned to the client when it boots, and
  this is how the client knows it is acting as a shared host. We could
  probably compute this value in tmcd, but I did not want to change
  that now.

* Add new slot to interfaces called "trunk". This is set when an
  interface should be set to trunking by "snmpit -t". We previously
  used the vinterfaces table to determine this, but there are other
  cases when we want an interface to be trunked.

  Aside: there is a problem with this approach I realized later;
  trunked interfaces are essentially shared resources of multiple
  vlans, which does not play well with the vlan optimizations in
  swapmod.

* Add new slot to vinterfaces called "bandwidth"; this is the amount
  of bandwidth that link is defined to use (via the NS file). This is
  used in resource scheduling of link bandwidth.

* Add new slot to interface_state called "remaining_bandwidth". For a
  shared node, we have to reserve both the nodes and the link bandwidth.
  We use this slot (with locking) from the mapper_wrapper to make sure that
  the link has not been consumed already by another assignment running at
  the same time.

* Add new tokens "veth-ne", "veth-en", and "vlans" to the osfeatures slot
  of the os_info table. veth-en (encap) and veth-ne (no-encap) are intended
  to replace the current "veths" feature. "vlans" is set for OSs that
  support vlan devices.

* Fix the primary key on the linkdelays table; it needed to have the
  exptidx in the key so that we can allocated linkdelays for different
  experiments to the same node.

the existing DB format.

changes in my devel tree.

Set the charset for all emulab pages to utf-8 which is needed to
display foreign names correctly.  Also added three rows to table_regex:
tinytext_utf8, text_utf8, fulltext_utf8

Enhanced form_defs.php:

  - Add Support for Textarea

  - Add Support for a vertical list, for example see Category in
    submitpub.php

  - Add support for dumping the form values in plain text, used in the
    email I send to testbed-ops.

  - Change Error Reporting text for subfields such as in a list so
    they also include the text from the outer field, for example in
    newosid "ping" will become "OS Features, ping".  I needed this
    since some of my subfields didn't have labels and it seamed like
    the right thing to do.

  - Set #return_value to 1 in FormRenderCheckBox if it not already
    specified

  - In FormRenderSelect, if the #value is not in the list add it to
    the end.

  - Add support for "display" type, to just display some text in the
    table

  - Possibly some other little things I forgot about.

New site variable PUBSUPPORT.

Implement FS#201: Provide users a way to add a paper that used Emulab
see (https://users.emulab.net/flyspray/?do=details&id=201), summary of
changes:

  - Controlled via the new site variable PUBSUPPORT.

  - The emulab_papers spread sheet is now frozen all new changes
    should be made using the new system.

  - New pages submitpub.php for users to submit a new publication
    with different views for admins and non-admins.

  - New page expubs.php to replace doc/expubs.html, lists publications
    in database.  Also allows admins to edit publications.

  - Project members can edit papers associated with the project as
    long as they have local_root or higher privileges. Papers for a
    project are listed under a new tab in the showproject page. I keep
    track of the last person to edit the paper submission in addition
    to who initially submitted it.

  - Users can modify papers they submitted which is needed if the
    project was N/A. Papers users submitted now appear in a new tab in
    the showuser page.

  - Script to import spreadsheet to the database is in
    utah/bibliography/emulab_pubs-import.pl.

  - Changes web pages that point to doc/expubs.html to instead point
    to expubs.php

the User Name (or id) in various tables.  This also involved adding
the field to the newproject, joinproject, and moduserinfo forms.

Note: also modified 4.149 database-migrate.txt entry to add a note
to add the necessary "slot" to table_regex.

lan_member_attributes, and ifaces. Making the world safe for protogenis.

  tb-fix-interface $node $link "iface"

         www/beginexp_form.php3 - No initial whitespace in plab Slice Description textarea.
         sql/database-fill.sql - Use default:fulltext for 'experiments','expt_name'
             to allow multi-line Slice Description in plab_ez->beginexp->batchexp.
         www/moduserinfo.php3 - Pass null string for optional usr_addr2 and notes fields.
         backend/moduserinfo.in - Null string attr comes from XML as an undef value.
         db/User.pm.in - Distinguish between undef and null string in ModUserInfo.

NewNewUser()/newuser would block and then unblock and get done; meanwhile the PHP
thread went away so we never returned to call NewNewProject/mkproj.  Move the call
on the newuser script from PHP into the back-end Perl newproj script for atomicity.

    www/newproject.php3 - When the project leader is a new user, pass two xml
        files to the newproj backend script, one describing the project and the
        second one (an optional) file describing the newuser.

    www/user_defs.php - Factor the xml-making part of NewNewUser into NewNewUserXML.

    www/project_defs.php - Remove the required $leader arg of NewNewProject.
        newproj may call newuser, which may generate the leader uid.

    backend/newproj.in - Call newuser with an optional 'newuser_xml' XML file.

    sql/database-fill.sql - Add 'projects','newuser_xml'.

      www/delaycontrol.php3 - Convert the form into SPITFORM/submit/formfields.
                Add a ChangeDelayConfig function to spit out XML to delay_config.
      account/delay_config.in - Add -X <xmlfile> in place of other command-line args.
      sql/database-fill.sql - Add a couple of 'virt_lans' entries for delay_config's use.

      www/showpubkeys.php3 - Add a NewPubKey function to spit out XML to addpubkey.
      account/addpubkey.in - Add -X <xmlfile> in place of other command-line args.
      sql/database-fill.sql - Add 'user_pubkeys' entries for addpubkey's use.

     www/newimageid_ez.php3 - The reworked PHP page.  Calls Image::NewImageId with ez=1.
     www/newimageid.php3 - Call Image::NewImageId with ez=0.
     www/imageid_defs.php - Re-use the Image::NewImageId class method, adding an 'ez' arg.
     backend/{newimageid_ez,GNUmakefile}.in configure configure.in - New backend script.
         After checks, calls OSinfo->Create and Image->Create with the same XML args array.
     db/Image.pm.in - Re-use the Image->Create method, adding an imageid over-ride arg.
     db/OSinfo.pm.in - Filter out extraneous db slot args from XML in the Create method.
     db/libdb.pm.in - Add TB_{OS,IMAGE}ID_* constants from dbdefs.php3 .
     sql/database-fill.sql - Add OS entries to the table_regex 'images' pattern set.

Also commit the DB changes.

     www/editgroup.php3 - The reworked PHP page.
     www/editgroup_form.php3 - Removed, form merged into editgroup.php3 .
     www/showgroup.php3 - Link to editgroup.php3, rather than editgroup_form.php3 .
     www/group_defs.php - Add an Image::EditGroup class method
                            bridging to the script via XML.
     backend/{editgroup,GNUmakefile}.in configure configure.in - New backend script.
     db/Group.pm.in - Add an EditGroup worker class method for script arg checking.
                      Also the missing NonMemberList and CheckTrustConsistency methods,
                      and a GETTRUST flag to MemberList, as in the PHP version.
     db/User.pm.in - Add the missing but tiny {Set,Get}TempData methods.
     sql/database-fill.sql - Add gid_idx to the table_regex 'groups' checking patterns.

     www/newimageid.php3 - The reworked PHP page.
     www/imageid_defs.php - Add an Image::NewImageId class method
                            bridging to the script via XML.
     backend/{newimageid,GNUmakefile}.in configure configure.in - New backend script.
     db/Image.pm.in - Add a Create class method for script arg checking.
     sql/database-fill.sql - Add to the table_regex 'images' checking patterns.

     www/editsitevars.php3 - The reworked PHP page, including
       a SetSiteVar function bridging to the script via XML.
     backend/{editsitevars,GNUmakefile}.in configure configure.in - New backend script.
     sql/database-fill.sql - Add table_regex 'sitevariables' checking patterns.

Enumerate the boolean slots of the node_types table in table_regex.  Use it in editnodetype.in, plus a little fix for new_type checking.

       Note: new_node attributes have not yet been converted from
       names for osid's and imageid's to new-style integer indices.
     www/editnodetype.php3 - The reworked PHP page, including
       an EditNodeType function bridging to the script via XML.
     backend/{editnodetype,GNUmakefile}.in configure configure.in - New backend script,
       including wildcarding on %xmlfields table entries to handle dynamic attributes.
     sql/database-fill.sql - Add table_regex 'node_types' checking patterns.

     www/editimageid.php3 - The reworked PHP page.
     www/imageid_defs.php - Add an Experiment::EditImageid class method
                            bridging to the script via XML.
     backend/{editimageid,GNUmakefile}.in configure configure.in - New backend script.
     db/Image.pm.in - Add an EditImageid worker class method for script arg checking.
     sql/database-fill.sql - Add to the table_regex 'images' checking patterns.

     www/editexp.php3 - The reworked PHP page.
     backend/{editexp,GNUmakefile}.in configure configure.in - New backend script.
     www/experiment_defs.php - Add an Experiment::EditExp class method
                               bridging to the script via XML.
     db/Experiment.pm.in - Add an EditExp worker class method for script arg checking,
                           And a missing description method.
     sql/database-fill.sql - Add to the table_regex 'experiments' checking patterns.

     www/newmmlist.php3 - The reworked PHP page.
     backend/{newmmlist,GNUmakefile}.in configure configure.in - New backend script.
     sql/database-fill.sql - Add table_regex 'mailman_lists' checking patterns.

     www/newgroup.php3 - The reworked PHP page.
     www/newgroup_form.php3 - Removed, form merged into newgroup.php3 .
     www/group_defs.php - Add a Group::Create method bridging to the script via XML.
     www/showproject.php3 - Link to newgroup.php3, rather than newgroup_form.php3 .
     configure configure.in  - Add the newgroup backend script.
     backend/{newgroup,GNUmakefile}.in - Add the Perl script.
     db/Group.pm.in - Update the AccessCheck method to allow TB_PROJECT_LEADGROUP.
     sql/database-fill.sql - Add table_regex 'groups' checking patterns.

 GNUmakefile.in configure configure.in  - Add the testbed/backend directory.
 www/moduserinfo.php3 - The reworked PHP page.
 www/user_defs.php - Add a ModUserInfo method bridging to the script via XML,
     and remove the ChangeProfile method that is being replaced.
 backend/{moduserinfo,GNUmakefile}.in - Add the Perl script.
 db/User.pm.in - Add a ModUserInfo worker class method for script arg checking.
     Also SetUserInterface, SetWindowsPassword, and AccessCheck methods,
     and a copy of the escapeshellarg fn.
 sql/database-fill.sql - Add some to the table_regex 'users' checking patterns.

Support stuff:
 account/tbacct.in - Update the UpdateWindowsPassword() function.
 db/libdb.pm.in - Add TBDB_USER_INTERFACE_EMULAB and TBDB_USER_INTERFACE_PLAB().
 tbsetup/libtestbed.pm.in - Add TB*EMAIL, TBMAIL_* vars (OPS, WWW, AUDIT).

form processing to be done.

The gist is that I have moved all of the data checking and DB work to
the backend into a new script called utils/newosid. This script does
all the field checking that used to be done in php. It takes a simple
XML file as input and returns a set of strings to format as errors (if
there are any).

The overall goal to make a big push to move this code out of PHP and
perl.  A nice side effect is that many operations that are current
only available via the web interface will also become available
command line (and also XMLRPC with a little moew work).