Small changes to image access permissions checks. Root can get any image it wants, and frisbeelauncher only requires READINFO permissions, so that users can os_load shared images still. Also, have os_load pass its debug flag to frisbeelauncher if set.

experiment. Here is mail to tbops:

* Moved the working directory for experiment setup/swap/end to a new
  directory located on boss instead of over NFS to /proj/$pid/$eid. This
  new location is /usr/testbed/expwork/$pid/$eid.

* Changed the name of the directories we create in /usr/testbed/expinfo to
  $pid-$eid.$index where $index is a new autoincrement field in the DB
  table. I really hated the names that were created before.

* Changed where logs are written from /tmp to the new location in
  /usr/testbed/expwork/$pid/$eid.

Okay, why.

* We no longer operate on NFS mounted directories that might hang. Its
  easier to catch the situation where a copy of the log file over at the
  end of experiment creation fails cause of an NFS problem.

* We no longer have user writable files that are inputs to other parts of
  the system (like top and ptop files).  Not that a user would be bad, but
  it closes a hole.

* We no longer copy user writable files from /proj to boss where we might
  fill up an important filesystem cause the user put a .ndz file in the the
  working directory. Not that a user would be bad, but it closes a hole.

* Its easier to save all the log files this way, for each swap in and
  out.

* Removing a directory over NFS is a royal irritant when someone is CD'ed
  into that directory or looking at a file on the other side (the astute
  observer will peg this as the reason I went down this idiotic path in the
  first place!).

* About 6 other reasons that I can no longer remember. Seriously, I really
  had more reasons I can no longer remember! :-)

exhibiting. Sheesh!

flag in the nodes table when a user changes his info. Two routines,
one to do it by type (as for widearea nodes) and another to do it by
project (as for local) nodes. This last is kinda inefficient, but
probably not too big a deal.

mkacct: Two changes.
1. Use the above changes in libdb when a user changes his info. With
   this change, no longer need to do an account update in the
   experiment page for the ron/wa nodes. The nodes are marked as
   needing the update in mkacct, based on the nodes the user has
   access to. Note, this change applies only to widearea nodes; still
   need to use the update option in the experiment menu for local
   nodes, although I plan to change that to at some point by adding a
   watchdog on the local nodes.

2. ssh2 key support. The DB can now store both ssh1 and ssh2 keys,
   however those keys are handled differently when creating the auth
   keys files for users. There are actually two files created now, the
   second being the ssh2 key file call authorized_keys2. This change
   is mirrored in the client side code as well.

files so that they can be viewed later from the web interface.

person logged in is an admin, and doesn't have adminoff set, set it.
(We have to prepend an HTTP_ so that it will make it through suexec.)

This is needed by, for example Mac's/Kirk's idlecheck/slothd stuff.

admin privs.  The idea is that you have to be explicit about when you
want to make use of your super powers, to prevent accidents.

Use the new withadminprivs script to get your admin privs, as in:
withadminprivs inuse
or
withadminprivs node_reboot -e testbed,foo

Add a taint check for \-tcsh, per Leigh's suggestion. It fixes the bug we were seeing, oddly enough.

name of the script, and the name of the user who ran it, in the form
"script:user" . Should be useful for accounting.

returns the node ID that should be used in the widearea_* tables.
This is mainly so that we don't have to hardcode 'boss' as the local
representative in too many places (all local node are assumed to
have the same network characteristics as boss.)

happened (in the web page listing).

node_types table (isvirtnode).

two benefits: (1) More general (2) Regains ability to run without the
event system. Previously, since programs that watned to set node state
had to 'use event', this broke our ability to run without the event
system. Now, we can do a check in libdb for the event system, and not
use it if EVENTSYS is not set. If not, we update state in the database
directly rather than sending an event.

Also added equivalent calls for node operational mode, as well as new
constants for both state and mode.

Converted power and node_reboot to use this new scheme.

experiment log file to the user as it gets generated. The web page
does not redraw, it just never exits until the backend sees that the
experiement transition is done, and then it exists, which terminates
the script. I added a DB field to hold the logfile name and some
routines in libdb, with the idea that this might be more generally
useful at some point. Next time you create an experiment, look for the
last sentence, and click on "realtime".

* tmcd/ron: A new directory of client code, based on the freebsd
  client code, but scaled back to the bare minimum. Does only account
  and group file maintenance. I redid the account stuff so that only
  emulab accounts are operated on. Does not require a stub file, but
  instead keeps a couple of local dbm files recording what groups and
  accounts were added by Emulab. There is a ton of paranoia checking
  to make sure that local accounts are not touched.

  The update script that runs on the client node detaches so that the
  ssh from boss returns immediately. update can also be run from the
  node periodically and at boottime. The script is installed setuid
  root, but checks to make sure that *only* root or "emulabman" has
  invoked it.

* utils/sshremote: New file. For remote nodes, instead of using sshtb,
  use sshremote, which ssh's in as "emulabman", which needs to be a
  local non-root user, but with an authorized_keys file containing
  boss' public key.

* web interface changes: Allow user to specify his own public key in
  addition to the emulab key.

  Add option in showexp page to update accounts on nodes in the
  experiment. I was originally intending to do this from approveuser,
  but this was easier and faster. I will add an option to do it on the
  approveuser page later.

* libdb.pm: Add a TBIsNodeRemote() query to see if a node is in the
  local testbed or a pcRemote node. Currently, this test is hardwired
  to a check for class=pcRemote, but this will need to change to a
  node_types property at some point.

* node_update: Reorg so that there is a maximum number of children
  created. Previously, a child was forked for each node, but that
  could chew up too many processes, especially for remote nodes which
  might hang up. For the same reason, we need to "lock" the experiment
  so that it cannot be terminated while a node_update is in progress.
  Might be to relax that, but this was easy for now. Also add
  distinction between local and remote, since for remote we use
  sshremote insted of sshtb. Various cleanup stuff

* mkacct; When generating a new account, include user supplied pub key
  in the authorized keys file, in addition to the eumlab generated
  key. Both keys are stored in the DB in the users table. Anytime we
  update an account, get a fresh copy of the emulab pub key, in case
  user changes it.

nodeid.

recreating the connection to the DB across a fork. It appears that
with the connection shared, DB queries can fail. It would be nice if
PERL had fork handlers.

Add TBSetNodeEventState() and TBGetNodeEventState() library routines,
and some constants for the event tags.

Beef up the experiment access check code to handle destroy as a
distinct case.

line in all email from the system. Remove all of the TESTBED: tags
and modify the email function in the web server and perl library to
prepend @DOMAIN@: to the message.

supporting autocreating and autoloading images. The imageid form now
sports a field to specify a nodeid to create the image from; If set,
the backend create_image script is invoked. Thats the easy part.
Slightly harder is autoloading images based on the osid specified in
the NS file. To support this, I have added a new DB table called
osidtoimageid, which holds the mapping from osid/pctype to imageid.
When users create images, they must specify what node types that image
is good for. Obviously, the mappings have to be unique or it would be
impossible to figure it out! Anyway, once that image mapping is
in place and the image created, the user can specify that ID in the NS
file. I've changed os_setup to to look for IDs that are not loaded,
and to try and find one in the osidtoimageid. If found, it invokes
os_load. To keep things running in parallel as much as possible,
os_setup issues all the loads/reboots (could be more than a single set
of loads is multiple IDs are in the NS file) at once, and waits for
all the children to exit. I've hacked up os_load a bit to try and be
more robust in the face of PXE failures, which still happen and are
rather troublsesome. Need an event system!

Contained in this revision are unrelated changed to make the OS and
Image IDs per-project unique instead of globally unique, since thats a
pain for the users. This turns out to be very messy, since underneath
we do not want to pass around pid/ID in all the various places its
used. Rather, I create a globally unique name and extened the OS and
Image tables to include pid/name/ID. The user selects pid/name, and I
create the globally unique ID. For the most part this is invisible
throughout the system, except where we interface with the user, say in
the web pages; the user should see his chosen name where possible, and
the should invoke scripts (os_load, create_image, etc) using his/her
name not the internal ID. Also, in the front end the NS file should
use the user name not the ID. All in all, this accounted for a number
of annoying changes and some special cases that are unavoidable.

Not done yet.

extra unix groups (unixgroup_membership) for special local users that
need more groups than just their project membership (ie: flux, wheel,
etc). In mkacct-ctrl, no longer use the admin bit to determine extra
groups (which were hardwired in), but get the extra group list from
the DB. This applies to accounts on boss/users; experimental nodes
still use the admin bit (via tmcd) to get wheel added to the group
set. Might be worth doing at some point.

to its current value returns an error.

a '-n' option to use netdisk, and will respond properly to changing
TB_DEFAULT_RELOADTYPE in libdb.

os_load also got some fixes for the -w flag when used with Frisbee - it fires
off all nodes at once, rather than two at once.

stuff so that the web page did not need to do anything except display
and form processing. Add tbsetup/node_control for backend so that it
can be called from the command line too. The virt_nodes table is also
updated (for those values that have virt_nodes equivalents), and this
mostly implies that changes can be applied only to swapped in
experiments since we use the reserved table to map pcXXX to its vname
so that the virt_nodes table can be updated. It is an easy extension
to allow changes based on the pid/eid/vname, but I do not see a reason
to support this ability yet. Note usage:

    Usage: node_control name=value [name=value ...] node [node ...]
           node_control -e pid,eid name=value [name=value ...]
           node_control -l
    For multiword values, use name='word0 ... wordN'
    Use -l to get a list of operational parameters you can change.
    Use -e to change parameters of all nodes in an experiment.

    {824} stoller$ /build/testbed/install//bin/node_control -l
      next_boot_osid            - (administrators only)
      startup_command
      bios_version              - (administrators only)
      rpms                      - (multiple options allowed)
      default_boot_cmdline
      default_boot_path
      default_boot_osid
      next_pxe_boot_path        - (administrators only)
      tarfiles                  - (multiple options allowed)
      pxe_boot_path             - (administrators only)
      next_boot_cmdline         - (administrators only)
      deltas                    - (multiple options allowed)
      next_boot_path            - (administrators only)

snmpit.

pages now show the lastlogin info that is gathered from sshd syslog
reporting to users. That info is parsed by security/genlastlog.c, and
entered into the DB in the nodeuidlastlogin and uidnodelastlogin
tables. If not obvious from the names, for each user we want the last time
they logged in anyplace, and for each node we want the last time anyone
logged into it. The latter is obviously more useful for scheduling
purposes. All of the various images have new /etc/syslog.conf files,
and the 6.2 got new sshd_configs (all cvsup'ed with kill -HUP). There
is an entry in boss:/etc/crontab and users:/etc/syslog.conf. All of
this is decribed in greater detail in security/genlastlog.c.

default OSID from the node_types table, to a specific OSID from the
partition table on the actual node. This is to avoid setting the boot
OSID to RHL_STD when the node is released, which causes a boot
failure. Okay, so I added a library routine to do this (yanked out of
os_setup where I did the code originally). This would solve most of
the problems, except where there was no OS loaded that would satisfy
the mapping, in which case the user must have done an os_load, and now
that auto schedules a reload. Anyway, seemed like this should work.
Ha! Mysql locking is downright dumb; all tables used within a lock
region must be locked. nfree was already locking 9 tables, and in
order to call out to library routines (which might use anything) I
would have to lock the world, which is not actually possible anyway.
Why all this locking in nfree in the first place? The idea is that
there is a race between releasing the node from reserved, and cleaning
up all those tables (interfaces, delays, nodes, etc). We don't want to
free a node, and have it get allocated to another experiment before
the cleanup is done, since that would mess up the state of the node.
The solution (albiet a crufty one) was to lock just the reserved table
(which guards against multiple people trying to nfree the same node at
the same time) and switch the reservation out of the pid,eid and into
a holding reservation. This effectively removes the node from the
users control, but keeps it reserved. Then I unlock the reserved
table. With that done, I can clean up all those tables without any
locking, since the node is still reserved. After cleanup, I can either
delete the reservation, or move it to the next reserve or reload
reservation if those were pending. No locking is needed at this point
since single table changes are atomic (and nalloc locks reserved
anyway). Okay, so now we sit back and see if this was a good idea.

reloads for nodes in an experiment.
Change os_load to schedule a default image reload whenever a mereuser
loads an image that is not the default image for that node type.
Add some support stuff in libdb (TBSetSchedReload) and some constant
definitions for sched_reload and for nodelog.

viewing the log.

experiment code. No longer uses another table. Rather, the experiment
record contains a couple of extra fields for the batch system. Also
combined some of the backend code (no longer a killbatch script).
Also added scriptable experiments; the batchexp program in the bin
directory can start an experiment from the command line, and in fact
is used from the web page for both batch experiments and immediate
experiments (-i option). All of the DB code that was in the web
interfaces was moved to batchexp.