GitLab

The major functional change in this revision is converting from user
selected UIDs to system selected UIDs. This is controlled by the
variable $USERSELECTUIDS in defs/defs.php3.in which is now set to
zero, so system selected UIDs is the default.

The algo for creating the uid is to take the email address, strip the
@whatever from it, squeeze out dots and dashes and underlines, and
make sure any +foo tokens are removed. Then make sure it is unique by
taking the first 5 characters and then adding a 3 digit number,
derived by checking the DB to see what exists.

Since we will want to (more often) change the UID selected, there is a
new admin only menu option on the Show User page. It calls the backend
script to do the work (sbin/changeuid).

The login page now defaults to storing and showing the email address
for login, rather then the UID. It will still accept either one though
(has for a long time).

Along the way I also reorg'ed a number of pages to use the new user,
group, and project classes and moved some common functionality into
the class defs.

Also changed the way addpubkey is called, to avoid some confusion.

term archiving of firstclass objects like users, projects, and of
course templates.

* Projects, Users, and Groups are now uniquely identified inside the
  DB by a index value that will not be reused. If necessary, this
  could easily be a globally unique identifier, but without federation
  there is no reason to do that yet.

* Currently, pid, gid, and uid still need to be locally unique until
  all of the changes are in place (which is going to take a fairly
  long time since the entire system operates in terms of those, except
  for the few places that I had to change to get the ball rolling).

* We currently archive deleted users to the deleted_users table (their
  user_stats are kept forever since they are indexed by the new index
  column). Eventually do the same with projects (not sure about
  groups) but since we rarely if ever delete a project, there is no
  rush on this one.

* At the same time, I have started a large reorg of the code, to move
  all of the user, group, project code into modules, both in php and
  perl, turning them into first class "objects" (as far as that goes
  in php and perl). Eventually, the number of query statements
  scattered around the code will be manageable, or so I hope.

* Another related part of this reorg is to make it easier to move the
  new user/project/group code in the perl backend so that it can be
  made available via the xmlrpc interface (without duplication of the
  code).

of people who put "Dr." for title!)

would be no big deal, except that we want to retain user_stats for
deleted users, and rather then a deleted_user_stats table, I want to
retain stats for deleted users in the user_stats table, since that
is a more natural place for them.

The main problem is that we use the login (uid) as the cross table
reference slot all over the DB, which is fundamentally incorrect, if
we want to be able reuse uids and still know what historical data
refers to.

So, I have taken a few baby steps towards weaning us off the uid, and
towards permanently unique key for users, using the unix_uid integer
for now, but probably something slightly different later.

The user_stats is now indexed on this new key (called uid_idx in the
users_stats table) instead of the plain uid.

The unix_uid slot in the users table is no longer an auto_increment
field, but instead uses the emulab_indicies table for the next
available index.

they don't by default destroy the partially completed project request data
in the form.

Jay has "comments"), but I do not want it hanging around in my source
tree. Here is my mail message:

* The "My Mailing Lists" is context sensitive (copied from Tim's
  changes to the My Bug Databases). It takes you to the *archives* for
  the current project (or subgroup) list. Or it takes you to your
  first joined project.

* The showproject and showgroup pages have direct links to the project
  and group specific archives. If you are in reddot mode, you also
  get a link to the admin page for the list. Note that project and
  group leaders are just plain members of these lists.

* The interface to create a new "user" list is:

	https://www.emulab.net/dev/stoller/newmmlist.php3

  We do not store the password, but just fire it over in the list
  creation process.

  Anyone can create their own mailing lists. They are not associated
  with projects, but just the person creating the list. That person
  is the list administrator and is given permission to access the
  configuration page.

  This page is not hooked in yet; not sure where.

* Once you have your own lists, you user profile page includes a link
  in the sub menu: Show Mailman Lists. From this page you can delete
  lists, zap to the admin page, or change the admin password (which is
  really just a subpage of the admin page).

* As usual, in reddot mode you can mess with anyone else's mailman lists,
  (via the magic of mailman cookies).

* Note on cross machine login. The mailman stuff has a really easy way
  to generate the right kind of cookie to give users access. You can
  generate a cookie to give user access, or to the admin interface for
  a list (a different cookie). Behind the scenes, I ssh over and get
  the cookie, and set it in the user's browser from boss. When the
  browser is redirected over to ops, that cookie goes along and gives
  the user the requested access. No passwords need be sent around,
  since we do the authentication ourselves.

for the near future. Two big changes:

* Add WikiOnly accounts. An external user can register for an account on
  the wiki. Rather then use the registration stuff that comes with TWiki,
  redirect to new Emulab web page so we can manage all of the wiki accounts
  from one place. I modified the joinproject page to spit out a subset of
  the required fields so that its simple to get a wiki only account (just a
  few things to fill in).

  In keeping with current security practices, we still generate a
  verification email message to ensure the email address works. However,
  when the user completes the verification, the wiki account is created right
  away, rather then waiting for someone to approve it (since that would
  defeat the entire point of the wiki).

  Aside: I have not thought much about the conversion from a wiki-only
  account to a real account. That is going to happen, and it would be nice
  if that step did not require one of use to go in and hack the DB. Will
  cross that moat later.

  Aside: Rather beat up on the modify user info page too much, I continue
  to spit out the same form, but mark most of the fields as not required,
  and allow wiki-only people to not specify them.

* Both the joinproject and newproject pages sport a new WikiName field so
  that users can select their own WikiName. I added some JavaScript to
  both pages that generate a suitable wikiname from the FullName field, so
  that as soon as the user clicks out of the FullName, a default wikiname is
  inserted in the field.

  Both pages verify the wikinames by checking to make sure it is not
  already in use, and that it meets the WikiRules for WikiTopic names.
  (someone please shoot me if I continue to use WikiNotation).

hopefully understand as First and Last name!

which should encourage them to verify quickly.
(This passed lbs review, but someone ought to test it after install.)

case lbs doesn't like it.

prevent people from asking for 'root', 'toor', etc. as usernames, and
will hopefully help with new installations, which may have created
accounts by hand. Note that this checks boss only, not ops.

Also fixed a bug in newproject.php3 that was incorrectly letting
through duplicate usernames.

error - it's too hard to destingish a string from 0 in php, and every
place we call this function, we've already made sure the group exists
anyway.

account, and direct user to Forgot Username/Password page.

Changed the Forgot page to allow user to *either* reset password, or
just find out their username if that is what they forgot.

usually handled via uid cookie we get back from the browser, but if
the user Clicks stop or maybe has cookies off, we don't that info.

zillions of DB fields that we have to set. My solution was to add a
meta table that describes what is a legal value for each table/slot
for which we take from user input. The table looks like this right
now, but is likely to adapt as we get more experience with this
approach (or it might get tossed if it turns out to be a pain in the
ass!).

	CREATE TABLE table_regex (
	  table_name varchar(64) NOT NULL default '',
	  column_name varchar(64) NOT NULL default '',
	  column_type enum('text','int','float') default NULL,
	  check_type enum('regex','function','redirect') default NULL,
	  check tinytext NOT NULL,
	  min int(11) NOT NULL default '0',
	  max int(11) NOT NULL default '0',
	  comment tinytext,
	  UNIQUE KEY table_name (table_name,column_name)
	) TYPE=MyISAM;

Entries in this table look like this:

	('virt_nodes','vname','text','regex','^[-\\w]+$',1,32,NULL);

Which says that the vname slot of the virt_nodes table (which we trust the
user to give us in some form) is a text field to be checked with the given
regex (perlre of course), and that the min/max length of the text field is
1 and 32 chars respectively.

Now, you wouldn't want to write the same regex over and over, and since we
use the same fields in many tables (like pid, eid, vname, etc) there is an
option to redirect to another entry (recursively). So, for "PID" I do this:

        ('eventlist','pid','text','redirect','projects:pid',0,0,NULL);

which redirects to:

	('projects','pid','text','regex','^[a-zA-Z][-\\w]+$',2,12,NULL);

And, for many fields you just want to describe generically what could go
into it. For that I have defined some default fields. For example, a user
description:

        ('experiment,'usr_name','text','redirect','default:tinytext',0,0,NULL);

which redirects to:

	('default','tinytext','text','regex','^[\\040-\\176]*$',0,256,NULL);

and this says that a tinytext (in our little corner of the database
universe) field can have printable characters (but not a newline), and
since its a tinytext field, its maxlen is 256 chars.

You also have integer fields, but these are little more irksome in the
details.

	('default','tinyint,'int,'regex','^[\\d]+$',-128,127,NULL);

and you would use this anyplace you do not care about the min/max values
being something specific in the tinyint range. The range for a float is of
course stated as an integer, and thats kinda bogus, but we do not have many
floats, and they generally do not take on specific values anyway.

A note about the min/max fields and redirecting. If the initial entry has
non-zero min/max fields, those are the min mac fields used. Otherwise they
come from the default. So for example, you can do this:

    ('experiments','mem_usage','int','redirect','default:tinyint',0,5,NULL);

So, you can redirect to the standard "tinyint" regular expression, but you
still get to define min/max for the specific field.

Isn't this is really neat and really obtuse too? Sure, you can say it.

Anyway, xmlconvert now sends all of its input through these checks (its
all wrapped up in library calls), and if a slot does not have an entry, it
throws an error so that we are forced to define entries for new slots as we
add them.

In the web page, I have changed all of the public pages (login, join
project, new project, and a couple of others) to also use these checks.
As with the perl code, its all wrapped up in a library. Lots more code
needs to be changed of course, but this is a start.

of the CHECKEMAIL function as a result.

for key upload). Add more regexes and addslashes().

Users can, via, moduserinfo, set a preferred shell.
One of {tcsh, bash, csh, sh}.
When users are created, they are given tcsh.
All users which already exist have been given tcsh.

The first three are aggregate tables, while the experiment stats table
gets a record for each new experiment, and is updated when an
experiment is swapped in/out/modify or terminated. Look at the table
to see what is tracked. Once the experiment_stats record is updated,
the aggregate tables are updated as necessary. There are a bunch of
ugly changes to assign_wrapper to get the stats. Note that pnodes is
not incremented until an experiment sucessfully swaps in. This is in
leu of getting status codes; I'm not tracking failed operations yet,
nor creating the log file that Jay wants. I'll do that in the next
round of changes when we see how useful these numbers are.

Most of the changes are to create/delete table entries where
appropriate, and to display the records. Display is only under admin
mode, and the display is raw; just a dump of the assoc tables in php.
The last 100 experiment stats records are available via the Experiment
List page, using the "Stats" show option at the top. Bad place, but
will do for now.

our foreign clientele.

Nit; remove legend for "+" (recommended) on fields,
since there are no recommended fields!

- Added 'Country' to users table
- Changed "Zip" to "ZIP/Postal Code"
- Reformatted Postal Address Forms

finally! These have been in the DB for a long time, but never used.
As Eric pointed out, its easier to get realistic address info from
people if we provide the right forms. The downside is that all current
users will have to adjust their info the next time they edit their
info. Thats okay. I thought about forcing all users to do it the next
time they log in, but I figured people would scream.

we can send links that cause those fields to be filled in for people.
Add those links to email generated by newproject and newgroup pages
so that they can be saved by the leaders.

Add a bit of referrer magic to login page. When clicked from the join
or new project pages, tell login page to pass along the referrer page
so that when login is complete, user is zapped back to the original
page. This is especially nice when combined with the above change to
joinproject, whereby we send along the target pid/gid, but the user
has not yet logged in and remembers to do so via the link at the top
of the page.

Account to make it more clear.

a certain person who shall remain nameless unhappy!

protected page except those that are explicitly deemed okay for a
webonly user. This makes me feel better and safer!

see if user actually has an account (by checking user status user
table). Avoids trying to run suexec as a user that does not actuall
exist on boss cause they do not have an account (since we allow users
to edit personal info before being approved and getting an account).
For addpubkey, we have to run the program as someone, so when the user
does not have an account, run it as nobody.

things. Now whenever someone joins a project, it goes to everyone that
can approve them, and when an approval happens, it gets cc'd to the same
list, so they all know it happened and what permissions were given.

of a random number, as suggested in the php manual. This number
is stashed in the database, in the new verify_key column in the
users table.

Rename the functions that generate and get the keys, and move from
defs.php3 to dbdefs.php3, since they're now DB operations.

getting into the system.

Improved error reporting style.. also added image for uky,
though right now it is the same as the standard image
(will edit it soon.)