GitLab

help nodes in reload_pending get sucked into reloading faster. If it
doesn't do enough, we'll need to do more batching of stuff, so we get some
parallelism in os_load instead of forcing it to serialize by calling
os_load one node at a time.

I was tempted to nuke all the stuff that was in there from the netdisk
reload type, but decided not to. It won't be too long (relatively
speaking) before we have freed, the new "free node manager" that will
replace/supersede our current reload_daemon anyway.

events. This may delay handling of other stuff that happens in my main
loop, but not by too much. To prevent skew, everything (including reload
frequency) is done strictly by seconds elapsed, not by iterations or
anything.

I found that even polling for multiple events without sleeping, I could
only handle a little over 1 per second when I was calling inuse/statetime
for additional info on every event. Even though this only happens in the
worst case (every event is wrong), it won't do. So I took that out. I'll
probably end up adding a faster lookup of the info I need (mostly
reservation, and what osid it thinks it is running). That change took it
up to at least 4 per second (as fast as I could send them manually), more
than 4x our previous performance. So we should be able to keep up now.

Also, add the support for "announcements" to testbed ops when I die and
such. (Been in a few days, but this is the first commit of it)

netbed/RON machines.

Eliminate the second pass over the file to fill in blockindex/total fields.
Blockindex fields are filled in on the first pass, we don't bother with
blocktotal since we don't use it any more.

passed to the frisbee init routine.

Instead of failing if the new image doesn't fit in the target slice,
just warn and truncate.

Back to aggressive adjustment of BSD partition table sizes.  Warn about
and truncate partitions that won't fit in the target slice and adjust the
'c' partition to exactly match the BSD slice.

This is going to be used to sign the stuff we send out to widearea
nodes (images, scripts, etc). The passphrase as the one I used on the
SSH priv keys for widearea nodes.

problem isn't fixed in the latest port or else there is a kernel problem too...

Add some additional debugging for bad packets (since I have seen some)

	add -Wall to CFLAGS and clean up lint
	update the TODO file
	explicitly size the header fields (e.g., int32_t not int)

imagezip:
	Version 2.

	Adds two ints to the header to help track free space.  Each chunk
	now has a first and last sector number which can describe any free
	block before or after the data contained in the chunk.  This is
	needed in order to properly zero all free space when laying down
	an image.  In practice: the first chunk describes any free space
	before the first allocated range and any free space after its
	contained ranges and before the first allocated range in the second
	chunk.  Every other chunk then describes just free space following
	itself (since the previous chunk has already described the space
	before this chunk).  The point being, we only describe each free
	range once.

	Added "relocation" information.  Relocation entries go in the chunk
	header along with region descriptors.  This allows us to identify
	chunks of data which need to be absolute disk blocks instead of
	offsets from the containing partition.  This is now used for BSD-slice
	partition tables which contain absolute disk blocks.  We can now
	create an image in one slice and reload it into another slice.

	Allow zlib compression level 0 (no compression).  This might be
	useful on machines that have slow CPUs: do just FS-compression and
	transfer the image elsewhere faster where it could be re-zipped
	with regular compression.

	Fix goof.  Previously we were not saving any DOS partition with
	an unrecognized type.  We should be naively compressing it instead.
	This is what we now do.  We continue to skip partitions of type 0
	("unused").

	mikeism: add handler for SIGINFO (^T) to report progress of
	a zip-age.

	Added everybody's favorite "dots" mode for reporting progress.

	Eliminate some excess copies left over from the conversion from
	write-every-little-piece to buffer-up-a-full-chunk-and-then-write.

	Eliminaged the special case handling of no skips (ranges) in
	compress_image by creating a single allocated range describing
	the whole disk/partition in this case.

	For NTFS, make the behavior of calling missing unicode routines
	be to return an error rather than exit.  These calls happen,
	but their failing doesn't seem to be fatal.

	Lots of typical mike-pissing on everything else.

imageunzip:

	Modify to handle both V1 and V2 images.

	In slice mode, make sure we don't write past the bounds of
	the slice.  ES&D if we try.

	Make output to unseekable devices work again (broken when
	pwrite was added)

	Add debug -F (Frisbee) option to randomize the presentation of
	chunks to the unzip/write threads.  Used to simulate frisbee.

	Add "-T DOS-type" option to tell imageunzip, when in slice mode,
	to set the type of the slice in the DOS partition table.
	This is useful if you are dropping say a BSD filesystem into
	an unused slice, you don't have to go back later and set this
	with fdisk.  Considered making this info part of the image
	itself (recorded by imagezip when creating a slice image),
	but decided against it.

	writezero takes an off_t for the size, we can be asked to write
	many gigabytes of zero at the end of a disk.

	Turn off dots mode by default.  Ya wanna see spots?  Ya gotta
	turn it on!

	Lots of typical mike-pissing on everything else.

imagedump:

	New tool for checking/dumping the structure of an image and
	reporting stats about it.

Server: make file readsize independent of burstsize (previously
readsize had to be a divisor of burstsize).  A subtle side-effect
is that the dynamic burst rate is recalcluated at the conslusion
of every burst instead of after every readsize count of blocks has
been sent (less than a burst)  This just seems to be more logical.

Client: add "-T DOS-type" option to tell frisbee, when in slice
mode, to set the type of the slice in the DOS partition table.
This is useful if you are dropping say a BSD filesystem into
an unused slice, you don't have to go back later and set this
with fdisk.  Considered making this info part of the image
itself (recorded by imagezip when creating a slice image),
but decided against it.

the different node types

Needed for the frisbee environment, so might as well use it everywhere.

a temporary class for testing new images.

Modified the timeout logic in create_image to track the image creation progress
(size) rather than simply waiting a certain amount of time.  Also changed the
code to report progress at regularly spaced intervals (adjustable), and
to indicate when the timeout timer has been activated, or halted due to
progress.  The changes also include an NFS cache slack factor, which makes
the effective non-progress timeout equal to the sum of the slack time, plus
the non-progress time (currently 3 + 5 = 8 minutes).

Some changes were made to the error and cleanup logic to help revert the
state of the DB and node as much as possible (node is not rebooted if the
DB state cannot first be reverted) prior to exit.

protected page except those that are explicitly deemed okay for a
webonly user. This makes me feel better and safer!

A user can request a local account on the machine he/she has dedicated
to netbed. In fact, multiple people can request local accounts. They
just need to fill in the form, supplying the usual personal data we
require, and then some info about the node. This includes the IP and
the CDKey as verification (we now save the original CDKey in the DB
forever), as well as info about the node (location, processor type,
connection type). They must fill out the node info for the first
account request. Otherwise, it can be ignored (that is, if an entry is
already in widearea_nodeinfo, we do not require those fields).
Once submitted, the user has to go through the usual verification
step.

To approve the user, admin people get a new link on the menu to
approve widearea accounts. That page looks a lot like the normal
join project approval page, only is tailored slightly for widearea
accounts instead of projects.

Once approved, widearea users get a webonly account. Note that it can
be a normal account, say if the user is also in a normal project, or
if we just want to give out an account on ops/boss to this person.
Just need to clear the webonly flag in the DB, and the account will be
built as normal, except they are put in the "guest" group on boss/ops
if not a member of any projects.

There are two new tables. widearea_accounts and widearea_nodeinfo. The
accounts table maps uid's to specific nodes they get an account on
(see changes in tmcd). The mapping also includes a trust value (user
or root, although it should be rare to give out root access) for the
account.

a new slot so that we remember it. The user needs to enter that key in
the local account request page. This is how we enforce some semblance
of security; the user has to know the IP of the node, and the CDKey.
Also works to weed out the bozos who like to fill in random web
forms.

personal info. Of course, they can still load the other pages if they
now the page names, but since they won't be a member of any projects
they will not have permission to actully do anything or look at
anything.

determine if user has a real account.

see if user actually has an account (by checking user status user
table). Avoids trying to run suexec as a user that does not actuall
exist on boss cause they do not have an account (since we allow users
to edit personal info before being approved and getting an account).
For addpubkey, we have to run the program as someone, so when the user
does not have an account, run it as nobody.

force audit mode when a non-admin mucks with another persons account.
Add check for "webonly" accounts and treat like other users that do
not get an account on boss/ops.
Check for users without any project membership, and create account
with the guest group. This won't actually happen, but I made this
change in case we decide to give widearea owners a real account.
I think setgroups should get an equiv change at some point.

nodes.

be like the later stuff I did in nodecontrol_list, which was cleaner.
Also, make the default view "active" users so that I do not have to
wait 5 seconds for the page to download!

in his profile output.

accounts and do not die. Also, no longer rename the home dir; just
leave it where it is; not sure that served any purpose at all.
Also mark nodes as needing account updates (this requires that the
group_membership remain intact until after the script runs).

In the php script, also purge the sfskeys table and the new
widearea_accounts table.