Commit 77dbad39 authored by Mike Hibler's avatar Mike Hibler

Support image PUT (aka, "upload") and assorted minor changes.

1. Support for PUT.

The big change is support for uploading via the master server, based heavily
on the prototype that Grant did. Currently only host-based (IP-based)
authentication is done as is the case with download. Grant's SSL-based
authentication code is "integrated" but has not even been compiled in.

The PUT protocol allows for assorted gewgaws, like specifying a maximum size,
setting a timeout value, returning size and signature info, etc.

There is a new, awkwardly-named client utility "frisupload" which, like the
download client, takes an "image ID" as an argument and requests to upload
(PUT) that image via the master server. As with download, the image ID can
be either of the form "<pid>/<emulab-image-name>", to upload/update an actual
Emulab image or it can start with a "/" in which case it is considered to
be a pathname on the server.

On the server side, the master server takes PUT requests, verifies permission
to upload the image, fires up a separate instance of an upload daemon (with
the even catchier moniker "frisuploadd"), and returns the unicast addr/port
info to the client which then begins the upload. The master server also acts
as a traffic cop to make sure that downloads and uploads (or uploads and
uploads) don't overlap.

This has been integrated into the Emulab "create image" process in a
backward-compatible way (i.e., so old admin MFSes will continue to work).
Boy, was that fun. One not-so-desirable effect of this integration is that
images now traverse our network twice, once to upload from node to boss and
once for boss to write out the image file across NFS to ops. This is not
really something that should be "fixed" in frisbee, it is only "undesirable"
because we have a crappy NFS server.

What has NOT been done includes: support of hierarchical PUT operations
(we don't need it for either the elabinelab or subboss case), support for
uploading standard images stored on boss (we really want something better
than host-based authentication here), and the aforementioned support of
SSL-based authentication.

2. Other tidbits that got mixed in with PUT support:

Added two new site variables:
    images/frisbee/maxrate_std
    images/frisbee/maxrate_usr
which replace the hardwired (in mfrisbeed and frisbeelauncher before that)
bandwidth limits for image download. mfrisbeed reads these (and the
images/create/* variables) when it starts up or receives a HUP signal.
These could be read from the DB on every GET/PUT, but they really don't change
much and I needed something to test the reread-the-config-on-a-HUP code!

Fixed avoidance of "problematic multicast addresses" so it would actually
work as intended.

Lots of internal "refactoring" to make up for things I did wrong the first
time and to give the general impression that "Wow, Mike did a LOT!"
parent 3ba294d9
......@@ -192,6 +192,18 @@ while (my ($uid,$uid_idx,$admin,$status) = $users_result->fetchrow_array()) {
mysystem("$TB/sbin/genelists -a");
mysystem("$TB/sbin/exports_setup");
#
# Set some sitevars that are different for inner elabs
#
if (TBSiteVarExists("images/frisbee/maxrate_std")) {
# XXX traditional value for elabinelab
TBSetSiteVar("images/frisbee/maxrate_std", "54000000");
}
if (TBSiteVarExists("images/frisbee/maxrate_usr")) {
# XXX traditional value for elabinelab
TBSetSiteVar("images/frisbee/maxrate_usr", "54000000");
}
#
# Fixup pxeboot and MFS info.
#
......
......@@ -2,7 +2,7 @@
#
# EMULAB-COPYRIGHT
# Copyright (c) 2000-2004, 2007 University of Utah and the Flux Group.
# Copyright (c) 2000-2011 University of Utah and the Flux Group.
# All rights reserved.
#
......@@ -10,14 +10,14 @@ use English;
use Getopt::Std;
#
# Create a disk image. Caller must have sudo permission!
# Client-side to create a disk image. Caller must have sudo permission!
#
sub usage()
{
print STDOUT "Usage: create-image [-s slice] <device file> <filename>\n";
print STDOUT "Usage: create-image [-S image-server] [-F imageid] [-s slice] <device file> <filename>\n";
exit(-1);
}
my $optlist = "rs:";
my $optlist = "F:S:s:";
#
# Turn off line buffering on output
......@@ -35,10 +35,15 @@ delete @ENV{'IFS', 'CDPATH', 'ENV', 'BASH_ENV'};
#
my $sudo = "/usr/local/bin/sudo";
my $zipper = "/usr/local/bin/imagezip";
my $uploader = "/usr/local/bin/frisupload";
my $slice = "";
my $device;
my $filename;
# Frisbee master server params
my $iserver = "boss"; # XXX
my $imageid;
#
# Parse command arguments. Once we return from getopts, all that should be
# left are the required arguments.
......@@ -51,6 +56,23 @@ if (@ARGV != 2) {
usage();
}
if (defined($options{"S"})) {
$iserver = $options{"S"};
if ($iserver =~ /^([-\w\.]+)$/) {
$iserver = $1;
} else {
die("Bad -S hostname: '$iserver'");
}
}
if (defined($options{"F"})) {
$imageid = $options{"F"};
if ($imageid =~ /^(\S+)$/) {
$imageid = $1;
} else {
die("Bad -F imageid: '$imageid'");
}
}
if (defined($options{"s"})) {
my $num = $options{"s"};
......@@ -68,7 +90,11 @@ if (defined($options{"s"})) {
$slice = "-N $slice";
}
$device = $ARGV[0];
$filename = $ARGV[1];
if (defined($imageid)) {
$filename = "-";
} else {
$filename = $ARGV[1];
}
#
# Untaint the arguments.
......@@ -87,13 +113,22 @@ else {
die("Tainted output filename: $filename");
}
#
# If imageid is defined, we use the frisbee uploader.
#
my $cmd = "$zipper $slice $device $filename";
if (defined($imageid)) {
$cmd .= " | $uploader -S $iserver -F $imageid -";
}
#
# Run the command using sudo, since by definition only testbed users
# with proper trust should be able to zip up a disk. sudo will fail
# if the user is not in the proper group.
#
if (system("$sudo $zipper $slice $device $filename")) {
if (system("$sudo $cmd")) {
print STDERR "*** Failed to create image!\n";
print STDERR " command: '$sudo $cmd'\n";
exit 1;
}
......
......@@ -38,7 +38,7 @@ WITH_MSERVER_EMULAB = 1
include $(OBJDIR)/Makeconf
all: frisbee frisbeed mfrisbeed
all: frisbee frisbeed mfrisbeed frisupload frisuploadd
include $(TESTBED_SRCDIR)/GNUmakerules
......@@ -74,6 +74,16 @@ MSERVERFLAGS = -DUSE_NULL_CONFIG $(CFLAGS) -I$(OBJDIR)
MSERVEROBJS = mserver.o $(SHAREDOBJS) config.o config_null.o
MSERVERLIBS =
# Master server based image uploader client
UPLOADFLAGS = $(CFLAGS)
UPLOADLIBS =
UPLOADOBJS = upload.o uploadio.o log.o network.o utils.o
# Master server based image uploader server
UPLOADDFLAGS = $(CFLAGS)
UPLOADDLIBS =
UPLOADDOBJS = frisuploader.o uploadio.o log.o
ifeq ($(WITH_MSERVER_EMULAB),1)
# Emulab master server config
MYSQLCFLAGS = -I/usr/local/include
......@@ -130,11 +140,27 @@ frisbeed-debug: $(SERVEROBJS)
# cp frisbeed frisbeed.debug
# strip frisbeed
frisupload-debug: $(UPLOADOBJS)
$(CC) $(LDFLAGS) $(UPLOADFLAGS) $(UPLOADOBJS) $(UPLOADLIBS) -o $@
# cp frisupload frisupload.debug
# strip frisupload
frisuploadd-debug: $(UPLOADDOBJS)
$(CC) $(LDFLAGS) $(UPLOADDFLAGS) $(UPLOADDOBJS) $(UPLOADDLIBS) -o $@
# cp frisuploadd frisuploadd.debug
# strip frisuploadd
mfrisbeed: $(MSERVEROBJS)
$(CC) $(LDFLAGS) $(MSERVERFLAGS) $(MSERVEROBJS) $(MSERVERLIBS) -o mfrisbeed
cp mfrisbeed mfrisbeed.debug
strip mfrisbeed
frisuploader.o: $(SRCDIR)/frisuploader.c decls.h uploadio.h
$(CC) -c $(UPLOADDFLAGS) $(SRCDIR)/frisuploader.c
upload.o: $(SRCDIR)/upload.c decls.h utils.h uploadio.h
$(CC) -c $(UPLOADFLAGS) $(SRCDIR)/upload.c
uploadio.o: $(SRCDIR)/uploadio.c decls.h utils.h uploadio.h
$(CC) -c $(UPLOADFLAGS) $(SRCDIR)/uploadio.c
mserver.o: $(SRCDIR)/mserver.c decls.h configdefs.h utils.h
$(CC) -c $(MSERVERFLAGS) $(SRCDIR)/mserver.c
config.o: $(SRCDIR)/config.c configdefs.h log.h
......@@ -164,12 +190,13 @@ log.o: decls.h log.h
network.o: decls.h utils.h
trace.o: decls.h trace.h log.h
install: $(INSTALL_SBINDIR)/mfrisbeed $(INSTALL_SBINDIR)/frisbeed $(INSTALL_SBINDIR)/frisbee
install: $(INSTALL_SBINDIR)/mfrisbeed $(INSTALL_SBINDIR)/frisbeed $(INSTALL_SBINDIR)/frisbee $(INSTALL_SBINDIR)/frisuploadd
client: frisbee
client: frisbee frisupload
client-install: client
$(INSTALL_PROGRAM) frisbee $(DESTDIR)$(CLIENT_BINDIR)
$(INSTALL_PROGRAM) frisupload $(DESTDIR)$(CLIENT_BINDIR)
clean:
/bin/rm -f *.o *.a frisbee frisbeed mfrisbeed *.debug
......
......@@ -11,7 +11,9 @@
#include <stdlib.h>
#include <signal.h>
#include <string.h>
#include <errno.h>
#include <assert.h>
#include <sys/stat.h>
#include "decls.h"
#include "configdefs.h"
#include "log.h"
......@@ -30,6 +32,7 @@ config_signal(int sig)
if (myconfig == NULL)
return;
log("Reading new configuration.");
savedconfig = myconfig->config_save();
if (myconfig->config_read() != 0) {
warning("WARNING: could not load new configuration, "
......@@ -154,12 +157,13 @@ config_free_host_authinfo(struct config_host_authinfo *ai)
* on success, an error code otherwise.
*/
int
config_auth_by_IP(struct in_addr *reqip, struct in_addr *hostip, char *imageid,
struct config_host_authinfo **aip)
config_auth_by_IP(int isget, struct in_addr *reqip, struct in_addr *hostip,
char *imageid, struct config_host_authinfo **aip)
{
struct config_host_authinfo *ai;
if (config_get_host_authinfo(reqip, hostip, imageid, &ai, 0))
if (config_get_host_authinfo(reqip, hostip, imageid,
isget ? &ai : 0, isget ? 0 : &ai))
return MS_ERROR_FAILED;
if (ai->hostid == NULL) {
config_free_host_authinfo(ai);
......@@ -199,3 +203,324 @@ config_dump(FILE *fd)
myconfig->config_dump(fd);
signal(SIGHUP, config_signal);
}
/*
* Utility functions for all configurations.
*/
/*
* Return 'dir' if 'path' is in 'dir'; i.e., is 'dir' a prefix of 'path'?
* Return NULL if 'path' is not in 'dir'.
* Note: returns NULL if path == dir.
* Note: assumes realpath has been run on both.
*/
char *
isindir(char *dir, char *path)
{
int len = dir ? strlen(dir) : 0;
char *rv = dir;
if (dir == NULL || path == NULL || *dir == '\0' || *path == '\0' ||
strncmp(dir, path, len) || path[len] != '/')
rv = NULL;
#if 0
fprintf(stderr, "isindir(dir=%s, path=%s) => %s\n",
dir ? dir : "", path ? path : "", rv ? rv : "NULL");
#endif
return rv;
}
/*
* Account for differences between BSD and Linux realpath.
*
* BSD realpath() apparently doesn't even test the final component.
* It will return success even if the final component doesn't exist.
* Settle on the Linux behavior, which does return an error if the
* final component does not exist.
*
* BSD realpath() is documented to, on error, return the canonicalized
* version of the path up to and including the component that caused the
* error. While this behavior is not documented in the Linux version,
* it is the observed behavior, so we assume it (making only a feeble
* check to verify).
*
* Returns a pointer to the 'rpath' buffer if 'path' fully resolves,
* and 'rpath' is filled with the complete canonicalized path.
*
* Returns NULL if there was an error resolving any component of 'path'
* with errno set to the cause of the failure (e.g., ENOENT if a component
* is missing) and 'rpath' is filled with the canonicalized path up to and
* including the component that caused the error.
*/
char *
myrealpath(char *path, char rpath[PATH_MAX])
{
char *rv;
assert(path[0] != '\0');
rpath[0] = '\0';
rv = realpath(path, rpath);
assert(rpath[0] != '\0');
#ifndef linux
if (rv != NULL) {
struct stat sb;
/* also sets errno correctly */
if (stat(path, &sb) < 0)
rv = NULL;
}
#endif
return rv;
}
#define INDIR(d, dl, p) ((p)[dl] == '/' && strncmp(p, d, dl) == 0)
/*
* "Resolve" a pathname.
*
* Makes sure that 'path' falls within the 'dir' after application of
* realpath to resolve "..", symlinks, etc. If 'create' is set, it will
* create missing intermediate directories. In detail:
*
* The path (and dir) must be absolute, or it is an error.
* If 'create' is zero, then all components of the path must exist
* or it is an error.
* If 'create' is non-zero, then missing components except the last
* will be created.
* If 'dir' is non-null, the existing part of the path must resolve
* within this directory or it is an error.
* If 'dir' is null, then the path is resolved as far as it exists
* ('create' is ignored and assumed to be zero).
*
* Returns NULL on an error, a malloc'ed pointer to a canonicalized
* path otherwise.
*/
char *
resolvepath(char *path, char *dir, int create)
{
char rpath[PATH_MAX], *next, *ep, *pathcopy;
struct stat sb;
int pathlen = strlen(path);
int dirlen = 0;
int rpathlen;
mode_t omask, cmask;
char *npath;
if (debug > 1)
info("resolvepath '%s' in dir '%s'", path, dir);
/* validate path */
if (path == NULL) {
if (debug > 1)
info(" null path");
return NULL;
}
if (path[0] != '/') {
if (debug > 1)
info(" path is not absolute");
return NULL;
}
if (pathlen >= sizeof rpath) {
if (debug > 1)
info(" path is too long");
return NULL;
}
/* validate dir */
if (dir) {
dirlen = strlen(dir);
if (dir[0] != '/') {
if (debug > 1)
info(" path and dir (%s) must be absolute",
dir);
return NULL;
}
/* XXX make dir=='/' work */
if (dirlen == 1)
dirlen = 0;
} else {
create = 0;
}
/*
* If the full path resolves, make sure it falls in dir (if given)
* and is a regular file.
*/
if (myrealpath(path, rpath) != NULL) {
if (dir && !INDIR(dir, dirlen, rpath)) {
if (debug > 1)
info(" resolved path (%s) not in dir (%s)",
rpath, dir);
return NULL;
}
if (stat(rpath, &sb) < 0 || !S_ISREG(sb.st_mode)) {
if (debug > 1)
info(" not a regular file");
return NULL;
}
return strdup(rpath);
}
if (debug > 1) {
int oerrno = errno;
info(" partially resolved to '%s' (errno %d)", rpath, errno);
errno = oerrno;
}
/*
* If create is not set or we failed for any reason other than
* a non-existent component, we return an error.
*/
if (!create || errno != ENOENT) {
if (debug > 1)
info(" realpath failed at %s with %d",
rpath, errno);
return NULL;
}
/*
* Need to create intermediate directories.
*
* First, check that what does exist of the path falls within
* the directory given.
*/
assert(dir != NULL);
if (!INDIR(dir, dirlen, rpath)) {
if (debug > 1)
info(" resolved path (%s) not in dir (%s)",
rpath, dir);
return NULL;
}
/*
* Establish permission (mode) for new directories.
* Start with permission of 'dir'; this will get updated with
* every component that resolves, so that new directories will
* get created with the permission of the most recent ancestor.
*/
if (stat(dir, &sb) < 0) {
if (debug > 1)
info(" stat failed on dir (%s)?!", dir);
return NULL;
}
cmask = (sb.st_mode & (S_IRWXU|S_IRWXG|S_IRWXO)) | S_IRWXU;
omask = umask(0);
if (debug > 1)
info(" umask=0 (was 0%o), initial cmask=0%o", omask, cmask);
/*
* Find the first component of the original path that does not
* exist (i.e., what part of the original path maps to what realpath
* returned) and create everything from there on.
*
* Note that if what realpath returned is a prefix of the original
* path then nothing was translated and we are already at the first
* component that needs creation. So we find the appropriate location
* in the original string and go from there.
*/
npath = NULL;
if ((pathcopy = strdup(path)) == NULL)
goto done;
rpathlen = strlen(rpath);
assert(rpathlen <= pathlen);
if (rpathlen > 1 && strncmp(pathcopy, rpath, rpathlen) == 0 &&
(pathcopy[rpathlen] == '\0' || pathcopy[rpathlen] == '/')) {
/* same string, start at last slash in path */
if (pathcopy[rpathlen] == '\0') {
next = rindex(pathcopy, '/');
assert(next != NULL);
}
/* rpath is a prefix, start at last slash in that prefix */
else {
next = rindex(rpath, '/');
assert(next != NULL);
next = &pathcopy[next-rpath];
}
}
/* rpath is not a prefix, must scan entire path */
else {
next = pathcopy;
}
assert(next >= pathcopy && next < &pathcopy[pathlen]);
assert(*next == '/');
next++;
if (debug > 1)
info(" pathscan: path='%s', rpath='%s', start at '%s'",
pathcopy, rpath, next);
while ((ep = index(next, '/')) != NULL) {
*ep = '\0';
if (debug > 1)
info(" testing: %s", pathcopy);
/*
* We use realpath here instead of just stat to make
* sure someone isn't actively tweaking the filesystem
* and messing with our head.
*
* If realpath fails, it should fail with ENOENT and
* the realpath-ified part should fall in our directory.
* Otherwise, someone really is playing games.
*/
if (myrealpath(pathcopy, rpath) == NULL) {
if (errno != ENOENT ||
(dir && !INDIR(dir, dirlen, rpath))) {
if (debug > 1)
info(" resolves bad (%s)\n",
rpath);
goto done;
}
/*
* We have hit a missing component.
* Create the component and carry on.
*/
if (mkdir(rpath, cmask) < 0) {
if (debug > 1)
info(" create failed (%s)\n",
rpath);
goto done;
}
if (debug > 1)
info(" created (%s)", rpath);
} else {
if (debug > 1)
info(" exists (%s)", rpath);
/*
* Update the creation permission; see comment above.
*/
if (stat(rpath, &sb) < 0) {
if (debug > 1)
info(" stat failed (%s)?!", rpath);
goto done;
}
cmask = (sb.st_mode & (S_IRWXU|S_IRWXG|S_IRWXO)) |
S_IRWXU;
}
*ep = '/';
next = ep+1;
}
/*
* We are down to the final component of the original path.
* It should either exist and be a regular file or not
* exist at all.
*/
if (myrealpath(pathcopy, rpath) == NULL &&
(errno != ENOENT || !INDIR(dir, dirlen, rpath))) {
if (debug > 1)
info(" final resolved path (%s) bad (%d)",
rpath, errno);
goto done;
}
/*
* We are in our happy place. rpath is the canonicalized path.
*/
npath = strdup(rpath);
if (debug > 1)
info("resolvepath: '%s' resolved to '%s'", path, npath);
done:
umask(omask);
if (pathcopy)
free(pathcopy);
return npath;
}
......@@ -16,6 +16,7 @@
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <errno.h>
#include <assert.h>
#include <mysql/mysql.h>
#include "log.h"
......@@ -25,6 +26,21 @@
#include "tbdb.h" /* the DB library */
#include "config.h" /* the defs-* defines */
extern int debug;
/*
* Private configuration state that can be saved/restored.
*
* For Emulab, there are only a couple of DB values that don't change
* that often.
*/
struct emulab_configstate {
int image_maxsize; /* sitevar:images/create/maxsize (in GB) */
int image_maxwait; /* sitevar:images/create/maxwait (in min) */
int image_maxrate_std; /* sitevar:images/frisbee/maxrate_std (in MB/s) */
int image_maxrate_usr; /* sitevar:images/frisbee/maxrate_usr (in MB/s) */
};
/* Extra info associated with a image information entry */
struct emulab_ii_extra_info {
int DB_imageid;
......@@ -47,12 +63,20 @@ static char *PROJDIR = PROJROOT_DIR;
static char *GROUPSDIR = GROUPSROOT_DIR;
static char *USERSDIR = USERSROOT_DIR;
static char *SCRATCHDIR = SCRATCHROOT_DIR;
#ifndef ELABINELAB
/* XXX should be autoconfiged as part of Emulab build */
static char *IMAGEDIR = "/usr/testbed/images";
#ifdef ELABINELAB
static int INELABINELAB = ELABINELAB;
#else
static int INELABINELAB = 0;
#endif
static uint64_t put_maxsize = 10000000000ULL; /* zero means no limit */
static uint32_t put_maxwait = 2000; /* zero means no limit */
static uint32_t get_maxrate_std = 72000000; /* zero means no limit */
static uint32_t get_maxrate_usr = 54000000; /* zero means no limit */
/* Standard image directory: assumed to be "TBROOT/images" */
static char *STDIMAGEDIR;
/* Emit aliases when dumping the config info; makes it smaller */
static int dump_doaliases = 1;
......@@ -64,6 +88,37 @@ static void *mymalloc(size_t size);
static void *myrealloc(void *ptr, size_t size);
static char *mystrdup(const char *str);
/*
* Return the value of a sitevar as a string.
* Returns in order: the value if set, default_value if set, NULL
*/
static char *
emulab_getsitevar(char *name)
{
MYSQL_RES *res;
MYSQL_ROW row;
char *value = NULL;
res = mydb_query("SELECT value,defaultvalue FROM sitevariables "
" WHERE name='%s'", 2, name);
if (res == NULL)
return NULL;
if (mysql_num_rows(res) == 0) {
mysql_free_result(res);
return NULL;
}
row = mysql_fetch_row(res);
if (row[0])
value = mystrdup(row[0]);
else if (row[1])
value = mystrdup(row[1]);
mysql_free_result(res);
return value;
}
static void
emulab_deinit(void)
{
......@@ -73,66 +128,129 @@ emulab_deinit(void)
static int
emulab_read(void)
{
/* "Reading" the config file is a no-op. */
char *val;
int ival;
/*
* Grab a couple of image creation related sitevars that don't
* change that often. We don't want to be reading these everytime
* set_put_values() gets called, so we do it explicitly here.
*/
val = emulab_getsitevar("images/create/maxsize");
if (val) {
ival = atoi(val);
/* in GB, allow up to 10TB */
if (ival >= 0 && ival < 10000)
put_maxsize = (uint64_t)ival * 1024 * 1024 * 1024;
free(val);
}
log(" image_put_maxsize = %d GB",
(int)(put_maxsize/(1024*1024*1024)));
val = emulab_getsitevar("images/create/maxwait");
if (val) {
ival = atoi(val);
/* in minutes, allow up to about 10TB @ 10MB/sec */
if (ival >= 0 && ival < 20000)
put_maxwait = (uint32_t)ival * 60;
free(val);
}
log(" image_put_maxwait = %d min",
(int)(put_maxwait/60));
val = emulab_getsitevar("images/frisbee/maxrate_std");
if (val) {
ival = atoi(val);
/* in bytes/sec, allow up to 2Gb/sec */
if (ival >= 0 && ival < 2000000000)
get_maxrate_std = (uint32_t)ival;
free(val);
}
log(" image_get_maxrate_std = %d MB/sec",
(int)(get_maxrate_std/1000000));
val = emulab_getsitevar("images/frisbee/maxrate_usr");
if (val) {
ival = atoi(val);
/* in bytes