Commit 54f55585 authored by Leigh B. Stoller's avatar Leigh B. Stoller
Browse files

Here is a checkpoint of the admission control stuff I have been working on.

The last part is the stuff to hook it in from assign_wrapper, and some
additional support in assign that Rob is adding for me. This comment is
from the top of new file db/libadminctrl.pm.in and describes everything in
detail.

# Admission control policies. These are the ones I could think of, although
# not all of these are implemented.
#
#  * Number of experiments per type/class (only one expt using robots).
#
#  * Number of experiments per project
#  * Number of experiments per subgroup
#  * Number of experiments per user
#
#  * Number of nodes per project      (nodes really means pc testnodes)
#  * Number of nodes per subgroup
#  * Number of nodes per user
#
#  * Number of nodes of a class per project
#  * Number of nodes of a class per group
#  * Number of nodes of a class per user
#
#  * Number of nodes of a type per project
#  * Number of nodes of a type per group
#  * Number of nodes of a type per user
#
#  * Number of nodes with attribute(s) per project
#  * Number of nodes with attribute(s) per group
#  * Number of nodes with attribute(s) per user
#
# So we have group (pid/gid) policies and user policies. These are stored
# into two different tables, group_policies and user_policies, indexed in
# the obvious manner. Each row of the table defines a count (experiments,
# nodes, etc) and a type of thing being counted (experiments, nodes, types,
# classes, etc). When we test for admission, we look for each matching row
# and test each condition. All conditions must pass. No conditions means a
# pass. There is also some "auxdata" which holds extra information needed
# for the policy (say, the type of node being restricted).
#
#      uid:     a uid
#   policy:     'experiments', 'nodes', 'type', 'class', 'attribute'
#    count:     a number
#  auxdata:     a string (optional)
#
# Example: A user policy of ('mike', 'nodes', 10) says that poor mike is
# not allowed to have more 10 nodes at a time, while ('mike', 'type',
# '10', 'pc850') says that mike cannot allocate more than 10 pc850s.
#
# The group_policies table:
#
#      pid:     a pid
#      gid:     a gid
#   policy:     'experiments', 'nodes', 'type', 'class', 'attribute'
#    count:     a number
#  auxdata:     a string (optional)
#
# Example: A project policy of ('testbed', 'testbed', 'experiments', 10)
# says that the testbed project may not have more then 10 experiments
# swapped in at a time, while ('testbed', 'TG1', 'nodes', 10) says that the
# TG1 subgroup of the testbed project may not use more than 10 nodes at
# time.
#
# In addition to group and user policies (which are policies that apply to
# specific users/projects/subgroups), we also need policies that apply to
# all users/projects/subgroups (ie: do not want to specify a particular
# restriction for every user!). To indicate such a policy, we use a special
# tag in the tables (for the user or pid/gid):
#
#      '+'  -  The policy applies to all users (or project/groups).
#
# Example: ('+','experiments',10) says that no user may have more then 10
# experiments swapped in at a time. The rule overrides anything more
# specific (say a particular user is restricted to 20 experiments; the above
# rule overrides that and the user (all users) is restricted to 10.
#
# Sometimes, you want one of these special rules to apply to everyone, but
# *allow* it to be overridden by a more specific rule. For that we use:
#
#      '-'  -  The policy applies to all users (or project/groups),
#              but can be overridden by a more specific rule.
#
# Example: The rules:
#
#	('-','type',0, 'garcia')
#       ('testbed', 'testbed', 'type', 10, 'garcia')
#
# says that no one is allowed to allocate garcias, unless there is specific
# rule that allows it; in this case the testbed project can allocate them.
#
# There are other global policies we would like to enforce. For example,
# "only one experiment can be using the robot testbed." Encoding this kind
# of policy is harder, and leads down a path that can get arbitrarily
# complex. Tha path leads to ruination, and so we want to avoid it at
# all costs.
#
# Instead we define a simple global policies table that applies to all
# experiments currently active on the testbed:
#
#   policy:     'nodes', 'type', 'class', 'attribute'
#     test:     'max', others I cannot think of right now ...
#    count:     a number
#  auxdata:     a string
#
# Example: A global policy of ('nodes', 'max', 10, '') say that the maximum
# number of nodes that may be allocated across the testbed is 10. Thats not
# a very realistic policy of course, but ('type', 'max', 1, 'garcia') says
# that a max of one garcia can be allocated across the testbed, which
# effectively means only one experiment will be able to use them at once.
# This is of course very weak, but I want to step back and give it some
# more thought before I redo this part.
#
# Is that clear? Hope so, cause it gets more complicated. Some admission
# control tests can be done early in the swap phase, before we really do
# anything (before assign_wrapper). Others (type and class) tests cannot
# be done here; only assign can figure out how an experiment is going to map
# to physical nodes (remember virtual types too), and in that case we need
# to tell assign what the "constraints" are and let it figure out what is
# possible.
#
# So, in addition to the simple checks we can do, we also generate an array
# to return to assign_wrapper with the maximum counts of each node type and
# class that is limited by the policies. assign_wrapper will dump those
# values into the ptop file so that assign can enforce those maximum values
# regardless of what hardware is actually available to use. As per discussion
# with Rob, that will look like:
#
#	set-type-limit <type> <limit>
#
# and assign will spit out a new type of violation that assign_wrapper will
# parse.
#
# NOTES:
#
#  1) Admission control is skipped in admin mode; returns okay.
#  2) Admission control is skipped when the pid is emulab-ops; returns okay.
#  3) When calculating current usage, nodes reserved to emulab-ops are
#     ignored.
#  4) The sitevar "swap/use_admission_control" controls the use of admission
#     control; defaults to 1 (on).
#  5) The current policies can be viewed in the web interface. See
#     https://www.emulab.net/showpolicies.php3
#  6) The global policy stuff is weak. I plan to step back and think about it
#     some more before redoing it, but it will tide us over for now.
#
parent 602e1c6d
......@@ -1890,7 +1890,7 @@ outfiles="$outfiles Makeconf GNUmakefile \
db/libdb.pm db/inuse db/avail db/nodeip db/showgraph \
db/dhcpd_makeconf db/nodelog db/webnodelog db/unixgroups \
db/dbcheck db/interswitch db/dbboot db/schemacheck \
db/sitevarscheck db/dbfillcheck \
db/sitevarscheck db/dbfillcheck db/libadminctrl.pm \
db/grabron db/webnfree db/stategraph db/readycount \
db/idletimes db/idlemail db/webidlemail db/xmlconvert \
db/webnewwanode db/libdb.py db/elabinelab_bossinit \
......
......@@ -597,7 +597,7 @@ outfiles="$outfiles Makeconf GNUmakefile \
db/libdb.pm db/inuse db/avail db/nodeip db/showgraph \
db/dhcpd_makeconf db/nodelog db/webnodelog db/unixgroups \
db/dbcheck db/interswitch db/dbboot db/schemacheck \
db/sitevarscheck db/dbfillcheck \
db/sitevarscheck db/dbfillcheck db/libadminctrl.pm \
db/grabron db/webnfree db/stategraph db/readycount \
db/idletimes db/idlemail db/webidlemail db/xmlconvert \
db/webnewwanode db/libdb.py db/elabinelab_bossinit \
......
#
# EMULAB-COPYRIGHT
# Copyright (c) 2000-2004 University of Utah and the Flux Group.
# Copyright (c) 2000-2005 University of Utah and the Flux Group.
# All rights reserved.
#
SRCDIR = @srcdir@
......@@ -18,7 +18,7 @@ SBIN_SCRIPTS = avail inuse showgraph if2port backup webcontrol node_status \
idletimes idlemail setsitevar audit changeuid \
elabinelab_bossinit
LIBEXEC_SCRIPTS = webnodelog webnfree webnewwanode webidlemail xmlconvert
LIB_SCRIPTS = libdb.pm Node.pm libdb.py
LIB_SCRIPTS = libdb.pm Node.pm libdb.py libadminctrl.pm
# Stuff installed on plastic.
USERSBINS = genelists.proxy
......
#!/usr/bin/perl -w
#
# EMULAB-COPYRIGHT
# Copyright (c) 2000-2005 University of Utah and the Flux Group.
# All rights reserved.
#
#
# Admission control policies. These are the ones I could think of, although
# not all of these are implemented.
#
# * Number of experiments per type/class (only one expt using robots).
#
# * Number of experiments per project
# * Number of experiments per subgroup
# * Number of experiments per user
#
# * Number of nodes per project (nodes really means pc testnodes)
# * Number of nodes per subgroup
# * Number of nodes per user
#
# * Number of nodes of a class per project
# * Number of nodes of a class per group
# * Number of nodes of a class per user
#
# * Number of nodes of a type per project
# * Number of nodes of a type per group
# * Number of nodes of a type per user
#
# * Number of nodes with attribute(s) per project
# * Number of nodes with attribute(s) per group
# * Number of nodes with attribute(s) per user
#
# So we have group (pid/gid) policies and user policies. These are stored
# into two different tables, group_policies and user_policies, indexed in
# the obvious manner. Each row of the table defines a count (experiments,
# nodes, etc) and a type of thing being counted (experiments, nodes, types,
# classes, etc). When we test for admission, we look for each matching row
# and test each condition. All conditions must pass. No conditions means a
# pass. There is also some "auxdata" which holds extra information needed
# for the policy (say, the type of node being restricted).
#
# uid: a uid
# policy: 'experiments', 'nodes', 'type', 'class', 'attribute'
# count: a number
# auxdata: a string (optional)
#
# Example: A user policy of ('mike', 'nodes', 10) says that poor mike is
# not allowed to have more 10 nodes at a time, while ('mike', 'type',
# '10', 'pc850') says that mike cannot allocate more than 10 pc850s.
#
# The group_policies table:
#
# pid: a pid
# gid: a gid
# policy: 'experiments', 'nodes', 'type', 'class', 'attribute'
# count: a number
# auxdata: a string (optional)
#
# Example: A project policy of ('testbed', 'testbed', 'experiments', 10)
# says that the testbed project may not have more then 10 experiments
# swapped in at a time, while ('testbed', 'TG1', 'nodes', 10) says that the
# TG1 subgroup of the testbed project may not use more than 10 nodes at time.
#
# In addition to group and user policies (which are policies that apply to
# specific users/projects/subgroups), we also need policies that apply to
# all users/projects/subgroups (ie: do not want to specify a particular
# restriction for every user!). To indicate such a policy, we use a special
# tag in the tables (for the user or pid/gid):
#
# '+' - The policy applies to all users (or project/groups).
#
# Example: ('+','experiments',10) says that no user may have more then 10
# experiments swapped in at a time. The rule overrides anything more
# specific (say a particular user is restricted to 20 experiments; the above
# rule overrides that and the user (all users) is restricted to 10.
#
# Sometimes, you want one of these special rules to apply to everyone, but
# *allow* it to be overridden by a more specific rule. For that we use:
#
# '-' - The policy applies to all users (or project/groups),
# but can be overridden by a more specific rule.
#
# Example: The rules:
#
# ('-','type',0, 'garcia')
# ('testbed', 'testbed', 'type', 10, 'garcia')
#
# says that no one is allowed to allocate garcias, unless there is specific
# rule that allows it; in this case the testbed project can allocate them.
#
# There are other global policies we would like to enforce. For example,
# "only one experiment can be using the robot testbed." Encoding this kind
# of policy is harder, and leads down a path that can get arbitrarily
# complex. Tha path leads to ruination, and so we want to avoid it at
# all costs.
#
# Instead we define a simple global policies table that applies to all
# experiments currently active on the testbed:
#
# policy: 'nodes', 'type', 'class', 'attribute'
# test: 'max', others I cannot think of right now ...
# count: a number
# auxdata: a string
#
# Example: A global policy of ('nodes', 'max', 10, '') say that the maximum
# number of nodes that may be allocated across the testbed is 10. Thats not
# a very realistic policy of course, but ('type', 'max', 1, 'garcia') says
# that a max of one garcia can be allocated across the testbed, which
# effectively means only one experiment will be able to use them at once.
# This is of course very weak, but I want to step back and give it some
# more thought before I redo this part.
#
# Is that clear? Hope so, cause it gets more complicated. Some admission
# control tests can be done early in the swap phase, before we really do
# anything (before assign_wrapper). Others (type and class) tests cannot
# be done here; only assign can figure out how an experiment is going to map
# to physical nodes (remember virtual types too), and in that case we need
# to tell assign what the "constraints" are and let it figure out what is
# possible.
#
# So, in addition to the simple checks we can do, we also generate an array
# to return to assign_wrapper with the maximum counts of each node type and
# class that is limited by the policies. assign_wrapper will dump those
# values into the ptop file so that assign can enforce those maximum values
# regardless of what hardware is actually available to use. As per discussion
# with Rob, that will look like:
#
# set-type-limit <type> <limit>
#
# and assign will spit out a new type of violation that assign_wrapper will
# parse.
#
# NOTES:
#
# 1) Admission control is skipped in admin mode; returns okay.
# 2) Admission control is skipped when the pid is emulab-ops; returns okay.
# 3) When calculating current usage, nodes reserved to emulab-ops are
# ignored.
# 4) The sitevar "swap/use_admission_control" controls the use of admission
# control; defaults to 1 (on).
# 5) The current policies can be viewed in the web interface. See
# https://www.emulab.net/showpolicies.php3
# 6) The global policy stuff is weak. I plan to step back and think about it
# some more before redoing it, but it will tide us over for now.
#
package libadminctrl;
use strict;
use Exporter;
use vars qw(@ISA @EXPORT);
@ISA = "Exporter";
@EXPORT =
qw ( TBAdmissionControlCheck );
# Must come after package declaration!
use lib '@prefix@/lib';
use English;
use libdb;
use libtestbed;
# Configure variables
my $TB = "@prefix@";
# Locals
my $debug = 0;
my $expt_min;
my $expt_max;
my %virt_types = (); # Indexed by virt type, gives number desired
my %virt_classes = (); # Indexed by virt class, gives number desired
my %node_types = (); # Indexed by type, gives class
my %node_classes = (); # Indexed by class, gives class
my $assignflag = 0;
my %assign_classes= (); # For assign (wrapper).
# Constants.
sub TBADMINCTRL_TYPE_USER() { "user"; }
sub TBADMINCTRL_TYPE_PROJECT() { "project"; }
sub TBADMINCTRL_TYPE_GROUP() { "group"; }
sub TBADMINCTRL_POLICY_EXPT() { "experiments"; }
sub TBADMINCTRL_POLICY_NODES() { "nodes"; }
sub TBADMINCTRL_POLICY_TYPE() { "type"; }
sub TBADMINCTRL_POLICY_CLASS() { "class"; }
sub TBADMINCTRL_POLICY_ATTR() { "attribute"; }
#
# The current usage data structure, filled in below.
#
my %curusage =
("experiments" => {"user" => 0,
"project" => 0,
"group" => 0},
"nodes" => {"user" => 0,
"project" => 0,
"group" => 0},
# Arrays of user/project/group counts, indexed by type and class.
"class" => {},
"type" => {},
);
# Output useful info.
sub Declare($)
{
my ($msg) = @_;
print "*** Admission Control: $msg\n";
}
# Debug stuff
sub Debug($)
{
my ($msg) = @_;
if ($debug) {
print "*** - $msg\n";
}
}
# Update assign number.
sub UpdateForAssign($$)
{
my ($typeclass, $count) = @_;
$count = 0
if ($count < 0);
$assign_classes{$typeclass} = 999999
if (!defined($assign_classes{$typeclass}));
# Max for assign is the global minus current number
$assign_classes{$typeclass} = $count
if ($count < $assign_classes{$typeclass});
}
#
# Test a user policy.
#
sub TestUserPolicy($$$$$)
{
my ($uid, $policy, $count, $auxdata, $global) = @_;
my $query_result;
my $current = 0;
my $gstring = ($global ? "global" : "user");
if ($policy eq TBADMINCTRL_POLICY_EXPT()) {
#
# Simple check first.
#
if (!$count) {
Declare("$uid is not allowed to swap any experiments in!");
return 0;
}
$current = $curusage{"experiments"}->{'user'};
Debug("$uid has $current active experiments; ".
"${gstring} limit is $count");
if ($current >= $count) {
Declare("$uid has too many experiments swapped in!");
return 0;
}
}
elsif ($policy eq TBADMINCTRL_POLICY_NODES()) {
#
# Simple check first.
#
if (!$count) {
Declare("$uid is not allowed to allocate any nodes!");
return 0;
}
$current = $curusage{"nodes"}->{'user'};
Debug("$uid has $current nodes allocated; ".
"${gstring} limit is $count");
if ($current + $expt_min > $count) {
Declare("$uid has too many nodes allocated! ".
"Needs $expt_min more.");
return 0;
}
}
elsif ($policy eq TBADMINCTRL_POLICY_CLASS()) {
$current = $curusage{"class"}->{$auxdata}->{'user'}
if (exists($curusage{"class"}->{$auxdata}));
if ($assignflag) {
UpdateForAssign($auxdata, $count - $current);
# return 1;
}
#
# Check first to see if the experiment even wants this class.
#
return 1
if (! $virt_classes{$auxdata});
#
# If it does, then simple check first.
#
if (!$count) {
Declare("$uid is not allowed to allocate any nodes of ".
"class $auxdata!");
return 0;
}
Debug("$uid has $current nodes of class $auxdata allocated; ".
"${gstring} limit is $count");
if ($current + $virt_classes{$auxdata} > $count) {
Declare("$uid has too many nodes of class $auxdata allocated! ".
"Needs ". $virt_classes{$auxdata} ." more.");
return 0;
}
}
elsif ($policy eq TBADMINCTRL_POLICY_TYPE()) {
$current = $curusage{"type"}->{$auxdata}->{'user'}
if (exists($curusage{"type"}->{$auxdata}));
if ($assignflag) {
UpdateForAssign($auxdata, $count - $current);
# return 1;
}
#
# Check first to see if the experiment even wants this class.
#
return 1
if (! $virt_types{$auxdata});
#
# If it does, then simple check first.
#
if (!$count) {
Declare("$uid is not allowed to allocate any nodes of ".
"type $auxdata!");
return 0;
}
Debug("$uid has $current nodes of type $auxdata allocated; ".
"${gstring} limit is $count");
if ($current + $virt_types{$auxdata} > $count) {
Declare("$uid has too many nodes of type $auxdata allocated! ".
"Needs ". $virt_types{$auxdata} ." more.");
return 0;
}
}
else {
warn("*** WARNING: Unknown user policy '$policy'!\n");
}
return 1;
}
#
# Test a group policy.
#
sub TestGroupPolicy($$$$$$)
{
my ($pid, $gid, $policy, $count, $auxdata, $global) = @_;
my $query_result;
my $current = 0;
my $gstring = ($global ? "global" : "group");
if ($policy eq TBADMINCTRL_POLICY_EXPT()) {
#
# Simple check first.
#
if (!$count) {
Declare("$pid/$gid is not allowed to swap any experiments in!");
return 0;
}
$current = ($pid eq $gid ?
$curusage{"experiments"}->{'project'} :
$curusage{"experiments"}->{'group'});
Debug("$pid/$gid has $current active experiments; ".
"${gstring} limit is $count");
if ($current >= $count) {
Declare("$pid/$gid has too many experiments swapped in!");
return 0;
}
}
elsif ($policy eq TBADMINCTRL_POLICY_NODES()) {
#
# Simple check first.
#
if (!$count) {
Declare("$pid/gid is not allowed to allocate any nodes!");
return 0;
}
$current = ($pid eq $gid ?
$curusage{"nodes"}->{'project'} :
$curusage{"nodes"}->{'group'});
Debug("$pid/$gid has $current nodes allocated; ".
"${gstring} limit is $count");
if ($current + $expt_min > $count) {
Declare("$pid/$gid has too many nodes allocated! ".
"Needs $expt_min more.");
return 0;
}
}
elsif ($policy eq TBADMINCTRL_POLICY_CLASS()) {
$current = ($pid eq $gid ?
$curusage{"class"}->{$auxdata}->{'project'} :
$curusage{"class"}->{$auxdata}->{'group'})
if (exists($curusage{"class"}->{$auxdata}));
if ($assignflag) {
UpdateForAssign($auxdata, $count - $current);
# return 1;
}
#
# Check first to see if the experiment even wants this class.
#
return 1
if (! $virt_classes{$auxdata});
#
# If it does, then simple check first.
#
if (!$count) {
Declare("$pid/$gid is not allowed to allocate any nodes of ".
"class $auxdata!");
return 0;
}
Debug("$pid/$gid has $current nodes of class $auxdata allocated; ".
"${gstring} limit is $count");
if ($current + $virt_classes{$auxdata} > $count) {
Declare("$pid/$gid has too many nodes of class $auxdata ".
"allocated! Needs ". $virt_classes{$auxdata} ." more.");
return 0;
}
}
elsif ($policy eq TBADMINCTRL_POLICY_TYPE()) {
$current = ($pid eq $gid ?
$curusage{"type"}->{$auxdata}->{'project'} :
$curusage{"type"}->{$auxdata}->{'group'})
if (exists($curusage{"type"}->{$auxdata}));
if ($assignflag) {
UpdateForAssign($auxdata, $count - $current);
# return 1;
}
#
# Check first to see if the experiment even wants this class.
#
return 1
if (! $virt_types{$auxdata});
#
# If it does, then simple check first.
#
if (!$count) {
Declare("$pid/$gid is not allowed to allocate any nodes of ".
"type $auxdata!");
return 0;
}
Debug("$pid/$gid has $current nodes of type $auxdata allocated; ".
"${gstring} limit is $count");
if ($current + $virt_types{$auxdata} > $count) {
Declare("$pid/$gid has too many nodes of type $auxdata ".
"allocated! Needs ". $virt_types{$auxdata} ." more.");
return 0;
}
}
else {
warn("*** WARNING: Unknown group policy '$policy'!\n");
}
return 1;
}
#
# Test a Global policy.
#
sub TestGlobalPolicy($$$$)
{
my ($policy, $test, $count, $auxdata) = @_;
my $current = 0;
if ($policy eq TBADMINCTRL_POLICY_EXPT()) {
#
# Simple check first.
#
if (!$count) {
Declare("No one is allowed to swap any experiments in!");
return 0;
}
#
# Get number of current experiments.
#
my $query_result =
DBQueryWarn("select count(eid) from experiments ".
"where (state='" . EXPTSTATE_ACTIVE() . "' or ".
" state='" . EXPTSTATE_ACTIVATING() . "') and ".
" pid!='" . TBOPSPID() . "'");
return -1
if (!$query_result);
$current = ($query_result->fetchrow_array())[0];
Debug("There are $current active experiments; ".
"global limit is $count");
if ($current >= $count) {
Declare("There are too many experiments swapped in!");
return 0;
}
}
elsif ($policy eq TBADMINCTRL_POLICY_NODES()) {
#
# Simple check first.
#
if (!$count) {
Declare("No one is not allowed to allocate any more nodes!");
return 0;
}
my $query_result =
DBQueryFatal("select count(r.node_id) from reserved as r ".
"left join nodes as n on n.node_id=r.node_id ".
"left join node_types as nt on nt.type=n.type ".
"where r.pid!='" . TBOPSPID() . "' and ".
" n.role='testnode'");
return -1
if (!$query_result);
$current = ($query_result->fetchrow_array())[0];
Debug("The testbed has $current nodes allocated; ".
"global limit is $count");
if ($current + $expt_min > $count) {
Declare("The testbed has too many nodes allocated! ".
"Needs $expt_min more.");
return 0;
}
}
elsif ($policy eq TBADMINCTRL_POLICY_CLASS()) {
#
# Get current count.
#
my $query_result =
DBQueryFatal("select count(r.node_id) from reserved as r ".
"left join nodes as n on n.node_id=r.node_id ".
"left join node_types as nt on nt.type=n.type ".
"where r.pid!='" . TBOPSPID() . "' and ".
" nt.class='$auxdata' and ".
" n.role='testnode'");
return -1
if (!$query_result);
$current = ($query_result->fetchrow_array())[0];