Commit 4b5e17b0 authored by Mike Hibler's avatar Mike Hibler

Remaining infrastructure for control network "ARP lockdown".

It works like this. Certain nodes that are on the node control net
(right now just subbosses, but ops coming soon) can set static ARP entries
for the nodes they serve. This raises the bar for (but does not eliminate
the possibility of) nodes spoofing servers. Currently this is only for
FreeBSD.

When such a server boots, it will early on run /etc/rc.d/arplock.sh
which will in turn run /usr/local/etc/emulab/fixarpinfo. fixarpinfo
asks boss via an SSL tmcc call for "arpinfo" (using SSL ensures that the
info coming back is really from boss). Tmcd on boss returns such arpinfo
as appropriate for the node (subboss, ops, fs, etc.) along with the type
of lockdown being done. The script uses this info to update the ARP
cache on the machine, adding, removing, or making permanent entries
as appropriate.

fixarpinfo is intended to be called not just at boot, but also whenever
we might need to update the ARP info on a server. The only other use right
now is in subboss_dhcpd_makeconf which is called whenever DHCP info may
need to be changed on a subboss (we hook this because a call to this script
might also indicate a change in the set of nodes served by the subboss).
In the future, fixarpinfo might be called from the newnode path (for ops/fs,
when a node is added to the testbed), the deletenode path, or maybe from
the watchdog (if we started locking down arp entries on experiment nodes)

The type of the lockdown is controlled by a sitevar on boss,
general/arplockdown, which can be set to 'none', 'static' or 'staticonly'.
'none' means do nothing, 'static' means just create static arp entries
for the given nodes but continue to dynamically arp for others, and
'staticonly' means use only this set of static arp entries and disable
dynamic arp on the control net interface. The last implies that the server
will only be able to talk to the set of nodes for which it got ARP info.

As mentioned, tmcd is responsible for returning the correct set of arp
info for a given request. The logic currently is:

 * Only return ARP info to nodes which are on the CONTROL_NETWORK.
   If the requester is elsewhere (e.g., Utah's boss and ops are currently
   segregated on different IP subnets) then this whole infrastructure
   does not apply and nothing is returned.

 * If the requester is a subboss, return info for all other servers that
   are on the node control network as well as for the set of nodes
   which the subboss serves.

 * If the requester is an ops or fs node, again return info for all
   other servers and info for all testnodes or virtnodes whose control
   net IP is on the node control net.

 * Otherwise, return nothing.

One final note is that the ARP info for servers such as boss/ops/fs or
the gateway router is not readily available in most Emulab instances
since those machines are not in the DB nodes or interfaces tables.
Eventually we will fix that, but for now the info must come from new
site variables. To help initially populate those variables, I added
the utils/update_sitevars script which attempts to determine which
servers are on the node control net and gathers the appropriate IP and
MAC info from them.
parent 16e2665d
......@@ -100,6 +100,7 @@ common-script-install: dir-install
$(INSTALL) -m 755 $(SRCDIR)/bootvnodes $(BINDIR)/bootvnodes
$(INSTALL) -m 755 $(SRCDIR)/startcmddone $(BINDIR)/startcmddone
$(INSTALL) -m 755 $(SRCDIR)/getblob $(BINDIR)/getblob
$(INSTALL) -m 755 $(SRCDIR)/fixarpinfo $(BINDIR)/fixarpinfo
(cd config; $(MAKE) DESTDIR=$(DESTDIR) script-install)
subboss-common-script-install: dir-install
......@@ -116,6 +117,7 @@ subboss-common-script-install: dir-install
$(INSTALL) -m 755 $(SRCDIR)/ifdynconfig $(BINDIR)/ifdynconfig
$(INSTALL) -m 755 $(SRCDIR)/startcmddone $(BINDIR)/startcmddone
$(INSTALL) -m 755 $(SRCDIR)/getblob $(BINDIR)/getblob
$(INSTALL) -m 755 $(SRCDIR)/fixarpinfo $(BINDIR)/fixarpinfo
(cd config; $(MAKE) DESTDIR=$(DESTDIR) subboss-script-install)
symlinks: dir-install
......@@ -164,6 +166,7 @@ control-script-install: dir-install bossnode
$(INSTALL) -m 755 $(SRCDIR)/ctrlnode.sh $(SYSRCDIR)/ctrlnode.sh
$(INSTALL) -m 755 $(SRCDIR)/rc.ctrlnode $(RCDIR)/rc.ctrlnode
$(INSTALL) -m 755 $(SRCDIR)/config/librc.pm $(BINDIR)/librc.pm
$(INSTALL) -m 755 $(SRCDIR)/fixarpinfo $(BINDIR)/fixarpinfo
$(INSTALL) bossnode $(ETCDIR)/bossnode
PGENIFILES = rc.ifconfig rc.topomap rc.progagent rc.pgeni \
......
......@@ -3185,6 +3185,11 @@ sub getlocalevserver()
return $evserver;
}
#
# Return a hash of arpinfo provided by boss in $rptr.
# Note that the hash key is the IP address and not the name.
# Function returns the type of the arp configuration or undef on error.
#
sub getarpinfo($)
{
my ($rptr) = @_;
......@@ -3195,11 +3200,17 @@ sub getarpinfo($)
if (tmcc(TMCCCMD_ARPINFO, undef, \@tmccresults, %opthash) < 0) {
warn("*** WARNING: Could not get arpinfo from server!\n");
return -1;
return undef;
}
#
# Example:
# First line should be the type:
# ARPTYPE=(none|static|staticonly)
#
my $atype = "none";
#
# The remaining lines are entries for hosts and servers, e.g.:
# SERVER=gw CNETIP=155.98.36.1 CNETMAC=00d0bcf414f8
# SERVER=subboss CNETIP=155.98.38.162 CNETMAC=001f29329224
# HOST=pc271 CNETIP=155.98.39.71 CNETMAC=001143e43be6
......@@ -3207,21 +3218,54 @@ sub getarpinfo($)
my $pat = q((HOST|SERVER)=([-\w]+) CNETIP=([\d\.]*) CNETMAC=([\da-f]{12}));
foreach my $line (@tmccresults) {
if ($line =~ /ARPTYPE=([-\w]+)/) {
$atype = $1;
if ($atype eq "static" || $atype eq "staticonly") {
next;
}
if ($atype ne "none") {
warn("*** WARNING: arpinfo: invalid type '$atype', assuming 'none'!\n");
$atype = "none";
}
last;
}
if ($line =~ /$pat/) {
my $type = $1;
my $ntype = $1;
my $name = $2;
my $ip = $3;
my $mac = $4;
$arpinfo{$name}{'type'} = $type;
$arpinfo{$name}{'ip'} = $ip;
$arpinfo{$name}{'mac'} = $mac;
# canonicalize the MAC
$mac = lc($mac);
if ($mac =~ /^(..)(..)(..)(..)(..)(..)$/) {
$mac = "$1:$2:$3:$4:$5:$6";
}
if (exists($arpinfo{$ip})) {
# XXX subbosses may appear twice since they are testnodes
# XXX boss may appear as both boss and gw in elabinelab
if ($arpinfo{$ip}{'mac'} ne $mac) {
warn("*** WARNING: Conflicting arpinfo for $ip: '$line'\n");
} else {
$arpinfo{$ip}{'type'} = $ntype
if ($ntype eq "SERVER");
}
} else {
$arpinfo{$ip}{'type'} = $ntype;
$arpinfo{$ip}{'name'} = $name;
$arpinfo{$ip}{'mac'} = $mac;
}
} else {
warn("*** WARNING: Bad arpinfo info line ignored: '$line'\n");
}
}
%$rptr = %arpinfo;
return 0;
if ($atype eq "none") {
%$rptr = ();
} else {
%$rptr = %arpinfo;
}
return $atype;
}
#
......
......@@ -39,6 +39,8 @@ use Exporter;
os_groupdel os_getnfsmounts os_islocaldir os_mountextrafs
os_fwconfig_line os_fwrouteconfig_line os_config_gre os_nfsmount
os_find_freedisk os_get_ctrlnet_ip
os_getarpinfo os_createarpentry os_removearpentry
os_getstaticarp os_setstaticarp
);
sub VERSION() { return 1.0; }
......@@ -86,6 +88,7 @@ $RPMCMD = "/usr/local/bin/rpm";
$HOSTSFILE = "/etc/hosts";
$WGET = "/usr/local/bin/wget";
$CHMOD = "/bin/chmod";
$ARP = "/usr/sbin/arp";
#
# These are not exported
......@@ -977,6 +980,13 @@ sub os_get_ctrlnet_ip
my $iface;
my $address;
# just use recorded IP if available
if (-e "$BOOTDIR/myip") {
$myip = `cat $BOOTDIR/myip`;
chomp($myip);
return $myip;
}
if (!open CONTROLIF, "$BOOTDIR/controlif") {
return undef;
}
......@@ -1269,4 +1279,116 @@ sub os_find_freedisk($$)
return %diskinfo;
}
#
# Read the current arp info and create a hash for it.
#
sub os_getarpinfo($$)
{
my ($diface,$airef) = @_;
my %arpinfo = ();
if (!open(ARP, "$ARP -a|")) {
print "os_getarpinfo: Cannot run arp command\n";
return 1;
}
while (<ARP>) {
if (/^(\S+) \(([\d\.]+)\) at (..:..:..:..:..:..) on (\S+) (.*)/) {
my $name = $1;
my $ip = $2;
my $mac = $3;
my $iface = $4;
my $stuff = $5;
# this is not the interface you are looking for...
if ($diface ne $iface) {
next;
}
if (exists($arpinfo{$ip})) {
if ($arpinfo{$ip}{'mac'} ne $mac) {
print "os_getarpinfo: Conflicting arpinfo for $ip:\n";
print " '$_'!?\n";
return 1;
}
}
$arpinfo{$ip}{'name'} = $name;
$arpinfo{$ip}{'mac'} = $mac;
$arpinfo{$ip}{'iface'} = $iface;
if ($stuff =~ /permanent/) {
$arpinfo{$ip}{'static'} = 1;
} else {
$arpinfo{$ip}{'static'} = 0;
}
}
}
close(ARP);
%$airef = %arpinfo;
return 0;
}
#
# Create a static ARP entry given info in the supplied hash.
# Returns zero on success, non-zero otherwise.
#
sub os_createarpentry($$$)
{
my ($iface, $ip, $mac) = @_;
return system("$ARP -s $ip $mac >/dev/null 2>&1");
}
sub os_removearpentry($;$)
{
my ($iface, $ip) = @_;
if (!defined($ip)) {
return system("$ARP -i $iface -da >/dev/null 2>&1");
}
return system("$ARP -d $ip >/dev/null 2>&1");
}
#
# Returns whether static ARP is enabled or not in the passed param.
# Return zero on success, an error otherwise.
#
sub os_getstaticarp($$)
{
my ($iface,$isenabled) = @_;
my $info = `$IFCONFIGBIN $iface 2>/dev/null`;
return $?
if ($?);
if ($info =~ /STATICARP/) {
$$isenabled = 1;
} else {
$$isenabled = 0;
}
return 0;
}
#
# Turn on/off static ARP on the indicated interface.
# Return zero on success, an error otherwise.
#
sub os_setstaticarp($$)
{
my ($iface,$enable) = @_;
my $curenabled = 0;
os_getstaticarp($iface, \$curenabled);
if ($enable && !$curenabled) {
return system("$IFCONFIGBIN $iface staticarp >/dev/null 2>&1");
}
if (!$enable && $curenabled) {
return system("$IFCONFIGBIN $iface -staticarp >/dev/null 2>&1");
}
return 0;
}
1;
......@@ -40,6 +40,8 @@ use Exporter;
os_fwconfig_line os_fwrouteconfig_line os_config_gre
os_get_disks os_get_disk_size os_get_partition_info os_nfsmount
os_get_ctrlnet_ip
os_getarpinfo os_createarpentry os_removearpentry
os_getstaticarp os_setstaticarp
);
sub VERSION() { return 1.0; }
......@@ -95,6 +97,7 @@ $RPMCMD = "/bin/rpm";
$HOSTSFILE = "/etc/hosts";
$WGET = "/usr/bin/wget";
$CHMOD = "/bin/chmod";
$ARP = "/sbin/arp";
#
# These are not exported
......@@ -2149,7 +2152,7 @@ sub os_config_gre($$$$$$$)
return 0;
}
sub os_get_disks
sub os_get_disks()
{
my @blockdevs;
......@@ -2158,11 +2161,18 @@ sub os_get_disks
return @blockdevs;
}
sub os_get_ctrlnet_ip
sub os_get_ctrlnet_ip()
{
my $iface;
my $address;
# just use recorded IP if available
if (-e "$BOOTDIR/myip") {
$myip = `cat $BOOTDIR/myip`;
chomp($myip);
return $myip;
}
if (!open CONTROLIF, "$BOOTDIR/controlif") {
return undef;
}
......@@ -2287,4 +2297,126 @@ sub os_mountextrafs($)
return $mntpt;
}
#
# Read the current arp info and create a hash for it.
#
sub os_getarpinfo($$)
{
my ($diface,$airef) = @_;
my %arpinfo = ();
if (!open(ARP, "$ARP -a|")) {
print "os_getarpinfo: Cannot run arp command\n";
return 1;
}
while (<ARP>) {
if (/^(\S+) \(([\d\.]+)\) at (..:..:..:..:..:..) (.*) on (\S+) /) {
my $name = $1;
my $ip = $2;
my $mac = lc($3);
my $stuff = $4;
my $iface = $5;
# this is not the interface you are looking for...
if ($diface ne $iface) {
next;
}
if (exists($arpinfo{$ip})) {
if ($arpinfo{$ip}{'mac'} ne $mac) {
print "os_getarpinfo: Conflicting arpinfo info for $ip:\n";
print " '$_'!?\n";
return 1;
}
}
$arpinfo{$ip}{'name'} = $name;
$arpinfo{$ip}{'mac'} = $mac;
$arpinfo{$ip}{'iface'} = $iface;
if ($stuff =~ /PERM/) {
$arpinfo{$ip}{'static'} = 1;
} else {
$arpinfo{$ip}{'static'} = 0;
}
}
}
close(ARP);
%$airef = %arpinfo;
return 0;
}
#
# Create a static ARP entry given info in the supplied hash.
# Returns zero on success, non-zero otherwise.
#
sub os_createarpentry($$$)
{
my ($iface, $ip, $mac) = @_;
return system("$ARP -i $iface -s $ip $mac >/dev/null 2>&1");
}
sub os_removearpentry($;$)
{
my ($iface, $ip) = @_;
#
# XXX ugh, Linux arp doesn't support clearing all entries.
# Do it the hard way!
#
if (!defined($ip)) {
my %info = os_getarpinfo($iface);
my $err = 0;
foreach my $_ip (keys %arpinfo) {
if (system("$ARP -i $iface -d $_ip >/dev/null 2>&1")) {
$err++;
}
}
return $err;
}
return system("$ARP -i $iface -d $ip >/dev/null 2>&1");
}
#
# Returns whether static ARP is enabled or not in the passed param.
# Return zero on success, an error otherwise.
#
sub os_getstaticarp($$)
{
my ($iface,$isenabled) = @_;
my $info = `$IFCONFIGBIN $iface 2>/dev/null`;
return $?
if ($?);
if ($info =~ /NOARP/) {
$$isenabled = 1;
} else {
$$isenabled = 0;
}
return 0;
}
#
# Turn on/off static ARP on the indicated interface.
# Return zero on success, an error otherwise.
#
sub os_setstaticarp($$)
{
my ($iface,$enable) = @_;
my $curenabled = 0;
os_getstaticarp($iface, \$curenabled);
if ($enable && !$curenabled) {
return system("$IFCONFIGBIN $iface -arp >/dev/null 2>&1");
}
if (!$enable && $curenabled) {
return system("$IFCONFIGBIN $iface arp >/dev/null 2>&1");
}
return 0;
}
1;
#
# Copyright (c) 2000-2011 University of Utah and the Flux Group.
# Copyright (c) 2000-2012 University of Utah and the Flux Group.
#
# {{{EMULAB-LICENSE
#
......@@ -89,7 +89,7 @@ endif
subboss-install: $(addprefix $(INSTALL_SBINDIR)/, $(SUBBOSS_SBIN_SCRIPTS))
ln -sf $(INSTALL_SBINDIR)/subboss_dhcpd_makeconf \
$(CLIENT_BINDIR)/subboss_dhcpd_makeconf
$(DESTDIR)$(CLIENT_BINDIR)/subboss_dhcpd_makeconf
boss-install: $(addprefix $(INSTALL_BINDIR)/, $(BIN_SCRIPTS)) \
$(addprefix $(INSTALL_SBINDIR)/, $(SBIN_SCRIPTS)) \
......
#!/usr/bin/perl -wT
#
# Copyright (c) 2000-2011 University of Utah and the Flux Group.
# Copyright (c) 2000-2012 University of Utah and the Flux Group.
#
# {{{EMULAB-LICENSE
#
......@@ -46,6 +46,8 @@ my $restart = 0;
# Configure variables
#
my $TBOPS = "@TBOPSEMAIL@";
my $CBINDIR = "@CLIENT_BINDIR@";
my $LOGDIR = "@prefix@/log";
# un-taint path
$ENV{'PATH'} = '/bin:/usr/bin:/usr/sbin:/usr/local/bin';
......@@ -126,6 +128,22 @@ if ($restart) {
fatal("Could not kill(TERM) process $dpid (dhcpd): $!");
}
}
#
# If we are also locking down ARP entries for the nodes we service,
# reconfig that info here in case there has been a change in which
# nodes we manage.
#
# XXX yes, this has nothing to do with DHCP other than it is a handy
# hook point since we know any change to the mapping of nodes to subbosses
# will result in a call to this routine.
#
if (-x "$CBINDIR/fixarpinfo") {
if (system("$CBINDIR/fixarpinfo -u >>$LOGDIR/fixarpinfo.log 2>&1")) {
fatal("Could not reconfigure static ARP setup");
}
}
TBScriptUnlock();
exit(0);
......
......@@ -36,8 +36,8 @@ include $(OBJDIR)/Makeconf
RC_SCRIPTS = 2.mysql-server.sh 3.mfrisbeed.sh 3.testbed.sh \
2.dhcpd.sh 1.mysql-client.sh
SUBBOSS_SCRIPTS = 2.dhcpd.sh 3.mfrisbeed-subboss.sh
OPS_SCRIPTS = 3.and.sh 1.mysql-client.sh 1.mysql-server.sh
SUBBOSS_SCRIPTS = 2.dhcpd.sh 3.mfrisbeed-subboss.sh arplock.sh
OPS_SCRIPTS = 3.and.sh 1.mysql-client.sh 1.mysql-server.sh arplock.sh
ifeq ($(ELVINCOMPAT),1)
OPS_SCRIPTS += 2.elvind.sh 3.elvin_gateway.sh
endif
......
#!/bin/sh
# PROVIDE: arplock
# REQUIRE: netif
# BEFORE: pf ipfw routing
# KEYWORD: shutdown
#
# Copyright (c) 2012 University of Utah and the Flux Group.
#
# {{{EMULAB-LICENSE
#
# This file is part of the Emulab network testbed software.
#
# This file is free software: you can redistribute it and/or modify it
# under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or (at
# your option) any later version.
#
# This file is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public
# License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this file. If not, see <http://www.gnu.org/licenses/>.
#
# }}}
#
. /etc/emulab/paths.sh
if [ ! -x $BINDIR/fixarpinfo ]; then
echo "*** fixarpinfo script missing, ARP lockdown not done"
exit 0
fi
#
# XXX create some missing state if we are the ops node
#
if [ ! -r $BOOTDIR/controlif -a -x $BINDIR/findif ]; then
iface=`$BINDIR/findif -i @USERNODE_IP@`
if [ -n "$iface" ]; then
echo $iface > $BOOTDIR/controlif
echo @USERNODE_IP@ > $BOOTDIR/myip
fi
fi
#
# ARP lockdown script. Has to run early, after network setup but before
# we start firing up daemons.
#
case "$1" in
start|faststart)
echo "Setting up static ARP entries."
$BINDIR/fixarpinfo -s >$LOGDIR/fixarpinfo.log 2>&1
;;
restart)
echo "Updating static ARP entries."
$BINDIR/fixarpinfo -u >$LOGDIR/fixarpinfo.log 2>&1
;;
stop)
echo "Removing static ARP entries."
$BINDIR/fixarpinfo -c >$LOGDIR/fixarpinfo.log 2>&1
;;
*)
echo "Usage: `basename $0` {start|stop|restart}" >&2
;;
esac
exit 0
......@@ -48,7 +48,8 @@ SBIN_SCRIPTS = vlandiff vlansync withadminprivs export_tables cvsupd.pl \
management_iface sharevlan check-shared-bw \
addspecialdevice addspecialiface imagehash clone_image \
addvpubaddr imageinfo ctrladdr image_import \
prereserve_check tcppd addexternalnetwork
prereserve_check tcppd addexternalnetwork \
update_sitevars
WEB_SBIN_SCRIPTS= webnewnode webdeletenode webspewconlog webarchive_list \
webwanodecheckin webspewimage webdumpdescriptor
......
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment