Commit 938d5fe0 authored by David Johnson's avatar David Johnson

Improvements in ovs/linuxbridge iface config and hosts file setup.

Also, I had to change the linuxbridge configuration around.  I'd been
using statically-configured linux bridges, and I'd assumed I could tell
Neutron to use them.  Unfortunately, this is sort of true, but not
enough; from the comments:

 NB: We can only control the name of the external br-ex bridge,
 because only the Neutron linuxbridge driver accepts both a map of
 physical networks to physical interfaces; and physical network to
 bridge names.  Nova assumes that the bridge it must plug a device
 into is named according to the physical network uuid.  Thus, for
 the linuxbridge case, we only setup bridge_mappings for
 br-ex... modulo a flag.  Hopefully in the future they will see the
 sense in allowing static bridge configurations.

So for now, only br-ex is static; the others are dynamic.  What a pain
for debugging!  Stupid.

More importantly, this commit also takes integrates correctly with the
new Emulab clientside improvements that let the user customize interface
config and /etc/hosts file generation.  To handle hosts, we create a
static manifest file that tells Emulab to call an rc.hostnames
pre-hook.  This hook basically grabs our latest special openstack hosts
entries, and ensures they make it into /etc/hosts.head, so that
genhostsfile prepends our special names.  To handle interfaces, we
further customize /etc/network/interfaces so that Emulab's rc.ifconfig
only tries to configure interfaces we haven't handled.

Thus, if the clientside of the disk image the scripts are operating on
includes the new clientside hookable support, we no longer move
rc.ifconfig and rc.hostnames out of our way --- we let them run, secure
in the knowledge that our customizations won't get trampled.

(All this improvement was necessary so that blockstores and event system
stuff would work.)
parent 19a6adfe
......@@ -49,23 +49,28 @@ if [ "$HOSTNAME" = "$NETWORKMANAGER" ]; then
# Get our hosts files setup to point to the new management network.
# (These were created one-time in setup-lib.sh)
#
cat $OURDIR/mgmt-hosts > /etc/hosts
echo "127.0.0.1 localhost" >> /etc/hosts
cat $OURDIR/mgmt-hosts > /etc/hosts.tmp
cp -p /etc/hosts $OURDIR/hosts.orig
cat $OURDIR/hosts.orig >> /etc/hosts.tmp
mv /etc/hosts.tmp /etc/hosts
for node in $NODES
do
[ "$node" = "$NETWORKMANAGER" ] && continue
if unified ; then
continue
fi
#if unified ; then
# continue
#fi
fqdn=`getfqdn $node`
$SSH $fqdn mkdir -p $OURDIR
#scp -p -o StrictHostKeyChecking=no \
#$SETTINGS $OURDIR/mgmt-hosts $OURDIR/mgmt-netmask \
#$OURDIR/data-hosts $OURDIR/data-netmask \
#$fqdn:$OURDIR
scp -p -o StrictHostKeyChecking=no \
$SETTINGS $OURDIR/mgmt-hosts $OURDIR/mgmt-netmask \
$OURDIR/data-hosts $OURDIR/data-netmask \
$fqdn:$OURDIR
$SSH $fqdn cp $OURDIR/mgmt-hosts /etc/hosts
$SSH $fqdn 'echo 127.0.0.1 localhost | tee -a /etc/hosts'
$OURDIR/mgmt-hosts $fqdn:$OURDIR
# For now, just insert the new hosts in front of the existing ones.
# setup-{ovs,linuxbridge}-node.sh may do differently.
$SSH $fqdn "cp -p /etc/hosts $OURDIR/hosts.orig ; cat $OURDIR/mgmt-hosts > /etc/hosts.tmp ; cat $OURDIR/hosts.orig >> /etc/hosts.tmp ; mv /etc/hosts.tmp /etc/hosts"
done
echo "*** Setting up the Management Network"
......
......@@ -109,6 +109,7 @@ COMPUTE_EXTRA_NOVA_DISK_SPACE="1"
ML2PLUGIN="openvswitch"
MANILADRIVER="generic"
EXTRAIMAGEURLS=""
LINUXBRIDGE_STATIC=0
#
# We have an 'adminapi' user that gets a random password. Then, we have
......@@ -1125,17 +1126,29 @@ if [ ! -f $OURDIR/neutron.vars ]; then
#
# Figure out the bridge mappings
#
if [ "${ML2PLUGIN}" = "openvswitch" ]; then
# NB: We can only control the name of the external br-ex bridge,
# because only the Neutron linuxbridge driver accepts both a map of
# physical networks to physical interfaces; and physical network to
# bridge names. Nova assumes that the bridge it must plug a device
# into is named according to the physical network uuid. Thus, for
# the linuxbridge case, we only setup bridge_mappings for
# br-ex... modulo a flag. Hopefully in the future they will see the
# sense in allowing static bridge configurations.
#
bridge_mappings="bridge_mappings=external:br-ex"
else
bridge_mappings="physical_interface_mappings=external:${EXTERNAL_NETWORK_INTERFACE}"
extra_mappings=""
if [ "${ML2PLUGIN}" = "linuxbridge" ]; then
extra_mappings="physical_interface_mappings=external:${EXTERNAL_NETWORK_INTERFACE}"
fi
for lan in $DATAFLATLANS ; do
. $OURDIR/info.${lan}
if [ "${ML2PLUGIN}" = "openvswitch" ]; then
if [ "${ML2PLUGIN}" = "linuxbridge" ]; then
extra_mappings="${extra_mappings},${lan}:${DATADEV}"
if [ $LINUXBRIDGE_STATIC -eq 1 ]; then
bridge_mappings="${bridge_mappings},${lan}:${DATABRIDGE}"
fi
else
bridge_mappings="${bridge_mappings},${lan}:${DATADEV}"
bridge_mappings="${bridge_mappings},${lan}:${DATABRIDGE}"
fi
done
for lan in $DATAVLANS ; do
......@@ -1145,15 +1158,19 @@ if [ ! -f $OURDIR/neutron.vars ]; then
if [ $? = 0 ] ; then
continue;
else
if [ "${ML2PLUGIN}" = "openvswitch" ]; then
if [ "${ML2PLUGIN}" = "linuxbridge" ]; then
extra_mappings="${extra_mappings},${lan}:${DATADEV}"
if [ $LINUXBRIDGE_STATIC -eq 1 ]; then
bridge_mappings="${bridge_mappings},${DATAVLANDEV}:${DATABRIDGE}"
fi
else
bridge_mappings="${bridge_mappings},${lan}:${DATADEV}"
bridge_mappings="${bridge_mappings},${DATAVLANDEV}:${DATABRIDGE}"
fi
fi
done
echo "bridge_mappings=\"${bridge_mappings}\"" >> $OURDIR/neutron.vars
echo "extra_mappings=\"${extra_mappings}\"" >> $OURDIR/neutron.vars
#
# Figure out the network_vlan_ranges
......
#!/bin/sh
#
# This sets up openvswitch networks (on neutron, the external and data
# This sets up linuxbridge networks (on neutron, the external and data
# networks). The networkmanager and compute nodes' physical interfaces
# have to get moved into br-ex and br-int, respectively -- on the
# moonshots, that's eth0 and eth1. The controller is special; it doesn't
......@@ -32,9 +32,6 @@ logtstart "linuxbridge-node"
# new DATAIP iff USE_EXISTING_IPS was set) to br-int
#
EXTERNAL_NETWORK_BRIDGE="br-ex"
#DATA_NETWORK_INTERFACE=`ip addr show | grep "inet $MYIP" | sed -e "s/.*scope global \(.*\)\$/\1/"`
DATA_NETWORK_BRIDGE="br-data"
INTEGRATION_NETWORK_BRIDGE="br-int"
#
# If this is the controller, we don't have to do much network setup; just
......@@ -68,6 +65,21 @@ else
. $OURDIR/ctlnet.vars
fi
modprobe bridge
#
# Setup the external network
#
brctl addbr ${EXTERNAL_NETWORK_BRIDGE}
brctl addif ${EXTERNAL_NETWORK_BRIDGE} ${EXTERNAL_NETWORK_INTERFACE}
#
# Now move the $EXTERNAL_NETWORK_INTERFACE and default route config to ${EXTERNAL_NETWORK_BRIDGE}
#
ifconfig ${EXTERNAL_NETWORK_INTERFACE} 0 up
ifconfig ${EXTERNAL_NETWORK_BRIDGE} $ctlip netmask $ctlnetmask up
route add default gw $ctlgw
#
# Make the configuration for the $EXTERNAL_NETWORK_INTERFACE be static.
#
......@@ -89,13 +101,28 @@ iface lo inet loopback
auto ${EXTERNAL_NETWORK_INTERFACE}
iface ${EXTERNAL_NETWORK_INTERFACE} inet static
address 0.0.0.0
auto ${EXTERNAL_NETWORK_BRIDGE}
iface ${EXTERNAL_NETWORK_BRIDGE} inet static
bridge_ports ${EXTERNAL_NETWORK_INTERFACE}
address $ctlip
netmask $ctlnetmask
gateway $ctlgw
dns-search $DNSDOMAIN
dns-nameservers $DNSSERVER
up echo "${EXTERNAL_NETWORK_BRIDGE}" > /var/run/cnet
up echo "${EXTERNAL_NETWORK_BRIDGE}" > /var/emulab/boot/controlif
EOF
# Also restart slothd so it listens on the new control iface.
echo "${EXTERNAL_NETWORK_BRIDGE}" > /var/run/cnet
echo "${EXTERNAL_NETWORK_BRIDGE}" > /var/emulab/boot/controlif
/usr/local/etc/emulab/rc/rc.slothd stop
pkill slothd
sleep 1
/usr/local/etc/emulab/rc/rc.slothd start
#
# Add the management network config if necessary (if not, it's already a VPN)
#
......@@ -106,6 +133,8 @@ auto ${MGMT_NETWORK_INTERFACE}
iface ${MGMT_NETWORK_INTERFACE} inet static
address $MGMTIP
netmask $MGMTNETMASK
up mkdir -p /var/run/emulab
up echo "${MGMT_NETWORK_INTERFACE} $MGMTIP $MGMTMAC" > /var/run/emulab/interface-done-$MGMTMAC
EOF
if [ -n "$MGMTVLANDEV" ]; then
cat <<EOF >> /etc/network/interfaces
......@@ -121,13 +150,40 @@ for lan in $DATAFLATLANS ; do
# suck in the vars we'll use to configure this one
. $OURDIR/info.$lan
if [ $LINUXBRIDGE_STATIC -eq 1 ]; then
brctl addbr ${DATABRIDGE}
brctl addif ${DATABRIDGE} ${DATADEV}
ifconfig ${DATADEV} 0 up
ifconfig ${DATABRIDGE} $DATAIP netmask $DATANETMASK up
# XXX!
#route add -net 10.0.0.0/8 dev ${DATA_NETWORK_BRIDGE}
cat <<EOF >> /etc/network/interfaces
auto ${DATADEV}
iface ${DATADEV} inet static
address 0.0.0.0
auto ${DATABRIDGE}
iface ${DATABRIDGE} inet static
bridge_ports ${DATADEV}
address $DATAIP
netmask $DATANETMASK
up mkdir -p /var/run/emulab
up echo "${DATABRIDGE} $DATAIP $DATAMAC" > /var/run/emulab/interface-done-$DATAMAC
EOF
else
cat <<EOF >> /etc/network/interfaces
auto ${DATADEV}
iface ${DATADEV} inet static
address $DATAIP
netmask $DATANETMASK
up mkdir -p /var/run/emulab
up echo "${DATADEV} $DATAIP $DATAMAC" > /var/run/emulab/interface-done-$DATAMAC
EOF
fi
if [ -n "$DATAVLANDEV" ]; then
cat <<EOF >> /etc/network/interfaces
vlan-raw-device ${DATAVLANDEV}
......@@ -148,6 +204,28 @@ for lan in $DATAVLANS ; do
ifconfig $DATADEV down
vconfig rem $DATADEV
if [ $LINUXBRIDGE_STATIC -eq 1 ]; then
# If the bridge exists, we've already done it (we might have
# multiplexed (trunked) more than one vlan across this physical
# device).
brctl addbr ${DATABRIDGE}
brctl addif ${DATABRIDGE} ${DATAVLANDEV}
grep "^auto ${DATAVLANDEV}$" /etc/network/interfaces
if [ ! $? -eq 0 ]; then
cat <<EOF >> /etc/network/interfaces
auto ${DATAVLANDEV}
iface ${DATAVLANDEV} inet static
#address 0.0.0.0
up mkdir -p /var/run/emulab
# Just touch it, don't put iface/inet/mac into it; the vlans atop this
# device are being used natively by openstack. So just let Emulab setup
# to not setup any of these vlans.
up touch /var/run/emulab/interface-done-$DATAPMAC
EOF
fi
fi
done
#else
......@@ -167,14 +245,37 @@ done
# fi
#fi
# Flush the routing cache
ip route flush cache
#
# Set the hostname for later after reboot!
#
echo `hostname` > /etc/hostname
echo "*** Removing Emulab rc.hostnames and rc.ifconfig boot scripts"
mv /usr/local/etc/emulab/rc/rc.hostnames /usr/local/etc/emulab/rc/rc.hostnames.NO
mv /usr/local/etc/emulab/rc/rc.ifconfig /usr/local/etc/emulab/rc/rc.ifconfig.NO
grep -q DYNRUNDIR /etc/emulab/paths.sh
if [ $? -eq 0 ]; then
echo "*** Hooking Emulab rc.hostnames boot script..."
mkdir -p $OURDIR/bin
touch $OURDIR/bin/rc.hostnames-openstack
chmod 755 $OURDIR/bin/rc.hostnames-openstack
cat <<EOF >$OURDIR/bin/rc.hostnames-openstack
#!/bin/sh
cp -p $OURDIR/mgmt-hosts /var/run/emulab/hosts.head
exit 0
EOF
mkdir -p /etc/emulab/run/rcmanifest.d
touch /etc/emulab/run/rcmanifest.d/0.openstack-rcmanifest.sh
cat <<EOF >> /etc/emulab/run/rcmanifest.d/0.openstack-rcmanifest.sh
HOOK SERVICE=rc.hostnames ENV=boot WHENCE=every OP=boot POINT=pre FATAL=0 FILE=$OURDIR/bin/rc.hostnames-openstack ARGV=""
EOF
else
echo "*** Nullifying Emulab rc.hostnames and rc.ifconfig services!"
mv /usr/local/etc/emulab/rc/rc.hostnames /usr/local/etc/emulab/rc/rc.hostnames.NO
mv /usr/local/etc/emulab/rc/rc.ifconfig /usr/local/etc/emulab/rc/rc.ifconfig.NO
fi
if [ ! ${HAVE_SYSTEMD} -eq 0 ] ; then
# Maybe this is helpful too
......
......@@ -171,6 +171,7 @@ crudini --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup \
cat <<EOF >> /etc/neutron/plugins/ml2/linuxbridge_agent.ini
[linux_bridge]
${bridge_mappings}
${extra_mappings}
[vxlan]
enable_vxlan = True
......
......@@ -44,7 +44,9 @@ fi
#
echo "*** Copying updated network metadata files to $CONTROLLER and $NEWNODE ..."
cat $OURDIR/mgmt-hosts > /etc/hosts
cat $OURDIR/mgmt-hosts > /etc/hosts.tmp
cat $OURDIR/hosts.orig >> /etc/hosts.tmp
mv /etc/hosts.tmp /etc/hosts
fqdn=`getfqdn $CONTROLLER`
$SSH $fqdn mkdir -p $OURDIR
......@@ -55,7 +57,7 @@ $SCP $SETTINGS \
$OURDIR/nextsparesubnet $OURDIR/router-ipaddr.* \
$OURDIR/manifests*.xml $OURDIR/topomap* $OURDIR/fqdn.map \
$fqdn:$OURDIR
$SSH $fqdn cp $OURDIR/mgmt-hosts /etc/hosts
$SSH $fqdn "cat $OURDIR/mgmt-hosts > /etc/hosts.tmp ; cat $OURDIR/hosts.orig >> /etc/hosts.tmp ; mv /etc/hosts.tmp /etc/hosts"
#
# XXX: also copy the manifests and parameters. This is because if the
......@@ -71,7 +73,7 @@ $SCP $SETTINGS $OURDIR/parameters \
$OURDIR/nextsparesubnet $OURDIR/router-ipaddr.* \
$OURDIR/manifests*.xml $OURDIR/topomap* $OURDIR/fqdn.map \
$fqdn:$OURDIR
$SSH $fqdn cp $OURDIR/mgmt-hosts /etc/hosts
$SSH $fqdn "cp -p /etc/hosts $OURDIR/hosts.orig ; cat $OURDIR/mgmt-hosts > /etc/hosts.tmp ; cat $OURDIR/hosts.orig >> /etc/hosts.tmp ; mv /etc/hosts.tmp /etc/hosts"
#
# Update the management network if necessary
......@@ -107,7 +109,7 @@ do
fqdn=`getfqdn $node`
scp -p -o StrictHostKeyChecking=no $OURDIR/mgmt-hosts $fqdn:$OURDIR
$SSH $fqdn cp -p $OURDIR/mgmt-hosts /etc/hosts
$SSH $fqdn "cat $OURDIR/mgmt-hosts > /etc/hosts.tmp ; cat $OURDIR/hosts.orig >> /etc/hosts.tmp ; mv /etc/hosts.tmp /etc/hosts"
done
##
......
......@@ -49,7 +49,9 @@ fi
#
echo "*** Copying updated network metadata files to $CONTROLLER ..."
cat $OURDIR/mgmt-hosts > /etc/hosts
cat $OURDIR/mgmt-hosts > /etc/hosts.tmp
cat $OURDIR/hosts.orig >> /etc/hosts.tmp
mv /etc/hosts.tmp /etc/hosts
fqdn=`getfqdn $CONTROLLER`
$SSH $fqdn mkdir -p $OURDIR
......@@ -60,7 +62,7 @@ $SCP $SETTINGS \
$OURDIR/nextsparesubnet $OURDIR/router-ipaddr.* \
$OURDIR/manifests*.xml $OURDIR/topomap* $OURDIR/fqdn.map \
$fqdn:$OURDIR
$SSH $fqdn cp $OURDIR/mgmt-hosts /etc/hosts
$SSH $fqdn "cat $OURDIR/mgmt-hosts > /etc/hosts.tmp ; cat $OURDIR/hosts.orig >> /etc/hosts.tmp ; mv /etc/hosts.tmp /etc/hosts"
#
# Now copy the updated $OURDIR/mgmt-hosts to all the other nodes and
......@@ -73,7 +75,7 @@ do
fqdn=`getfqdn $node`
scp -p -o StrictHostKeyChecking=no $OURDIR/mgmt-hosts $fqdn:$OURDIR
$SSH $fqdn cp -p $OURDIR/mgmt-hosts /etc/hosts
$SSH $fqdn "cat $OURDIR/mgmt-hosts > /etc/hosts.tmp ; cat $OURDIR/hosts.orig >> /etc/hosts.tmp ; mv /etc/hosts.tmp /etc/hosts"
done
exit 0
......@@ -112,6 +112,8 @@ iface ${EXTERNAL_NETWORK_BRIDGE} inet static
gateway $ctlgw
dns-search $DNSDOMAIN
dns-nameservers $DNSSERVER
up echo "${EXTERNAL_NETWORK_BRIDGE}" > /var/run/cnet
up echo "${EXTERNAL_NETWORK_BRIDGE}" > /var/emulab/boot/controlif
auto ${EXTERNAL_NETWORK_INTERFACE}
iface ${EXTERNAL_NETWORK_INTERFACE} inet static
......@@ -124,6 +126,14 @@ route add default gw $ctlgw
service_restart openvswitch-switch
# Also restart slothd so it listens on the new control iface.
echo "${EXTERNAL_NETWORK_BRIDGE}" > /var/run/cnet
echo "${EXTERNAL_NETWORK_BRIDGE}" > /var/emulab/boot/controlif
/usr/local/etc/emulab/rc/rc.slothd stop
pkill slothd
sleep 1
/usr/local/etc/emulab/rc/rc.slothd start
#
# Add the management network config if necessary (if not, it's already a VPN)
#
......@@ -134,6 +144,8 @@ auto ${MGMT_NETWORK_INTERFACE}
iface ${MGMT_NETWORK_INTERFACE} inet static
address $MGMTIP
netmask $MGMTNETMASK
up mkdir -p /var/run/emulab
up echo "${MGMT_NETWORK_INTERFACE} $MGMTIP $MGMTMAC" > /var/run/emulab/interface-done-$MGMTMAC
EOF
if [ -n "$MGMTVLANDEV" ]; then
cat <<EOF >> /etc/network/interfaces
......@@ -164,6 +176,8 @@ auto ${DATABRIDGE}
iface ${DATABRIDGE} inet static
address $DATAIP
netmask $DATANETMASK
up mkdir -p /var/run/emulab
up echo "${DATABRIDGE} $DATAIP $DATAMAC" > /var/run/emulab/interface-done-$DATAMAC
auto ${DATADEV}
iface ${DATADEV} inet static
......@@ -201,6 +215,20 @@ for lan in $DATAVLANS ; do
ovs-vsctl add-br ${DATABRIDGE}
ovs-vsctl add-port ${DATABRIDGE} ${DATAVLANDEV}
fi
grep "^auto ${DATAVLANDEV}$" /etc/network/interfaces
if [ ! $? -eq 0 ]; then
cat <<EOF >> /etc/network/interfaces
auto ${DATAVLANDEV}
iface ${DATAVLANDEV} inet static
#address 0.0.0.0
up mkdir -p /var/run/emulab
# Just touch it, don't put iface/inet/mac into it; the vlans atop this
# device are being used natively by openstack. So just let Emulab setup
# to not setup any of these vlans.
up touch /var/run/emulab/interface-done-$DATAPMAC
EOF
fi
done
#else
......@@ -232,9 +260,29 @@ ip route flush cache
# Just wait a bit
#sleep 8
echo "*** Removing Emulab rc.hostnames and rc.ifconfig boot scripts"
mv /usr/local/etc/emulab/rc/rc.hostnames /usr/local/etc/emulab/rc/rc.hostnames.NO
mv /usr/local/etc/emulab/rc/rc.ifconfig /usr/local/etc/emulab/rc/rc.ifconfig.NO
grep -q DYNRUNDIR /etc/emulab/paths.sh
if [ $? -eq 0 ]; then
echo "*** Hooking Emulab rc.hostnames boot script..."
mkdir -p $OURDIR/bin
touch $OURDIR/bin/rc.hostnames-openstack
chmod 755 $OURDIR/bin/rc.hostnames-openstack
cat <<EOF >$OURDIR/bin/rc.hostnames-openstack
#!/bin/sh
cp -p $OURDIR/mgmt-hosts /var/run/emulab/hosts.head
exit 0
EOF
mkdir -p /etc/emulab/run/rcmanifest.d
touch /etc/emulab/run/rcmanifest.d/0.openstack-rcmanifest.sh
cat <<EOF >> /etc/emulab/run/rcmanifest.d/0.openstack-rcmanifest.sh
HOOK SERVICE=rc.hostnames ENV=boot WHENCE=every OP=boot POINT=pre FATAL=0 FILE=$OURDIR/bin/rc.hostnames-openstack ARGV=""
EOF
else
echo "*** Nullifying Emulab rc.hostnames and rc.ifconfig services!"
mv /usr/local/etc/emulab/rc/rc.hostnames /usr/local/etc/emulab/rc/rc.hostnames.NO
mv /usr/local/etc/emulab/rc/rc.ifconfig /usr/local/etc/emulab/rc/rc.ifconfig.NO
fi
if [ ! ${HAVE_SYSTEMD} -eq 0 ] ; then
# Maybe this is helpful too
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment