GitLab

Drop 99-emulab-networkd.rules into /etc/udev/rules.d,
emulab-networkd@.service into /lib/systemd/system, and do
`systemctl enable systemd-networkd.service`.  I would also do
`ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf`; we don't
want any part of systemd-resolved.

This is the only way I could come up with to make systemd-networkd
listen on all wired ethernet ifaces, and get it into the right state.
systemd-networkd is like NetworkManager in some ways.  I don't know if
it's better or worse.  The main problem for us is that it doesn't
support hooks, and that it runs its own dhcp client (which is crappy and
inflexible compared to dhclient).  We don't need anything super flexible
though since we don't use custom options, and it has an option to ignore
the lease lifetime and leave the iface up even if expiry happens before
renewal can take place.

Anyway, here's the flow: udev rules tell systemd to require a unit
(emulab-networkd@$iface.service) for each physical wired ethernet
device.  These files write runtime .network files into
/run/systemd/network, which are read by systemd-networkd.service if it
is enabled.  We just tell systemd-networkd to listen on them all, then
we use its waiter script.  However, its wait script is broken and/or the
documentation is contradictory, so we don't trust it to tell us which
interface comes up; we invoke its networkctl client to tell us.  We then
write the up iface to /run/emulab/cnet, rm -f the .network files for all
ifaces other than the control net device (cause we don't want
systemd-networkd to manage expt network devices), and restart
systemd-networkd so it can appropriately report that it is "managing"
the control net device, but not the expt devices.  It is very
unfortunate that we have to restart systemd-networkd, but there is no
other way to tell it about new configuration.  If you HUP it, it dies.
These systemd folks must live in a box where all use cases look alike.

Note that if the user overrides with their own .network scripts in
/etc/systemd/network that Match any of the physical NICs, those take
precedence over our per-interface scripts, which is what we want.

Note also that to do something like move the real control net into a
bridge, you can just remove or update the .network file in
/run/systemd/network, and restart systemd-networkd.  It should re-dhcp
on the bridge.  It does not remove NIC configuration at exit; it just
"catches up" with any NIC address when it is restarted.

I have not done all the dhclient-exit-hook normal goo (i.e. writing the
control net controlif/IP into the right place, etc), but that should all
be possible.

Finally: I haven't yet decided that this is the path for Ubuntu 18.  We
might well ignore this and fall back to the old /etc/network/interfaces
ifupdown strategy.  At least there it is possible to use a real
dhclient.

Currently these are just copies of the 2.2 file, but I am anticipating changes.

and it throws an error.

and it throws an error.

This worked under perl 5.24 and before as '.' was implicitly in @INC.
No longer true in 5.26.

with no rspec, nor try to start an experiment with no rspec.

checking node list.

Alpine's shadow is like archlinux; it does not support uppercase
chars in user/group names.  So, fix that along with the runit install.

Fixing errors in emulabizing Docker images

See merge request emulab-devel!36

Fixed issue with cmd and entrypoint values being set
incorrectly

Fixed issue with rcS init.d process negatively hurting
our networking settings.

Better handling of whitespace in Environment variable
values.

Squashed commit of the following:

commit 9aff9b12aa6a60ebb66f10f0a044bf2288b492be
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Wed May 9 19:15:40 2018 -0600

    Handling environment variable values with whitespace

commit c9a16bbd53c183d7f8259d878efb0be07bc8703c
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Wed May 9 17:07:43 2018 -0600

    Fixing racing issue and rcs issue

commit 10bf4517c419fc82f89d084c4f45d60c05a67834
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Wed May 9 16:00:45 2018 -0600

    Setting cmd to be empty and entrypoint to be runit

Fixing cmd/entrypoint issues and adding support for debian:sid

See merge request emulab-devel!35

Fixed issue involving alternate users being set in
the image's Dockerfile.

Fixed issue with accurate permissions to run runit service.

Fixed issue with quotes to properly handle sh commands.

Fixed issue where Docker images based on scratch are not in
an array like other Docker images, so config attributes are
preserved.

Modified the analyze script that if it sees a debian
without a version, it should update it to debianS.
Then added a symlink that points debianS to debian9.
This with some other changes David made should result
in functioning support of debian:sid in Docker

Squashed commit of the following:

commit 7af0b39cbcd9059f1af35e2b778b3879019aa06e
Merge: dff5441da 9b7e1a39
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Wed May 9 13:07:22 2018 -0600

    Merge remote-tracking branch 'origin/master' into entrypoint-bug-fix

commit dff5441da8d938ec2b27290b3fdd3ceba56fc502
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Wed May 9 11:24:14 2018 -0600

    Adding support for debian sid

commit 5b4e102b77aae312dc26cf54c0760af59211f974
Merge: ef1f1102b 34aa0d23
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Tue May 8 19:18:13 2018 -0600

    Merge remote-tracking branch 'origin/master' into entrypoint-bug-fix

commit ef1f1102bee796ad83c51b01a319c226518ff503
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Tue May 8 16:42:12 2018 -0600

    Removed ubuntu18 references

commit ecded73c3bb6340e6bed0032f602a98397577d0c
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Tue May 8 16:40:59 2018 -0600

    Adding missing suppress flags for install

commit 4235897954d59c1c473b71c34a4c666f2079d009
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Tue May 8 14:49:10 2018 -0600

    Fixing bad fd issue with libssl install

commit 40528b133f7ab3b23d1e533b36bcf41d406c5f90
Merge: a8061efea e468cc49
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Tue May 8 12:58:40 2018 -0600

    Merge remote-tracking branch 'origin/master' into entrypoint-bug-fix

commit a8061efea8f9a6762ad628ebb8071c5397cd4f8f
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Tue May 8 09:53:55 2018 -0600

    Update path reference to ubuntu18

commit 4305ad261edd5be0714c9b000d8a3919fddc4300
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Tue May 8 09:42:10 2018 -0600

    Adding support for ubuntu 18.04 lts

commit e21c64a378025ee2e970ff731a30479eac8efd8e
Merge: 9ed6c0e55 642d15f8
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Tue May 8 08:41:09 2018 -0600

    Merge remote-tracking branch 'upstream/master' into entrypoint-bug-fix

commit 9ed6c0e55c48aa8dfe74d4d2823a185a59d281a0
Merge: 7fdb8351b ee3694f4
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Mon May 7 22:27:03 2018 -0600

    Merge remote-tracking branch 'origin/master' into entrypoint-bug-fix

commit 7fdb8351b35bd4a02c725ba9757a2ed5cb5a6e6e
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Mon May 7 17:56:16 2018 -0600

    Reorganizing user account support for cmd/entrypoint

commit 608afa03b154e0e0d70c9994765ab6bdb8f4eed1
Merge: 44c4ca770 faf2368f
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Mon May 7 17:10:22 2018 -0600

    Merge remote-tracking branch 'origin/master' into entrypoint-bug-fix

commit 44c4ca77017bf6733db619e55192133f45977156
Merge: fe063cc22 186c6b7b
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Mon May 7 16:19:47 2018 -0600

    Merge branch 'master' into entrypoint-bug-fix

commit fe063cc22def6a08e468f882e6d4a34f66b9bb48
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Mon May 7 16:17:31 2018 -0600

    Persist environment variables

commit 8c8e206d04b6d07932fc5fe9519735cdabd7bed0
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Mon May 7 15:55:00 2018 -0600

    Adding support for dockerfiles that use alt users

commit d6082d11bcce75a99c7fe9ec1bf2f46970ca4f0d
Merge: f5a5b0e4c d633f6a2
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Fri May 4 10:54:00 2018 -0600

    Merge remote-tracking branch 'origin/master' into entrypoint-bug-fix

commit f5a5b0e4ca6c31a395e3f909e8c107b7f3bdb58e
Merge: 6ef666cd2 0bf7e0b3
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Fri May 4 00:25:20 2018 -0600

    Resolved libvnode_docker conflict

commit 6ef666cd220327845174fa8d0d4897755d7c8786
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Thu May 3 18:27:48 2018 -0600

    Removing debugging statements

commit 72336705c35493849496e1529ea39e8c944e45ec
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Thu May 3 18:06:24 2018 -0600

    Fixed issue where os images aren't in an array

commit ab48322165fb02701ac25acbfb23a4da0e99ddf2
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Wed May 2 23:49:39 2018 -0600

    more debugging

commit 88e7a74f3f4054ab3b1e87d5640cd4afda498efa
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Wed May 2 21:02:35 2018 -0600

    some debugging work

commit cb107aaaf1f7431c32b4b34575f0a67fecf3e07e
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Wed May 2 17:55:08 2018 -0600

    Fixing permission issue

commit 55f07ba52a8e7998413896a0d0fec4d79cf68b0d
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Wed May 2 09:20:23 2018 -0600

    Forgot semicolons are a thing in perl

commit d667575e474f25012a4d5ac8177506af5dd08f2d
Author: Elijah Grubb <u0894728@utah.edu>
Date:   Wed May 2 08:45:22 2018 -0600

    Fixing quotes bug

Particularly, override Entrypoint and User.

This is probably true for Xen too, but in some cases, the
vnodesetup early-release hackwaitandexit timeout of 30 seconds
causes a race condition.  Normally, the first node sets up
significant network state, and sometimes flips MAC addresses
around from interface to interface -- OR puts a physical interface
into a bridge, then changes the bridge's MAC address.  There is a
short window of time where both the bridge and the new member
interface share a MAC address -- and if the tmcc ifconfig assembly
process for vnodes following the first vnode resolves
the wrong device's MAC address and uses that to flesh out the
ifconfig info, the vnodesetup will be in a world of hurt (i.e., you
might see an attempt to make a vlan device out of a vlan device).
The chance of this happening is miniscule, but I've seen it.

So, at least for docker for now, we protect the first vnode against
the 30-second timeout in vnodesetup hackwaitandexit, and we wait for the
actual running file to be written, or error.

This is probably applicable to any linux mkvnode.pl path, but I suppose
it would have been another hundred thousand vnode creates before I saw
it again.

Under high load, of course we can have DNS problems.  However, perl
seems to get stuck on retry; it's like the nak gets cached (which would
be extremely odd, but can't argue with the evidence).

Anyway, if resolution continues to fail, give up and feed the name to
iptables, and let it try :).

(All we need to do while holding the global lock is allocated IFBs; the
generation of routing scripts and traffic shaping scripts is both
unlikely to fail and potentially slow due to running djikstra.  So, also
let the vnode early release prior to those things, immediately after IFB
allocation.)

Docker vnodes require the full image path anytime the vnode is created,
even if the image in question already exists on the vhost.  This is
because emulab custom docker images are fully-qualified with their
hosting private registry, so we need that detail even if we're not
re-pulling the image; i.e., if the vnode gets destroyed and recreated
after its initial reload.

Prior to this commit, I had reused all the libvnode:: network functions,
but that requires these system-wide indexes to be built (i.e. bridges,
ifaces) before using the helpers -- and on any network state change.
Those indexing functions take a long time on heavily-loaded systems
(i.e., 5k processes, hundreds of ifaces).  The helpers become very fast;
but for the case of large numbers of vnodes on one vhost, it is better
to hold the global lock for less time, and pay a higher per-helper cost,
instead of making use of a nicely-indexed cache.

So now we cache nothing and (hopefully) use /sys intelligently to avoid
forking unnecessarily.

and so we have to look through the image versions for each image
in the alias.