emulab-devel issueshttps://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues2018-07-12T13:37:55-06:00https://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/96Upgrade Mothership experimental network interconnects2018-07-12T13:37:55-06:00Mike HiblerUpgrade Mothership experimental network interconnectsWe have the parts to finish the CRIOPS2 proposed interconnect upgrade. This includes:
1. Connecting arista1 with 16x10Gb fibers. The fibers are run and the optics are installed. "Just" need to reconfigure arista1, z9500, and the Emulab ...We have the parts to finish the CRIOPS2 proposed interconnect upgrade. This includes:
1. Connecting arista1 with 16x10Gb fibers. The fibers are run and the optics are installed. "Just" need to reconfigure arista1, z9500, and the Emulab DB. Supposedly both the Arista and Dell switches will support 16 ports in an aggregation, but maybe we don't want to do it that way.
2. Connecting procurve[1345] to z9500 with 4x10Gb each. Again, the fibers have been run and I have the necessary procurve linecards and optics. We need to upgrade the firmware on procurve[345] (procurve1 has been done already). Then we need to configure the switches and Emulab DB.Finish Mothership upgradehttps://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/438VPN concentrator for control network2019-03-26T11:24:39-06:00Robert Ricciricci@cs.utah.eduVPN concentrator for control networkThis is about setting up a physical machine, probably in MEB or the DDC, to act as an openvpn server to the various POWDER aggregates; configuring its openvpn software; and setting up appropriate routing/firewalling to the mothership (et...This is about setting up a physical machine, probably in MEB or the DDC, to act as an openvpn server to the various POWDER aggregates; configuring its openvpn software; and setting up appropriate routing/firewalling to the mothership (et al). (The aggregate (client) side of this issue is being discussed in https://gitlab.flux.utah.edu/emulab/emulab-devel/issues/439).
Subtasks:
- [x] @hibler is going to obtain a new /22 from campus and have them route it to the MEB firewall
- [x] @hibler or @kwebb configure the firewall with the routes for the concentrated /29s to point to a gateway address on the VPN outside all those /29s
- [x] @johnsond will setup a physical VPN concentrator box, probably running Ubuntu 18.04.
- [x] @mike or @kwebb will setup a path from the firewall to the concentrator, and from the concentrator to the mothership control net.
- [x] @johnsond is going to write a profile that is a mockup of (most of) the software, including the failover stuff (wired to start, then wireless using a nuc), to validate the design (this is happening in https://gitlab.flux.utah.edu/johnsond/powder-vpn)
- [x] @johnsond needs to turn the scripts from https://gitlab.flux.utah.edu/johnsond/powder-vpn into a single script on the concentrator; this is trivial.
~~- [ ] @johnsond needs to tweak the concentrator's configuration to move to the "scalable", one openvpn server process per client (aggregate) -- and adapt his profile's scripts to add configuration for each new aggregate.~~ (Given that UConnect bandwidth is what it is, we decided that there is currently no need to move to the scalable design.)Dan ReadingDan Readinghttps://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/430RF allocation / shutdown daemon2018-08-21T16:12:56-06:00Robert Ricciricci@cs.utah.eduRF allocation / shutdown daemonGary WongGary Wonghttps://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/442Disk image caching on AMs2020-07-14T16:16:08-06:00Robert Ricciricci@cs.utah.eduDisk image caching on AMsThe control network connectivity and bandwidth of different types of end points and base stations requires rethinking of how we distribute and collect the potentially large images for experiment nodes. If and how images are cached on the...The control network connectivity and bandwidth of different types of end points and base stations requires rethinking of how we distribute and collect the potentially large images for experiment nodes. If and how images are cached on the end point control nodes is an issue.Mike HiblerMike Hiblerhttps://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/404Bring up CloudLab install at Rutgers2019-08-29T11:05:29-06:00Mike HiblerBring up CloudLab install at RutgersThey have a rack of equipment (see attached figure) that will be part of CloudLab. Would be simple except that they want to federate/integrate/ram-it-together-at-high-velocity with ORBIT.
[Rutgers_CloudLab_Rack.pdf](/uploads/5cc487d611e...They have a rack of equipment (see attached figure) that will be part of CloudLab. Would be simple except that they want to federate/integrate/ram-it-together-at-high-velocity with ORBIT.
[Rutgers_CloudLab_Rack.pdf](/uploads/5cc487d611efc42466a3af6eca9b85cc/Rutgers_CloudLab_Rack.pdf)Mike HiblerMike Hiblerhttps://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/686Handle blockstores "correctly" during experiment modify2023-10-11T11:45:50-06:00Mike HiblerHandle blockstores "correctly" during experiment modifySince @stoller got experiment modify working via the portal (have I mentioned how awesome this is?) I need to figure out how to handle blockstores in a sane manner.
First, it should be pretty easy to make this work for remote blockstore...Since @stoller got experiment modify working via the portal (have I mentioned how awesome this is?) I need to figure out how to handle blockstores in a sane manner.
First, it should be pretty easy to make this work for remote blockstores, we just have to detach from them before modify and reattach to whatever is in the experiment after the modify.
Local blockstores are the issue. The cleanest approach would just be to destroy any existing blockstores before modify and then let it recreate blockstores after the modify. However, this would likely not sit well with users unless they were explicitly making a change to the blockstore configuration as part of their modify operation. If they were just say, adding a node, then wiping out their local blockstores on all other nodes is probably not a reasonable thing to do. Unfortunately, this may be what I do for the current "reconfig" target, I had better go fix that!
There is code in place today to save the current config at boot time and on reboot read in that config, adding or removing any blockstores that appear of disappear. But that is not going to work unless the reconfig involves a reboot. It also won't work if the boot disk is reloaded as part of the modify. We either need to store the configuration info in /proj, of maybe take advantage of the fact that volume managers like LVM and ZFS can reconstruct their config from on-disk info.Mike HiblerMike Hiblerhttps://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/671Enable SEV on Cloudlab AMD nodes2022-12-22T12:21:14-07:00Mike HiblerEnable SEV on Cloudlab AMD nodesThis only applies to Clemson `r6525` nodes right now, but I am creating this ticket to make note of what to do for future nodes.
To enable AMD SEV, enable the following (on Dell machines):
* IOMMU
* Kernel DMA Protection
* Secure Memory...This only applies to Clemson `r6525` nodes right now, but I am creating this ticket to make note of what to do for future nodes.
To enable AMD SEV, enable the following (on Dell machines):
* IOMMU
* Kernel DMA Protection
* Secure Memory Encryption
* Secure Nested Paging
* SNP Memory Coverage
and set "Minimum SEV non-ES ASID" to a value greater than one. It appears to actually be a maximum based on the description in [the vSphere docs](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-757E2B37-C9D0-416A-AA38-088009C75C56.html). They say set it to N+1 if you want N VMs, so we could set it to something like 17 or 33 maybe.https://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/666Periodic retries for nodes in `hwdown`2022-07-11T11:12:17-06:00Mike HiblerPeriodic retries for nodes in `hwdown`We spend a considerable amount of time on an ongoing basis dealing with nodes in the `hwdown` experiment. By the time we diagnose such a node, quite often the problem has disappeared or we discover the problem is easily fixable. Worse, i...We spend a considerable amount of time on an ongoing basis dealing with nodes in the `hwdown` experiment. By the time we diagnose such a node, quite often the problem has disappeared or we discover the problem is easily fixable. Worse, if we don't notice a node in there quickly enough it can get pushed down the list by more recent failures and can fall off our radar, winding up in `hwdown` for months.
So... @eeide asks, "Can we do better?" Maybe periodically releasing nodes from `hwdown` to give them a second chance (which is often times what we do manually when we don't have time to diagnose).
Note that this is mostly an issue for nodes of the less popular types (e.g. `pc3000`s). `hwdown`ing of nodes of frequently used types (e.g., those with GPUs) will cause overbook problems in the reservation system or users will complain and we will act quickly.https://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/659Fix up Apt node BIOSes2022-03-02T08:58:19-07:00Mike HiblerFix up Apt node BIOSesWhile attempting to collect hwinfo from all Apt nodes, I noticed two anomalies with the pt (in particular `r320`) nodes.
* No setup password set
* The lifecycle controller is disabled, did we intentionally do that?
Because of the first,...While attempting to collect hwinfo from all Apt nodes, I noticed two anomalies with the pt (in particular `r320`) nodes.
* No setup password set
* The lifecycle controller is disabled, did we intentionally do that?
Because of the first, we should make a pass over the BIOSes and make sure they are configured as we expect.https://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/658Time synchronization at Cloudlab clusters2022-02-24T12:14:21-07:00Mike HiblerTime synchronization at Cloudlab clustersA recent question on the users list asked about time synchronization between the clusters which got me thinking about this again.
All of the nodes at a cluster use a local (`ntp1`) NTP time server which by convention is `ops`. We also ...A recent question on the users list asked about time synchronization between the clusters which got me thinking about this again.
All of the nodes at a cluster use a local (`ntp1`) NTP time server which by convention is `ops`. We also stash away the "drift" value from each node (via the watchdog) and use the latest saved value to initialize the drift file when a node is imaged. The various cluster NTP servers use a range of upstream servers and NTP pools, but are not directly connected ("peers"). We seem to keep reasonable time between the cluster NTP servers at least, generally around 1-5ms.
Some questions:
* Is saving/restoring the drift value still a good thing to do?
* Should we be using PTP?
* Any chance of getting a GPS receiver at the main clusters?
* Should we use `chrony` which is ```aimed at ordinary computers, which are unstable, go into sleep mode or have intermittent connection to the Internet. chrony is also designed for virtual machines, a much more unstable environment.```? I think current Ubuntu images already use it.
At the very least, we should probably move the `ntp1` alias off of `ops`, which is a VM at all but Emulab, and onto the control node instead where there would be a more stable clock.https://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/640Fix clientside scripts to work with python3.2021-09-03T13:54:54-06:00Mike HiblerFix clientside scripts to work with python3.We have fixed up the server side (#611) along with a few of the client-side scripts that are used on `ops`, but we should finish the job.We have fixed up the server side (#611) along with a few of the client-side scripts that are used on `ops`, but we should finish the job.https://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/632Get our content out of the Plone wiki2021-09-03T13:52:31-06:00Mike HiblerGet our content out of the Plone wikiWe have been keeping Plone on life support through a couple of boss/ops upgrades. After the latest, it is time to pull the plug since nobody wants to convert it to python3. So we need to get our content out of there and loaded somewhere ...We have been keeping Plone on life support through a couple of boss/ops upgrades. After the latest, it is time to pull the plug since nobody wants to convert it to python3. So we need to get our content out of there and loaded somewhere else. (gitlab wiki?) If only I had remembered this *before* we converted ops...
So now we have to move the current installation over to a machine with python2, either an elabinelab or else just move it somewhere like ops.utah.cloudlab.us. Then figure out a way to extract the useful content. Then figure out a way to get that content into something else in a reasonable form.
I expect this falls on @hibler or @stoller.https://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/620Do something with DriveScale hardware2021-04-13T13:51:49-06:00Mike HiblerDo something with DriveScale hardwareSince DriveScale got bought and we can no longer use their software, we have a whole bunch of disk space that we should make use of somehow:
* **2 x 72 drive JBOD boxes**. These are just disk boxes with external SAS interfaces that we s...Since DriveScale got bought and we can no longer use their software, we have a whole bunch of disk space that we should make use of somehow:
* **2 x 72 drive JBOD boxes**. These are just disk boxes with external SAS interfaces that we should be able to hook up to a server and use in the traditional Emulab storage server way. The one box we have mounted is fully populated with 1-3TB drives and has around 120TB, the other is fully populated and a spot check showed 1-2 TB drives, so probably upwards of 100TB again.
* **24 drive NVMe chassis**. This is actually a server with 24 NVMe slots, and 4 x 100Gb network interfaces. But we have not been able to figure out exactly what it is or how to boot it into anything but its internal OS drive. But we have not tried very hard. It is currently populated with 16 x 6.4TB NVMe drives that were purchased with CARES funding. So we need to at least get the drives into use.https://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/567More storage for CloudLab storage servers2022-10-06T12:05:57-06:00Mike HiblerMore storage for CloudLab storage serversAfter spending quite a lot of time scrambling to come up with 10TB on one of our blockstore servers, I came to the conclusion that we are "under-storaged" if we are serious about the blockstore mechanism and allocation of 10+TB datasets....After spending quite a lot of time scrambling to come up with 10TB on one of our blockstore servers, I came to the conclusion that we are "under-storaged" if we are serious about the blockstore mechanism and allocation of 10+TB datasets. The situation:
* Utah: 43.5TB in one server, **6.7TB free**. We have a couple of options for more space here. One is the DriveScale chassis which have over 100TB between them but needs more work to integrate with the model. The other is to wire up the Apt half of the Dell storage box. This would add a second server with 36TB. Requires running a 40Gb link from the Apt side of the pod over to `bighp1`.
* Clemson: 50TB on one server, but in two zpools of 43 and 7TB. **5.8TB and 1.3TB free**. Again a couple of possibilities. The first would be to commandeer another one of the first-gen storage node, giving us another 50TB. A more intriguing possibility would be to take over one of the `dss7500` nodes with 45 HDDs and 270TB. That would solve the space issue for some time to come but would take one of only two of those machines. I only suggest it because those machines are almost never used right now, which is a waste.
* Wisc: 32TB on one server. **10TB free**. The only option for more space here would be to take over another one or two `c240g1` nodes, the only ones with significant storage. That would get another 32TB per server.
* UMass: no storage servers, no current plan.
* OneLab: no storage servers, no current plan.
* Emulab: 92TB on two storage servers. Each server has 40TB for persistent and 6TB for ephemeral. There is about **9.7TB free on each**.
* Apt: 36TB in one server, all available, have been using for testing. The intent was to move this storage server + disk to CloudLab Utah.Mike HiblerMike Hiblerhttps://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/560Ensure datasets are not busy when taking a snapshot2020-07-03T09:46:00-06:00Mike HiblerEnsure datasets are not busy when taking a snapshotThis is a very common failure mode for image-backed datasets:
```
About to: '/usr/testbed/bin/sshtb -n -host c220g5-111012 /usr/local/bin/create-versioned-image METHOD=frisbee SERVER=128.104.222.9 IMAGENAME=praxis-PG0/bench-setup:0 BSNAM...This is a very common failure mode for image-backed datasets:
```
About to: '/usr/testbed/bin/sshtb -n -host c220g5-111012 /usr/local/bin/create-versioned-image METHOD=frisbee SERVER=128.104.222.9 IMAGENAME=praxis-PG0/bench-setup:0 BSNAME=bs IZOPTS=N' as uid 0
c220g5-111012: started image capture for '/.amd_mnt/ops.wisc.cloudlab.us/proj/praxis-PG0/images/bench-setup/bench-setup.ndz.tmp', waiting up to 90 minutes total or 8 minutes idle.
umount: /benchdata: target is busy.
Could not unmount /dev/mapper/emulab-bs!
Could not parse all arguments
FAILED: Returned error code 2 generating image ...
```
We want to unmount the filesystem to get a consistent snapshot of the filesystem, but the user has a process active on the dataset at that time.
Things to look at:
* attempting to locate all such processes and killing them
* doing a forcible unmount
* shutting down the machine to single-user
* identifying the situation in advance and refusing to snapshotMike HiblerMike Hiblerhttps://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/554The FreeBSD `metis` port has undergone a major, possibly incompatible revision2020-06-03T09:17:26-06:00Mike HiblerThe FreeBSD `metis` port has undergone a major, possibly incompatible revisionFrom the Slack thread:
> FYI, the FreeBSD `metis4` port is gone is the latest quarterly port set. There is now a `metis` port, which is Metis version 5. Anyone care to speculate whether that is going to cause problems? I thought it was ...From the Slack thread:
> FYI, the FreeBSD `metis4` port is gone is the latest quarterly port set. There is now a `metis` port, which is Metis version 5. Anyone care to speculate whether that is going to cause problems? I thought it was used with `assign`, but apparently it is only `assign_prepass` and `ipassign`.
@ricci thought them no longer used, though I verified that we have classic experiments (none in the last 10 years), that will use them do to magical DB settings in the `experiments` table. @stoller says `ipassign` is not used through the portal path, though `assign_prepass` (via `mapper`) might be.
The new port does cause problems:
> It looks like at least `assign_prepass` is still used. It uses the `kmetis` command line tool and that tool is now gone. According to the manual (http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/manual.pdf, search for "kmetis"), `gpmetis` is the direct replacement command, but I don't have any idea if it behaves the same without any additional options.
> I should add that `ipassign` is also technically still used, but it has to be specified explicitly in an ns file and there are only a handful of experiments that do that--none swapped in in the last 10 years. But we don't have a better solution for large, complex topos, we just punt on them now. `ipassign` links with `metis` libraries and we have already discovered that they moved include files around which breaks the build. Who knows what has changed in the API.
> Looks like mapper has to be invoked with "-x", or the experiment flagged with "useprepass", in order for assign_prepass to be called. There are no instances of the former, and though there are three experiments with the "useprepass" column set, I see no path in our code through which that field can ever be set!https://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/530create_image -s on boss fails2020-03-26T14:20:26-06:00chuck cranorcreate_image -s on boss failsThis is Mike's favorite bug! It predates flux gitlab, so I couldn't file an issue on it when i first encountered it. Now I can...
running "create_image -s" (uses ssh) on boss causes the new disk image binary to be emailed to your acco...This is Mike's favorite bug! It predates flux gitlab, so I couldn't file an issue on it when i first encountered it. Now I can...
running "create_image -s" (uses ssh) on boss causes the new disk image binary to be emailed to your account instead of saved on disk. its too large, so the mail system rejects it with:
<pre>
The original message was received at Wed, 29 Jan 2020 14:54:46 -0500 (EST)
from localhost [127.0.0.1]
----- The following addresses had permanent fatal errors -----
<chuck@ece.cmu.edu>
(reason: 552 5.2.3 Message size exceeds fixed maximum message size
+(52428800))
----- Transcript of session follows -----
... while talking to dept-mx-03.andrew.cmu.edu.:
>>> MAIL From:<chuck@boss.narwhal.pdl.cmu.edu> SIZE=928020875
<<< 552 5.2.3 Message size exceeds fixed maximum message size (52428800)
554 5.0.0 Service unavailable
</pre>
one possible solution is just to remove "-s" from create_image, as the non-ssh options work.
another solution is this:
<pre>
diff -r -u baseline/utils/create_image.in orca/utils/create_image.in
--- baseline/utils/create_image.in 2019-05-22 17:03:13.000000000 -0400
+++ orca/utils/create_image.in 2019-05-22 17:02:25.000000000 -0400
@@ -1224,7 +1224,7 @@
#
my $SAVEUID = $UID;
$EUID = $UID = 0;
- $result = run_with_ssh($command, undef);
+ $result = run_with_ssh($command, $filename);
$EUID = $UID = $SAVEUID;
if ($result eq "setupfailed") {
goto done;
</pre>
but Mike was worried that it might break(?) something else if applied.Mike HiblerMike Hiblerhttps://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/519SonicWall firewall issues with vnode to pnode communication2024-01-02T11:11:09-07:00Mike HiblerSonicWall firewall issues with vnode to pnode communicationSince the firmware upgrade (I think), we are once again having problems with routing between the control net (155.98.36.x) and the vnode control net (172.16.x.x). We are once again dropping these packets:
```
DROPPED, Drop Code: 710(Pack...Since the firmware upgrade (I think), we are once again having problems with routing between the control net (155.98.36.x) and the vnode control net (172.16.x.x). We are once again dropping these packets:
```
DROPPED, Drop Code: 710(Packet dropped - drop bounce same link pkt)
```
Packets necessarily need to bounce off of the firewall to route between the two. The vnode control net is a "secondary subnet" in the control net zone on the firewall. We had a problem with this initially, but after putting in the needed ARP and routing hacks, it started working. As far as I can tell, those hacks are still in place.https://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/516Delay-and-loss tolerant interactive access to endpoint nodes2020-09-15T15:56:36-06:00Robert Ricciricci@cs.utah.eduDelay-and-loss tolerant interactive access to endpoint nodesUsers will inevitably want interactive access to endpoints: both fixed and mobile. Trying to run protocols like VNC already performs poorly to the fixed endpoints (over wifi), and this will be much worse for the mobile ones.
Some possib...Users will inevitably want interactive access to endpoints: both fixed and mobile. Trying to run protocols like VNC already performs poorly to the fixed endpoints (over wifi), and this will be much worse for the mobile ones.
Some possible avenues to pursue:
* Mosh for shell access (downside: requires user to install client-side software)
* Low-bandwidth alternatives to VNC or X11Robert Ricciricci@cs.utah.eduRobert Ricciricci@cs.utah.eduhttps://gitlab.flux.utah.edu/emulab/emulab-devel/-/issues/509Frisbee-ing from subbosses to virtual nodes with non-public IPs2020-01-27T08:08:19-07:00Mike HiblerFrisbee-ing from subbosses to virtual nodes with non-public IPsSince subbosses are on the node control net segment and may need to served vnodes with 172.16.0.0 adresses, they need to have an alias on their control net for that network. Can we get away with simply adding a route for 172.16.0.0/12 to...Since subbosses are on the node control net segment and may need to served vnodes with 172.16.0.0 adresses, they need to have an alias on their control net for that network. Can we get away with simply adding a route for 172.16.0.0/12 to the control net interface? Will IGMP snooping on the switch work in that case?Mike HiblerMike Hibler