Making the event system great again
We have had a variety of issues recently affecting the stability and performance of the event system.
Of particular interest is the UPenn peoples' use of rapid
tevc calls (or even direct-to-pubsub calls) to deliver link change events to 30+ nodes in their experiments. And there are typically 2-3 instances of this experiment workflow going on at once. They have discovered that they are limited to about 50 events/sec (I assume this is in aggregate and not to each node in the experiment). Beyond that they have problems with the event scheduler crashing and/or the entire infrastructure melting down, either immediately or after 30+ minutes.
So maybe it is time to revisit our NCR-era scaling plans. We discussed the problems in the Big Book of Emulab, chapter 7 (there is a copy at bas:~mike/ncr/report-0.6.pdf), but it is not clear whether the goals then were the same as what we care about now.
clusterd was part of the proposed solution, though we have not moved very rapidly on that yet. In particular we had plans to use it to replace
pubsubd on every experiment node (we have deployed it on vnode hosts) and even talked about adding another level--running an instance on every subboss (though that would violate Mike's First Law of Hierarchy). https://wiki.emulab.net/wiki/clusterd contains old thoughts on
clusterd in particular.
Some possible actions we can pursue:
- Optimize the UPenn case. What can we do to allow a single client to deliver, say, 100 events/sec to one or more agents? For true, dynamic generation of "do it now" events this might involve bypassing the event scheduler or even bypassing the central
pubsubdand talking straight to the node(s) pubsubd(s). Or if the event timeline can be worked out in advance, we might be able to pre-stage the events on the end nodes using some bulk event send version of
- Optimize the current architecture for greater system-wide throughput. This is maybe just taking advantage of
clusterdand doing some engineering on the components (e.g., multithread
- (Re)investigate off-the-shelf pubsub solutions. There should be things out there that receive constant attention and are known to scale.
nanomsgare two packages I have seen in a quick google. They both have pubsub "patterns".
What we do here will depend on what we think the event model will look like in the Portalized world of Emulab too.
Maybe we can find something here that is well-formed enough to make it a project for Kobus's class!