GitLab

No idea why rpc exchanges started failing after updating boss/ops to
FreeBSD 6.1 or, for that matter, why this really fixes the problem.
Monitoring the exchanges between client and the RPC serial line didn't
show any rhyme or reason why some exchanges elicited "Input error" and
others didn't.  But I noticed that every \r we send to the RPC gets
echoed back as \r\r\n (and no, it isn't the serial line CR<->NL mapping).
So I reduced the number of \r (actually \n) we send it and things got better.

This "fix" has been running every 5 minutes for the last hour without
failure, so I declare it: An Improvement.

Something subtle changed with the upgrade to FreeBSD 6.1,
maybe something to do with perl buffering?

couple of MFS booting problems:
 * in the RPC power controller, make sure that an "on" command succeeds
   by checking the status, retrying if it failed (we already did this for
   "off")
 * if nodes fail to boot up the MFS after a power on, try again with a
   power cycle.  I have seen "power on" leave pc600s hung, and a power
   cycle seems to cure it.

Now we keep reading after we send the command, til we get the next prompt.
That way we will catch any "Input error" messages and can try again.

The downside is that, for reboots, we wait through the whole 10 second
countdown before returning to the caller.  Previously we would return
immediately after issuing the command.

read matches the prompt, i.e., don't wait to get a newline.  With the
new power controller on the rocketport muxes on a pc1500 tipserver,
even though we send two newlines, we just get back a single prompt
with no newline.  This would cause us to hang forever waiting for the
newline.

Note that it did not do this on the same RPCs with the same muxes on
our ops node (a pc3000).  This could be a bad sign for things to come...

(eg. "power status rpc14-1" or "power status all").

In rpc module, add a "status" function that is called to collect,
well..status.  This is used by the generic power code, but more importantly,
by the new "powermon" script (which could have just called "power status",
but no point in parsing output twice).

the controller.

Read characters from the device one at a time.  GaTech has one of these hooked
to the built-in PC serial port, and when reading 1024 chars at a time, it will
not read complete lines.  This would confuse our primitive expect-like logic.

limited to 31 characters, so we can't give commands with more than 8
outlets. So, we split up the outlets into groups of 8, and issue
multiple commands, if we are passed too many outlets.

Took some minor re-structuring, since we now may need to sync up with
the RPC prompt more than once.

power controller, and controls all nodes on the same controller at
once. This is particularly helpful when rebooting nodes connected to
the RPCs, as you now longer have to wait through the RPCs 'rebooting'
junk for each node.

power_rpc27.pm and snmpit_apc.pm received some minor changes to
support passing multiple outlets to one function call.

the connection handshake. If the capture is active, it is going to
write status and close the socket on its end, which may or may not be
propogated back to the client side before/after/during the
sysread/syswrite. I guess that a miserable way of saying there is a
lot of asynchrony involved.

ack/nak for a connection so that the connecting process knows what the
hell is going on. Turned out to be necessary for power control since
we do that in parallel, and because it stays busy for 10 seconds on
each power control. I think we will end up revisiting this at some
point, adding blocking connections instead of connect/fail status.

goo whenever it goes inactive for more than a couple of minutes. Also,
require that we sync up so that we know for sure that the power
control failed. Previously, I was blindly spitting out the command
without being sure we had a command prompt.

Note, that we still have a problem with an active tip session messing
up power control since only one connection can be active at a time.
At present, there is no positive way to know that the connection was
refused since capture accepts the connection and then closes it if
there is an active tip session. I need to change the little handshake
so that there is a response Yes/Know response to the secret key
exchange.

distributed tiplines (and capture processes).