Skip to content
Snippets Groups Projects
Commit 87dd2e60 authored by Leigh B. Stoller's avatar Leigh B. Stoller
Browse files

The panic button ...

* tbsetup/panic.in: New backend script to implement the panic button
  feature. When used, it will cut the severe the connection to the
  firewall node by using snmpit to disable the port. Sets the panic
  bit (and date) in the experiments table, and changes the state of
  the experiment from "active" to "paniced" to ensure that the
  experiment cannot be messed with (swapped out or modified). Sends
  email to tbops when the panic button is pressed.

  Used with -r option, reverses the above. State is set back to
  active, the panic bit is cleared, and the port is renabled with
  snmpit.

* tbsetup/tbswap.in: During swapout, a firewalled experiment that has
  been paniced will get a cleaning; The nodes are powered off, then
  the osids for all the nodes are reset (with os_select) so that they
  will boot the MFS, and then the nodes are powered on. Then the
  control network is turned back on, and then I wait for the nodes to
  reboot (this is simply cause we do not record in the DB that a node
  is turned off, and if I do not wait, the reload daemon will end
  hitting the power button again if they do not reboot in time. We can
  fix this later.

  I am not planning to apply this to general firewalled experiments
  yet as the power cycling is going to be hard on the nodes, so would
  rather that we at least have a 1/2 baked plan before we do that.

* www/showexp.php3: If experiment is firewalled, show the Panic
  Button, linked to the panic button web script. If the experiment has
  already had the panic button pressed, show a big warning message and
  explain that user must talk to tbops to swap the experiment out.
  Also fiddle with menu options so that the terminate link is gone,
  and the swap link is visible only in admin mode. In other words, only
  an admin person can swap an experiment once it is paniced. And of
  course, an admin person can the backend panic script above with the
  -r option, but thats not something to be done lightly.

* db/libdb.pm.in: Add "paniced" as an experiment state (EXPTSTATE_PANICED).
  Add utility functions: TBExptSetPanicBit(), TBExptGetPanicBit(), and
  TBExptClearPanicBit().

* tbsetup/swapexp.in: Minor state fiddling so that an experiment can
  be swapped while in paniced state, but only when in admin mode. Also
  clear the panic bit when experiment is swapped out.

* www/dbdefs.php3.in: Add "paniced" as an experiment state. Add a
  utility function TBExptFirewall() to see if experiment is firewalled.

* www/panicbutton.php3: New web script to invoke the backend panic
  script mentioned above, after the usual confirm song and dance.

* www/panicbutton.gif: New gif of a red panic button that I stole off
  the net. If anyone has sees/has a better one, feel free to replace
  this one.

* utils/node_statewait.in: Add -s option so that I can pass in the
  state I want to wait for (used from tbswap above to wait for nodes
  to reach ISUP after power on).
parent 9e7b2287
No related branches found
No related tags found
No related merge requests found
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment