Commit 3dd64b0b authored by Kirk Webb's avatar Kirk Webb

Added text describing error handling.

parent 3a2741da
......@@ -174,4 +174,34 @@ used for the corresponding slice.
\subsection{Failure Handling and Recovery}
Given two large distributed systems such as Emulab and \plab, failures
are a given and have many possible modes. We approach this by ...
are a given and have many possible modes. We apply several mechanisms
to cope with these. The \plab backend defines wrapper functions that
call a requested remote API function and handle error conditions
encountered. There are three types of errors the handler can cope
with: fatal, retryable and continuable. On detecting a fatal error,
the backend halts the current operation and reports failure back to
the caller. For retryable error types, the wrapper will try the RPC
again; by default, the RPC wrapper will attempt a remote procedure
three times before giving up. Continueable errors are cases where the
error indicates that the goal has already been acheived (e.g., when a
node deallocation RPC reports that a node is no longer allocated).
The classification of these errors is defined in software; there is no
heuristic to determine when to continue or give up. The default
error classification is retryable.
The outer Emulab infrastructure combined with the \plab backend track
the resources that are in use at any given time (swapin, active,
swapout). The \plab backend gaurantees not to leave slices or nodes
allocated when their allocation ultimately fails. When a setup fails
or is canceled further into swapin, the Emulab infrastructure takes
care to call the appropriate \plab backend commands to free any
allocated resources. For example, when a \plab experiment setup fails
because some nodes fail to allocate or load and run the Emulab
client-side startup scripts (and setup failure is set to fatal for
these nodes), a full Emulab experiment termination will be activated.
This will result in the deallocation of any resources; \plab nodes
will be freed by whichever backend module is appropriate, and the
slice will be destroyed. No resources are leaked, and namespaces are
cleared so that future setups will not collide.
\xxx{Talk about timeout handling}
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment