#lang scribble/manual

@(require "defs.rkt")

@(parse-sphinx-inventory (string-append (geni-lib-dir) "/docs/build/html"))

@title[#:tag "geni-lib" #:style main-style #:version apt-version]{Describing a profile with python and @tt{geni-lib}}

@(geni-lib) is a tool that allows users to generate @seclink["rspecs"]{RSpec} files from Python
code.  @(tb) offers the ability to use @(geni-lib)
scripts as the definition of a profile, rather then the more primitive
RSpec format. When you supply a @(geni-lib) script on the
@seclink["creating-profiles"]{Create Profile} page, your script is uploaded
to the server so that it can be executed in the @(geni-lib) environment. This
allows the script to be verified for correctness, and also produces the 
equivalent RSpec representation that you can view if you so desire.

@screenshot["create-geni-lib-empty.png"]

When you provide a @(geni-lib) script, you will see a slightly different set
of buttons on the @seclink["creating-profiles"]{Create Profile} page; next
to the ``Source'' button there is an ``XML'' button that will pop up the
RSpec XML for you to look at. The XML is read-only; if you want to change
the profile, you will need to change the python source code that is
displayed when you click on the ``Source'' button. Each time you change the
python source code, the script is uploaded to the server and processed. Be
sure to save your changes if you are 
@seclink["updating-profiles"]{updating an existing profile}.

The following examples demonstrate basic @(geni-lib) usage. More information
about @(geni-lib) and additional examples, can be found in the
@hyperlink["https://bitbucket.org/emulab/geni-lib/src"]{@tt{geni-lib} repository}.
The current version of @(geni-lib) used by @(tb) can be found in the @tt{0.9-EMULAB} branch.
Its full documentation is online as
@link["geni-lib/index.html"]{part of this manual}.

@section[#:tag "geni-lib-example-single-vm"]{A single XEN VM node}

@profile-code-sample["PortalProfiles" "single-vm"]

This example demonstrates the two most important objects: the @bold{portal
context} (accessed through the @geni-lib["portal.context"] object in the
@geni-lib["geni.portal"] module), and the @bold{request RSpec} created by calling
@geni-lib["portal.Context.makeRequestRSpec" 'func] on it.
These fundamental objects
are central to essentially all @(tb) @(geni-lib) profiles.

@margin-note{Another way to create a @geni-lib["rspec.pg.Request" 'id] RSpec
object is to call its constructuor, @geni-lib["geni.rspec.pg.Request"]
directly. We ask the @geni-lib["portal.Context" 'id] to create it for us so it
it is "bound" to the context and does not need to be explicitly passed to
other functions on the context}

Once the request object has been created, resources may be added to it by
calling methods on it like @geni-lib["rspec.pg.RawPC" 'func] or
@geni-lib["rspec.pg.LAN"].
In this example,
just a single node (created with the @geni-lib["rspec.pg.XenVM" 'func] constructor,
asking for a single VM identified by the name "node") is requested.

@margin-note{Most functions called on @geni-lib["rspec.pg.Request" 'id] objects
are not directly members of that class. Rather, they are loaded as "extensions"
by modules such as @geni-lib["geni.rspec.emulab"].}

The final action the @(geni-lib) script performs is to generate the
XML representation of the request RSpec, with the @geni-lib["portal.Context.printRequestRSpec" 'func]
call on the last line.  This has the effect of communicating the
description of all the resources requested by the profile back to
@(tb).

You will also notice that the profile begins with a string literal
(to be precise, it is a Python
@link["http://www.python.org/dev/peps/pep-0257/"]{docstring}).  The initial
text will also be used as the
@seclink["creating-the-profile"]{profile description}; the text following
the @tt{Instructions:} line will be used as the corresponding
@seclink["creating-the-profile"]{instructions}.  This documentation
is so important that adding the description to the profile is mandatory.
(Using a docstring like this is not the only way to produce the description
and instructions, although it is the most convenient.)

This simple example has now demonstrated all the important elements of
a @(geni-lib) profile.  The portal context and request RSpec objects,
the final @geni-lib["portal.Context.printRequestRSpec" 'func] call, and the docstring description
and instructions are ``boilerplate'' constructions, and you will probably
include similar or identical versions of them in every @(geni-lib)
profile you create unless you are doing something quite unusual. 

@section[#:tag "geni-lib-example-single-pc"]{A single physical host}

@profile-code-sample["PortalProfiles" "single-pc"]

As mentioned above, most of these simple examples consist of boilerplate
@(geni-lib) fragments, and indeed the portal context and request RSpec
operations are unchanged from the previous script.  The big difference,
though (other than the updated documentation) is that in this case the
@geni-lib["rspec.pg.RawPC" 'func] method is invoked on the
@geni-lib["rspec.pg.Request" 'id] object instead of
@geni-lib["rspec.igext.XenVM" 'func].  As you might expect, the new profile will request a
physical host instead of a virtual one.  (A side effect of using a
real machine is that it automatically comes with a unique public IP address,
where the VM used in the earlier example did not.  Profiles can
@seclink["public-ip-access"]{request public IP addresses} for VMs too,
though it does not happen by default.)

@section[#:tag "geni-lib-example-two-vm-lan"]{Two XenVM nodes with a link between them}

@profile-code-sample["PortalProfiles" "two-vm-lan"]

This example demonstrates two important @(geni-lib) concepts: first,
adding more than a single node to the request (which is a relatively
straightforward matter of calling more than one node object constructor,
being careful to use a different name each time).
It also shows how to add @bold{links} between nodes. It is possible to 
construct links and LANs in a more complicated manner (such as explicitly
creating @geni-lib["rspec.pg.Interface" 'id] objects to control interfaces),
but the simplest case is to supply the member nodes at the time the link
is created.

@section[#:tag "geni-lib-example-two-arm-lan"]{Two ARM64 servers in a LAN}

@profile-code-sample["PortalProfiles" "two-arm-lan"]

We now come to demonstrate requesting particular properties of nodes---until
now, all nodes had been either @geni-lib["rspec.igext.XenVM" 'id]s or @geni-lib["rspec.pg.RawPC" 'id]s and nothing further was said about them.
@(geni-lib) allows the user to
specify various details about the nodes, and this example makes use
of the @tt{hardware_type} property.  The @tt{hardware_type} can be set
to a string describing the type of physical machine onto which the logical
node can be mapped: in this case, the string is @tt{"m400"}, which means
a ProLiant Moonshot m400 host (an ARM64 server).  Obviously, such a
profile cannot be instantiated on a cluster without a sufficient quantity
of appropriate machines!  (This profile was written with the
Utah CloudLab cluster in mind.)  @(tb)
will indicate a list of suitable clusters when the user attempts to
instantiate the profile, so he or she is not forced to find one by
trial and error.

@section[#:tag "geni-lib-example-single-vm-sized"]{A VM with a custom size}

@profile-code-sample["PortalProfiles" "single-vm-sized"]

The earlier examples requesting VMs used the default number of cores, quantity
of RAM, and disk size. It's also possible to customize these value, as this
example does by setting the @geni-lib["rspec.igext.XenVM.cores" 'id], @geni-lib["rspec.igext.XenVM.ram" 'id],
and @geni-lib["rspec.igext.XenVM.disk" 'id] properties of the @geni-lib["rspec.pg.XenVM" 'id] class (which is a subclass of @geni-lib["rspec.pg.Node"].)


@section[#:tag "geni-lib-example-node-ips"]{Set a specific IP address on each node}

@code-sample["geni-lib-node-ips.py"]

This code sample assigns specific IP addresses to interfaces on the
nodes it

Some of the available qualifiers on requested nodes are specified by
manipulating attributes within the node (or interface) object directly.  The
@tt{hardware_type} in the
@seclink["geni-lib-example-two-arm-lan"]{previous example} is one
such case, as is the @tt{component_id} here.  (Note that the @tt{component_id}
in this example is applied to an interface, although it is also possible to
specify @tt{component_id}s on nodes, too, to request a particular
physical host.)

Other modifications to requests require dedicated methods.  For instance,
see the @geni-lib["rspec.pg.Interface.addAddress" 'func] calls made on each of the two interfaces above.
In each case, an @geni-lib["rspec.pg.IPv4Address" 'id] object is obtained from the appropriate
constructor (the parameters are the address and the netmask, respectively),
and then added to the corresponding interface.

@wireless-only{
    @section[#:tag "geni-lib-openepc"]{OpenEPC extensions}

    @(geni-lib) includes convenience functions to assist constructing
    profiles including OpenEPC core networks.  Although it is possible
    to instantiate OpenEPC using @(geni-lib) primitives only, the
    @tt{geni.rspec.emulab.pnext"} module hides some of the OpenEPC details, and
    allows more concise profile scripts.  An example of a profile
    using @tt{geni.rspec.emulab.pnext} is given below.

    @profile-code-sample["PhantomNet" "Basic-OpenEPC"]

    While the @tt{geni.portal} and @tt{geni.rspec.pg} modules will seem
    familiar from their use in general @(geni-lib) profiles, the
    @tt{pnext} module is new, and provides two main classes: @tt{EPCNode}
    and @tt{EPClan}.  These convenience classes provide facilities for
    adding OpenEPC nodes and networks, respectively.  Both classes
    are also added as extensions to @tt{Request}, so that (assuming
    @tt{rspec} is a valid @tt{Request} object) a simple call like
    @tt{mgmt = rspec.EPClan( PN.EPCLANS.MGMT )} will both create a LAN
    and add it to the request.

    For the @tt{EPCNode} class only, a factory method @tt{mkepcnode}
    is also defined.  (The advantage of using the factory method instead
    of invoking the @tt{EPCNode} constructor directly is that it allows
    specifying default hardware types and disk images to be used by
    all EPC nodes in the profile.  By default, @tt{mkepcnode} will
    use a binary OpenEPC disk image.)

    The @tt{mkepcnode} call has two mandatory parameters:
    the node identifier (just as seen in previous generic @(geni-lib)
    examples), and a new @italic{role}, specific to OpenEPC nodes.
    The role must be chosen from the following, each defined within
    @tt{geni.rspec.pnext.EPCROLES}:

    @(tabular #:style 'boxed #:sep (hspace 3) (list
	(list "ENABLERS")
	(list "PGW")
	(list "SGW_MME_SGSN")
	(list "CLIENT")
	(list "ENODEB")))

    Please consult the
    @hyperlink["https://wiki.phantomnet.org/wiki/phantomnet/openepc-general-information"]{OpenEPC documentation}
    for details about specific OpenEPC nodes.

    Once nodes are assigned, they should be connected with appropriate
    core network links.  This is the job of the @tt{EPClan} class, and
    each LAN should be created by invoking the @tt{EPClan} constructor
    with a single mandatory parameter chosen from the following list
    of network identifiers:

    @(tabular #:style 'boxed #:sep (hspace 3) (list
	(list "MGMT")
	(list "NET_A")
	(list "NET_B")
	(list "NET_C")
	(list "NET_D")
	(list "AN_LTE")))

    The
    @hyperlink["https://wiki.phantomnet.org/wiki/phantomnet/oepc-protected/openepc-tutorial-profile"]{OpenEPC Tutorial}
    gives an overview of the purpose and topology of each network.  Once
    the LAN has been created, its @tt{addMember} method may be invoked to
    describe the nodes it should connect.

    @section[#:tag "geni-lib-rflinks"]{RF communication}

    @(tb) provides facilities for radio frequency links well suited
    for wireless protocols, and @(geni-lib) support is provided for
    requesting them in the form of the @tt{pnext} module @tt{RFLink}
    class.  @tt{RFLink} behaves much like standard LANs, except that all
    connections are made point-to-point (and so any @tt{RFLink} must
    connect exactly two interfaces).

    @code-sample["trivialrf.py"]
}

@section[#:tag "geni-lib-example-os-install-scripts"]{Specify an operating system and set install and execute scripts}

@code-sample["geni-lib-os-install-scripts.py"]

This example demonstrates how to request @bold{services} for a node,
where @(tb) will automate some task as part of the profile instance
setup procedure.  In this case, two services are described (an
@bold{install} and an @bold{execute}).  This is a very common pair of services
to request together: the @geni-lib["geni.rspec.pg.Install" 'id] object describes a service which
retrieves a tarball from the location given in the @tt{url} parameter,
and installs it into the local filesystem as specified by @tt{path}.
(The installation occurs during node setup, upon the first boot after the
disk image has been loaded.)  The second service, described by the
@geni-lib["rspec.pg.Execute" 'id] object, invokes a @tt{shell} process to run the given
@tt{command}.  In this example (as is common), the command refers directly
to a file saved by the immediately preceding @geni-lib["geni.rspec.pg.Install" 'id] service.  This
behaviour works, because @(tb) guarantees that all @geni-lib["geni.rspec.pg.Install" 'id] services
complete before any @geni-lib["rspec.pg.Execute" 'id] services are started. The command executes
every time the node boots, so you can use it start daemons, etc. that are necessary for your
experiment.

@section[#:tag "geni-lib-example-parameters"]{Profiles with user-specified parameters}

@code-sample["geni-lib-parameters.py"]

Until now, all of the @(geni-lib) scripts have described profiles
which could also have been generated with @seclink["jacks"]{the Jacks GUI},
or even by writing a @seclink["rspecs"]{raw XML RSpec} directly.  However,
@(geni-lib) profiles offer an important feature unavailable by the
other methods: the ability to describe not a static request, but a
request ``template'' which is dynamically constructed based on a
user's choices at the time the profile is instantiated.  The
mechanism for constructing such profiles relies on profile @bold{parameters};
the @(geni-lib) script describes the set of parameters it will accept,
and then retrieves the corresponding values at instantiation time
and is free to respond by constructing arbitrarily different resource
requests based on that input.

The profile above accepts exactly one parameter---the number of VMs it
will instantiate.  You can see that the parameter is described via the
portal @geni-lib["portal.context"] object, using the @geni-lib["portal.Context.defineParameter" 'func] call shown
for the first time in this example. @geni-lib["portal.Context.defineParameter" 'func] must be
invoked once per profile parameter, and requires the parameter symbol,
parameter description, type, and default value respectively.  The
parameter symbol (@tt{"n"} in this example) must be unique within the
profile, and is used to retrieve the parameter's value during script
execution.  The description (@tt{"Number of VMs"}, in this case) will
be shown to prompt the user to supply a corresponding value when the
the profile is instantiated.  The type is used partly to constrain the
parameters to valid values, and partly to assist the instantiating
user by suggesting appropriate choices.  The list of valid types is:

@(tabular #:style 'boxed #:sep (hspace 3) (list
	(list (geni-lib "portal.ParameterType.INTEGER")
	      "Simple integer")
	(list (geni-lib "portal.ParameterType.STRING")
	      "Arbitrary (uninterpreted) string")
	(list (geni-lib "portal.ParameterType.BOOLEAN")
	      "True or False")
	(list (geni-lib "portal.ParameterType.IMAGE")
	      "URN to a disk image")
	(list (geni-lib "portal.ParameterType.AGGREGATE")
	      "URN of a GENI Aggregate Manager")
	(list (geni-lib "portal.ParameterType.NODETYPE")
	      "String specifying a type of node")
	(list (geni-lib "portal.ParameterType.BANDWIDTH")
	      "Floating-point number specifying bandwidth in kbps")
	(list (geni-lib "portal.ParameterType.LATENCY")
	      "Floating-point number specifying delay in ms")
	(list (geni-lib "portal.ParameterType.SIZE")
	      "Integer used for memory or disk size (e.g., MB, GB, etc.)")))

The last field is the default value of the parameter, and is required: not
only must the field itself contain a valid value, but the set of
@italic{all} parameters must be valid when each of them assumes the
default value.  (This is partly so that the portal can construct a
default topology for the profile without any manual intervention, and
partly so that unprivileged users, who may lack permission to supply
their own values, might still be able to instantiate the profile.)

After all parameters have been defined, the profile script may retrieve
the runtime values with the @geni-lib["portal.Context.bindParameters" 'func] method.  This will return
a Python class instance with one attribute for each parameter (with the
name supplied during the appropriate @geni-lib["portal.Context.defineParameter" 'func] call).  In the
example, the instance was assigned to @tt{params}, and therefore the
only parameter (which was called @tt{"n"}) is accessible as @tt{params.n}.

Of course, it may be possible for the user to specify nonsensical values
for a parameter, or perhaps give a set of parameters whose combination
is invalid.  A profile should detect error cases like these, and respond
by constructing a @geni-lib["portal.ParameterError"] object, which can be passed to
the portal context's @geni-lib["portal.Context.reportError" 'func] method to abort generation of
the RSpec.

@section[#:tag "local-diskpace"]{Add temporary local disk space to a node}

@code-sample["geni-lib-local-diskspace.py"]

This example demonstrates how to request @bold{extra temporary diskspace}
on a node.  The extra disk space is allocated from unused disk partitions,
and is mounted at a directory of your choosing. In the example code above,
we are asking for a 30GB file system mounted at @tt{"/mydata"}. Anything
you store in this file system is @bold{temporary}, and will be @bold{lost}
when your experiment is terminated.

The total size of the file systems you can ask for on a node, is obviously
limited to the amount of unused disk space available. The system does its
best to find nodes with enough space to fulfill the request, but in general
you are limited to temporary file systems in the 10s of, or a few hundred
GB.

@section[#:tag "local-dataset"]{Creating a reusable dataset}

@code-sample["geni-lib-local-dataset.py"]

In this example, we demonstrate how to create and use a @italic{dataset}. A
dataset is simply a snapshot of a temporary file system (see the previous
example) that has been saved to permanent storage, and reloaded on a node
(or nodes) in a different experiment. This type of dataset must be
explicitly saved (more on this below) in order to make changes permanent
(and available later). In the example code above, the temporary file system
will be loaded with the dataset specified by the URN.

But before you can use a dataset, you first have to create one using the
following steps:

@itemlist[ #:style 'ordered
    @instructionstep["Create an experiment"]{Create an experiment using the
    @seclink["local-diskpace"]{local diskspace example} above.}

    @instructionstep["Add your data"]{Populate the file system mounted at
    @tt{/mydata} with the data you wish to use in other experiments.}

    @instructionstep["Fill out the Create Dataset form"]{Click on the
    @tt{"Create Dataset"} option in the @tt{Actions} menu. This will bring
    up the form to create a new dataset. Choose a name for your dataset and
    optionally the project the dataset should be associated with. Be sure
    to select @bold{Image Backed} for the type. Then choose the the
    experiment, which node in the experiment, and which blockstore on the
    node. 

    @screenshot["create-imdataset.png"]
    }

    @instructionstep["Click ``Create" #:screenshot "snapshot-dataset.png"]{
        When you click the ``Create'' button, the file system on your node
        will be unmounted so that we can take a consistent snapshot of the
        contents. This process can take several minutes or longer,
        depending on the size of the file system. You can watch the
        progress on this page. When the progress bar reaches the ``Ready''
        stage, your new dataset is ready! It will now show up in your
        ``List Datasets'' list, and can be used in new experiments.}

    @instructionstep["Use your dataset" #:screenshot "show-dataset.png"]{
        To use your new dataset, you will need to reference it in your geni
        lib script (see the example code above). The @italic{name} of your
        dataset is a URN, and can be found on the information page for the
        dataset. From the @tt{Actions} menu, click on @tt{"List Datasets"},
        find the name of your dataset in the list, and click on it.}

    @instructionstep["Update your dataset"]{If you need to make
        changes to your dataset, simply start an experiment that uses your
        dataset. Make the changes you need to the file system mounted at
	@tt{/mydata}, and then use the @tt{"Modify"} button as shown in the
	previous step.}
]

@section[#:tag "geni-lib-debugging"]{Debugging @tt{geni-lib} profile scripts}

It is not necessary to instantiate the profile via the portal web interface
to test it.  Properly written profile scripts should work perfectly well
independent of the normal portal---the same @(geni-lib) objects will
behave sensibly when invoked from the command line.  As long as
@hyperlink["https://bitbucket.org/barnstorm/geni-lib/src"]{@tt{geni-lib} is
installed}, then invoking the Python interpreter on the profile script
should simply write the corresponding RSpec to standard output.  (Parameters,
if any, will assume their default values.)  For instance, if the script
in the previous example is saved as @tt{geni-lib-parameters.py}, then
the command:

@(elem #:style code-sample-style "python geni-lib-parameters.py")

will produce an RSpec containing three nodes (the default value for @tt{n}).
It is also possible to override the defaults on the command line by
giving the parameter name as an option, followed by the desired value:

@(elem #:style code-sample-style "python geni-lib-parameters.py --n 4")

The option @tt{--help} will list the available parameters and their
descriptions.