geni-lib.scrbl 22.3 KB
Newer Older
1 2 3 4
#lang scribble/manual

@(require "defs.rkt")

5
@(parse-sphinx-inventory (string-append (geni-lib-dir) "/docs/build/html"))
6

7
@title[#:tag "geni-lib" #:style main-style #:version apt-version]{Describing a profile with python and @tt{geni-lib}}
8

9 10
@(geni-lib) is a tool that allows users to generate @seclink["rspecs"]{RSpec} files from Python
code.  @(tb) offers the ability to use @(geni-lib)
11
scripts as the definition of a profile, rather then the more primitive
12
RSpec format. When you supply a @(geni-lib) script on the
13
@seclink["creating-profiles"]{Create Profile} page, your script is uploaded
14
to the server so that it can be executed in the @(geni-lib) environment. This
15 16 17 18 19
allows the script to be verified for correctness, and also produces the 
equivalent RSpec representation that you can view if you so desire.

@screenshot["create-geni-lib-empty.png"]

20
When you provide a @(geni-lib) script, you will see a slightly different set
21 22 23 24 25 26 27 28 29
of buttons on the @seclink["creating-profiles"]{Create Profile} page; next
to the ``Source'' button there is an ``XML'' button that will pop up the
RSpec XML for you to look at. The XML is read-only; if you want to change
the profile, you will need to change the python source code that is
displayed when you click on the ``Source'' button. Each time you change the
python source code, the script is uploaded to the server and processed. Be
sure to save your changes if you are 
@seclink["updating-profiles"]{updating an existing profile}.

30 31
The following examples demonstrate basic @(geni-lib) usage. More information
about @(geni-lib) and additional examples, can be found in the
32
@hyperlink["https://bitbucket.org/emulab/geni-lib/src"]{@tt{geni-lib} repository}.
33
The current version of @(geni-lib) used by @(tb) can be found in the @tt{0.9-EMULAB} branch.
34
Its full documentation is online as
35
@link["geni-lib/index.html"]{part of this manual}.
36

37
@section[#:tag "geni-lib-example-single-vm"]{A single XEN VM node}
38

39
@profile-code-sample["PortalProfiles" "single-vm"]
40

41
This example demonstrates the two most important objects: the @bold{portal
42
context} (accessed through the @geni-lib["portal.context"] object in the
43 44 45
@geni-lib["geni.portal"] module), and the @bold{request RSpec} created by calling
@geni-lib["portal.Context.makeRequestRSpec" 'func] on it.
These fundamental objects
46
are central to essentially all @(tb) @(geni-lib) profiles.
47

48 49 50 51 52 53 54 55 56 57
@margin-note{Another way to create a @geni-lib["rspec.pg.Request" 'id] RSpec
object is to call its constructuor, @geni-lib["geni.rspec.pg.Request"]
directly. We ask the @geni-lib["portal.Context" 'id] to create it for us so it
it is "bound" to the context and does not need to be explicitly passed to
other functions on the context}

Once the request object has been created, resources may be added to it by
calling methods on it like @geni-lib["rspec.pg.RawPC" 'func] or
@geni-lib["rspec.pg.LAN"].
In this example,
58
just a single node (created with the @geni-lib["rspec.pg.XenVM" 'func] constructor,
59 60 61 62 63
asking for a single VM identified by the name "node") is requested.

@margin-note{Most functions called on @geni-lib["rspec.pg.Request" 'id] objects
are not directly members of that class. Rather, they are loaded as "extensions"
by modules such as @geni-lib["geni.rspec.emulab"].}
64

65
The final action the @(geni-lib) script performs is to generate the
66
XML representation of the request RSpec, with the @geni-lib["portal.Context.printRequestRSpec" 'func]
67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82
call on the last line.  This has the effect of communicating the
description of all the resources requested by the profile back to
@(tb).

You will also notice that the profile begins with a string literal
(to be precise, it is a Python
@link["http://www.python.org/dev/peps/pep-0257/"]{docstring}).  The initial
text will also be used as the
@seclink["creating-the-profile"]{profile description}; the text following
the @tt{Instructions:} line will be used as the corresponding
@seclink["creating-the-profile"]{instructions}.  This documentation
is so important that adding the description to the profile is mandatory.
(Using a docstring like this is not the only way to produce the description
and instructions, although it is the most convenient.)

This simple example has now demonstrated all the important elements of
83
a @(geni-lib) profile.  The portal context and request RSpec objects,
84
the final @geni-lib["portal.Context.printRequestRSpec" 'func] call, and the docstring description
85
and instructions are ``boilerplate'' constructions, and you will probably
86
include similar or identical versions of them in every @(geni-lib)
87
profile you create unless you are doing something quite unusual. 
88

89
@section[#:tag "geni-lib-example-single-pc"]{A single physical host}
90

91
@profile-code-sample["PortalProfiles" "single-pc"]
92

93
As mentioned above, most of these simple examples consist of boilerplate
94
@(geni-lib) fragments, and indeed the portal context and request RSpec
95 96
operations are unchanged from the previous script.  The big difference,
though (other than the updated documentation) is that in this case the
Robert Ricci's avatar
Robert Ricci committed
97 98 99
@geni-lib["rspec.pg.RawPC" 'func] method is invoked on the
@geni-lib["rspec.pg.Request" 'id] object instead of
@geni-lib["rspec.igext.XenVM" 'func].  As you might expect, the new profile will request a
100 101 102 103 104 105
physical host instead of a virtual one.  (A side effect of using a
real machine is that it automatically comes with a unique public IP address,
where the VM used in the earlier example did not.  Profiles can
@seclink["public-ip-access"]{request public IP addresses} for VMs too,
though it does not happen by default.)

106
@section[#:tag "geni-lib-example-two-vm-lan"]{Two XenVM nodes with a link between them}
107

108
@profile-code-sample["PortalProfiles" "two-vm-lan"]
109

110
This example demonstrates two important @(geni-lib) concepts: first,
111 112
adding more than a single node to the request (which is a relatively
straightforward matter of calling more than one node object constructor,
113
being careful to use a different name each time).
114 115 116 117 118
It also shows how to add @bold{links} between nodes. It is possible to 
construct links and LANs in a more complicated manner (such as explicitly
creating @geni-lib["rspec.pg.Interface" 'id] objects to control interfaces),
but the simplest case is to supply the member nodes at the time the link
is created.
119

120
@section[#:tag "geni-lib-example-two-arm-lan"]{Two ARM64 servers in a LAN}
121

122
@profile-code-sample["PortalProfiles" "two-arm-lan"]
123

124
We now come to demonstrate requesting particular properties of nodes---until
125 126
now, all nodes had been either @geni-lib["rspec.igext.XenVM" 'id]s or @geni-lib["rspec.pg.RawPC" 'id]s and nothing further was said about them.
@(geni-lib) allows the user to
127 128 129 130 131 132 133
specify various details about the nodes, and this example makes use
of the @tt{hardware_type} property.  The @tt{hardware_type} can be set
to a string describing the type of physical machine onto which the logical
node can be mapped: in this case, the string is @tt{"m400"}, which means
a ProLiant Moonshot m400 host (an ARM64 server).  Obviously, such a
profile cannot be instantiated on a cluster without a sufficient quantity
of appropriate machines!  (This profile was written with the
134
Utah CloudLab cluster in mind.)  @(tb)
135 136 137 138
will indicate a list of suitable clusters when the user attempts to
instantiate the profile, so he or she is not forced to find one by
trial and error.

139 140
@section[#:tag "geni-lib-example-single-vm-sized"]{A VM with a custom size}

141
@profile-code-sample["PortalProfiles" "single-vm-sized"]
142 143 144 145 146 147 148

The earlier examples requesting VMs used the default number of cores, quantity
of RAM, and disk size. It's also possible to customize these value, as this
example does by setting the @geni-lib["rspec.igext.XenVM.cores" 'id], @geni-lib["rspec.igext.XenVM.ram" 'id],
and @geni-lib["rspec.igext.XenVM.disk" 'id] properties of the @geni-lib["rspec.pg.XenVM" 'id] class (which is a subclass of @geni-lib["rspec.pg.Node"].)


149
@section[#:tag "geni-lib-example-node-ips"]{Set a specific IP address on each node}
150

151
@profile-code-sample["PortalProfiles" "node-ips"]
152

Robert Ricci's avatar
Robert Ricci committed
153
This code sample assigns specific IP addresses to interfaces on the
Robert Ricci's avatar
Robert Ricci committed
154
nodes it requests.
Robert Ricci's avatar
Robert Ricci committed
155

156 157 158 159 160 161 162 163 164 165
Some of the available qualifiers on requested nodes are specified by
manipulating attributes within the node (or interface) object directly.  The
@tt{hardware_type} in the
@seclink["geni-lib-example-two-arm-lan"]{previous example} is one
such case, as is the @tt{component_id} here.  (Note that the @tt{component_id}
in this example is applied to an interface, although it is also possible to
specify @tt{component_id}s on nodes, too, to request a particular
physical host.)

Other modifications to requests require dedicated methods.  For instance,
166
see the @geni-lib["rspec.pg.Interface.addAddress" 'func] calls made on each of the two interfaces above.
Robert Ricci's avatar
Robert Ricci committed
167
In each case, an @geni-lib["rspec.pg.IPv4Address" 'id] object is obtained from the appropriate
168 169
constructor (the parameters are the address and the netmask, respectively),
and then added to the corresponding interface.
170

171
@wireless-only{
172 173
    @section[#:tag "geni-lib-openepc"]{OpenEPC extensions}

174
    @(geni-lib) includes convenience functions to assist constructing
175
    profiles including OpenEPC core networks.  Although it is possible
176
    to instantiate OpenEPC using @(geni-lib) primitives only, the
177
    @tt{geni.rspec.emulab.pnext"} module hides some of the OpenEPC details, and
178
    allows more concise profile scripts.  An example of a profile
179
    using @tt{geni.rspec.emulab.pnext} is given below.
180

181
    @profile-code-sample["PhantomNet" "Basic-OpenEPC"]
182 183

    While the @tt{geni.portal} and @tt{geni.rspec.pg} modules will seem
184
    familiar from their use in general @(geni-lib) profiles, the
185 186
    @tt{pnext} module is new, and provides two main classes: @tt{EPCNode}
    and @tt{EPClan}.  These convenience classes provide facilities for
187 188 189 190 191 192 193 194 195 196 197 198 199 200
    adding OpenEPC nodes and networks, respectively.  Both classes
    are also added as extensions to @tt{Request}, so that (assuming
    @tt{rspec} is a valid @tt{Request} object) a simple call like
    @tt{mgmt = rspec.EPClan( PN.EPCLANS.MGMT )} will both create a LAN
    and add it to the request.

    For the @tt{EPCNode} class only, a factory method @tt{mkepcnode}
    is also defined.  (The advantage of using the factory method instead
    of invoking the @tt{EPCNode} constructor directly is that it allows
    specifying default hardware types and disk images to be used by
    all EPC nodes in the profile.  By default, @tt{mkepcnode} will
    use a binary OpenEPC disk image.)

    The @tt{mkepcnode} call has two mandatory parameters:
201
    the node identifier (just as seen in previous generic @(geni-lib)
202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218
    examples), and a new @italic{role}, specific to OpenEPC nodes.
    The role must be chosen from the following, each defined within
    @tt{geni.rspec.pnext.EPCROLES}:

    @(tabular #:style 'boxed #:sep (hspace 3) (list
	(list "ENABLERS")
	(list "PGW")
	(list "SGW_MME_SGSN")
	(list "CLIENT")
	(list "ENODEB")))

    Please consult the
    @hyperlink["https://wiki.phantomnet.org/wiki/phantomnet/openepc-general-information"]{OpenEPC documentation}
    for details about specific OpenEPC nodes.

    Once nodes are assigned, they should be connected with appropriate
    core network links.  This is the job of the @tt{EPClan} class, and
219 220 221 222
    each LAN should be created by invoking the @tt{EPClan} constructor
    with a single mandatory parameter chosen from the following list
    of network identifiers:

223
    @(tabular #:style 'boxed #:sep (hspace 3) (list
224
	(list "MGMT")
225 226 227 228 229 230 231 232
	(list "NET_A")
	(list "NET_B")
	(list "NET_C")
	(list "NET_D")
	(list "AN_LTE")))

    The
    @hyperlink["https://wiki.phantomnet.org/wiki/phantomnet/oepc-protected/openepc-tutorial-profile"]{OpenEPC Tutorial}
233 234 235 236 237 238 239
    gives an overview of the purpose and topology of each network.  Once
    the LAN has been created, its @tt{addMember} method may be invoked to
    describe the nodes it should connect.

    @section[#:tag "geni-lib-rflinks"]{RF communication}

    @(tb) provides facilities for radio frequency links well suited
240
    for wireless protocols, and @(geni-lib) support is provided for
241 242 243 244
    requesting them in the form of the @tt{pnext} module @tt{RFLink}
    class.  @tt{RFLink} behaves much like standard LANs, except that all
    connections are made point-to-point (and so any @tt{RFLink} must
    connect exactly two interfaces).
245

246
    @profile-code-sample["PhantomNet" "trivialrf"]
247 248
}

249
@section[#:tag "geni-lib-example-os-install-scripts"]{Specify an operating system and set install and execute scripts}
250

251
@profile-code-sample["PortalProfiles" "os-install-scripts"]
252 253 254 255 256

This example demonstrates how to request @bold{services} for a node,
where @(tb) will automate some task as part of the profile instance
setup procedure.  In this case, two services are described (an
@bold{install} and an @bold{execute}).  This is a very common pair of services
257
to request together: the @geni-lib["geni.rspec.pg.Install" 'id] object describes a service which
258 259 260 261
retrieves a tarball from the location given in the @tt{url} parameter,
and installs it into the local filesystem as specified by @tt{path}.
(The installation occurs during node setup, upon the first boot after the
disk image has been loaded.)  The second service, described by the
262
@geni-lib["rspec.pg.Execute" 'id] object, invokes a @tt{shell} process to run the given
263
@tt{command}.  In this example (as is common), the command refers directly
264 265
to a file saved by the immediately preceding @geni-lib["geni.rspec.pg.Install" 'id] service.  This
behaviour works, because @(tb) guarantees that all @geni-lib["geni.rspec.pg.Install" 'id] services
266 267 268
complete before any @geni-lib["rspec.pg.Execute" 'id] services are started. The command executes
every time the node boots, so you can use it start daemons, etc. that are necessary for your
experiment.
269 270 271

@section[#:tag "geni-lib-example-parameters"]{Profiles with user-specified parameters}

272
@profile-code-sample["PortalProfiles" "parameterized"]
273

274
Until now, all of the @(geni-lib) scripts have described profiles
275 276
which could also have been generated with @seclink["jacks"]{the Jacks GUI},
or even by writing a @seclink["rspecs"]{raw XML RSpec} directly.  However,
277
@(geni-lib) profiles offer an important feature unavailable by the
278 279 280 281
other methods: the ability to describe not a static request, but a
request ``template'' which is dynamically constructed based on a
user's choices at the time the profile is instantiated.  The
mechanism for constructing such profiles relies on profile @bold{parameters};
282
the @(geni-lib) script describes the set of parameters it will accept,
283 284 285 286 287
and then retrieves the corresponding values at instantiation time
and is free to respond by constructing arbitrarily different resource
requests based on that input.

The profile above accepts exactly one parameter---the number of VMs it
288
will instantiate.  You can see that the parameter is described via the
289
portal @geni-lib["portal.context"] object, using the @geni-lib["portal.Context.defineParameter" 'func] call shown
290
for the first time in this example. @geni-lib["portal.Context.defineParameter" 'func] must be
291 292 293 294 295 296 297 298 299 300 301
invoked once per profile parameter, and requires the parameter symbol,
parameter description, type, and default value respectively.  The
parameter symbol (@tt{"n"} in this example) must be unique within the
profile, and is used to retrieve the parameter's value during script
execution.  The description (@tt{"Number of VMs"}, in this case) will
be shown to prompt the user to supply a corresponding value when the
the profile is instantiated.  The type is used partly to constrain the
parameters to valid values, and partly to assist the instantiating
user by suggesting appropriate choices.  The list of valid types is:

@(tabular #:style 'boxed #:sep (hspace 3) (list
302
	(list (geni-lib "portal.ParameterType.INTEGER")
303
	      "Simple integer")
304
	(list (geni-lib "portal.ParameterType.STRING")
305
	      "Arbitrary (uninterpreted) string")
306
	(list (geni-lib "portal.ParameterType.BOOLEAN")
307
	      "True or False")
308
	(list (geni-lib "portal.ParameterType.IMAGE")
309
	      "URN to a disk image")
310
	(list (geni-lib "portal.ParameterType.AGGREGATE")
311
	      "URN of a GENI Aggregate Manager")
312
	(list (geni-lib "portal.ParameterType.NODETYPE")
313
	      "String specifying a type of node")
314
	(list (geni-lib "portal.ParameterType.BANDWIDTH")
315
	      "Floating-point number specifying bandwidth in kbps")
316
	(list (geni-lib "portal.ParameterType.LATENCY")
317
	      "Floating-point number specifying delay in ms")
318
	(list (geni-lib "portal.ParameterType.SIZE")
319 320 321 322 323 324 325 326 327 328 329
	      "Integer used for memory or disk size (e.g., MB, GB, etc.)")))

The last field is the default value of the parameter, and is required: not
only must the field itself contain a valid value, but the set of
@italic{all} parameters must be valid when each of them assumes the
default value.  (This is partly so that the portal can construct a
default topology for the profile without any manual intervention, and
partly so that unprivileged users, who may lack permission to supply
their own values, might still be able to instantiate the profile.)

After all parameters have been defined, the profile script may retrieve
330
the runtime values with the @geni-lib["portal.Context.bindParameters" 'func] method.  This will return
331
a Python class instance with one attribute for each parameter (with the
332
name supplied during the appropriate @geni-lib["portal.Context.defineParameter" 'func] call).  In the
333 334 335 336 337 338
example, the instance was assigned to @tt{params}, and therefore the
only parameter (which was called @tt{"n"}) is accessible as @tt{params.n}.

Of course, it may be possible for the user to specify nonsensical values
for a parameter, or perhaps give a set of parameters whose combination
is invalid.  A profile should detect error cases like these, and respond
339 340
by constructing a @geni-lib["portal.ParameterError"] object, which can be passed to
the portal context's @geni-lib["portal.Context.reportError" 'func] method to abort generation of
341 342
the RSpec.

Leigh Stoller's avatar
Leigh Stoller committed
343 344
@section[#:tag "local-diskpace"]{Add temporary local disk space to a node}

345
@profile-code-sample["PortalProfiles" "local-diskspace"]
Leigh Stoller's avatar
Leigh Stoller committed
346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361

This example demonstrates how to request @bold{extra temporary diskspace}
on a node.  The extra disk space is allocated from unused disk partitions,
and is mounted at a directory of your choosing. In the example code above,
we are asking for a 30GB file system mounted at @tt{"/mydata"}. Anything
you store in this file system is @bold{temporary}, and will be @bold{lost}
when your experiment is terminated.

The total size of the file systems you can ask for on a node, is obviously
limited to the amount of unused disk space available. The system does its
best to find nodes with enough space to fulfill the request, but in general
you are limited to temporary file systems in the 10s of, or a few hundred
GB.

@section[#:tag "local-dataset"]{Creating a reusable dataset}

362
@profile-code-sample["PortalProfiles" "longterm-dataset"]
Leigh Stoller's avatar
Leigh Stoller committed
363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415

In this example, we demonstrate how to create and use a @italic{dataset}. A
dataset is simply a snapshot of a temporary file system (see the previous
example) that has been saved to permanent storage, and reloaded on a node
(or nodes) in a different experiment. This type of dataset must be
explicitly saved (more on this below) in order to make changes permanent
(and available later). In the example code above, the temporary file system
will be loaded with the dataset specified by the URN.

But before you can use a dataset, you first have to create one using the
following steps:

@itemlist[ #:style 'ordered
    @instructionstep["Create an experiment"]{Create an experiment using the
    @seclink["local-diskpace"]{local diskspace example} above.}

    @instructionstep["Add your data"]{Populate the file system mounted at
    @tt{/mydata} with the data you wish to use in other experiments.}

    @instructionstep["Fill out the Create Dataset form"]{Click on the
    @tt{"Create Dataset"} option in the @tt{Actions} menu. This will bring
    up the form to create a new dataset. Choose a name for your dataset and
    optionally the project the dataset should be associated with. Be sure
    to select @bold{Image Backed} for the type. Then choose the the
    experiment, which node in the experiment, and which blockstore on the
    node. 

    @screenshot["create-imdataset.png"]
    }

    @instructionstep["Click ``Create" #:screenshot "snapshot-dataset.png"]{
        When you click the ``Create'' button, the file system on your node
        will be unmounted so that we can take a consistent snapshot of the
        contents. This process can take several minutes or longer,
        depending on the size of the file system. You can watch the
        progress on this page. When the progress bar reaches the ``Ready''
        stage, your new dataset is ready! It will now show up in your
        ``List Datasets'' list, and can be used in new experiments.}

    @instructionstep["Use your dataset" #:screenshot "show-dataset.png"]{
        To use your new dataset, you will need to reference it in your geni
        lib script (see the example code above). The @italic{name} of your
        dataset is a URN, and can be found on the information page for the
        dataset. From the @tt{Actions} menu, click on @tt{"List Datasets"},
        find the name of your dataset in the list, and click on it.}

    @instructionstep["Update your dataset"]{If you need to make
        changes to your dataset, simply start an experiment that uses your
        dataset. Make the changes you need to the file system mounted at
	@tt{/mydata}, and then use the @tt{"Modify"} button as shown in the
	previous step.}
]

416 417 418 419
@section[#:tag "geni-lib-debugging"]{Debugging @tt{geni-lib} profile scripts}

It is not necessary to instantiate the profile via the portal web interface
to test it.  Properly written profile scripts should work perfectly well
420
independent of the normal portal---the same @(geni-lib) objects will
421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438
behave sensibly when invoked from the command line.  As long as
@hyperlink["https://bitbucket.org/barnstorm/geni-lib/src"]{@tt{geni-lib} is
installed}, then invoking the Python interpreter on the profile script
should simply write the corresponding RSpec to standard output.  (Parameters,
if any, will assume their default values.)  For instance, if the script
in the previous example is saved as @tt{geni-lib-parameters.py}, then
the command:

@(elem #:style code-sample-style "python geni-lib-parameters.py")

will produce an RSpec containing three nodes (the default value for @tt{n}).
It is also possible to override the defaults on the command line by
giving the parameter name as an option, followed by the desired value:

@(elem #:style code-sample-style "python geni-lib-parameters.py --n 4")

The option @tt{--help} will list the available parameters and their
descriptions.