creating-profiles.scrbl 18.6 KB
Newer Older
Robert Ricci's avatar
Robert Ricci committed
1 2
#lang scribble/manual

Robert Ricci's avatar
Robert Ricci committed
3 4
@(require "defs.rkt")

Robert Ricci's avatar
Robert Ricci committed
@title[#:tag "creating-profiles" #:version apt-version]{Creating Profiles}
Robert Ricci's avatar
Robert Ricci committed

7 8 9 10 11 12 13 14 15 16 17 18
    Using @(tb), you can share your research artifacts with others by creating
    your own profiles.

    In @(tb), a profile captures an entire cloud environment---the software
    needed to run a particular cloud, plus a description of the hardware
    (including network topology) that the cloud software stack should run on.

When you create a new profile, you are creating a new
Robert Ricci's avatar
Robert Ricci committed
@seclink["rspecs"]{RSpec}, and, usually, creating one or more
20 21
@seclink["disk-images"]{disk images} that are referenced by that RSpec.  When
someone uses your profile, they will get their own
Robert Ricci's avatar
Robert Ricci committed
@seclink["experiments"]{experiment} that boots up the resources (virtual or
23 24 25
physical) described by the RSpec. It is common to create the resource specifification 
for profiles using a @seclink["jacks"]{GUI} or by writing a
@seclink["geni-lib"]{python script} rather than dealing with RSpecs directly.

Robert Ricci's avatar
Robert Ricci committed
@section[#:tag "creating-from-existing"]{Creating a profile from an existing one}

The easiest way to create a new profile is by cloning or copying an existing one and
30 31
customizing it to your needs. The basic steps are:

32 33 34
When you @bold{clone} an experiment, you are taking an existing experiment, including a snapshot of the disk, and creating a new profile based on it. The new profile will be identical to the profile that experiment was based on in all other respects. Cloning only works on experiments with a single node.

If you @bold{copy} a profile, you are creating a new profile that is identical in every way to an existing profile. You may or may not have a running experiment using the source profile. And if you do have a running experiment, it does not impact the copy. After copying a profile, you can then modify it for your own use. And if you instantiate the copy, you can then take snapshots of disk images and use them in future version of your copy. Any profile that you have access to may be copied.

Robert Ricci's avatar
Robert Ricci committed
@subsection[#:tag "profile-creation-preparation"]{Preparation and precautions}
37 38 39

To create profiles, you need to be a @seclink["register"]{registered user}.

Cloning a profile can take a while, so we recommend that you
Robert Ricci's avatar
Robert Ricci committed
41 42
@seclink["experiments"]{extend your experiment} while creating it, and contact
us if you are worried your experiment might expire before you're done creating
43 44 45
your profile. We also strongly recommend testing your profile fully before
terminating the experiment you're creating it from.

46 47 48 49 50 51 52 53
When cloning, your home directory is @bold{not} included in the disk image snapshot! You will need to install your code and data elsewhere in the image. We recommend /local/. Keep in mind that others who use your profile are going to have their own accounts, so make sure that nothing in your image makes assumptions about the username, home directory, etc. of the user running it.

When cloning, be aware the only the contents of disk (not running process, etc.) are stored as part of the profile, and as part of the creation process, your node(s) will be rebooted in order to take consistent snapshots of the disk.

For the time being, cloning only works for single-node profiles; we will add support for multi-node profiles in the future.

When copying a profile, remember that the disk images of a currently running experiment are not saved. If you want to customize the disk images using copy, you must copy the profile first, then instantiate your copy, then take snapshots of the modified disk image in your experiment.

54 55 56 57 58 59 60 61 62
    If you want your profile to be usable by @seclink["guest-users"]{guest
    users}, keep in mind that there are @seclink["guest-users"]{several
    restrictions} placed on them; among these is the fact that they are not
    allowed to make outgoing connections, meaning that the experiment must be
    self-contained. For example, they will not be able to download software or
    datasets from within the experiment---it should all be contained as part of
    the profile.

@subsection[#:tag"cloning-a-profile"]{Cloning a Profile}
65 66

@itemlist[ #:style 'ordered

68 69 70 71 72 73 74 75 76 77 78
    @instructionstep["Create an experiment"]{Create an experiment using the
    profile that is most similar to the one you want to build. Usually, this
    will be one of our facility-provided profiles with a generic installation
    of Linux.}

    @instructionstep["Set up the node the way you want it"]{Log into the node
    and install your software, datasets, packages, etc. Note the
    @seclink["profile-creation-preparation"]{caveat above} that it needs to be
    installed somewhere outside of your home directory, and should not be tied
    to your user account.}

Robert Ricci's avatar
Robert Ricci committed
    @instructionstep["Clone the experiment to create a new profile"
80 81 82 83 84
                     #:screenshot "clone-button.png" ]{
    While you are logged in, the experiment page for your active experiments
    will have a ``clone'' button. Clicking this button will create a new
    profile based on your running experiment.

Robert Ricci's avatar
Robert Ricci committed
85 86 87 88
    Specifically, the button creates a copy of the @seclink["rspecs"]{RSpec}
    used to create the experiment, passes it to the form used to create new
    profiles, and helps you create a @seclink["disk-images"]{disk image} from
    your running experiment.
89 90

91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123
    @instructionstep["Create Profile"]{
      You will be taken to a complete profile form and should fill it out as described below.

@subsection[#:tag"copying-a-profile"]{Copying a Profile}

@itemlist[ #:style 'ordered

    @instructionstep["Choose a profile"
                     #:screenshot "begin-experiment.png" ]{

    Find the profile you wish to clone using the “Start Experiment” selector. Then you can click “Show Profile” to clone the profile directly or you can instantiate the profile if you wish to create an experiment first. Both profiles themselves and active experiments can be copied.

    @instructionstep["Copy the profile or experiment"
                     #:screenshot "copy-button.png" ]{
    While logged in, both your experiment page and the show profile page will have a copy button. Clicking this button will create a profile based on that profile or experiment.

    This button only copies the rspec or genilib script. No state in the active experiment is preserved.

    @instructionstep["Create Profile"]{
      You will be taken to a complete profile form and should fill it out as described below.

@subsection[#:tag "creating-the-profile"]{Creating the Profile}

    After copying or cloning a profile (see above) or selecting the menu option to create a new profile from scratch, you will need to fill out the profile creation form in order to complete the creation process.

@itemlist[ #:style 'ordered

    @instructionstep["Fill out information for the new profile"]{
125 126 127 128
    After clicking on the ``clone'' button, you will see a form that
    allows you to view and edit the basic information associated with your

129 130

Robert Ricci's avatar
Robert Ricci committed
131 132 133
    Each profile must be associated with a @seclink["projects"]{project}. If
    you're a member of more than one project, you'll need to select which one
    you want the profile to belong to.
134 135 136 137

    Make sure to edit the profile's Description and Instructions.

    The ``Description'' is the text that users will see when your profile is
    listed in @(tb), when the user is selecting which profile to use. It is also
Robert Ricci's avatar
Robert Ricci committed
    displayed when @seclink["sharing-profiles"]{following a direct link to your
140 141 142 143 144 145 146 147 148 149 150
    profile}. It should give the reader a brief description of what they will
    get if they create an experiment with this profile. If the profile is
    associated with a paper or other publication, this is a good place to
    mention that.  @seclink["markdown"]{Markdown} markup, including hyperlinks,
    are allowed in the profile description.

    The ``Instructions'' text is displayed on the experiment page after the
    user has created an experiment using the profile. This is a good place
    to tell them where the code and data can be found, what scripts they
    might want to run, etc. Again, @seclink["markdown"]{Markdown} is allowed.

Robert Ricci's avatar
Robert Ricci committed
    The ``Steps'' section allows you to create a @seclink["tours"]{tour} of your
152 153 154 155 156 157
    profile, which is displayed after a user creates an experiment with it.
    This feature is mostly useful if your profile contains more than one
    node, and you wish to explain to the user what the purpose of each node

    You have the option of making your profile usable to anyone, only
158 159
    registered @(tb) users, or members of your project. Regardless of the
    setting you chose here, @(tb) will also give you a
Robert Ricci's avatar
Robert Ricci committed
    @seclink["sharing-profiles"]{direct link} that you can use to share your
Robert Ricci's avatar
Robert Ricci committed
    profile with others of your choosing.
162 163

Robert Ricci's avatar
Robert Ricci committed
164 165 166 167 168 169 170 171 172
    @instructionstep["Click ``Create''" #:screenshot "image-creating.png"]{
        When you click the ``Create'' button, your node will be rebooted,
        so that we can take a consistent snapshot of the disk contents. This
        process can take several minutes or longer, depending on the size
        of your disk image. You can watch the progress on this page. When
        the progress bar reaches the ``Ready'' stage, your new profile is
        ready! It will now show up in your ``My Profiles'' list.}

    @instructionstep["Test your profile"]{Before terminating your experiment
    (or letting it expire), we strongly recommend testing out the new profile. If
Robert Ricci's avatar
Robert Ricci committed
    you elected to make it publicly visible, it will be listed in the profile
Robert Ricci's avatar
Robert Ricci committed
    selection dialog on the front page of @url[(apturl)]. If not,
Robert Ricci's avatar
Robert Ricci committed
176 177 178 179
    you can instantiate it from the listing in your ``My Profiles'' page. If
    the profile will be used by guest users, we recommend testing it as one
    yourself: log out, and instantiate it using a different username (you will
    also have to use an alternate email address.}
Robert Ricci's avatar
Robert Ricci committed

181 182 183 184 185
    @instructionstep["Share your profile"]{Now that your profile is working,
    you can @seclink["sharing-profiles"]{share it} with others by sending
    them direct links, putting links on your webpage or in papers, etc. See
    ``@secref["sharing-profiles"]'' for more details.}
Robert Ricci's avatar
Robert Ricci committed

Robert Ricci's avatar
Robert Ricci committed
@subsection[#:tag "updating-profiles"]{Updating a profile}

189 190 191 192 193 194 195 196 197 198 199 200 201 202 203
You can update the metadata associated with a profile at any time by going to
the ``My Profiles'' page and clicking on the name of the profile to go to the
profile page. On this page, you can edit any of the text fields (Description,
Instructions, etc.), change the permissions, etc.

@margin-note{As with cloning a profile, this snapshot feature currently only
works with single-node profiles.}

If you need to update the contents of the disk image in the profile, simply
create a new experiment from the profile. (You will only see this button on
experiments created from profiles that you own.) Once your experiment is ready,
you will see a ``Snapshot'' button on the experiment page. Log into your node,
get the disk changed the way you want, and click the button.


This button kicks off the same image creation process that occurs during
Robert Ricci's avatar
Robert Ricci committed
206 207
@seclink["creating-from-existing"]{cloning a profile}. Once it's finished, any
new experiments created from the profile will use the new image.
208 209 210 211

As with creating a new profile, we recommend testing the profile before letting
your experiment expire. If something goes wrong, we do keep one previous image
file for each profile; currently, the only way to get access to this backup
Robert Ricci's avatar
Robert Ricci committed
is to @seclink["getting-help"]{contact us}.

214 215 216
@section[#:tag "jacks"]{Creating a profile with a GUI}

@(tb) embeds the Jacks GUI for simple creation of small profiles. Jacks can
Robert Ricci's avatar
Robert Ricci committed
217 218
be accessed by clicking the ``topology'' button on the profile creation or
editing page. Jacks is designed to be simple, and to ensure that the topologies
219 220 221 222 223 224
drawn can be instantiated on the @seclink["hardware"]{hardware available}.
Thus, after making certain choices (such as picking an operating system image)
you may find that other choices (such as the node type) become limited.


Robert Ricci's avatar
Robert Ricci committed
Jacks has a ``palette'' on the left side, giving the set of node types 
(such as physical or virtual machines) that are available. Dragging a node
Robert Ricci's avatar
Robert Ricci committed
from this palette onto the larger canvas area on the right adds it to
228 229 230 231 232 233 234 235 236 237 238 239
the topology. To create a link between nodes, move the mouse near the first
node, and a small black line will appear. Click and drag to the second node
to complete the link. To create a LAN (multi-endpoint link), create a link
between two nodes, then drag links from other nodes to the small grey box
that appears in the middle of the original link.


To edit the properties of a node or link, select it by clicking on its icon
on the canvas. The panel on the left side will be replace by a property
editor that will allow you to to set the @seclink["disk-images"]{disk image}
for the node, set commands to be run when the node boots, etc. To unselect 
Robert Ricci's avatar
Robert Ricci committed
the current node or link, and return to the palette on the left, simply
241 242
click a blank area of the canvas.

243 244 245
@section[#:tag "repo-based-profiles"]{Repository-Based Profiles}

You can turn any public @tt{git} repository (including those hosted on GitHub)
into a @(tb) profile. Simply place a @tt{geni-lib} script
247 248 249 250 251 252 253 254
named @tt{} into the top-level directory of your repository. When
you create a new profile, you can provide the URL for your repository. The URL
needs to be a @tt{http://} or @tt{https://} URL, and the CloudLab portal needs
to be able to clone the repository without authentication.

    Note that CloudLab is not a @tt{git} hosting service; while we do keep a
    cache of your repository, we don't guarantee that the profile will continue
255 256
    to work if the original repository becomes unavailable. We also have limits
    on the size of the repositories that we will clone.
257 258 259 260 261 262 263 264 265 266 267

When you intantiate a repository-based profile, the repository will be cloned
into the directory @tt{/local/repository} on all nodes in the experiment. This
means that you can keep source code, startup scripts, etc. in your repository
and refrence them from @tt{}. @(tb) sets the @tt{pull} URL for all
of these clones to be your ``upstream'' repository, and attempts to set 
a suitable @tt{push} URL for (it assumes that the hosting service uses @tt{ssh}
for pushes, and uses the @tt{git@"@"<hostname>:user/repo} convention). As a
result, @tt{git pull} and @tt{git push} should be connected to your repository.

268 269 270 271 272
There is an example repository on GitHub at
@hyperlink[""]{@tt{}}; if you don't already have a @tt{git} repository created, a good
way to get started is to for this one and crate a new profile pointing at your

273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292
    Pushing to your repository is still govered by the authentication and
    permissions of your @tt{git} hosting service, so others using your profile
    will not be able to push to your repository.

@subsection{Updating Repository-Based Profiles}

By default, the CloudLab profile does @bold{not} automatically update whenever
you push to your upstream repository; this means that people instantiating your
profile see the repository as it existed at the time @(tb) last pulled from it.

You can @bold{manually} cause @(tb) to pull from your repository using the
``Update'' button on the profile management page.

You can also set up @(tb) to @bold{automatically} pull from your repository
whever it is updating. To do so, you will need to set up a ``web hook'' on 
the service that hosts your @tt{git} repository. @(tb) currently supports
webhooks for @tt{}, @tt{}, and sites hosted using
GitLab (including both @tt{} and self-hosted GitLab installations.)
Robert Ricci's avatar
Robert Ricci committed
293 294
See the ``push URL'' in the Repository Info panel on the left side of the
profile page for the webhook
295 296 297 298 299 300
URL, and use the ``information'' icon next to it to get specific instructions
for setting webhook on each service. Once you have set the webhook up, every
time you push to your repository, your hosting service will let @(tb) know
that it should automatically initiate a pull. (This will not be instantaneous,
but should complete quickly in most cases.)

@subsection{Branches and Tags in Repository-Based Profiles}
302 303 304 305 306 307 308 309 310

By default, repository-based profiles will be instaniated from the @tt{master}
branch. At the bottom of the profile page, you will also find a list of all
branches and tags in the repository, and can instantiate the version contained
in any of them. Branches can be used for development work that is not yet ready
to be come the @tt{master} (default) version of the profile, and tags can be
used to mark specific versions of the profiles that were used for specific
papers or course assignments, for example.

@section[#:tag "creating-from-scratch"]{Creating a profile from scratch}

@(tb) profiles are described by
314 315 316 317 318
RSpecs}. You can create a profile directly from an RSpec by using the 
``Create Profile'' option from the ``Actions'' menu. Note that you cannot edit
the text fields until you upload an RSpec, as these fields edit (in-browser)
fields in the RSpec.
319 320 321 322

@section[#:tag "sharing-profiles"]{Sharing Profiles}

If you chose to make your profile publicly visible, it will show up in the main
``Select Profile'' list on @url[(apturl)].  @(tb) also gives you direct links to
324 325 326
your profiles so that you can share them with others, post them on your
website, publish them in papers, etc. The link can be found on the profile's
detail page, which is linked for your ``My Profiles'' page. If you chose to
Robert Ricci's avatar
Robert Ricci committed
make your profile accessible to anyone, the link will take the form
Robert Ricci's avatar
Robert Ricci committed
@code[(apturl "/p/<project-id>/<profile-id>")]. If you didn't make the profile
public, the URL will have the form @code[(apturl "/p/<UUID>")], where
330 331 332
@code{UUID} is a 128-bit number so that the URL is not guessable. You can still
share this URLs with anyone you want to have access to the profile---for
example, to give it to a collaborator to try out your work before publishing.
333 334 335 336 337 338 339 340 341 342 343 344 345 346

@section[#:tag "versioned-profiles"]{Versioned Profiles}

Profiles are @italic{versioned} to capture the evolution
of a profile over time. When @seclink["updating-profiles"]{updating profiles},
the result is be a new version that does not (entirely) replace the profile
being updated.

When @seclink["sharing-profiles"]{sharing a profile}, you are given two links
to share. One link will take the user to the most recent version of the profile
that exists at the time they click the link. This is the most appropriate
option in most cases. There is also a link that takes one to a specific version
of the profile. This link is most useful for publication in papers or other
cases in which reproducability with the exact same environment is a concnern.