#lang scribble/manual @(require "defs.rkt") @title[#:tag "chef-tutorial" #:version apt-version]{@(tb) Chef Tutorial} This tutorial will walk you through the process of creating and using an instance of the Chef configuration management system on @(tb). @not-clab{ Since this tutorial was originally developed for CloudLab, you will find the screenshots below that have the CloudLab name and logo in them. When you use @(tb) to build Chef, the process is exactly the same; the only difference is that you will see the @(tb) name and logo instead in the portal pages you will open and use. } @margin-note{Chef is both the name of a company and the name of a popular modern configuration management system written in Ruby and Erlang. A large variety of tutorials, articles, technical docs and training opportunities is available at the @link["https://learn.chef.io/"]{Chef official website}. Also, refer to the @link["https://www.chef.io/customers/"]{Customer Stories} page to see how Chef is used in production environments, including very large installations (e.g., at Facebook, Bloomberg, and Yahoo!).} @section{Objectives} In the process of taking this tutorial, you will learn to: @itemlist[ @item{Create your own instance of Chef using a pre-defined profile} @item{Explore profile parameters allowing to customize components of your Chef deployments} @item{Access monitoring and management capabilities provided by Chef} @item{Use Chef to perform two exercises: @itemlist[ @item{Install and configure NFS, the Network File System, on experiment nodes} @item{Install and configure an instance of the Apache web server, as well as run a benchmark against it} ] } @item{Terminate your experiment} @item{Learn where to get more information} ] @section{Motivation} This tutorial will demonstrate how experiments can be managed on @(tb), as well as show how experiment resources can be administered using Chef. By following the instructions provided below, you will learn how to take advantage of the powerful features of Chef for configuring multi-node software environments. The exercises included in this tutorial are built around simple but realistic configurations. In the process of recreating these configurations on nodes running default images, you will explore individual components of Chef and follow the configuration management workflow applicable to more complex configurations and experiments. @section{Prerequisites} This tutorial assumes that: @itemlist[ @item{You have an existing account on @bold{either}: @itemlist[ @item{@(tb) (Instructions for getting an account can be found @seclink["register"]{here}.)} @item{The @link["https://portal.geni.net"]{GENI portal}. (Instructions for getting an account can be found @link["http://groups.geni.net/geni/wiki/SignMeUp"]{here}.)} ]} ] @include-section["tutorial-login-common.scrbl"] @section[#:tag "chef-tutorial-body"]{Launching Chef Experiments} Once you have logged in to @(tb), you will ``instantiate'' a @seclink["profiles"]{``profile''} to create an @seclink["experiments"]{experiment}. (An experiment in @(tb) is similar to a @link["http://groups.geni.net/geni/wiki/GENIConcepts#Slice"]{``slice''} in GENI.) Profiles are @(tb)'s way of packaging up configurations and experiments so that they can be shared with others. Each experiment is separate: the experiment that you create for this tutorial will be an instance of a profile provided by the facility, but running on resources that are dedicated to you, which you have complete control over. This profile uses local disk space on the nodes, so anything you store there will be lost when the experiment terminates. @margin-note{The Chef cluster we are building in this tutorial is very small, but @(tb) has @seclink["hardware"]{large clusters} that can be used for larger-scale experiments.} For this tutorial, we will use a profile that launches a Chef cluster --- a set of interconnected nodes running Chef components such as Chef server, workstation, clients. The @(tb) staff have built this profile by scripting installation procedures for these components. The developed scripts will run on the experiment nodes after they boot and customize them (create necessary user accounts, install packages, establish authentication between the nodes, etc.) to create a fully functional Chef cluster with a multi-node, production-like structure. See this manual's @seclink["profiles"]{section on profiles} for more information about how they work. @itemlist[#:style 'ordered @instructionstep["Start Experiment"]{ @clab-screenshot["tutorial/start-experiment-menu.png"] After logging in, you are taken to your main status @link[@apt-url["user-dashboard.php"]]{dashboard}. Select ``Start Experiment'' from the ``Experiments'' menu. } @instructionstep["Select a profile"]{ By default, the ``Start an Experiment'' page suggests launching the OpenStack profile which is discribed in detail in the @seclink["openstack-tutorial"]{OpenStack tutorial}. Go to the list of available profile by clicking ``Change Profile'': @clab-screenshot["chef-tutorial/change-profile.png"] Find the profile by typing @bold{ChefCluster} in the search bar. Then, select the profile with the specified name in the list displayed below the search bar. A 2-node preview should now be shown along with high-level profile information. Click ``Select Profile'' at the bottom of the page: @clab-screenshot["chef-tutorial/find-chefcluster.png"] After you select the correct profile, click ``Next'': @clab-screenshot["chef-tutorial/instantiate-next.png"] } @instructionstep["Set parameters" #:screenshot "chef-tutorial/params-next.png" #:screenshot-where "clab"]{ Profiles in @(tb) can have @emph{parameters} that affect how they are configured; for example, this profile has parameters that allow you to set the number of client nodes, specify the repository with the infrastructure code we plan to use, obtain copies of the application-specific infrastructure code developed by the global community of Chef developers, etc. For this tutorial, we will leave all parameters at their defaults and just click ``Next''. } @instructionstep["Choose experiment name"]{ You may optionally give your experiment a meaningful name, e.g., ``chefdemo''. This is useful if you have many experiments running at once. @clab-screenshot["chef-tutorial/experiment-name.png"] } @instructionstep["Select a cluster" #:screenshot "chef-tutorial/select-cluster.png" #:screenshot-where "clab"]{ @(tb) has multiple clusters available to it. Some profiles can run on any cluster, some can only run on specific ones due to specific hardware constraints. @bold{ChefCluster} can only run on the x86-based clusters. This excludes the @(tb) Utah cluster which is built on ARMv8 nodes. Refer to the @seclink["hardware"]{Hardware} section for more information. @bold{Note:} If you are at an in-person tutorial, the instructor will tell you which cluster to select. Otherwise, you may select any compatible and available cluster. @margin-note{The dropdown menu for the clusters shows you both the health (outer ring) and available resources (inner dot) of each cluster. The ``Check Cluster Status'' link opens a page (in a new tab) showing the current utilization of all @(tb) clusters.} } @instructionstep["Click Finish!"]{ When you click the ``Finish'' button, @(tb) will start provisioning the resources that you requested on the cluster that you selected. } @instructionstep["@(tb) instantiates your profile"]{ @(tb) will take a few minutes to bring up your experiment, as many things happen at this stage, including selecting suitable hardware, loading disk images on local storage, booting bare-metal machines, re-configuring the network topology, etc. While this is happening, you will see the topology with yellow node icons: @clab-screenshot["chef-tutorial/booting.png"] @margin-note{Provisioning is done using the @link["http://groups.geni.net/geni/wiki/GeniApi"]{GENI APIs}; it is possible for advanced users to bypass the @(tb) portal and call these provisioning APIs from their own code. A good way to do this is to use the @link["https://geni-lib.readthedocs.org"]{@tt{geni-lib} library for Python.}} As soon as a set of resources have been assigned to you, you will see details about them if you switch to the "List View" tab (though you will not be able to log in until they have gone through the process of imaging and booting.) While you are waiting for your resources to become available, you may want to have a look at the @link[apt-doc-url]{@(tb) user manual}, or use the ``Sliver'' button to watch the logs of the resources (``slivers'') being provisioned and booting. } @instructionstep["Your resources are ready!"]{ Shortly, the web interface will report the state as ``Booted''. @clab-screenshot["chef-tutorial/booted.png"] @bold{Important:} A ``Booted'' status indicates that resources are provisioned and booted; this particular profile runs scripts to complete the Chef setup, and it will be a few more minutes before Chef becomes fully functional. You will be able to tell that the configuration process has finished when the status changes from ``Booted'' to ``Ready''. Until then, you will likely see that the startup scripts (i.e. programs that run at the beginning of the experiment to set it up) run longer on @tt{head} than on @tt{node-0} --- a lot more work is required to install the Chef server than the client. In the Topology View tab, mouse over the circle in the @tt{head}'s icon, to confirm the current state of the node. @clab-screenshot["chef-tutorial/head-running.png"] } ] @section{Exploring Your Experiment} While the startup scripts are still running, you will have a few minutes to look at various parts of the @(tb) experiment page and learn what resources you now have access to and what you can do with them. @subsection{Experiment Status} The panel at the top of the page shows the status of your experiment --- you can see which profile it was launched with, when it will expire, etc. The buttons in this area let you make a copy of the profile (so that you can @seclink["creating-profiles"]{customize it}), ask to hold on to the resources for longer, or release them immediately. @clab-screenshot["chef-tutorial/experiment-status.png"] Note that the default lifetime for experiments on @(tb) is less than a day; after this time, the resources will be reclaimed and their disk contents will be lost. If you need to use them for longer, you can use the ``Extend'' button and provide a description of why they are needed. Longer extensions require higher levels of approval from @(tb) staff. You might also consider @seclink["creating-profiles"]{creating a profile} of your own if you might need to run a customized environment multiple times or want to share it with others. You can click on the title of the panel to expand or collapse it. @subsection{Profile Instructions} Profiles may contain written instructions for their use. Clicking on the title of the ``Profile Instructions'' panel will expand or collapse it. In this case, the instructions provide a link to the Chef server web console. (Don't click on the link yet --- it is likely that Chef is still being configured; for now, let's keep exploring the @(tb) interface.) Also, notice the list of private and public hostnames of the experiment nodes included in the instructions. You will use the public hostnames (shown in bold) in the exercise with the Apache web server and apache benchmark. @clab-screenshot["chef-tutorial/experiment-instructions.png"] @subsection{Topology View} We have already used the topology viewer to see the node status; let's take a closer look at the topology of this experiment. This profile has two nodes connected by a 10 Gigabit LAN, which is represented by a gray box in the middle of the topology. The names given for each node are the names assigned as part of the profile; this way, every time you instantiate a profile, you can refer to the nodes using the same names, regardless of which physical hardware was assigned to them. The green boxes around each node indicate that they are up; click the ``Refresh Status'' button to initiate a fresh check. @margin-note{You can also run several networking tests by clicking the ``Run Linktest'' button at the bottom of the page. The available tests include: @itemlist[ @item{Level 1 --- Connectivity and Latency} @item{Level 2 --- Plus Static Routing} @item{Level 3 --- Plus Loss} @item{Level 4 --- Plus Bandwidth} ] Higher levels will take longer to complete and require patience. } @clab-screenshot["chef-tutorial/topology-view.png"] If an experiment has startup services, their statuses are indicated by small icons in the upper right corners of the node icons. You can mouse over this icon to see a description of the current status. In this profile, the startup services on the client node(s), such as @tt{node-0}, typically complete quickly, but the scripts on @tt{head} take much longer. The screenshot above shows the state of the experiment when all startup scripts complete. It is important to note that every node in @(tb) has at least two network interfaces: one ``control network'' that carries public IP connectivity, and one ``experiment network'' that is isolated from the Internet and all other experiments. It is the experiment net that is shown in this topology view. You will use the control network to @(ssh) into your nodes, interact with their web interfaces, etc. This separation gives you more freedom and control in the private experiment network, and sets up a clean environment for @seclink["repeatable-research"]{repeatable research}. @subsection{List View} The list view tab shows similar information to the topology view, but in a different format. It shows the identities of the nodes you have been assigned, and the full @(ssh) command lines to connect to them. In some browsers (those that support the @tt{ssh://} URL scheme), you can click on the SSH commands to automatically open a new session. On others, you may need to cut and paste this command into a terminal window. Note that only public-key authentication is supported, and you must have set up an @(ssh) keypair on your account @bold{before} starting the experiment in order for authentication to work. @clab-screenshot["chef-tutorial/list-view.png"] @subsection{Manifest View} The final default tab shows a @link["http://groups.geni.net/geni/wiki/GENIExperimenter/RSpecs#ManifestRSpec"]{manifest} detailing the hardware that has been assigned to you. This is the @seclink["rspecs"]{``request'' RSpec} that is used to define the profile, annotated with details of the hardware that was chosen to instantiate your request. This information is available on the nodes themselves using the @link["http://groups.geni.net/geni/wiki/GeniGet"]{@tt{geni-get}} command, enabling you to do rich scripting that is fully aware of both the requested topology and assigned resources. @margin-note{Most of the information displayed on the @(tb) status page comes directly from this manifest; it is parsed and laid out in-browser.} @clab-screenshot["chef-tutorial/manifest-view.png"] @subsection[#:tag "chef-tutorial-actions"]{Actions} In both the topology and list views, you have access to several actions that you may take on individual nodes. In the topology view, click on the node to access this menu; in the list view, it is accessed through the icon in the ``Actions'' column. Available actions include rebooting (power cycling) a node, and re-loading it with a fresh copy of its disk image (destroying all data on the node). While nodes are in the process of rebooting or re-imaging, they will turn yellow in the topology view. When they have completed, they will become green again. The @seclink["chef-tutorial-web-shell"]{shell} action is described in more detail below and will be used as the main method for issuing commands on the experiment nodes throughout this tutorial. @clab-screenshot["chef-tutorial/experiment-actions.png"] @section{Brief Introduction to Chef} While the startup scripts are running, let's take a quick look at the architecture of a typical Chef intallation. The diagram provided at the official @link["https://docs.chef.io/chef_overview.html"]{Overview of Chef} page demonstrates the relationships between individual Chef components. While there is a number of optional, advanced components that we don't use in this tutorial, it is worth noting the following: @itemlist[ @item{Chef clients can run on a variety of resources: virtual and physical servers, cloud instances, storage and networking devices, containers, etc. All clients connect and listen to the central, infrastructure-wide Chef server.} @item{The Chef server has a web interface. The server manages @bold{cookbooks}, which are fundamental units of configuration management that define configurations and contain everything that is required to instantiate those configurations. Additionally, the server manages @bold{run lists}, which can be viewed as mappings between collections of cookbooks and the clients on which they need to execute.} @item{A workstation represents a machine from which a Chef administrator connects to and controls the server. It typically has a copy of @bold{chef-repo}, the repository which contains cookbooks and other code artifacts. The administrator modifies cookbooks locally and submits them to the server to make them available to all clients.} @item{@bold{Recipes}, shown next to cookbooks on the workstation, are the "building blocks" from which cookbooks are assembled. Recipes can be viewed as individual scripts accomplishing specific fine-grained tasks, such as create a user, install a package, clone a repository, configure a network interface, etc. Cookbooks, which include one or more recipes, are responsible for "bigger" tasks: install and configure a database, install and run a web server, and so on.} @item{Another type of Chef code artifacts (not shown in the diagram) is called @bold{role}. A role can include one or more cookbooks and specific recipes (each cookbook and recipe can be used in several roles if necessary). Roles typically define complete node configurations; for instance, in a simple case, one role corresponds to a master node on a cluster with a set of specific services, while another role defines how the rest of the cluster nodes, so-called ``worker'' nodes, should be configured. After being created on the workstation, these roles are submitted to the server by the administrator and then assigned to specific nodes via the run lists.} ] In the experiment we have launched, @tt{head} runs all three: the server, the workstation, and the client (there is no conflict between these components), while @tt{node-0} runs only the client. If this profile is instantiated with the arbitrary number N of clients, there will be N ``client-only'' nodes named @tt{node-0},@tt{node-1},...,@tt{node-(N-1)}. Another profile parameter allows choosing the repository from which chef-repo is cloned; by default, it points to @link["https://github.com/emulab/chef-repo"]{emulab/chef-repo}. The startup scrips in this profile also obtain and install copies of public cookbooks hosted at the respository called @link["https://supermarket.chef.io"]{Supermarket}. Specifically, the exercises in this tutorial rely on the @link["https://supermarket.chef.io/cookbooks/nfs"]{nfs} and @link["https://supermarket.chef.io/cookbooks/apache2"]{apache2} cookbooks --- both should be installed on your experiment now in accordance with the default value of the corresponding profile parameter we have used. @section{Logging in to the Chef Web Console} As soon as the startup scripts complete, you will likely recieve an email confirming that Chef is installed and operational. @verbatim{Dear User, Chef server and workstataion should now be installed on head.chefdemo.utahstud.emulab.net. To explore the web management console, copy this hostname and paste it into your browser. Installation log can be found at /var/log/init-chef.log on the server node. To authenticate, use the unique credentials saved in /root/.chefauth on the server node. Below are several Chef commands which detail the launched experiment: # chef -v Chef Development Kit Version: 0.7.0 chef-client version: 12.4.1 berks version: 3.2.4 kitchen version: 1.4.2 # knife cookbook list apache2 3.1.0 apt 2.9.2 emulab-apachebench 1.0.0 emulab-nfs 0.1.0 ... nfs 2.2.6 # knife node list head node-0 # knife role list apache2 apachebench ... nfs # knife status -r 1 minute ago, head, [], ubuntu 14.04. 0 minutes ago, node-0, [], ubuntu 14.04. Happy automation with Chef! } In some cases, you will not be able to see this email --- email filters (e.g., if you are using a university email account) may classify it as spam. This will not be a problem since the email is supposed to provide useful but noncritical information. You can still access the Chef web console using the link included in the profile instructions and also obtain the credentials as described in the instructions. When you receive this email or see in the topology viewer that the startup scripts completed, you can proceed to the following step. @subsection[#:tag "chef-tutorial-web-shell"]{Web-based Shell} @(tb) provides a browser-based shell for logging into your nodes, which is accessed through the action menu described above. While this shell is functional, it is most suited to light, quick tasks; if you are going to do serious work, on your nodes, we recommend using a standard terminal and @(ssh) program. This shell can be used even if you did not establish an @(ssh) keypair with your account. Three things of note: @itemlist[ @item{Your browser may require you to click in the shell window before it gets focus.} @item{Depending on your operating system and browser, cutting and pasting into the window may not work. If keyboard-based pasting does not work, try right-clicking to paste.} @item{@bold{The recommended browsers are Chrome and Firefox.} The web-based shell does not work reliably in Safari. It is not guaranteed that it works smoothly in other browsers.} ] Create a shell tab for @tt{head} by choosing ``Shell'' in the Actions menu for @tt{head} in either the topology or the list view. The new tab will be labeled ``head''. It will allow typing shell commands and executing them on the node. All shell commands used throughout this tutorial should be issued using this shell tab. While you are here, switch to the @italic{root} user by typing: @verbatim{@bold{sudo su -}} @clab-screenshot["chef-tutorial/experiment-shell.png"] Your prompt, the string which is printed on every line before the cursor, should change to @tt{root@"@"head}. All commands in the rest of the tutorial should be executed under @italic{root}. @subsection{Chef Web Console} Type the following to print Chef credentials: @verbatim{@bold{cat /root/.chefauth}} You should see the unique login and password that have been generated by the startup scripts specifically for your instance of Chef: @clab-screenshot["chef-tutorial/shell-chefauth.png"] Expand the ``Profile Instructions'' panel and click on the ``Chef web console'' link. @bold{Warning:} When your browser first points to this link, you will see a warning about using a self-signed SSL certificate. Using self-signed certificates is not a good option in production scenarios, but it is totally acceptable in this short-term experimentation environment. We will ignore this warning and proceed to using the Chef console. The sequence of steps for ignoring it depends on your browser. In Chrome, you will see a message saying "Your connection is not private". Click ``Advanced'' at the bottom of the page and choose ``Proceed to (unsafe)''. In Firefox, the process is slightly different: click ``Advanced'', choose ``Add Exception'', and click ``Confirm Security Exception''. When the login page is finally displayed, use the credentials you have printed in the shell: @clab-screenshot["chef-tutorial/chef-wui-login.png"] @clab-screenshot["chef-tutorial/chef-wui-nodelist.png"] If you see a console like the one above, with both @tt{head} and @tt{node-0} listed on the ``Nodes'' tab, you have a working Chef cluster! You can now proceed to managing your nodes using Chef recipes, cookbooks, and roles. In the rest of the tutorial, we demonstrate how you can use several pre-defined cookbooks and roles. Development of cookbooks and roles is a subject of another tutorial (many such tutorials can be found online). Our goal in the following sections is to walk you through the process of using existing cookbooks and roles --- the process which is the same for simple and more complex configurations. You will learn how to modify run lists and apply cookbooks and roles to your nodes, run them, check their status, and explore individual components of cookbooks and roles. @section{Configuring NFS} Now that you have a working instance of Chef where both @tt{head} and @tt{node-0} can be controlled using Chef, let's start modifying the software stacks on these nodes by installing NFS and exporting a directory from @tt{head} to @tt{node-0}. @itemlist[#:style 'ordered @instructionstep["Modify the run list on head"]{ Click on the row for @tt{head} in the node list (so the entire row is highlighted in orange), and then click "Edit" in the Run List panel in the bottom right corner of the page: @clab-screenshot["chef-tutorial/head-edit-runlist.png"] } @instructionstep["Apply nfs role:"]{ In the popup window, find the role called @tt{nfs} in the Available Roles list on the left, drag and drop it into the "Current Run List" field on the right. When this is done, click "Save Run List". @clab-screenshot["chef-tutorial/head-nfs-drag.png"] } @instructionstep["Repeat the last two steps for node-0:"]{ @clab-screenshot["chef-tutorial/node0-edit-runlist.png"] @clab-screenshot["chef-tutorial/node0-nfs-drag.png"] At this point, @tt{nfs} role is assigned to both nodes, but nothing has executed yet. } @instructionstep["Check the status from the shell"]{ Before proceeding to applying these updates, let's check the role assignment from the shell by typing: @verbatim{@bold{knife status -r}} The run lists for your nodes should be printed inside square brackets: @clab-screenshot["chef-tutorial/shell-knife-status.png"] The output of this command also conveniently shows when Chef applied changes to your nodes last (first column) and what operating systems run on the nodes (last column). } @instructionstep["Trigger the updates"]{ Run the assigned role on @tt{head} by typing: @verbatim{@bold{chef-client}} @clab-screenshot["chef-tutorial/head-chef-client.png"] As a part of the startup procedure, passwordless @tt{ssh} connections are enabled between the nodes. Therefore, you can use @tt{ssh} to run commands remotely, on the nodes other than @tt{head}, from the same shell. Execute the same command on @tt{node-0} by typing: @verbatim{@bold{ssh node-0 chef-client}} @clab-screenshot["chef-tutorial/node0-chef-client.png"] When nodes execute the ``@code{chef-client}'' command, they contact the server and request the cookbooks, recipes, and roles have been assigned to them in their run lists. The server responds with those artifacts, and the nodes execute them in the specified order. } @instructionstep["Verify that NFS is working"]{ After updating both nodes, you should have a working NFS configuration in which the @code{/exp-share} directory is exported from @tt{head} and mounted on @tt{node-0}. @margin-note{The name of the NFS directory is one of the attributes that is set in the @tt{nfs} role. To explore other attributes and see how this role is implemented, take a look at the @code{/chef-repo/roles/nfs.rb} file on @tt{head}} The simplest way to test that this configuration is functioning properly is to create a file on one node and check that this file becomes available on the other node. Follow these commands to test it (the lines that start with the # sign are comments): @verbatim{ # List files in the NFS directory /exp-share on head; should be empty @bold{ls /exp-share/} # List the same directory remotely on node-0; also empty @bold{ssh node-0 ls /exp-share/} # Create an empty file in this derectory locally @bold{touch /exp-share/NFS-is-configured-by-Chef} # Find the same file on node-0 @bold{ssh node-0 ls /exp-share/} } @clab-screenshot["chef-tutorial/nfs-is-working.png"] If you can see the created file on @tt{node-0}, your NFS is working as expected. Now you can easily move files between your nodes using the @code{/exp-share} directory. } ] @bold{Summary:} You have just installed and configured NFS by assigning a Chef role and running it on your nodes. You can create much more complex software environments by repeating these simple steps and installing more software components on your nodes. Additionally, installing NFS on a set of nodes is not a subject of a research experiment but rather an infrastructure prerequisite for many distributed systems. You can automate installation and configuration procedures in those systems using a system like Chef and save a lot of time when you need to periodically recreate them or create multiple instances of those systems. @subsection{Exploring The Structure} It is worth looking at how the role you have just used is implemented. Stored at @code{/chef-repo/roles/nfs.rb} on @tt{head}, it should include the following code: @verbatim{ # # This role depends on the emulab-nfs cookbook; # emulab-nfs depends on the nfs cookbook # available at: https://supermarket.chef.io/cookbooks/nfs # Make sure it is installed; If it is not, try: knife cookbook site install nfs # name "nfs" description "Role applied to all NFS nodes - server and client" override_attributes( "nfs" => { "server" => "head", "dir" => "/exp-share", "export" => { "network" => "10.0.0.0/8", "writeable" => true } } ) run_list [ "emulab-nfs" ] } You can see a number of domain-specific Ruby attributes in this file. @code{name} and @code{description} are self-describing attributes. The @code{override_attribute} attribute allows you to control high-level configuration parameters, including (but not limited to) the name of the node running the NFS server, the directory being exported and mounted, the network in which this directory is shared, and the ``write'' permission in this directory (granted or not). The @code{run_list} attribute includes a single cookbook called @code{emulab-nfs}. You are probably wondering now about how the same cookbook can perform different actions on different nodes. Indeed, the @tt{emulab-nfs} cookbook has just installed NFS server on @tt{head} and NFS client on @tt{node-0}. You can take a look at the implementation of the @code{default.rb}, the default recipe in the @tt{emulab-nfs} cookbook which gets called when the cookbook is executed. You should find the following code at @code{/chef-repo/cookbooks/emulab-nfs/recipes/default.rb}: @verbatim{ # # Cookbook Name:: emulab-nfs # Recipe:: default # if node["hostname"] == node["nfs"]["server"] include_recipe "emulab-nfs::export" else include_recipe "emulab-nfs::mount" end } The @code{if} statement above allows comparing the hostname of the node on which the cookbook is running with one of the attributes specified in the role. Based on this comparison, the recipe takes the appropriate actions by calling the code from two other recipes in this cookbook. Optionally, you can explore @code{/chef-repo/cookbooks/emulab-nfs/recipes/export.rb} and @code{/chef-repo/cookbooks/emulab-nfs/recipes/mount.rb} to see how these recipes are implemented. Obviously, this is not the only possible structure for this configuration. You can alternatively create two roles, such as @tt{nfs-server} and @tt{nfs-client}, that will call the corresponding recipes without the need for the described comparison. Since a single model cannot fit perfectly all scenarios, Chef provides the developer with enough flexibility to organize the code into structures that match specific environment and application needs. @section{Apache Web Server and ApacheBench Benchmarking tool} In this exercise, we are going to switch gears and experiment with different software --- @code{apache2}, the Apache HTTP web server, and the @code{ab} benchmarking tool. You will explore more Chef capabilities and perform administrative tasks, primarily from the command line. We recommend you to continue using the shell tab for @tt{head} (again, make sure that your commands are executed as @italic{root}). Run the commands described below in that shell. @itemlist[#:style 'ordered @instructionstep["Add a role to the head's run list"]{ Issue the command in bold (the rest is the expected output): @verbatim{@bold{knife node run_list add head "role[apache2]"} head: run_list: role[nfs] role[apache2] } } @instructionstep["Add two roles to the node-0's run list"]{ Run the two @code{knife} commands listed below (also on @tt{head}) in order to assign two roles to @tt{node-0}: @verbatim{@bold{knife node run_list add node-0 "role[apache2]"} run_list: head: run_list: role[nfs] role[apache2] @bold{knife node run_list add node-0 "role[apachebench]"} run_list: head: run_list: role[nfs] role[apache2] role[apachebench] } Notice that we did not exclude the @tt{nfs} role from the run lists. Configuration updates in Chef are @italic{idempotent}, which means that an update can be applied to a node multiple times, and every time the update will yield identical configuration (regardless of the node’s previous state) Thus, this time, when you run ``@code{chef-client}'' again, Chef will do the right thing: the state of each individual component will be inspected, and all NFS-related tasks will be silently skipped because NFS is already installed. } @instructionstep["Trigger updates on all nodes"]{ In the previous section, we updated one node at a time. Try the ``batch'' update - run a single command that will trigger configuration procedures on all client nodes: @verbatim{@bold{knife ssh "name:*" chef-client} apt155.apt.emulab.net resolving cookbooks for run list: ["emulab-nfs", "apache2", "apache2::mod_autoindex", "emulab-apachebench"] apt154.apt.emulab.net resolving cookbooks for run list: ["emulab-nfs", "apache2", "apache2::mod_autoindex"] apt155.apt.emulab.net Synchronizing Cookbooks: .... apt155.apt.emulab.net Chef Client finished, 5/123 resources updated in 04 seconds } You should see interleaved output from the ``@code{chef-client}'' command running on both nodes at the same time. The last command uses a node search based on the ``@code{name:*}'' criteria. As a result, every node will execute ``@code{chef-client}''. In cases where it is necessary, you can use more specific search strings, such as, for instance, ``@code{name:head}'' and ``@code{name:node-*}''. } @instructionstep["Check that the webservers are running"]{ One of the attributes in the @code{apache2} role configures the web server such that it runs on the port @code{8080}. Let's check that the web servers are running via checking the status of that port: @verbatim{@bold{knife ssh "name:*" "netstat -ntpl | grep 8080" } pc765.emulab.net tcp6 0 0 :::8080 :::* LISTEN 9415/apache2 pc768.emulab.net tcp6 0 0 :::8080 :::* LISTEN 30248/apache2 } Note that this command sends an arbitrary command (unrelated to Chef) to a group of nodes. With this functionality, the @code{knife} command-line utility can be viewed as an orchestration tool for managing groups of nodes that is capable of replacing @code{pdsh}, the Parallel Distributed Shell utility, that is often used on computing clusters. The output that is similar to the one above indicates that both @code{apache2} web servers are running. The command that you have just issued uses @code{netstat}, a command-line tool that displays network connections, routing tables, interface statistics, etc. Using @code{knife}, you have run @code{netstat} in combination with a Linux pipe and a @code{grep} command for filtering output and displaying the information only on the port @code{8080}. } @instructionstep["Check benchmarking results on node-0"]{ Many different actions have taken place on @tt{node-0}, including: @itemlist[ @item{@code{apache2} has been installed and configured} @item{@code{ab}, a benchmarking tool from the @code{apache2-utils} package, has been installed and executed} @item{benchmarking results have been saved, and plots have been created using the @code{gnuplot} utility} @item{the plots have been made available using the installed web server} ] To see how the last three tasks are performed using Chef, you can take a look at the @link["https://github.com/emulab/chef-repo/blob/master/cookbooks/emulab-apachebench/recipes/default.rb"]{default.rb} recipe inside the @code{emulab-apachebench} cookbook. In the in-person tutorial, let's proceed to the following steps and leave the discussion of this specific recipe and abstractions used in Chef recipes in general to the end of the tutorial. Obtain the @tt{node-0}'s public hostname (refer to the list included in the profile instructions), and construct the following URL: @verbatim{http://:8080} With the right hostname, copy and paste this URL into your browser to access the web server running on @tt{node-0} and serving the benchmarking graphs. You should see a directory listing like this: @clab-screenshot["chef-tutorial/apache-dir-listing-0.png"] You can explore the benchmarking graphs by clicking at the listed links. So far, @code{ab} has run against the local host (i.e. @tt{node-0}), therefore the results may not be so interesting. Do not close the window with your browser since one of the following steps will ask you to go back to it to see more results. } @instructionstep["Modify a role"]{ Just like the @tt{nfs} role described in the previous section, @tt{apachebench} has a number of attributes that define its behavior. We will use the Chef web console to update the value for one of its attributes. In the console, click on the Policy tab at the top of the page, choose ``Roles'' in the panel on the left, and select ``apachebench'' in the displayed list of roles. Select the Attributes tab in the middle of the page. At the bottom of the page, you should now see a panel called ``Override Attributes''. Click the ``Edit'' button inside that panel: @clab-screenshot["chef-tutorial/console-apachebench-edit.png"] In the popup window, change the @code{target_host} attribute from @code{null} to the @tt{head}'s public hostname (refer to the hostnames listed in the profile instructions). Don't forget to use double quotes around the hostname. Also, let's modify the list of ports against which we will run the benchmark. Change @code{80} to @code{8080} since nothing interesting is running on port @code{80}, while the @code{apache2} server you have just installed is listening on the port @code{8080}. Leave the port @code{443} in the list --- this is the port on which the Chef web console is running. Here is an example of the recommended changes: @clab-screenshot["chef-tutorial/console-apachebench-change-attr.png"] When these changes are complete, click "Save Attributes". @bold{Alternative:} It turns out that you don't have to use the Chef web console for modifying this role. Take a look at the two steps described below to see how we can modify this role from the shell or @bold{skip} to running the benchmark. This is what you need to make the described changes without using the web console: @itemlist[#:style 'ordered @item{Edit the file @code{/chef-repo/roles/apachebench.rb} on @tt{head} using a text editor of your choice (e.g., @code{vi})} @item{After making the suggested changes, ``push'' the updated role to the server by typing: @code{knife role from file /chef-repo/roles/apachebench.rb}} ] } @instructionstep["Run the benchmark against head"]{ The updated version of the role is available on the server now. You can run it on @tt{node-0} using the following command: @verbatim{@bold{knife ssh "role:apachebench" chef-client}} This command, unlike the @code{knife} commands you issued in the previous steps, uses a search based on the assigned roles. In other words, it will find the nodes to which the @tt{apachebench} role has been assigned by now --- in our case, only @tt{node-0} --- and execute ``@code{chef-client}'' on them. } @instructionstep["Check benchmarking results again"]{ Go back to your browser window and update the page to see the directory listing with new benchmarking results. @clab-screenshot["chef-tutorial/apache-dir-listing-1.png"] The first two (the most recent) graphs represent the results of benchmarking of the web services running on @tt{head} performed from @tt{node-0}. Among many interesting facts that are revealed by these graphs, you will see that the response time is much higher on the port @code{443} than on the port @code{8080}. @clab-screenshot["chef-tutorial/bench-graphs.png"] } ] @bold{Summary:} You have just performed an experiment with @code{apache2} and @code{ab} in a very automated manner. The Chef role you have used performed many tasks, including setting up the infrastructure, running a benchmark, saving and processing results, as well as making them easily accessible. The following section will shed some light on how these tasks were accomplished and also how they can be customized. @subsection{Understanding the Internals} Below we describe some of the key points about the pieces of infrastructure code you have just used. The @tt{apache2} role is stored at @link["https://github.com/emulab/chef-repo/blob/master/roles/apache2.rb"]{roles/apache2.rb} as part of the @link["https://github.com/emulab/chef-repo"]{emulab/chef-repo}. Note that the run list specified in this role includes a cookbook @code{apache2} (the recipe called @code{default.rb} inside the cookbook is used) and also a recipe @code{mod_autoindex} from the same cookbook. This cookbook is one of the cookbooks that have been obtained from @link["https://supermarket.chef.io"]{Chef Supermarket}. All administrative work for installing and configuring the web server is performed by the recipes in this cookbook, while the role demonstrates an example of how that cookbook can be ``wrapped'' into a specification that satisfy specific application needs. The @tt{apachebench} role is stored at @link["https://github.com/emulab/chef-repo/blob/master/roles/apachebench.rb"]{roles/apachebench.rb} It specifies values for several attributes and adds a custom cookbook @tt{emulab-apachebench} to the run list. We use the ``@code{emulab}'' prefix in the names of the cookbooks that have been developed by the @(tb) staff to emphasize that they are developed for @link["https://www.emulab.net/"]{Emulab} and derived testbeds such as @(tb). This also allows distinguishing them from the Supermarket cookbooks, which are installed in the same directory on @tt{head}. Inside the @link["https://github.com/emulab/chef-repo/blob/master/cookbooks/emulab-apachebench/recipes/default.rb"]{default.rb} recipe in @tt{emulab-apachebench} you can find many key words, including @code{package}, @code{log}, @code{file}, @code{execute}, @code{template}, etc. They are called Chef @bold{resources}. These elements, which allow defining fine-grained configuration tasks and actions, are available for many common administrative needs. You can refer to the @link["https://docs.chef.io/resources.html#resources"]{list of supported Chef resources} and see examples of how they can be used. Another item that is worth mentioning is the ``@code{ignore_failure true}'' attribute used in some of the resources. It allows the recipe to continue execution even when something does not go as expected (shell command fail, necessary files do not exist, etc.). Also, when relying on specific files in your recipes, you can augment resources with additional checks like ``@code{only_if{::File.exists?()}}'' and ``@code{not_if{::File.exists?()}}'' to make your infrastructure code more reliable and repeatable (this refers to the notion of @italic{idempotent} code mentioned earlier). @section{Final Remarks about Chef on @(tb)} In this tutorial, you had a chance to explore the Chef configuration management system and used some of its powerful features for configuration management in a multi-node experiment on @(tb). Even though the instructions walked you through the process of configurating only two nodes, you can use the demonstrated code artifacts, such as roles, cookbooks, recipes, and resources, and apply them to infrastructure in larger experiments. With the @code{knife} commands like the ones shown above, you can ``orchestrate'' diverse and complex applications and environments. When establishing such orchestration in your experiments, you can take advantage of the relevant pieces of intrastructure code, e.g., available through Chef @link["https://supermarket.chef.io"]{Supermarket}. In cases when you have to develop your own custom code, you may adopt the structure and the abstractions supported by Chef and aim to develop infrastructure code that is modular, flexible, and easy to use. The @bold{ChefCluster} profile is available to all users on @(tb). If you are interested in learning more about Chef and developing custom infrastructure code for your experiments, this profile will spare you from the need to set up necessary components every time and allow you to run a fully functional Chef installation for your specific needs. @section{Terminating Your Experiment} Resources that you hold in @(tb) are real, physical machines and are therefore limited and in high demand. When you are done, you should release them for use by other experimenters. Do this via the ``Terminate'' button on the @(tb) experiment status page. @clab-screenshot["chef-tutorial/terminate.png"] @bold{Note:} When you terminate an experiment, all data on the nodes is lost, so make sure to copy off any data you may need before terminating. If you were doing a real experiment, you might need to hold onto the nodes for longer than the default expiration time. You would request more time by using the ``Extend'' button the on the status page. You would need to provide a written justification for holding onto your resources for a longer period of time. @section{Future Steps} Now that you've got a feel for how Chef can help manage experiment resources, there are several things you might try next: @itemlist[ @item{Learn more about Chef at the @link["https://learn.chef.io/"]{Chef official website}} @item{Explore existing cookbooks and roles in the @link["https://github.com/emulab/chef-repo"]{emulab/chef-repo} repository} @item{Follow the steps in the @link["http://jtimberman.housepub.org/blog/2011/09/03/guide-to-writing-chef-cookbooks/"]{Guide to Writing Chef Cookbooks} blog post by Joshua Timberman, a Chef developer and one of the active members of the Chef community } @item{Configure a LAMP (Linux, Apache, MySql, and PHP) stack using Chef by following the @link["https://www.linode.com/docs/applications/chef/creating-your-first-chef-cookbook"]{Creating Your First Chef Cookbook} tutorial } ]