ChangeLog 63.4 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
stoller     2003/10/06 14:27:46 MDT

  Modified files:
    tmcd                 libsetup.pm 
    tmcd/common          GNUmakefile.in bootsubnodes bootvnodes 
                         ntpstart rc.progagent sendevent update 
                         vnodesetup watchdog 
    tmcd/freebsd         GNUmakefile.in delaysetup rc.delayagent 
                         tbshutdown 
    tmcd/linux           GNUmakefile.in delaysetup ixpboot 
                         supfile.in 
    tmcd/plab            GNUmakefile.in 
    tmcd/ron             GNUmakefile.in emulabctl webserver.pl 
  Added files:
    tmcd/common          libtmcc.pm tmcc.pl 
    tmcd/ron             makefile 
  Log:
  * New libtmcc.pm module that encapsulates the tmcc interface. Most of the
    code that was in libsetup has moved into this library, and underwent a
    giant cleaning and pumping up. The interface from your typical perl
    script now looks like this:
  
    use libtmcc;
  
    if (tmcc(TMCCCMD_STATUS, "optional arguments", \@tmccresults) < 0) {
        warn("*** WARNING: Could not get status from server!\n");
        return -1;
    }
    foreach my $me (@tmccresults) {
  	print "bite $me";
    }
  
    The arguments and results are optional values. There is a fourth optional
    value that is a hash of config options (basically converted to command
    line switches passed to tmcc). For example, to set the timeout on an
    individual call, pass a fourth argument like:
  
  	("timeout" => 5)
  
    There is also a way to set global options so that all subsequent tmcc
    calls are affected:
  
  	configtmcc("timeout", 5);
  
    I'll probably clean this up a bit to avoid the direct strings.
  
    The result list is a list of strings. Since we are trending away from
    using tmcc to transfer large amounts of data, I think this is okay.
  
  * A new tmcc.pl which does little more than load libtmcc and use it.
    This will become the new tmcc, with the existing C version becoming a
    backend binary for it.
  
  * All of the perl scripts in tmcd have been changed to use the new
    library. I left the few uses of tmcc in shell scripts alone since they
    were of the simple variety (mostly "state" command).
  
  * And again, if you have read this far, you will learn why I bothered with
    all this. Well, the existing code was really bad and it was getting out
    of control. Sort of like a squid that was getting harder to control as
    its rotting tenticles slithered into more and more scripts. Anyway ...
  
    More important, my goal is to use the libtmcc library to add caching.  I
    have not worked out the details yet, but I am envisioning a configuration
    file, perhaps generated initially by tmcd, of all of the config
    values. If the library finds that file, it sucks the info out of the file
    instead of going to tmcd. Eventually, this config file would be generated
    as part of experiment swapping and stored in the DB, but thats a longer
    term project, and perhaps orthogonal (how we fill the cache is not as
    important as adding the ability to use a cache, right?).
  
    Note that certain operations (like "state" and "ready") are flagged by
    the library to always bypass the "cache".
  
  Revision  Changes    Path
  1.83      +322 -446  testbed/tmcd/libsetup.pm
  1.14      +2 -0      testbed/tmcd/common/GNUmakefile.in
  1.2       +14 -10    testbed/tmcd/common/bootsubnodes
  1.10      +13 -9     testbed/tmcd/common/bootvnodes
  1.4       +17 -13    testbed/tmcd/common/ntpstart
  1.4       +2 -1      testbed/tmcd/common/rc.progagent
  1.2       +4 -3      testbed/tmcd/common/sendevent
  1.4       +1 -1      testbed/tmcd/common/update
  1.18      +10 -7     testbed/tmcd/common/vnodesetup
  1.8       +17 -11    testbed/tmcd/common/watchdog
  1.40      +1 -1      testbed/tmcd/freebsd/GNUmakefile.in
  1.10      +18 -13    testbed/tmcd/freebsd/delaysetup
  1.7       +2 -1      testbed/tmcd/freebsd/rc.delayagent
  1.3       +4 -2      testbed/tmcd/freebsd/tbshutdown
  1.36      +1 -1      testbed/tmcd/linux/GNUmakefile.in
  1.2       +12 -8     testbed/tmcd/linux/delaysetup
  1.5       +23 -12    testbed/tmcd/linux/ixpboot
  1.3       +1 -1      testbed/tmcd/linux/supfile.in
  1.4       +5 -2      testbed/tmcd/plab/GNUmakefile.in
  1.9       +1 -1      testbed/tmcd/ron/GNUmakefile.in
  1.7       +4 -3      testbed/tmcd/ron/emulabctl
  1.4       +49 -12    testbed/tmcd/ron/webserver.pl


Leigh B. Stoller's avatar
Update  
Leigh B. Stoller committed
100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190
stoller     2003/09/30 09:30:09 MDT

  Modified files:
    db                   libdb.pm.in 
    tbsetup              assign_wrapper.in batch_daemon.in 
                         batchexp.in endexp.in nscheck.in 
                         startexp.in swapexp.in tbswap.in 
    www                  beginexp.php3 boot.php3 dbdefs.php3.in 
                         defs.php3.in delaycontrol.php3 
                         editexp.php3 endexp.php3 modifyexp.php3 
                         nscheck.php3 showexp.php3 showstuff.php3 
                         swapexp.php3 
  Log:
  Up to now we have had two state variables associated with an experiment,
  plus a lock field. The lock field was a simple "experiment locked, go away"
  slot that is easy to use when you do not care about the actual state that
  an experiment is in, just that it is in "transition" and should not be
  messed with.
  
  The other two state variables are "state" and "batchstate". The former
  (state) is the original variable that Chris added, and was used by the tb*
  scripts to make sure that the experiment was in the state each particular
  script wanted them to be in. But over time (and with the addition of so
  much wrapper goo around them), "state" has leaked out all over the place to
  determine what operations on an experiment are allowed, and if/when it
  should be displayed in various web pages. There are a set of transition
  states in addition to the usual "active", "swapped", etc like "swapping"
  that make testing state a pain in the butt.
  
  I added the other state variable ("batchstate") when I did the batch
  system, obviously! It was intended as a wrapper state to control access to
  the batch queue, and to prevent batch experiments from being messed with
  except when it was really okay (for example, its okay to terminate a
  swapped out batch experiment, but not a swapped in batch experiment since
  that would confuse the batch daemon). There are fewer of these states, plus
  one additional state for "modifying" experiments.
  
  So what I have done is change the system to use "batchstate" for all
  experiments to control entry into the swap system, from the web interface,
  from the command line, and from the batch daemon. The other state variable
  still exists, and will be brutally pushed back under the surface until its
  just a vague memory, used only by the original tb* scripts. This will
  happen over time, and the "batchstate" variable will be renamed once I am
  convinced that this was the right thing to do and that my changes actually
  work as intended.
  
  Only people who have bothered to read this far will know that I also added
  the ability to cancel experiment swapin in progress. For that I am using
  the "canceled" flag (ah, this one was named properly from the start!), and
  I test that at various times in assign_wrapper and tbswap. A minor downside
  right now is that a canceled swapin looks too much like a failed swapin,
  and so tbops gets email about it. I'll fix that at some point (sometime
  after the boss complains).
  
  I also cleaned up various bits of code, replacing direct calls to exec
  with calls to the recently improved SUEXEC interface. This removes
  some cruft from each script that calls an external script.
  
  Cleaned up modifyexp.ph3 quite a bit, reformatting and indenting.
  Also fixed to not run the parser directly! This was very wrong; should
  call nscheck instead. Changed to use "nobody" group instead of group
  flux (made the same change in nscheck).
  
  There is a script in the sql directory called newstates.pl. It needs
  to be run to initialize the batchstate slot of the experiments table
  for all existing experiments.
  
  Revision  Changes    Path
  1.131     +49 -18    testbed/db/libdb.pm.in
  1.128     +30 -1     testbed/tbsetup/assign_wrapper.in
  1.39      +108 -39   testbed/tbsetup/batch_daemon.in
  1.38      +19 -8     testbed/tbsetup/batchexp.in
  1.27      +74 -59    testbed/tbsetup/endexp.in
  1.8       +16 -5     testbed/tbsetup/nscheck.in
  1.57      +50 -46    testbed/tbsetup/startexp.in
  1.42      +200 -82   testbed/tbsetup/swapexp.in
  1.27      +17 -6     testbed/tbsetup/tbswap.in
  1.37      +29 -58    testbed/www/beginexp.php3
  1.7       +2 -2      testbed/www/boot.php3
  1.59      +10 -10    testbed/www/dbdefs.php3.in
  1.44      +32 -23    testbed/www/defs.php3.in
  1.6       +2 -2      testbed/www/delaycontrol.php3
  1.3       +7 -7      testbed/www/editexp.php3
  1.34      +44 -67    testbed/www/endexp.php3
  1.10      +174 -151  testbed/www/modifyexp.php3
  1.9       +22 -19    testbed/www/nscheck.php3
  1.71      +20 -12    testbed/www/showexp.php3
  1.116     +137 -123  testbed/www/showstuff.php3
  1.19      +36 -53    testbed/www/swapexp.php3


Leigh B. Stoller's avatar
Leigh B. Stoller committed
191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949
stoller     2003/09/24 17:23:40 MDT

  Modified files:
    .                    configure configure.in 
    db                   libdb.pm.in 
    tbsetup              os_setup.in 
    tbsetup/plab         GNUmakefile.in 
  Added files:
    tbsetup/plab         plabmonitord.in 
  Log:
  Commit my daemon to monitor the status of plab physnodes in hwdown,
  trying to bring them back from the dead periodically by trying to
  instantiate a vserver/vnode on them, and then tearing it down. If we
  can do that, then the node is usable, and it gets moved back into the
  normal holding experiment so that ptopgen will add it to ptop files.
  
  This deamon is not turned on yet; waiting for other little bits and
  pieces to be done.
  
  There is an equiv change in os_setup that moves physnodes into hwdown
  when a setup on a vnode fails.
  
  Lbs
  
  Revision  Changes    Path
  1.174     +2 -1      testbed/configure
  1.181     +2 -1      testbed/configure.in
  1.126     +32 -3     testbed/db/libdb.pm.in
  1.78      +6 -0      testbed/tbsetup/os_setup.in
  1.5       +2 -1      testbed/tbsetup/plab/GNUmakefile.in


stoller     2003/09/24 16:43:43 MDT

  Modified files:
    .                    configure configure.in 
    os                   install-rpm install-tarfile 
    tmcd                 libsetup.pm 
    www                  dbdefs.php3.in 
  Added files:
    www                  spewrpmtar.php3 
    tbsetup              spewrpmtar.in 
  Log:
  Convert install-rpm/install-tarfile to use the web server instead of
  tmcd (which is bad, since tying up the tmcd threads blocks all nodes
  in the testbed). The old functionality is left in tmcd for now.
  
  On the server side, a new web page (www/spewrpmtar.php3) receives a
  request for a file, along with the nodeid (pcXXX) making the request,
  and the secret key that is generated for each new experiment and
  transfered to the node via tmcd. If the key matches, the operation is
  handed off to tbsetup/spewrpmtar.in which verifies that the file is in
  the list of rpm/tar files for that node, and then spits it out to
  stdout. The web page uses fpassthru() to send the file out to the
  client. The client is using wget, and is required to use https (the
  web page checks).
  
  At present, the external script is run as the creator of the
  experiment, and gid of the experiment. Perhaps this is not a good
  idea. In any event, the file must be in the list of rpm/tarfiles,
  either owned by the experiment creator or with a group of the
  experiment, and the file must reside in either /proj or /groups.
  I use the realpath() function to make sure there are no symlink tricks
  pointing to outside those filesystems. I use the standard NFS read goo to
  prevent transient mount problems that we all know and love.
  
  Revision  Changes    Path
  1.173     +1 -0      testbed/configure
  1.180     +1 -0      testbed/configure.in
  1.2       +122 -81   testbed/os/install-rpm
  1.8       +126 -83   testbed/os/install-tarfile
  1.80      +100 -14   testbed/tmcd/libsetup.pm
  1.58      +19 -0     testbed/www/dbdefs.php3.in


mike        2003/09/23 16:50:23 MDT

  Modified files:
    tmcd/freebsd         control_interface cpuspeed.awk nodetype 
    tmcd/linux           chipset.awk control_interface 
                         cpuspeed.awk nodetype 
  Log:
  Prototype changes for Aero 2.4Ghz boxes.
  Also change awk scripts to exit after printing something.
  
  Revision  Changes    Path
  1.12      +3 -0      testbed/tmcd/freebsd/control_interface
  1.8       +17 -12    testbed/tmcd/freebsd/cpuspeed.awk
  1.8       +3 -0      testbed/tmcd/freebsd/nodetype
  1.4       +4 -4      testbed/tmcd/linux/chipset.awk
  1.12      +1 -1      testbed/tmcd/linux/control_interface
  1.7       +14 -10    testbed/tmcd/linux/cpuspeed.awk
  1.9       +3 -0      testbed/tmcd/linux/nodetype


stoller     2003/09/22 09:35:54 MDT

  Modified files:
    www                  plabmetrics.php3 
  Log:
  Pump up linecount with sorting by each metric.
  
  Revision  Changes    Path
  1.2       +122 -29   testbed/www/plabmetrics.php3


stoller     2003/09/22 07:56:18 MDT

  Modified files:
    tbsetup              os_setup.in vnode_setup.in 
  Log:
  Minor allocstate changes to try and cope with plab nodes that either
  fail in "plabnode alloc" or in the remote vnodesetup call. In the
  former case, we do not want to "plabnode free" it later. In the later,
  we want to plabnode free it right away, and make sure we do not try to
  remote vnode teardown or plabfree it later. In either case, os_setup
  needs to check so that it does not bother waiting for the node since
  it is wasted time. I use an alternate dead state for this, but the
  real solution is to move much of the vnode specific code from os_setup
  to vnode_setup.
  
  Note that this stuff is mostly untested since I need nodes to fail!
  The normal path works fine though.
  
  Revision  Changes    Path
  1.77      +21 -12    testbed/tbsetup/os_setup.in
  1.28      +70 -31    testbed/tbsetup/vnode_setup.in


stoller     2003/09/19 14:37:12 MDT

  Modified files:
    www                  nodecontrol_list.php3 
  Added files:
    www                  plabmetrics.php3 
  Log:
  Add very primitive plabmetrics page. Needs a bunch of work (mostly
  sorting options).
  
  Revision  Changes    Path
  1.37      +2 -1      testbed/www/nodecontrol_list.php3


ricci       2003/09/19 12:40:21 MDT

  Modified files:
    assign               anneal.cc score.cc score.h 
  Log:
  Some new behavior - features and desires do not generate violations
  when mapping fixed nodes.
  
  Revision  Changes    Path
  1.17      +5 -5      testbed/assign/anneal.cc
  1.38      +12 -10    testbed/assign/score.cc
  1.18      +1 -1      testbed/assign/score.h


stoller     2003/09/19 09:15:51 MDT

  Modified files:
    www                  dbdefs.php3.in showexp.php3 
                         showstuff.php3 toggle.php 
  Added files:
    www                  editexp.php3 
  Log:
  Redo the Edit Experiment Metadata page; turned it into a standard form
  based page that looks like the original Begin Experiment page. Be sure
  to look at the page in both admin mode and non-admin mode since I had
  some trouble determining how swappable is treated these days.
  
  Oh, added the ability to convert non-batch experiments into batch, and
  back. The experiment must be unlocked and in the swapped state to go
  in either direction.
  
  Also added the cpu_usage and mem_usage slots for editing. I added a
  comment about planetlab only, since otherwise we would just confuse
  normal users who have no idea what they mean. I could conditionalize
  them on having plab nodes, but thats difficult to figure out in the
  web page when the experiment is swapped out, so lets not worry about
  it.
  
  Revision  Changes    Path
  1.56      +12 -0     testbed/www/dbdefs.php3.in
  1.70      +2 -131    testbed/www/showexp.php3
  1.115     +28 -103   testbed/www/showstuff.php3
  1.11      +2 -73     testbed/www/toggle.php


mike        2003/09/18 23:23:04 MDT

  Modified files:
    tbsetup/plab         GNUmakefile.in 
  Added files:
    tbsetup/plab         plabstats.in 
  Log:
  First whack at a per-node stats program that will make a stats web page go.
  
  Usage: plabstats [-dfh] [-CDHILMS]
    -d    print debug diagnostics
    -f    fetch new data, else use what is in /tmp/plabxml
    -h    this help message
    -i    print IP address along with metrics
    -n    do not print hostname with metrics
  
    -C    print Ganglia CPU metrics, sorted by %CPU usage
    -D    print Ganglia disk metrics, sorted by %disk usage
    -L    print Ganglia load metrics, sorted by one minute load
    -M    print Ganglia memory metrics, sorted by %mem usage
    -S    print Emulab state info, summarizing per-node availability
  
  Default is to print a terse summary of per-node resource usage.
  Use "plabstats -f" to get fresh data or try something whacky like:
  	plabstats -S | grep accept_3
  to get the list of nodes which are currently available for mapping
  by a "level 3" (aka, average) resource consuming experiment, or:
  	plabstats -S | grep reject
  to get info about the nodes that cannot be used along with the reason(s) why.
  
  Needs some refinement:
    plabmetrics should store raw info into the DB where plabstats can get it
    presentation of Emulab state info should be improved
  
  Revision  Changes    Path
  1.3       +1 -1      testbed/tbsetup/plab/GNUmakefile.in


kwebb       2003/09/18 18:36:13 MDT

  Modified files:
    tbsetup/plab         libplab.py.in 
  Log:
  A couple of enhancements to libplab:
  
  - new SENDMAIL function that mirrors the perl lib's
  - retry logic for communicating with the dslice agent when a slice/sliver is
    being deleted (plabnode free)
  
  One thing I'd like to do is write a wrapper class around agent and node
  manager communication since it can be flaky - would clean up a and simplify
  things.  Maybe just wait since we're going to have to port over to dynamic
  slices soon enough.
  
  Revision  Changes    Path
  1.16      +75 -1     testbed/tbsetup/plab/libplab.py.in


stoller     2003/09/18 15:13:59 MDT

  Modified files:
    tbsetup              tbswap.in vnode_setup.in 
  Log:
  Changes to the allocstate machine to support swapmod on widearea nodes
  in general, and plab nodes in particular.
  
  Revision  Changes    Path
  1.26      +14 -9     testbed/tbsetup/tbswap.in
  1.27      +21 -11    testbed/tbsetup/vnode_setup.in


stoller     2003/09/18 14:04:35 MDT

  Modified files:
    tbsetup/ns2ir        sim.tcl.in 
  Log:
  Add a LINKTEST agent, that fires to all nodes. This is to support the
  link testing stuff, and will eventually be under user control, but for
  now we stick this agent into every experiment, although most of the
  time no one will be listening for it.
  
  Revision  Changes    Path
  1.52      +3 -0      testbed/tbsetup/ns2ir/sim.tcl.in


ricci       2003/09/18 11:18:28 MDT

  Modified files:
    assign               score.cc score.h 
  Log:
  Add some new ways to treat features.
  
  Features that begin with a '*' have some kind of global scope. Since
  that, by itself, doesn't mean much, the second character of the
  feature name must tell us what type of global feature it is. The two
  currently supported are:
  
  '&' - 'one is okay' - don't penalize the first pnode that has this
      feature, but penalize subsequent ones. Will be used to prefer
      not to give people multiple nodes at the sime widearea site.
  '!' - 'more than one is okay' - penalize the first pnode that has
      this feature, but not subsequent ones. This could be used to let
      assign pick experiments to swap out in order to let the current
      experiment map. The idea being that once you've decided that one
      of the experiment's nodes is getting stolen, you might as well
      swap the whole thing.
  
  For now, these global features should not have corresponding desires,
  because it's not clear what the right thing to do would be. But, we may
  decide to add these semantics someday.
  
  Revision  Changes    Path
  1.37      +86 -10    testbed/assign/score.cc
  1.17      +0 -1      testbed/assign/score.h


kwebb       2003/09/17 02:56:24 MDT

  Modified files:
    tbsetup/plab         libplab.py.in plabnode.in 
  Log:
  Several updates to libplab.py and plabnode.in
  
  - getfree daemon doesn't die anymore when communcation with the plab dslice
    agent fails.
  
  - the link classifier logic has been changed slightly to allow nodes
    to be classified as inet2 even if they don't reverse resolve.  The problem
    here is that intl nodes that don't resolve, but which go through abilene
    will look like inet2 nodes, which is wrong.  Manual verification of the
    node_auxtypes table is still recommended.
  
  - The fping verifier has been disabled for now (since some plab nodes
    block ICMP traffic).
  
  - made some error messages more descriptive
  
  - plabnodes script now handles more agent communication errors gracefully
   (retries when if encounters them).
  
  - rearranged plabnode's retry loops to be a little easier to read, and
    more general.
  
  Revision  Changes    Path
  1.12      +30 -19    testbed/tbsetup/plab/libplab.py.in
  1.4       +29 -19    testbed/tbsetup/plab/plabnode.in


stoller     2003/09/16 16:03:17 MDT

  Modified files:
    tmcd/linux           ixpboot rc.ixp 
  Log:
  More IXP changes:
  
  * Put a hosts file in /opt/config, which gets copied to /etc/hosts on
    the IXP.
  * Put a resolv.conf in /opt/config, which gets copied to /etc on
    the IXP.
  * Add support for startcmd, which will override the default boot.
  * Do not squash root in the /opt export.
  
  Revision  Changes    Path
  1.2       +88 -31    testbed/tmcd/linux/ixpboot
  1.2       +8 -6      testbed/tmcd/linux/rc.ixp


ricci       2003/09/16 15:54:01 MDT

  Modified files:
    tbsetup              vnode_setup.in 
  Log:
  Add some paralelism - do up to $MAX_CHILDREN (currently 8) vnodes at
  a time.
  
  Revision  Changes    Path
  1.24      +130 -41   testbed/tbsetup/vnode_setup.in


kwebb       2003/09/16 15:15:10 MDT

  Modified files:
    tbsetup/plab         libplab.py.in 
  Log:
  Small change to libplab to have it try to pull the plab rootball from
  a little local loopback httpd server which now runs on all slivers of
  the static utah1 slice.
  
  It falls back to pushing the tarball from Emulab central if the local
  fetch fails.  Also introduced the notion of versioning for the plab rootball.
  For now, just chang the ROOTBALL_NAME def at the beginning of this file; I'd
  like to move this into autoconf or the makefile eventually.
  
  Revision  Changes    Path
  1.11      +33 -15    testbed/tbsetup/plab/libplab.py.in


stoller     2003/09/16 14:00:33 MDT

  Modified files:
    tbsetup              named_setup.in 
  Log:
  Changes to allow for exporting subnodes/vnodes that have their own
  properly routable IPs addresses in the interfaces table.
  
  Revision  Changes    Path
  1.30      +69 -15    testbed/tbsetup/named_setup.in


mike        2003/09/16 12:31:10 MDT

  Modified files:
    tbsetup/plab         plabmetrics.in 
  Log:
  Don't use nodes with less than 10% free disk
  
  Revision  Changes    Path
  1.3       +68 -14    testbed/tbsetup/plab/plabmetrics.in


kwebb       2003/09/15 20:01:06 MDT

  Modified files:
    tbsetup/plab         libplab.py.in plabnode.in 
  Added files:
    tbsetup/plab/etc     fixsudo.sh 
  Log:
  Some PLAB dslice manager updates:
  
  - in addition to asking the dslice agent (on plab) for a list of available
    nodes, we now also fping them all to weed out unresponsive ones.  One
    problem here is that several plab nodes block ICMP; could be
    solved by pinging with nmap (tries both a ICMP, and TCP ping).  This
    affects the plabdaemon getfree command, and subsequently which plab nodes
    appear as "up" in the DB
  
  - Changed slice naming scheme:  we now append the experiment index onto the
    slice name to try to ensure uniqueness (emulab_<pid>_<eid>_<idx>)
  
  - Modified plabnode to try to cope with flaky nodes - there is some retry
    code in there now
  
  - Added the "fixsudo" shell script which is run very first as root (via the
    cumbersome "su" command) to fix sudoers for later sudo use on plab nodes.
  
  Revision  Changes    Path
  1.9       +39 -1     testbed/tbsetup/plab/libplab.py.in
  1.3       +52 -11    testbed/tbsetup/plab/plabnode.in


ricci       2003/09/15 19:19:45 MDT

  Modified files:
    assign               parse_ptop.cc physical.h score.cc 
  Log:
  Some fixes aimed at vnodes:
  
  Don't allow intraswitch links to vnodes on switches - they'll get
  scored as direct links. Since LANs are now directly on switches, there
  were situations where we could have scored the same link as either
  direct or intraswitch, which results in two different scores. This can
  cause those 'contact calfeld' warnings, and might have caused some
  strange scoring fluxuations as assign ran.
  
  When picking a link to a switch, for emulated links, pick the least
  loaded link, not the one with the fewest users. This helps in situations
  where the link bandwidths are heterogenous.
  
  Revision  Changes    Path
  1.30      +1 -0      testbed/assign/parse_ptop.cc
  1.21      +4 -1      testbed/assign/physical.h
  1.36      +46 -13    testbed/assign/score.cc


mike        2003/09/15 16:20:06 MDT

  Modified files:
    tbsetup/plab         plabmetrics.in 
  Log:
  Default to using 15-minute loadave and a cap of 5.0.
  Change to make nodes with stale data unusable (give em a loadave of 999)
  
  Revision  Changes    Path
  1.2       +36 -8     testbed/tbsetup/plab/plabmetrics.in


kwebb       2003/09/12 16:51:14 MDT

  Modified files:
    tbsetup/plab         libplab.py.in 
  Log:
  Several changes to support plab nodes on the mainbed:
  
  1) put linktype in the node_auxtypes table instead of having separate classes
  of vnodes (now they're all one type).
  
  2) Code added to emulabify() to fix the sudoers file pre-overlay.
     - turns out that only "su" is available to supply initial root access
  
  3) uncapped eligibility to allow all plab nodes to filter in.
  
  Other minor things too.
  
  Revision  Changes    Path
  1.8       +29 -23    testbed/tbsetup/plab/libplab.py.in


ricci       2003/09/12 13:08:31 MDT

  Modified files:
    assign               assign.cc anneal.cc solution.h 
                         solution.cc 
  Log:
  Add a new function to test for equality of two scores - just make
  sure they're within some tolerance of each other. Use this so that
  cumulative float point errors don't cause us to spit out scary
  looking false positive warnings.
  
  Revision  Changes    Path
  1.59      +1 -1      testbed/assign/assign.cc
  1.14      +1 -1      testbed/assign/anneal.cc
  1.2       +8 -0      testbed/assign/solution.h
  1.3       +8 -0      testbed/assign/solution.cc


ricci       2003/09/12 11:21:55 MDT

  Modified files:
    assign               anneal.cc 
  Log:
  Fix an infinite loop bug - could only show up in some unlikely
  circumstances with vnodes, but I managed to run into it.
  
  Revision  Changes    Path
  1.13      +13 -7     testbed/assign/anneal.cc


stoller     2003/09/12 10:14:58 MDT

  Modified files:
    tbsetup/plab         GNUmakefile.in 
  Added files:
    tbsetup/plab         plabmetrics.in 
  Log:
  New script to query the plab web site, download the gmetad XML file,
  and push load data into the node_features table. Once every 5 minutes
  from cron on boss.
  
  Revision  Changes    Path
  1.2       +1 -1      testbed/tbsetup/plab/GNUmakefile.in


ricci       2003/09/11 17:48:09 MDT

  Modified files:
    sql                  database-create.sql database-migrate.txt 
    tbsetup              ptopgen.in 
  Log:
  Add a node_features table - just like the node_type_features table,
  but it's per node, not per type. Will initially be used to store
  inforomation about planetlab nodes.
  
  Revision  Changes    Path
  1.181     +12 -2     testbed/sql/database-create.sql
  1.71      +8 -0      testbed/sql/database-migrate.txt
  1.44      +21 -6     testbed/tbsetup/ptopgen.in


ricci       2003/09/11 17:30:52 MDT

  Modified files:
    assign               parse_ptop.cc pclass.cc physical.h 
  Log:
  Add a new 'unique' flag to the ptop file - tells assign that there is
  some reason, which is invisible to assign, that a node should never
  be considered the same as another. This is for scoring, so that assign
  doesn't prefer to give you 2 DSL nodes if you ask for 1 DSL and one
  generic wide-area.
  
  Revision  Changes    Path
  1.29      +4 -0      testbed/assign/parse_ptop.cc
  1.20      +9 -0      testbed/assign/pclass.cc
  1.20      +6 -1      testbed/assign/physical.h


stoller     2003/09/11 14:23:18 MDT

  Modified files:
    tbsetup              assign_wrapper.in 
  Log:
  Widearea changes, mostly started for plab but affects all widearea
  nodes. The intent is to better support last mile types, which right
  now are a total mess cause the types are associated with the virtual
  nodes, and a node can have just a single type. This info has now been
  moved in the node_types_auxtypes table and the node_types table.
  
  Anyway, we no longer use wanassign on unconnected widearea nodes, but
  use assign directly, much like we use assign to allocate virtual nodes
  on local physical nodes. ptopgen inserts the physical widearea nodes
  (even though they are allocated), and the proper counts for the types
  that are available on them. assign will pick the nodes, and
  assign_wrapper will allocate the necessary vnodes on the pnodes (but
  in the case of wideare nodes, the underlying physical node does not
  need to be allocated).
  
  Also added some fixes for dealing with vtypes when used in conjunction
  with widearea nodes. Rather then generating an error like it used to,
  you can create a vtype in your NS file:
  
  	tb-make-soft-vtype mytype {pcvroninet pcvwainet}
  	tb-set-hardware $v0 mytype
  
  and assign_wrapper now looks at the underlying types to figure out
  what it needs. Note: No consistency checking yet; mixing a remote/virt
  and a local/real type will break.
  
  Revision  Changes    Path
  1.120     +94 -34    testbed/tbsetup/assign_wrapper.in


kwebb       2003/09/11 13:35:55 MDT

  Modified files:
    tbsetup/plab         libplab.py.in 
  Log:
  Small change to insertion of PLAB nodes into the DB.  We no longer classify
  the plab vnodes according to link type (they are all of generic "pcplab"
  type and class now), but add the linktype to the node_auxtypes field.  This
  data in this table is then used to tell assign what kinds of types/roles the
  node can take on.
  
  Revision  Changes    Path
  1.7       +6 -1      testbed/tbsetup/plab/libplab.py.in


stoller     2003/09/11 13:06:14 MDT

  Modified files:
    tbsetup              ptopgen.in 
  Log:
  Split out the widearea code, and rework. Added new option, -r, which
  says to include widearea physnodes, with the types set to those listed
  in the node_auxtypes table. Those types will include the last mile
  types that a vnode on a pnode can take on. The :count value for the
  types is set to the min of the number of *free* vnodes on the pnode,
  and the virtnode_capacity in the node_types table. Obviously, if all
  the vnodes are allocated, the physnode will not appear in the ptop
  file. For plab, virtnode_capacity will get set to one in the
  node_types table, thereby preventing "revisitation" of a plab physnode
  in an experiment.
  
  Revision  Changes    Path
  1.43      +91 -34    testbed/tbsetup/ptopgen.in


mike        2003/09/11 12:43:32 MDT

  Modified files:
    tbsetup              batchexp.in 
  Log:
  Use new sitevar that controls how long we let a batch job be idle before
  swapping.
  
  This is primarily for the "queue an interactive job" form of batching,
  but applies to all batch jobs at the moment.
  
  Revision  Changes    Path
  1.36      +12 -1     testbed/tbsetup/batchexp.in


ricci       2003/09/10 18:32:11 MDT

  Modified files:
    www                  newnode-defs.php3 newnodecheckin.php 
                         newnodes_list.php3 
  Log:
  Add support for Wisconsin's naming scheme, or more generally, any
  naming scheme that ends in -A, where A is a letter, not a number.
  
  Revision  Changes    Path
  1.5       +18 -2     testbed/www/newnode-defs.php3
  1.5       +105 -64   testbed/www/newnodecheckin.php
  1.4       +17 -4     testbed/www/newnodes_list.php3


ricci       2003/09/10 13:55:32 MDT

  Modified files:
    assign               pclass.cc 
  Log:
  Fix a bug/memory leak - assign should now stay at a constant memory
  footprint as it runs. All the credit for this one goes to Tim!
  
  Revision  Changes    Path
  1.19      +0 -2      testbed/assign/pclass.cc


stoller     2003/09/10 11:09:05 MDT

  Modified files:
    sql                  database-create.sql database-migrate.txt 
  Log:
  Add new table for mapping auxtypes back to their parent in the
  node_types table. This should make it easier to assign last mile
  types to widearea nodes since a given type can now have as many
  subtypes as you like.
  
  Also add cpu_usage and mem_usage to the experiments table. The
  user will be able to specify these in the NS file. Currently a
  tinyint, where the default is zero, which means ignore.
  
  Revision  Changes    Path
  1.180     +13 -0     testbed/sql/database-create.sql
  1.70      +16 -0     testbed/sql/database-migrate.txt


mike        2003/09/09 15:41:55 MDT

  Modified files:
    os                   install-tarfile 
  Log:
  "Fix" to deal with NFS hup-to-mountd-screws-us-all race:
  add -c option to force copying file over locally before untaring,
  the copy code knows how to retry read operations that fail
  
  Revision  Changes    Path
  1.6       +90 -33    testbed/os/install-tarfile


ricci       2003/09/09 14:44:48 MDT

  Modified files:
    sql                  database-create.sql database-migrate.txt 
    tbsetup              ptopgen.in 
  Log:
  Add a new table, node_auxtypes - this table contains additional types
  that should be associated with a node when it is put into the ptop
  file.
  
  Revision  Changes    Path
  1.179     +12 -1     testbed/sql/database-create.sql
  1.69      +5 -0      testbed/sql/database-migrate.txt
  1.42      +18 -0     testbed/tbsetup/ptopgen.in


ricci       2003/09/04 17:16:50 MDT

  Modified files:
    assign               anneal.cc common.h parse_ptop.cc 
                         parse_top.cc pclass.cc physical.h 
                         score.cc score.h virtual.h 
  Log:
  Teach assign about subnode relationships - on node lines in both the
  top and ptop files, you can put
  subnode_of:nodename
  Any vnode that is a subnode must be mapped to a pnode that is a a
  subnode of the pnode the vnode's parent is mapped to. Read that a few
  times slowly, I swear it makes sense.
  
  Revision  Changes    Path
  1.12      +1 -1      testbed/assign/anneal.cc
  1.19      +5 -0      testbed/assign/common.h
  1.27      +18 -0     testbed/assign/parse_ptop.cc
  1.27      +20 -6     testbed/assign/parse_top.cc
  1.17      +8 -0      testbed/assign/pclass.cc
  1.17      +7 -1      testbed/assign/physical.h
  1.33      +77 -2     testbed/assign/score.cc
  1.16      +4 -2      testbed/assign/score.h
  1.18      +8 -1      testbed/assign/virtual.h


950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991
stoller     2003/09/04 08:30:38 MDT

  Modified files:
    tbsetup/ns2ir        lanlink.tcl node.tcl parse-ns.in 
                         sim.tcl.in tb_compat.tcl.in 
  Log:
  Add variable netmask support to the parser so that we can handle lots
  more subnets; with vnodes and 1000 node experiments, we running out of
  subnets really quick with just 255 networks (255.255.255.0).
  You can now do this in your NS file:
  
  	tb-set-netmask $lan0  "255.255.240.0"
  	tb-set-netmask $link0 "255.255.255.248"
  	tb-set-netmask $link1 "255.255.255.240"
  
  Yep, more rope for the user to hang himself with. Notes:
  
  * You are restricted to 255.255.XXX.XXX. I did not see a reason to
    allow the user that much rope.
  
  * get_subnet can handle 10 or 192.168 addresses so that other sites
    can continue to operate without changing to 10 addresses, although
    they will still be able to change the netmask.
  
  * I've changed the handling for widearea networks to use 192.168, but
    I force the netmask to 255.255.255.248 so that we are not restricted
    to just 255 networks (not that it really matters). To avoid possible
    confusion, the user is not allowed to choose their own IPs for
    widearea networks, and I actually set them to 1.1.x.x, and then
    patch it up later. This is to avoid conflict with existing
    experiments where people may have used tb-set-ip in their NS files.
  
  * There are tmcd and staticroutes and image changes that are required
    to make this all work right!
  
  Revision  Changes    Path
  1.29      +61 -12    testbed/tbsetup/ns2ir/lanlink.tcl
  1.39      +8 -5      testbed/tbsetup/ns2ir/node.tcl
  1.13      +26 -13    testbed/tbsetup/ns2ir/parse-ns.in
  1.50      +75 -13    testbed/tbsetup/ns2ir/sim.tcl.in
  1.51      +25 -0     testbed/tbsetup/ns2ir/tb_compat.tcl.in

992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088
stoller     2003/08/05 11:41:09 MDT

  Modified files:
    .                    configure configure.in 
    db                   xmlconvert.in 
    os                   GNUmakefile.in 
    tbsetup/ns2ir        parse.tcl.in sim.tcl.in tb_compat.tcl.in 
    tmcd                 libsetup.pm 
    tmcd/common          rc.setup watchdog 
    www                  showstuff.php3 
  Added files:
    os/syncd             GNUmakefile.in decls.h emulab-sync.c 
                         emulab-syncd.c 
  Log:
  The rest of the sync server additions:
  
  * Parser: Added new tb command to set the name of the sync server:
  
  	tb-set-sync-server <node>
  
    This initializes the sync_server slot of the experiment entry to the
    *vname* of the node that should run the sync server for that
    experiment. In other words, the sync server is per-experiment, runs
    on a node in the experiment, and the user gets to chose which node
    it runs on.
  
  * tmcd and client side setup. Added new syncserver command which
    returns the name of the syncserver and whether the requesting node
    is the lucky one to run the daemon:
  
      SYNCSERVER SERVER='nodeG.syncserver.testbed.emulab.net' ISSERVER=1
  
    The name of the syncserver is written to /var/emulab/boot/syncserver
    on the nodes so that clients can easily figure out where the server
    is.
  
    Aside: The ready bits are now ignored (no DB accesses are made) for
    virtual nodes; they are forced to use the new sync server.
  
  * New os/syncd directory containing the daemon and the client. The
    daemon is pretty simple. It waits for TCP (and UDP, although that
    path is not complete yet) connections, and reads in a little
    structure that gives the name of the "barrier" to wait for, and an
    optional count of clients in the group (this would be used by the
    "master" who initializes barriers for clients). The socket is saved
    (no reply is made, so the client is blocked) until the count reaches
    zero. Then all clients are released by writting back to the
    sockets, and the sockets are closed. Obviously, the number of
    clients is limited by the numbed of FDs (open sockets), hence the
    need for a UDP variant, but that will take more work.
  
    The client has a simple command line interface:
  
      usage: emulab-sync [options]
      -n <name>         Optional barrier name; must be less than 64 bytes long
      -d                Turn on debugging
      -s server         Specify a sync server to connect to
      -p portnum        Specify a port number to connect to
      -i count          Initialize named barrier to count waiters
      -u                Use UDP instead of TCP
  
      The client figures out the server by looking for the file created
      above by libsetup (/var/emulab/boot/syncserver). If you do not
      specify a barrier "name", it uses an internal default. Yes, the
      server can handle multiple barriers (differently named of course)
      at once (non-overlapping clients obviously).
  
      Clients can wait before a barrier in "initialized." The count on
      the barrier just goes negative until someone initializes the
      barrier using the -i option, which increments the count by the
      count. Therefore, the master does not have to arrange to get there
      "first." As an example, consider a master and one client:
  
  	nodeA> /usr/local/etc/emulab/emulab-sync -n mybarrier
  	nodeB> /usr/local/etc/emulab/emulab-sync -n mybarrier -i 1
  
      Node A waits until Node B initializes the barrier (gives it a
      count).  The count is the number of *waiters*, not including the
      master. The master is also blocked until all of the waiters have
      checked in.
  
      I have not made an provision for timeouts or crashed clients. Lets
      see how it goes.
  
  Revision  Changes    Path
  1.166     +1 -0      testbed/configure
  1.173     +1 -0      testbed/configure.in
  1.5       +2 -1      testbed/db/xmlconvert.in
  1.14      +10 -2     testbed/os/GNUmakefile.in
  1.42      +3 -0      testbed/tbsetup/ns2ir/parse.tcl.in
  1.49      +6 -0      testbed/tbsetup/ns2ir/sim.tcl.in
  1.47      +14 -0     testbed/tbsetup/ns2ir/tb_compat.tcl.in
  1.60      +72 -2     testbed/tmcd/libsetup.pm
  1.9       +5 -0      testbed/tmcd/common/rc.setup
  1.5       +2 -1      testbed/tmcd/common/watchdog
  1.103     +8 -0      testbed/www/showstuff.php3

1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148
kwebb       2003/07/22 18:48:44 MDT

  Modified files:
    tmcd/linux           GNUmakefile.in 
  Added files:
    tmcd/linux           delaysetup 
  Log:
  Here we have the Linux delaysetup startup script - first revision.  This
  uses iproute2+tc, modprobe, and iptables to setup traffic shaping.
  A few notes are warranted:
  
  1) [g]red is not yet supported
     - need to make these modules classful in the kernel first
  
  2) sysctls are used here to up the amount of buffer space available for
     sk_bufs (socket buffers).  Couldn't see a place to do this in the kernel
     config.
  
  3) Only linkdelay support is implemented
     - probably could add normal delay-node support without too much trouble.
  
  4) reverse pipe numbers are (currently) ignored
     - not needed since the IMQ device used to shape incoming traffic
       is distinct from the actual interface - no namespace collision.
  
  5) Kernel selection is similar to FBSD: check running kernel, and reboot
     if the kernel version isn't what we expect (have to rerun lilo too)
  
kwebb       2003/07/22 18:40:34 MDT

  Added files:
    delay/linux          README 
    delay/linux/iptables_mods iptables-1.2.6a-imq.diff-3 
    delay/linux/kmods    patch-combined-linkdelays-2.4.18 
    delay/linux/tc_mods  
                         patch-combined-iproute2-2.4.7-now-ss020116-try 
  Log:
  This commit represents the culmination of my work on linux traffic shaping.
  Both changes to the kernel, and to userland control applications are included
  in this checkin.  At present, these files are combined patches; I'll be
  separating out the Emulab specific additions/changes in the near future.
  
  The best resource (currently) for understanding the Linux traffic control
  mechanisms is at the Linux Diffserv project home page (under Documentation):
  http://diffserv.sourceforge.net/
  
  The most noteworthy contributions I've made are the PLR module and DELAY
  module port and enhancement.  These two modules are classful (though they
  don't use classification filters) and can be chained together with other
  modules such as the Heirarchical Token Bucket (used for limiting bandwith).
  See: http://luxik.cdi.cz/~devik/qos/htb/
  
  Note that this checkin also includes the patches necessary to obtain
  Intermediate Queueing Devices (IMQ) and HTB control under iproute2+tc.  The
  IMQ devices allow the attachment of qdiscs on the incoming side of a real
  interface (can't do this without them). See: http://trash.net/~kaber/imq/
  
  The Linux Advanced Routing HOWTO is another good source of information:
  http://lartc.org

Leigh B. Stoller's avatar
Add  
Leigh B. Stoller committed
1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174
stoller     2003/07/15 15:25:03 MDT

  Modified files:
    db                   libdb.pm.in 
    tbsetup              assign_wrapper.in os_setup.in tbswap.in 
                         vnode_setup.in 
  Log:
  A set of changes to make swapmod work on jailed nodes (note, swapmod
  does not yet work with remove virtual nodes; that will take even more
  work).
  
  Added a new allocstate called RES_TEARDOWN. assign_wrapper no longer
  deallocates unused nodes, but rather moves them into the new state for
  the wrapper (tbswap) to deal with. Thats cause deleted vnodes need to
  be torn down, since its possible that the node on which they were
  living will not be deallocated (say, if there are other vnodes on
  it). We do not want to be doing that from assign_wrapper, so tbswap
  looks for those nodes.
  
  Made vnode_setup allocstate aware in the same way that os_setup is;
  do not reboot vnodes or try to set up vnodes when they are already in
  the RES_READY state, as they will be when doing a swapmod. In
  addition, if os_setup is going to reboot the underlying physnode, move
  the vnodes on that node into RES_READY too, since there they will
  setup automatically. Might need an interim state here, for correctness.
  
1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194
ricci       2003/07/14 18:27:27 MDT
  Modified files:
    utils                newnode.in
    www                  newnodecheckin.php
  Added files:
    www                  newnode-defs.php3 newnode_edit.php3
                         newnodes_list.php3
  Log:
  Web interface for autoconfiguring nodes. This consisits of two pages,
  one that lists and allows operations on a number of nodes at once, and
  another that allows editing of an individual node.

  Split much of the functionality of newnodecheckin.php into a new
  library, so it can get re-called from newnodes_list.php3 later.

  Move where we look for MACs on the switches - instead of doing in in
  newnode, call it from newnodecheckin.php - this provides the admins
  more feedback as to what's going on, and allows them to call it again
  later or put the information in by hand.

Leigh B. Stoller's avatar
Leigh B. Stoller committed
1195 1196 1197 1198 1199 1200 1201 1202 1203
stoller     2003/07/08 17:34:42 MDT

  Modified files:
    tbsetup              GNUmakefile.in 
  Log:
  Turn off mainbed exclusion of ns2ir build/install, now that parser
  has been working on minibed okay for a while. In other words, the new
  parser really is going to be installed on mainbed, any moment!
  
1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283
ricci       2003/07/01 10:14:42 MDT

  Modified files:
    assign               anneal.cc anneal.h assign.cc 
                         assign_todo.txt pclass.cc pclass.h 
                         physical.h score.cc 
  Log:
  Give assign the ability to make dynamic pclasses. Here's how it works:
  After building the set of pclasses normally, we make another pass
  through the vnodes. The goal to create a pclass for each individual
  node. We disable the node's 'own' pclass to begin with. Then, the
  _first_ time it gets a vnode mapped to it, we remove it from the
  'regular' pclass it belongs to, and enable it's own pclass. Then, if
  it goes empty, we put it back in its regular pclass and disable it's
  own.
  
  The point of this is to replace -p for use with multiplexed nodes.
  Instead of disabling pclasses altogheter, which has serious
  performance implications, we can instead be smart about which pnodes
  remain equivalent (because nothing's been mapped to them), and which
  are now different. The result is that on my test topoloy, the time to
  get a good mapping has gone from over 3 minutes to about 6 seconds.
  
  This feature is enabled with the -d option, and the -P option is
  pretty much mandatory when using it, since it greatly exacerbates the
  problem of cruft in the ptop file.
  
  This satisfies #14 from the todo file:
  14.  do dynamic pclasses
  
  Also bumped up the minimum neighborhood size from 500 to 1000. In some
  tests I was doing, this resulted in better solutions.
  
ricci       2003/06/26 13:13:16 MDT

  Modified files:
    assign               anneal.cc anneal.h assign.cc 
                         assign_internals.txt parse_ptop.cc 
                         pclass.cc physical.h score.cc solution.cc 
    tbsetup              assign_wrapper.in ptopgen.in 
  Log:
  Major changes to the way assign handles LAN nodes.
  
  LAN nodes are no longer treated specially. Instead, I've introduced
  the idea of 'static' types (old-style types retroactively become
  'dynamic' types). While a pnode can only satisfy one dynamic type at a
  time, it can always satisfy its static types (assuming it has enough
  capacity left.) Static types are flagged by prepending them with a '*'
  in the ptop file. So, for example, you may give switches the
  '*lan:10000' type so that they can satisfy virtual LAN nodes. Of
  course, other pnodes can have this type too, so that we can get
  'trivial LANs'.
  
  Actually, removing special treatment for LANs cleans up a lot of code.
  However, it may have some negative impacts on solutions, since we're
  not as smart about where to place LAN nodes as we used to be (they get
  annealed along with everything else, and not migrated.) I haven't seen
  any evidence of this yet, however.
  
  This leaves us with a single type of special pnode, a switch.
  
  Also added a new bit of syntax in ptop files - when '*' is given as a
  the maxiumum load for a type, the node is allowed to take on an
  infinite (well, actually, just a really big number of) vnodes of that
  type.
  
  ptopgen was modified to always report switches as being capable of
  hosting LANs, and assign_wrapper now understands direct links to LANs,
  which is what we get when the LAN is hosted directly on a switch.
  
  Fixed a bug in scoring direct links, in which the penatly was being
  added once when a direct link was mapped, but subtracted only once
  when it was freed.
  
  Added a '-T' option for doing simple self-testing. When adding a node
  to the solution, assign records the score, adds the node, removes it
  again, and checks to make sure that the resulting score is the same as
  the original score. The usefulness of this feature in debugging
  scoring problems cannot be understated...

1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323

stoller     2003/06/30 11:33:58 MDT

  Modified files:
    .                    GNUmakefile.in configure configure.in 
    db                   xmlconvert.in 
    tbsetup              GNUmakefile.in batchexp.in nscheck.in 
    tbsetup/ns2ir        GNUmakefile.in lanlink.tcl node.tcl 
                         parse-ns.in parse.tcl.in program.tcl 
                         sim.tcl.in tb_compat.tcl.in traffic.tcl 
                         vtype.tcl 
    testsuite/tests/db1  test 
  Added files:
    tbsetup/ns2ir        parse.proxy.in 
  Log:
  Make the new parser live on mini. New parser ssh'es over to ops to
  do the actual parse, as the user. The parser now spits out XML instead of
  DB queries, and the wrapper on boss converts that to DB insertions after
  verification, using the new xmlconvert script. There are some makefile
  changes as well to install the new parser on ops via NFS, since
  otherwise the parser could intolerably out of date on ops!


stoller     2003/06/26 15:30:02 MDT

  Modified files:
    db                   GNUmakefile.in 
  Added files:
    db                   xmlconvert.in 
  Log:
  New script. Convert experiment to "well formed" XML, and back again.
  The conversion from XML to an experiment representation updates the
  DB, although the experiment has to exist, and only certain fields can
  be updated. For the rest of the virtual toplogy, the old toplogy is
  deleted, and the new one installed. Use with caution. This is intended
  to be hooked to the parser, and later to the vis tool so that it does
  not need to have to do its own NS parsing, but can simply interpret the
  XML. 
  

1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335
stoller     2003/06/23 13:55:30 MDT

  Modified files:
    cdrom/waboot         netbed-setup.pl register.pl waipconfig.pl 
  Log:
  Add support for DHCP to the NetBed CDROM.
  
  Revision  Changes    Path
  1.5       +19 -8     testbed/cdrom/waboot/netbed-setup.pl
  1.11      +17 -0     testbed/cdrom/waboot/register.pl
  1.8       +77 -4     testbed/cdrom/waboot/waipconfig.pl

1336
stoller     2003/05/22 15:35:51 MDT
1337

1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354
  Modified files:
    db                   libdb.pm.in 
    tbsetup              batch_daemon.in batchexp.in endexp.in 
                         swapexp.in 
    www                  beginexp.php3 swapexp.php3 
  Log:
  Reorg the batch system slightly as per Eric's request that batch mode
  experiments look more like regular experiments. Batch mode experiments
  can now be preloaded and swapped. When preloaded, they go into a
  "Pause" state. Swapping a batch mode experiment in puts them into the
  "posted" state so the batch daemon will see them. Swapping out a
  batchmode experiment does the expected; it puts them back into the
  Pause state. Terminating a batch mode experiment does the expected;
  its gone. When a batch mode experiment finishes normally, it goes back
  into the pause state, which allows batches to be reinjected as many
  times as Eric likes.
	
1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376
stoller     2003/05/1 09:09:12 MDT
	
  Modified files:
    event/sched          event-sched.c event-sched.h 
    lib/libtb            tbdefs.c tbdefs.h 
    sql                  database-fill.sql 
    tbsetup              eventsys_control.in 
    tbsetup/ns2ir        sim.tcl.in 
  Log:
  Add the long desired halt/swap event directives. You can now put this
  in your NS file:
  
  	$ns at 2000.0 "$ns halt" (or terminate)
  or
  	$ns at 2000.0 "$ns swapout"
  
  The first causes the experiment to terminate, the later causes it to
  swap out. The units are seconds, as are all "at" statements.

  NOTE: You can use halt (terminate) to cancel a batch experiment, but you
	cannot schedule a swapout.

1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396
stoller     2003/04/16 11:41:12 MDT

  Modified files:
    .                    configure configure.in 
    tbsetup              GNUmakefile.in swapexp.in 
    www                  showexp.php3 swapexp.php3 
  Added files:
    tbsetup              idleswap.in webidleswap.in 
  Log:
  Add support for idleswapping an experiment as the creator of the
  experiment, rather than as an administrator, which presents group
  permission problems when the experiment is in a subgroup (requires two
  additional group, whereas suexec adds only one group). That aside, the
  correct approach is to run the swap as the creator. To do that, must
  flip to the user (from the admin person) in the backend using the new
  idleswap script, and then run the normal swapexp. Add new option to
  swapexp (-i) which changes the email slightly to make it clear that
  the experiment was idleswapped, and so that the From: is tbops not the
  user (again, to make it more clear).
  
Mac Newbold's avatar
Mac Newbold committed
1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430
newbold     2003/04/09 16:07:12 MDT

  Modified files:
    .                    configure configure.in defs-barb-emulab 
                         defs-barb-mini defs-calfeld-emulab 
                         defs-default defs-example 
                         defs-kwebb-emulab defs-mini 
                         defs-newbold-emulab defs-newbold-macdb 
                         defs-newbold-mini defs-rchriste-emulab 
                         defs-ricci-emulab defs-ricci-mini 
                         defs-shash-emulab defs-shash-mini 
                         defs-stoller-emulab defs-stoller-home 
                         defs-stoller-mini defs-uky defs-wide 
  Added files:
    account              quotamail.in 
  Log:
  Add new script to notify users when they are over their quota. Runs from
  cron daily, on the fs node (ops) as root. Uses lots of configure variables
  so that it does the right thing in any installation, without any
  customization.
  
  One possible caveat regarding quotas: If FS_NODE != USERNODE, they don't
  have a login on the fs machine. So checking their quota won't work unless
  the rpc.rquotad(8) daemon is running on FS_NODE, which we currently don't
  do.
  
  In order to do this right, I had to add a new configure var,
  FS_WITH_QUOTAS, that has a space separated list of file systems that have
  quotas enabled. (Ie the default is 'FS_WITH_QUOTAS="/q /users"'.) It
  doesn't have any default, since I couldn't come up with a reasonable one.
  All the defs files have been updated appropriately to define this new
  variable.


Mike Hibler's avatar
Mike Hibler committed
1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458
mike        2003/04/09 15:18:00 MDT
	
  Semi-major frisbee overhaul:

  Implement PREQUEST message which passes a bit map of desired blocks.
  We still use the REQUEST message (start block + number of blocks) for
  full chunk requests as that is more efficient.

  Allow dynamic number of variable-sized write buffers.  Total memory
  not to exceed the writebufmem limit.

  Eliminate a complete copy of the uncompressed data in the client.

  Fix a bug in the client where we would not keep our request-ahead window full.

  Fix a bug in the server where we would not merge incoming requests with
  the request currently being serviced.
	
  The above changes/bugfixes gave us about a 5% improvement in run time locally.

  Add some client/server convenience options.  For the server, -W allows 
  specification of a target bandwidth.  For the client, -C, -W and -M
  give various ways of specifying how much memory to use for net/disk
  buffering.

  Security: -S option tells client to only accept packets from a specific
  IP address.
	
1459 1460 1461 1462 1463
stoller     2003/04/08 03:03:01 MDT

  Update Linux image to Redhat 7.3. The old image is still available to
  users by using the OSID RHL71-STD in their NS files. 
	    
1464
newbold     2003/04/08 12:32:01 MDT
1465 1466

  Modified files:
1467 1468 1469 1470 1471 1472
    .                    configure configure.in 
    db                   GNUmakefile.in 
  Added files:
    db                   idlemail.in idletimes.in webidlemail.in 
  Removed files:
    db                   idlecheck.in 
1473
  Log:
1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489
  Remove idlecheck. Add idletimes and idlemail (and webidlemail).
  (webidlecheck was removed in a separate commit.)
  
  idletimes is the new commandline tool for checking idleness. It shows a
  table like this:
  
 PID          EID          Nodes  HrsIdle     Last Activity     Act. type
 ------------------------------------------------------------------------
 AVQ          Tunnel           4   15.18   2003-04-07 21:18:20  tty
 pces         toy              1    3.92   2003-04-08 08:34:02  tty
 Spinglass    fawaz            1   13.76   2003-04-07 22:43:48  tty
 TempleSCTP   FTP-REAL         1   26.40   2003-04-07 10:05:18  tty,net
 testbed      bvclass          4   24.58   2003-04-07 11:54:13  net
 testbed      ltest            1   14.37   2003-04-07 22:07:02  tty
 testbed      ron-image        1  501.88   2003-03-18 13:36:28  cpu,ext,tty,net
 testbed      Tone             1  501.88   2003-03-18 13:36:30  cpu,ext,tty,net
1490
  
1491 1492 1493 1494 1495 1496 1497
  By default it only shows things that are over an idleness threshold. It
  also has a mode to show all expts. Also can show idleness/activity by node
  instead of by expt, which has a slightly different table. (s/Nodes/NodeID/)
  
  idlemail isn't finished yet, but I don't want it to hold up idletimes and
  the other changes any longer. It doesn't get used by anything yet, of
  course.
1498

1499
stoller     2003/04/04 10:20:31 MST
1500 1501

  Modified files:
1502 1503 1504 1505 1506 1507
    sql                  database-migrate.txt 
    www                  approveproject_list.php3 
                         approveuser_form.php3 joinproject.php3 
                         moduserinfo.php3 newproject.php3 
                         showstuff.php3 verifyusr.php3 
                         widearea_register.php 
1508
  Log:
1509 1510 1511 1512 1513 1514 1515
  Add "Address Line 2", City, State, and Zip as independent entities,
  finally! These have been in the DB for a long time, but never used.
  As Eric pointed out, its easier to get realistic address info from
  people if we provide the right forms. The downside is that all current
  users will have to adjust their info the next time they edit their
  info. Thats okay. I thought about forcing all users to do it the next
  time they log in, but I figured people would scream.
1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545

newbold     2003/04/02 15:24:48 MST

  Modified files:
    .                    configure configure.in defs-barb-emulab 
                         defs-barb-mini defs-calfeld-emulab 
                         defs-default defs-example 
                         defs-kwebb-emulab defs-mini 
                         defs-newbold-emulab defs-newbold-macdb 
                         defs-newbold-mini defs-rchriste-emulab 
                         defs-ricci-emulab defs-ricci-mini 
                         defs-shash-emulab defs-shash-mini 
                         defs-stoller-emulab defs-stoller-home 
                         defs-stoller-mini defs-wide 
    www                  defs.php3.in 
  Removed files:
    www                  barb-emulab-defs.php3 barb-mini-defs.php3 
                         default-defs.php3 example-defs.php3 
                         kwebb-emulab-defs.php3 mini-defs.php3 
                         newbold-emulab-defs.php3 
                         newbold-mini-defs.php3 
                         ricci-emulab-defs.php3 
                         ricci-mini-defs.php3 
                         shash-emulab-defs.php3 
                         shash-mini-defs.php3 
                         stoller-emulab-defs.php3 
                         stoller-home-defs.php3 
                         stoller-mini-defs.php3 wide-defs.php3 
  Log:
  Massive reworking of our structure for defs files.
1546
  
1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571
  Before:
  The main defs file (ie for configure) had an entry for WWWDEFS that
  pointed to a <@WWWDEFS@>-defs.php3 file in the www/ directory. The www
  defs file loaded some values about web pages, URLs, and some web
  configuration parameters.
  
  Problem:
  Anything that was only in the www defs file was not accessible in the rest
  of the universe (ie perl, C, and any other non-web-page scripts). For
  instance, you couldn't have a perl script send an email to a user with a
  link to the web site.
  
  Solution:
  Nuke all the www-defs files, move any important values into the main
  configure, and change the web defs infrastructure to respect that. This
  also meant adding about 3 lines each to all of the configure defs files.
  (There really are about 10 new values you can change in your defs file,
  but in almost all cases, the default values are the right thing.)
  
  Upgrading:
  External sites will need to move a few variables from their www-defs file
  into their configure defs file. The example file should make it pretty
  obvious. They may also want to customize some of the other vars that are
  mentioned in configure.in and www/defs.php3.in .

1572
stoller     2003/03/29 10:23:30 MST
1573 1574

  Modified files:
1575 1576 1577
    www                  approveproject.php3 joinproject.php3 
                         login.php3 newgroup.php3 newproject.php3 
                         reqaccount.php3 
1578
  Log:
1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590
  Add target_pid,target_gid arguments to joinproject page so that
  we can send links that cause those fields to be filled in for people.
  Add those links to email generated by newproject and newgroup pages
  so that they can be saved by the leaders.
  
  Add a bit of referrer magic to login page. When clicked from the join
  or new project pages, tell login page to pass along the referrer page
  so that when login is complete, user is zapped back to the original
  page. This is especially nice when combined with the above change to
  joinproject, whereby we send along the target pid/gid, but the user
  has not yet logged in and remembers to do so via the link at the top
  of the page.
Mac Newbold's avatar
Mac Newbold committed
1591

1592
stoller     2003/03/27 17:01:52 MST
Mac Newbold's avatar
Mac Newbold committed
1593 1594

  Modified files:
1595 1596
    db                   libdb.pm.in 
    www                  dbdefs.php3.in 
Mac Newbold's avatar
Mac Newbold committed
1597
  Log:
1598 1599 1600 1601 1602 1603 1604
  Changes to permission checks for nodes, experiments, and images.
  In addition to the usual checks, project_root and group_root (in the
  project) get the same permission as if they were in the subgroups with
  group_root permission. This means they can swap/terminate/reboot etc
  in subgroups even though they are not members of the subgroups.
  There is still some copy problems with files, but leaving that till
  later to deal with.
Mac Newbold's avatar
Mac Newbold committed
1605
  
1606