xpimage-notes.txt 42.7 KB
Newer Older
1
# Directions for setting up an XP image from scratch.
2 3
# These are raw notes and commands to paste into a shell.
# Mostly Bash shell commands for Windows, some tcsh commands for Boss or Ops.
4 5
# Some (most?) of it could be scriptified with some work. 

6 7
# Notice that this file has spaces instead of tabs at the beginning of lines.
# A tab in either Bash or tcsh causes it to display all of the possible command completions!
8 9 10 11
# Here's a little Emacs keyboard macro to ease the copy-and-paste business:
    ; Copy a command line, leaving off the whitespace on the beginning of the line.
    (fset 'copy-command-line [?\M-m ?\C-  ?\C-e ?\C-f C-insert])
    (global-set-key "\^C\^E" 'copy-command-line)
12

13 14
# By convention, optional "informational" commands are indented a couple of spaces more.
## Debugging and problem-solving stuff is double-# commented.
15 16

alias v 'ls -lsF'               # "Verbose" listing
17 18
setenv en emulab.net
alias rootpc 'sudo ssh pc\!^.$en \!:2*'
19 20
alias rootrd 'rd  -K -g 1280x1024 -u root pc\!^.$en &'

21

22
    . Start with a clean XP image, as it comes from the CD.
23 24 25

      - Swap in experiment Windows-1-base, log in as Administrator.

26 27 28 29 30 31 32 33 34 35
         . The experiment should be behind a firewall, to avoid contamination.

            # Firewall while making Windows images.
            set fw [new Firewall $ns]
            $fw set-type ipfw2-vlan
            $fw set-style basic

            # Allow Cygwin setup and Windows Update to work.
            $fw add-rule "allow tcp from any to any 80,443 in via vlan0 setup keep-state"

36 37 38
      - Set the Windows "w32time" NTP client to connect to the Emulab NTP host.

         . Runs as a service, periodically contacts the time server.
39 40 41 42 43 44 45 46 47 48
           ntp1 is a DNS alias for Ops.
           Need to restart w32time before it sees the setsntp configuration. (?)
             net time /querysntp
             net stop w32time
             net time /setsntp:ntp1
             net time /querysntp
             net start w32time
             # May take a couple of minutes to take effect.
             date
             
49 50 51 52
        . Need an NTP client, or at least the semblence of one.
            cat /etc/ntp.drift
          echo 0.000 > /etc/ntp.drift

53
      - Disable the Messenger Service to keep annoying pop-ups away.
54 55 56
            cygrunsrv -VQ  Messenger
          sc config Messenger start= disabled
          sc stop Messenger
57 58

      - Disable the SSDP Discovery Service and Universal Plug and Play Device Host.
59 60 61 62 63 64 65 66 67 68 69 70
        This closes port 5000 to attacks.  Also disable the Remote Registry service.
            cygrunsrv -VQ  SSDPSRV
            cygrunsrv -VQ  upnphost
            cygrunsrv -VQ  RemoteRegistry

          sc config SSDPSRV start= disabled
          sc config upnphost start= disabled
          sc config RemoteRegistry start= disabled

          sc stop SSDPSRV
          sc stop upnphost
          sc stop RemoteRegistry
71

72 73 74 75 76 77
      - Set the workgroup name to EMULAB in Control Panel/System/Computer Name/Change...
        No need to reboot yet.

      - Make a "root" account in Control Panel/Administrative Tools/Computer Management/
        System Tools/Local Users and Groups/Users, put it in the Administrators and Users groups.

78
        . While you're there, Right-click Start/"Explore All Users" and copy the Computer Management
79
          shortcut from Administrative Tools into the All Users/Desktop folder.
80 81 82 83 84 85 86 87

      - Start IE, make "blank" the home page. Click Tools/Internet Options/Home page/Use Blank.

      - Show My Computer.  (Desktop Properties/Desktop/Customize Desktop...)
        Turn off "Run Desktop Cleanup Wizard every 60 days".

      - Create C:/Temp, C:/Software/CygWin
      - Install CygWin
88 89 90 91 92
        . (Try copying /etc/setup/* from an existing image to save the selection work.)
        . Download setup from www.cygwin.com/setup.exe to C:/Software/CygWin, OR:
           # [On ops.]
           set pc=109
           scp /share/windows/cygwin-setup.exe $pc":"/tmp/setup.exe
93
        . Run it.  Install dir is C:\cygwin, package dir is C:\Software\CygWin .
94 95 96
            # Ensure that upgrading SSH won't hang.
            net stop sshd
            C:/Software/CygWin/setup.exe &
97 98 99
        . Mirror is http://mirrors.xmission.com .
        . Click [View] to "Not Installed" (alphabetical.)  
          Click on the Skip in the "New" column to add a binary version of:
Russ Fish's avatar
Russ Fish committed
100 101
            agetty, bison, cvs, cygrunsrv, ed, file, flex, gcc, gdb, inetutils, 
            make, minires-devel, nano, openssh, openssl-devel, 
102
            patch, perl, perl-libwin32, python, rpm, rsync, 
Russ Fish's avatar
Russ Fish committed
103
            shutdown, tcsh, vim, wget, zip .
104
          Click in the "src" column for openssh, so patches can be applied.
105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130
        . Don't "Create an icon on the Desktop", do "Add icon to Start Menu".

        . Add ;C:\cygwin\bin to the end of the System PATH in 
          Control Panel/System/Advanced/Environment Variables.

        . Start up a Cygwin shell and fix the shell properties:
            Options QuickEdit Mode on, Layout/screen buffer height 3000, window height 55.
            Check "Modify shortcut that started this window".
          - Might as well fix the Start/Programs/Accessories/Command Prompt properties, too.
          - Copy the bash shortcut to the All Users/Desktop.  
          - Copy it to a tcsh icon as well, changing the path to c:\cygin\cygwin-tcsh.bat .
          - Copy the tcsh icon into All Users/Start Menu/Programs/Cygwin.
          - Create c:\cygin\cygwin-tcsh.bat as a copy of c:\cygin\cygwin.bat with
              bash --login -i
            changed to
              tcsh -l

        . Set up local homedirs under /home as a symlink.  ~root is already there.
            cd /tmp
            mv /home{,.orig}
            ln -s /cygdrive/c/Documents\ and\ Settings/ /home

        . Symlink the Windows hosts file into the Cygwin /etc.
            ln -s /cygdrive/c/WINDOWS/system32/drivers/etc/hosts /etc/hosts

        . Create a proper group file.  Make wheel an alias for Administrators.
131 132
            mkgroup -l | \
              awk '/^Administrators:/{print "wheel" substr($0, index($0,":"))} \
133
                   {print}' > /etc/group.new
134 135 136 137 138 139 140 141 142 143
            diff /etc/group{,.new}
            cp -p /etc/group{,.prev}
            mv /etc/group{.new,}

        . Update the passwd file after creating new accounts.  Make root uid 0 with /home/root.
            mkpasswd -l | awk -F: 'BEGIN{ OFS=":" } \
               { if ($1=="root") $3="0"; else sub("/home/", "/users/"); print }' > /etc/passwd.new
            diff /etc/passwd{,.new}
            cp -p /etc/passwd{,.prev}
            cp -p /etc/passwd{.new,}
144
            chown root /etc/{passwd,group}*
145

Russ Fish's avatar
Russ Fish committed
146 147 148 149 150 151 152 153 154 155
        . Set up the syslog daemon.  (See usr/share/doc/Cygwin/inetutils-1.3.2.README)
            # Make sure /etc isn't owned by SYSTEM, which will prevent making syslogd.conf .
            chown root /etc
            syslogd-config -y
            # Start the daemon.  It starts automatically at reboot.
            net start syslogd
              # Test.
              logger "Test syslogd."
              tail /var/log/messages

156
        . Set up sshd.  
157
          - Edit /bin/ssh-host-config to add a -i argument to the "cygrunsrv -I sshd" lines.
158
                grep cygrunsrv /bin/ssh-host-config | grep -e -I
159 160 161 162 163
              ed /bin/ssh-host-config
              /cygrunsrv -I sshd/s//& -i/p
              /cygrunsrv -I sshd/s//& -i/p
              w
              q
164 165
          - Then start a Cygwin shell, stop sshd and remove its entry, run ssh-host-config:
            (You must be logged in as root over RDP, not ssh when you do this!)
Russ Fish's avatar
Russ Fish committed
166 167 168
                cygrunsrv -VQ sshd
              cygrunsrv -E sshd
              cygrunsrv -R sshd
169 170 171 172 173 174 175

              # May need to do some unmounts before running ssh-host-config.
              # It does a mount, and there's a hard-wired limit of 31 mount table entries.
              mount | wc -l
                ## mount: /ssh-host-config.3048: Too many mount entries
                for s in /users/s*; do umount $s; done

Russ Fish's avatar
Russ Fish committed
176
              # Should be NO ssh processes running, not even ssh-agent!
177 178
              ps -Welf | grep ssh

179 180 181
              # Make sure /etc is writable by root.
                v -d /etc
              chown root /etc
182

183
              ssh-host-config -y -c "ntsec tty"
184 185 186
              # or run ssh-host-config without args and answer the following interactive questions:
              # Select privilege separation = yes, sshd user = yes, install as service = yes, 
              # CYGWIN=ntsec tty
187

188
                v /etc/ssh*_config
189 190 191
              chown SYSTEM /etc/ssh*_config
              chmod 644 /etc/ssh*_config

192 193 194 195
          - Check for -i flag: look for Interactive = 0x00000001 (1)
              regtool -v list /HKLM/SYSTEM/CurrentControlSet/Services/sshd/Parameters

          - Edit /etc/sshd_config
196
            . Add AuthorizedKeysFile paths under /sshkeys/%u .
197 198 199
                  grep AuthorizedKeysFile /etc/sshd_config
                # Make it writable to edit, then change it back.
                chmod g+w /etc/sshd_config
Russ Fish's avatar
Tweaks.  
Russ Fish committed
200
                ed /etc/sshd_config
201 202
/AuthorizedKeysFile
a
203
AuthorizedKeysFile /sshkeys/%u/authorized_keys
204
AuthorizedKeysFile2 /sshkeys/%u/authorized_keys2
205 206 207
.
w
q
208
                chmod g-w /etc/sshd_config
209 210
                # Get a running sshd to read the config file with SIGHUP.
                kill -HUP `cat /var/run/sshd.pid`
211

212
            . LogLevel defaults to INFO, can be set to VERBOSE, DEBUG1, etc.
Russ Fish's avatar
Russ Fish committed
213 214 215
              With the syslogd service running, debug events are logged to /var/log/messages .
              [Otherwise, they show up under Event Viewer / Application / sshd,
               with one line per event (ugh.)  Refresh to see new events with F5.]
216
              ## sshd service debugging.
217 218 219
                ls -l /etc/sshd_config
                # Check.
                grep LogLevel /etc/sshd_config
220
                # Make it writable to edit, then change it back.
221
                chmod g+w /etc/sshd_config
222 223
                ed /etc/sshd_config
/#LogLevel/a
224
LogLevel DEBUG2
225 226 227
.
w
q
228 229 230 231
                chmod g-w /etc/sshd_config
                # Get a running sshd to read the config file with SIGHUP.
                kill -HUP `cat /var/run/sshd.pid`

232 233 234 235 236 237 238
          - Check /var/empty to avoid this error:
              /var/empty must be owned by root and not group or world-writable.
            Actually, it must be owned by SYSTEM.
              v -d /var/empty
            chown SYSTEM /var/empty
            chmod go-w /var/empty

239 240
          - Start sshd.
              cygrunsrv -S sshd
241 242

          - Set up for root ssh access from Boss.
243 244 245
              chown root.wheel /home/root
              chmod 755 /home/root
              passwd root
246 247
daFluxGroup
daFluxGroup
248 249 250 251 252
              mkdir ~root/.ssh
              chown root.wheel ~root/.ssh
              # [On boss.]
              set pc=73
              set ssh_args='-o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null"'
253 254
              # This password isn't used for anything else, and doesn't need to be
              # very secure because all users are in the Administrators group on the node.
255
              eval sudo ssh "$ssh_args" root@pc$pc id
256
daFluxGroup
257
              eval sudo scp "$ssh_args" ~root/.ssh/{id_dsa,identity}.pub root@pc$pc":".ssh
258
daFluxGroup
259
              eval sudo ssh "$ssh_args" root@pc$pc
260
daFluxGroup
261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281
              # [On the target.]
              id
              cd ~root/.ssh
              cat {id_dsa,identity}.pub > authorized_keys
              chmod 644 *
              ls -ld /home /home/root /home/root/.ssh /home/root/.ssh/auth*
              mkdir -p /sshkeys/root
              v -d /sshkeys
              chmod 700 /sshkeys/root
              cp -p /home/root/.ssh/authorized_keys /sshkeys/root
              ls -lR /sshkeys/root
              exit

            # [Check back on Boss.]
            eval sudo ssh "$ssh_args" pc$pc id
            # The following will likely complain due to nonstandard host keys.
            rootpc $pc id

          - Install the standard host keys, dated Jun 21  2001.
            ls -l /etc/ssh*
            # [On boss.]
282
            set pc=109
283 284
            set ssh_args='-o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null"'
            eval sudo scp -rp "$ssh_args" /proj/testbed/fish/elab-host-keys root@pc$pc":"
285 286
            # Get the standard ssl certificates while we're at it.
            eval sudo scp -rp "$ssh_args" /proj/testbed/fish/elab-ssl-certs root@pc$pc":"
287 288

            eval sudo ssh "$ssh_args" root@pc$pc
289
            # [As root on the target.]
290 291 292 293 294 295 296 297 298 299 300 301 302 303
              ls -l ~/elab-host-keys
              ls -l /etc/ssh*key*
              ls -l /etc/orig-ssh-keys

              mkdir /etc/orig-ssh-keys
              chown root /etc/ssh*key*
              cp -p /etc/ssh*key* /etc/orig-ssh-keys
              chown SYSTEM /etc/orig-ssh-keys/*
              ls -l /etc/orig-ssh-keys

              cp -p ~/elab-host-keys/* /etc
              chown SYSTEM /etc/ssh*key*
              ls -l /etc/ssh*key*

304 305 306
              ls -l ~/elab-ssl-certs/* /etc/emulab/*.pem
              cp -p ~/elab-ssl-certs/* /etc/emulab

307
            # The following should no longer complain due to nonstandard host keys.
308
            # [On Boss.] 
309
            rootpc $pc id
310 311

      - Install tools: WinZip and Emacs.
312 313 314 315 316
            # [On boss:]
            sudo scp -rp /share/windows/emacs-21.3-fullbin-i386.tar.gz root@pc$pc":"/tmp
            sudo scp -rp /share/windows/winzip90.exe root@pc$pc":"/tmp

            # Log in as root via RDP.
317 318
            rootrd $pc
            # [On the node, as root.]
319 320 321 322 323 324 325 326 327 328 329 330 331
            # Graphical installer.  Start with WinZip Classic, custon setup, no desktop icon.
            /tmp/winzip90.exe

            cd C:
            # Don't worry about a plethora of "Cannot change ownership" warnings.
            tar xfz /tmp/emacs-21.3-fullbin-i386.tar.gz
            # Graphical, set up the registry, start menu, etc.
            C:/emacs-21.3/bin/addpm.exe
            # Then copy the Emacs shortcut to the All Users/Desktop folder.

            # Make "emacs" be the NTEmacs runemacs starter, with "emacs-exe" for a compiler.
            ln -s /cygdrive/c/emacs-21.3/bin/runemacs.exe /usr/local/bin/emacs
            ln -s /cygdrive/c/emacs-21.3/bin/emacs.exe /usr/local/bin/emacs-exe
332 333

      - Get other stuff that "make client" depends on.
334 335 336 337 338 339 340 341 342 343 344 345 346 347
                ## Collect the include files for mysql and the Boost Graph Library.
                cd /usr/local/include
                tar cfz /share/windows/mysql-include.tgz mysql
                tar cfz /share/windows/boost-include.tgz boost
            # [On Boss.]
            sudo scp -rp /share/windows/{mysql,boost}-include.tgz root@pc$pc":"/tmp
            sudo scp -rp /share/windows/{WSName,addusers,usrtogrp,setx}.exe root@pc$pc":"/tmp
            # [On the target.]
            mkdir /usr/local/include
            cd /usr/local/include
            tar xfz /tmp/mysql-include.tgz
            tar xfz /tmp/boost-include.tgz

            # Build Elvin libs with GCC for testbed client programs.  
348
            # [On Boss.]
349
            sudo scp -p /usr/testbed/www/distributions/*elvin*-4.0.3.tar.gz root@pc$pc":"/tmp
350
            # [On the node.]
351 352 353 354 355 356 357 358 359
            # Need a path without embedded spaces for the make actions to work.
            mkdir C:/elvin
            cd C:/elvin
            # Don't worry about a plethora of "Cannot change ownership" warnings.
            tar xfz /tmp/libelvin-4.0.3.tar.gz
            tar xfz /tmp/elvind-4.0.3.tar.gz

            cd C:/elvin/libelvin-4.0.3
              # configure: error: Elvin requires that doubles be IEEE 754 compliant
360 361 362 363 364 365 366 367
              # Edit configure, line 3547, add exit(0); to patch around it.
              ed configure
3546p
a
exit(0);
.
w
q
368 369 370 371 372
            ./configure >& configure.trace 
              tail configure.trace

            # Comment out #elif defined(HAVE_WINBASE_H)
                              FreeLibrary(cat);
373 374 375 376 377 378 379
                  in c:/elvin/libelvin-4.0.3/src/lib/i18n.c
            ed c:/elvin/libelvin-4.0.3/src/lib/i18n.c
            /HAVE_WINBASE/p
            .,.+1s|^|//|p
            w
            q

380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452
            make >& make.log1
              tail make.log1
            make install >& install.log1
              tail install.log1
              make clean

# SKIP[
            # Build Elvin for Windows on Coke, and tar it up for later installation.
            scp -p bos:"/usr/testbed/www/distributions/*elvin*-4.0.3.tar.gz" /tmp
            mkdir C:/elvin
            cd C:/elvin
            tar xfz /tmp/libelvin-4.0.3.tar.gz
            tar xfz /tmp/elvind-4.0.3.tar.gz
            # Rename lib dir for makefiles in elvind.
            mv libelvin-4.0.3 elvin4

            cd C:/elvin/elvin4
            nmake /k /f Makefile.win >& lib-make.winlog1
            mkdir -p C:/Program\ Files/elvin4/{bin,lib,doc}
            cp -p win32/bin/*.exe C:/Program\ Files/elvin4/bin
            cp -p win32/lib/{,*/}*.{dll,lib} C:/Program\ Files/elvin4/lib
            mkdir C:/Program\ Files/elvin4/include
            cp -p src/include/elvin/*.h C:/Program\ Files/elvin4/include

            cd C:/elvin/elvind-4.0.3        
            nmake /k /f Makefile.win >& program-make.winlog1
            cp -p *.exe *.pem C:/Program\ Files/elvin4/bin
            cp -p [A-Z][A-Z]* C:/Program\ Files/elvin4/doc
              scp -p ../*/*.winlog* ops:/proj/testbed/fish/elvin
            scp -p ops:/proj/testbed/fish/elvin-config /cygdrive/c/Program\ Files/elvin4/bin

            # Install dll's in the system so the server can be run.
            v C:/Program\ Files/elvin4/lib
            chmod -R g-w C:/Program\ Files/elvin4
            chmod a+x C:/Program\ Files/elvin4/lib/*
            cp -p C:/Program\ Files/elvin4/lib/* $nts

            elvin="C:/Program Files/elvin4/bin/elvinsvc.exe"
              v "$elvin"
            "$elvin" --help
            # Application Error - The application failed to initialize properly (0xc0000022).

            tar cfz /tmp/elvin4-windows.tar.gz -C /cygdrive/c Program\ Files/elvin4
            scp -p /tmp/elvin4-windows.tar.gz ops:/share/windows
# SKIP]

            # Install the Windows Elvin, built on Coke above.
            # [On Boss.]
            sudo scp -p /share/windows/elvin4-windows.tar.gz root@pc$pc":"/tmp
            sudo scp -p /share/windows/elvind.conf.windows root@pc$pc":"/tmp/elvind.conf

            # [On the experiment node as root (Bash shell):]
            rootpc $pc
              cd C:
                ls -ld Program\ Files/elvin*
              tar xvfz /tmp/elvin4-windows.tar.gz
              chown -R root Program\ Files/elvin4
              cp -p C:/Program\ Files/elvin4/lib/* C:/WINDOWS/system32
              cp -p C:/Program\ Files/elvin4/lib/* /usr/local/lib
                diff /usr/local/etc/elvind_ssl.pem C:/Program\ Files/elvin4/bin/elvind_ssl.pem
              cp -p C:/Program\ Files/elvin4/bin/elvind_ssl.pem /usr/local/etc/elvind_ssl.pem

              elvind="C:/Program Files/elvin4"
              elvin="$elvind/bin/elvinsvc.exe"
                ls -l "$elvind/bin"
              chmod -R g-w "C:/Program Files/elvin4"
                "$elvin" --help &
              # Install as a service.
              "$elvin" -r
              # Install a config file and set the path for the server.
                diff /usr/local/etc/elvind.conf /tmp/elvind.conf
              cp /tmp/elvind.conf /usr/local/etc/elvind.conf
                ls -l /usr/local/etc/elvind.conf
453
              # Do once to register the config file.
454 455 456
              "$elvin" -c `cygpath -w /usr/local/etc/elvind.conf`
                  ## Testing: start elvinsvc from the Services Manager now.
              # Make elvinsvc automatic in services manager, or use these commands:
457
                regtool -v list /HKLM/SYSTEM/CurrentControlSet/Services/elvinsvc.exe
458 459 460 461 462 463 464 465 466
              # (4 is Disabled, 3 is Manual, 2 is Automatic, 1 is only used for System services.)
              regtool -v set /HKLM/SYSTEM/CurrentControlSet/Services/elvinsvc.exe/Start 2

# SKIP[
              ## Use any Windows experiment with a Program object in it for testing.
              pid=testbed eid=Windows-1
                pid=testbed eid=Windows-1b
                pid=testbed eid=Windows-1c
              $BINDIR/evproxy -s event-server -e $pid/$eid
467 468
              
              ## program-agent debugging.
469 470 471 472 473 474 475 476 477 478 479 480
                ps -Welf | grep program-agent
                $rc/rc.progagent shutdown
              $rc/rc.progagent boot
                ## Debugging.
                tail $LOGDIR/progagent.debug
                program-agent -d -e $pid/$eid -s localhost -c /var/emulab/boot/progagents
                # [On ops.]
                tevc -e testbed/Windows-1c now prog0 start \
                    COMMAND="bash -c 'date; hostname' > /tmp/host.txt"
                # [On the node.]
                tail /tmp/host.txt
                cat /local/logs/prog0.status
481
              
482 483 484 485
                ## C:\cygwin\bin\tcsh.exe (2504): *** couldn't create window, Win32 error 5
                ## See http://comments.gmane.org/gmane.os.cygwin.patches/2559
                ## This is at cygwin-1.5.17-1-winsup/cygwin/window.cc:wininfo::winthread():96
                ## Try starting rc.progagent as a separate service with -i for a desktop.
486
                
487 488
                  ## Started up and stopped immediately.  Needs something else in rc.bootsetup.
                  --dep elvinsvc.exe \
489
                
490 491 492
                  ## Depend on EmulabStartup (rc.bootsetup), which depends on the elvin service,
                  ## and also starts evproxy.  But it stops rather than staying running...
                  --dep EmulabStartup \
493
                
494 495 496 497 498 499 500
                ## Make it manual, and explicitly start it after rc.bootsetup in EmulabStartup.
                ## Works, but stays in "starting" state, err in bootsetup.log:
                ##  cygrunsrv: Error starting a service: QueryServiceStatus:  Win32 error 1053:
                ##  The service did not respond to the start or control request in a timely fashion.
# SKIP]

              # For setuid() to work, Root must have these rights: Create a token object; Replace a
501 502 503 504 505 506 507 508
              # process level token; and Increase Quota rights.
              # http://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-switch,
              # http://msdn.microsoft.com/library/en-us/secauthz/security/authorization_constants.asp
              editrights -u root -l
              editrights -u root -a SeCreateTokenPrivilege -l
              editrights -u root -a SeAssignPrimaryTokenPrivilege -l
              editrights -u root -a SeIncreaseQuotaPrivilege -l

509
              # program-agent service start-up.
510
                cygrunsrv -R ProgAgent
511
              progagent=/usr/local/etc/emulab/rc/rc.progagent
512
              cygrunsrv -I ProgAgent -d "Emulab Program Agent" -i -p /cygdrive/c/cygwin/bin/bash \
513
                  --type manual \
514
                  -a "--norc --noprofile -c '$progagent >& /var/log/program-agent.log'"
515
                regtool -v list /HKLM/SYSTEM/CurrentControlSet/Services/ProgAgent/Parameters
516
                cygrunsrv -VQ ProgAgent
517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546
                cygrunsrv -S ProgAgent
                cygrunsrv -E ProgAgent

                  tail /var/log/{program-agent,ProgAgent}.log
                touch /var/log/{program-agent,ProgAgent}.log
                chmod 777 /var/log/{program-agent,ProgAgent}.log

# SKIP[
                # Little problem: "Must be root to run this script!"
                # Add this: 
                  # This runs as a separate Local System service on XP.  Change to root.
                  if (WINDOWS()) { $EUID = $UID = 0; }

                # Testing on ops.
                tevc -e testbed/bsd-1 now prog0 start
                tevc -e testbed/bsd-1 now prog0 start COMMAND='hostname >>& /users/fish/test.out'

                tevc -e testbed/Windows-1 now prog0 start COMMAND='hostname>>&/users/fish/test.out'
                  v /users/fish/test.out
                  tail /users/fish/test.out
                tevc -e testbed/Windows-1 now prog0 run COMMAND='touch /tmp/foo'
                tevc -e testbed/Windows-1 now prog0 run COMMAND='id'
                tevc -e testbed/Windows-1 now prog0 run COMMAND='ls -l /users/fish'
                tevc -e testbed/Windows-1 now prog0 run COMMAND='ls -l /proj/testbed/fish'

                # [On the node.]
                cat /local/logs/prog0.status
                cat /local/logs/prog0.err
                cat /local/logs/prog0.out
# SKIP]
547 548 549

      - Get the testbed client code via CVS, build, and install it.
            rootpc $pc
550
            # [As root, on the node.]
551 552 553 554
            login_name=fish ws_name=kzin domain=flux.utah.edu
            ws_login=$login_name@$ws_name.$domain
            cvs_login=$login_name@cvs.$domain

555 556 557 558 559 560 561
            # Start an agent and go to your workstation to get your ssh keys for the cvs server.
            eval `ssh-agent -s`
              ssh-add -l
            ssh -A $ws_login
              ssh-add -l
            kdsa
            exit
562 563 564

            ssh $cvs_login id
              ssh -v $cvs_login id
Russ Fish's avatar
Russ Fish committed
565
            export CVSROOT=$cvs_login:/usr/flux/CVS CVS_RSH=ssh
566 567 568

              mkdir ~/flux
            cd ~/flux
569 570
              # First time only
              mkdir CVS; touch CVS/Entries; echo . > CVS/Repository
571
            # Any time the testbed tree needs to be re-created.  (Takes a while.)
572
            cvs -Q co testbed
573 574 575
              # Updates After that.
              cat CVS/Entries
                cvs -n -q update testbed
576
              cvs -q update -d testbed
577

Russ Fish's avatar
Russ Fish committed
578
            # Install some dotfiles for Root.
579 580 581
            cp -p testbed/tmcd/cygwinxp/cygwin.root.bashrc ~root/.bashrc
            cp -p testbed/tmcd/cygwinxp/cygwin.root.bash_profile ~root/.bash_profile
            cp -p testbed/tmcd/cygwinxp/cygwin.root.emacs ~root/.emacs
582
            # No HOME envar is set for root's desktop, so Emacs defaults it to C:/ .
583 584
            cp -p ~root/.emacs C:/.emacs

Russ Fish's avatar
Russ Fish committed
585
            # Install site-lisp files for Emacs.
586 587
              v -t testbed/tmcd/cygwinxp/site-lisp
              v -t c:/emacs-21.3/site-lisp
588 589 590 591 592 593 594 595 596
            cp -rp testbed/tmcd/cygwinxp/site-lisp/* c:/emacs-21.3/site-lisp
            ls -l c:/emacs-21.3/site-lisp

            # Need a resolv.conf before tmcc will work.
              cat /etc/resolv.conf
            cp -p ~/flux/testbed/tmcd/cygwinxp/resolv.conf /etc/resolv.conf
             
            mkdir /usr/local/man/man8
              
597
            # Get the downloaded binary programs into the source tree for install.
598 599 600 601 602 603
              ls -l ~/flux/testbed/tmcd/cygwinxp/*.exe
            # [On boss:]
            sudo scp -rp /share/windows/{WSName,addusers,usrtogrp,setx}.exe root@pc$pc":"/tmp
            # [Back on the client:]
            cp -p /tmp/{WSName,addusers,usrtogrp,setx}.exe ~/flux/testbed/tmcd/cygwinxp

604
            # Finally ready to do the Emulab makes!
605 606
            mkdir ~/flux/obj-real
            cd ~/flux/obj-real
607 608

            # Configure takes a while...
609 610 611 612
              v configure.trace*
              mv configure.trace{,.1}
            ../testbed/configure --enable-windows --enable-windowsclient >& configure.trace
              # Should end with "creating config.h".
613
              tail -f configure.trace
614 615 616 617 618 619 620 621 622 623 624

            # The first make fails with "Cannot change ownership" warnings unpacking tg2.0 .
            make client-install >& make.log1
              tail make.log1
            # No worries.  Patch it explicitly, since the patch action gets skipped.
            (cd ~/flux/testbed/event/trafgen; patch -p0 < tg.patch)

            # If this is an update, evproxy is run by rc.bootsetup and nothing stops it.
            # The install of evproxy in the make will fail unless we stop it first.
            ps -Welf | grep evproxy
              kill `ps -Welf | grep evproxy | awk '{print $2}'`
625 626 627 628
            # Ditto emulab-syncd and slothd.
            $rc/rc.syncserver shutdown
            $rc/rc.slothd stop

629
            make client-install >& make.log2
630
              tail -f make.log2
631 632

                # Only needed if there are problems...
633
                  v -t make.log*
634
                make client-install >& make.log3
635 636
                make client-install >& make.log4
                make client-install >& make.log5
637 638 639 640
                make client-install >& make.log6
                make client-install >& make.log7
                make client-install >& make.log8
                make client-install >& make.log9
641 642

      . Patch the /etc/profile file to use /home dirs if the /users mounts are down.
643 644 645 646 647
        # IF THIS FILE IS MODIFIED IT WILL NOT BE UPDATED BY THE CYGWIN
        # SETUP PROGRAM.  IT BECOMES YOUR RESPONSIBILITY.
        #
        # The latest version as installed by the Cygwin Setup program can
        # always be found at /etc/defaults/etc/profile
648
            (cd ~/flux; cvs update testbed/tmcd/cygwinxp/profile)
649
          diff /etc/defaults/etc/profile ~/flux/testbed/tmcd/cygwinxp
650 651 652
            # If the diffs are right, just copy the Emulab one.
            cp ~/flux/testbed/tmcd/cygwinxp/profile /etc
          # Otherwise, edit the file.
653
          cp /etc/defaults/etc/profile /etc
654 655 656 657 658 659 660 661 662 663 664 665
          ed /etc/profile
  /^# If the home directory doesn't exist, create it./,/^if \[ ! -d "\${HOME}" \]; then/p
/^# If the home directory doesn't exist, create it./,/^if \[ ! -d "\${HOME}" \]; then/c
### Use a local dir under sshd if the mount failed.
if [ ! -d "$HOME" ]; then
        HOME=/home/$USER
fi
# If the home directory doesn't exist, create it.
if [ ]; then
###if [ ! -d "${HOME}" ]; then
.
  .-10,.+5p
666 667
  w
  q
668 669

      . Set up the tbshutdown script to run as a service, to get a shutdown signal.
670 671 672 673
            editrights -u root -l
          editrights -u root -a SeServiceLogonRight -l
          # Don't forget to set the root password to this.
          rootpwd='daFluxGroup'
674
          # EmulabShutdown is started manually later on from rc.cygwinxp .
675
          echo "$rootpwd"
676 677 678 679
            cygrunsrv -R EmulabShutdown
          cygrunsrv -I EmulabShutdown -u root -w "$rootpwd" -p /cygdrive/c/cygwin/bin/bash \
              --shutdown --type manual \
              -a "--norc --noprofile -c '/usr/local/etc/emulab/tbshutdown'"
680
          cygrunsrv -VQ EmulabShutdown
681

682
          # If you see the following, try running rc.accounts or rc.bootsetup below to 
683
          # clear it up.  Haven't figured this out yet...
684 685 686
          ##cygrunsrv: Error installing a service: CreateService:  Win32 error 1057:
          ##The account name is invalid or does not exist, or the password is invalid 
          ##for the account name specified.
687

688 689 690
          touch /var/log/EmulabShutdown.log
          chmod 666 /var/log/EmulabShutdown.log
          regtool -v list /HKLM/SYSTEM/CurrentControlSet/Services/EmulabShutdown/Parameters
691
          cygrunsrv -VQ EmulabShutdown
692
            # Manual start-up.
693
            cygrunsrv -S EmulabShutdown
694 695

      . See if rc.bootsetup works.
696 697 698
          ##Running os dependent initialization script rc.cygwin
          ##chmod: cannot access `/var/log/EmulabStartup.log': No such file or directory
          ##chmod: cannot access `/etc/emulab/iscygwin': No such file or directory
699 700
          touch /var/log/EmulabStartup.log
          chmod 666 /var/log/EmulabStartup.log
701 702 703 704 705 706 707
            tmcc nodeid
              ## Missing /etc/resolv.conf .
              tmcc -d nodeid
                    nodeid 
                    /usr/local/etc/emulab/tmcc.bin  -d nodeid 
                    Connection to TMCD refused. Waiting ...
            ## Should reboot, the first time, when it changes the node ID.
708
            $rc/rc.cygwin
709 710
          v -d /sshkeys
          mkdir /sshkeys
711
          chmod 777 /sshkeys
712
            $rc/rc.accounts
713 714 715 716
          $rc/rc.bootsetup

      . Set up the boot script to run as a service.

717
          # Start up after DHCP and Elvin, run ProgAgent afterwards.
718 719
              cygrunsrv -R EmulabStartup 
          rootpwd='daFluxGroup'
720 721 722 723 724 725 726 727
          cygwinrc=/usr/local/etc/emulab/rc/rc.cygwin
          bootsetup=/usr/local/etc/emulab/rc/rc.bootsetup
          progagent="cygrunsrv -S ProgAgent"
          bootlog=/var/log/bootsetup.log
          cygrunsrv -I EmulabStartup -u root -w $rootpwd --dep DHCP --dep elvinsvc.exe \
            -p /cygdrive/c/cygwin/bin/bash \
            -a "--norc --noprofile -c '( $cygwinrc; $bootsetup; $progagent ) >& $bootlog'"
          cygrunsrv -VQ EmulabStartup 
728

729 730 731 732 733 734 735 736 737 738 739 740 741
            cygrunsrv -S EmulabStartup 
            cygrunsrv --help
          regtool -v list /HKLM/SYSTEM/CurrentControlSet/Services/EmulabStartup/Parameters
          sc query EmulabStartup

      . Make a $HOME envar for everybody, so Emacs works on startup from the desktop.
        - Set a user environment variable: HOME = /users/%USERNAME%
        - Stored in HKCU/Environment, which is HKU/*/Environment based on the user SIDs.
        - The user registry key (folder) is created at first login, doesn't exist before that.
          Run setx after that at login time to set the HOME environment variable value.
                # Check.
                regtool get /HKLM/SOFTWARE/Microsoft/Windows/CurrentVersion/Run/SetHOME
          # Use the Windows command prompt rather than a script.
742 743
          regtool -s set /HKLM/SOFTWARE/Microsoft/Windows/CurrentVersion/Run/SetHOME \
            'cmd /C "if not %USERNAME% == root if not %USERNAME% == Administrator setx HOME //fs/%USERNAME%"'
744 745 746
                # Undo.
                regtool unset /HKLM/SOFTWARE/Microsoft/Windows/CurrentVersion/Run/SetHOME
          # Check that setx.exe is in system32.
747 748
          v C:/WINDOWS/system32/setx.exe

749
      . Patch sshd so that shares (including /users homedirs) work with public-key logins.
Russ Fish's avatar
Russ Fish committed
750
        Also touches a file when client input is received, so slothd will know.
751 752

        - RDP into a node as root and shut down all ssh processes before update.
753
          (Otherwise, installation of an openssh update can hang mysteriously.)
754 755 756
            net stop sshd
            ps -Welf | grep ssh

757 758 759 760 761 762 763
        - Go through Cygwin setup and make sure everything is updated.  
              cygcheck -c openssh
              /cygdrive/c/software/cygwin/setup.exe &
              cygcheck -c openssh
          . View "Partial" will show what it wants to download and install.
          . Also select src for openssh, which goes under /usr/src .
          . When base dll's are updated, it will tell you to reboot.  Do it.
764

765
        - Install the source patches.  (Go get CVSROOT and agent keys set above first.)
766 767 768
            (cd ~/flux; cvs update testbed/tmcd/cygwinxp)
              (cd ~/flux; cvs co testbed/tmcd/cygwinxp)
          cd /usr/src/openssh*
769
            v -t *.[ch] | head -30
Russ Fish's avatar
Russ Fish committed
770 771

          # Enable no-password ssh logins which can access shared homedirs.
772 773 774
          cp -p uidswap.c{,.orig}
            patch -p1 --dry-run < ~/flux/testbed/tmcd/cygwinxp/uidswap.c.patch
          patch -p1 -b < ~/flux/testbed/tmcd/cygwinxp/uidswap.c.patch
775
            diff uidswap.c{.orig,}
776

Russ Fish's avatar
Russ Fish committed
777
          # Enable slothd to know of the last SSH client input time.
778 779
          for f in channels.{h,c} serverloop.c; do cp -p $f{,.orig}; done
            v *.orig
Russ Fish's avatar
Russ Fish committed
780 781 782
            patch -p1 --dry-run < ~/flux/testbed/tmcd/cygwinxp/sshd-client-input-time.patch
          patch -p1 -b < ~/flux/testbed/tmcd/cygwinxp/sshd-client-input-time.patch

783 784 785 786 787 788 789 790 791 792 793
        - Configure.  Takes a while.
            # These are the options that contrib/cygwin/README specifies:
            prefix=/usr sbindir=/usr/sbin datadir=$prefix/share
            ./configure > configure.trace 2>&1 \
                --prefix=/usr \
                --sysconfdir=/etc \
                --libexecdir=${sbindir} \
                --localstatedir=/var \
                --datadir=${prefix}/share \
                --mandir=${datadir}/man \
                --infodir=${datadir}/info
794
              tail -f configure.trace
795 796 797

        - Just make and install sshd.exe, assuming everything else is up-to-date.
            make sshd.exe > make.log.1 2>&1
798
              tail -f make.log.1
799 800 801 802 803 804 805

            # Make sure sshd is closed down while installing.
            ps -Welf | grep sshd
            net stop sshd
            /usr/bin/install -c -m 0755 -s sshd /usr/sbin/sshd.exe
            net start sshd

806
      . Make a load average log for slothd, averaged over a 1 minute period.
Russ Fish's avatar
Russ Fish committed
807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826

        - /proc/loadavg is hard-wired to "0.00 0.00 0.00" on Cygwin now.

        - All attempts to script this setup to reproduce it on another computer have
          failed so far, including using its own "Save/Restore Settings" and
          transplanting the registry subtree.

       - Click into Computer Management / Performance Logs and Alerts / Counter Logs.
          Right-click "New Log Settings..." in the logs pane, 
            Name: "ldavg", OK.

          General tab, 
            Counters list, "Add Counters...",
              check "Use local computer counters",
              click "Add" to add % total processor (the default),
              click "Close".
            Sample data every: Interval: "60" seconds.

          Log Files tab, 
            Log file type: "Text File (Comma delimited)", 
827 828 829 830
            Uncheck "End file names with" so the result goes into ldavg.csv .
            Configure... 
              Location: "C:\cygwin\var\run",
              Log file size: "Limit of: 1 MB", OK.
Russ Fish's avatar
Russ Fish committed
831 832 833 834

          Schedule tab, 
            Start Log: Click "At" (which defaults to the current time, as well as the future.) 
            Stop Log: Click "When the 1-MB log file is full.",
835
              When a log file closes: "Start a new log file".
Russ Fish's avatar
Russ Fish committed
836

837
          Check all three tabs, click OK.
Russ Fish's avatar
Russ Fish committed
838

839 840
          # ldavg should start out red (stopped) and then turn green (started) if you
          # refresh with F5.  It will start again after reboot.
Russ Fish's avatar
Russ Fish committed
841 842
          # You can turn it off and on with the right-click menu on "ldavg" in the logs pane.  
            tail -f /var/run/ldavg.csv
843 844 845 846 847
          # The first one is always 99.999, etc.

      . Reboot to make sure it all works right.
          prepare
          /sbin/reboot
Russ Fish's avatar
Russ Fish committed
848

849

850 851 852
================================================================
Making images

853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874
    . Windows Update
      - This might be needed after each Microsoft "Patch Tuesday" (second tues of the month.)
      - Start up Internet Explorer and go to:
        . http://update.microsoft.com
           - The first time, just installs/updates the updater and asks to reboot.
           - I just choose the EXPRESS update, installing all high-priority updates.
           - Don't turn on Automatic Updates.

           - After rebooting, check again if there's more to install.

             To install SP2, you must have access to a console screen, because the
             Windows Firewall defaults to block both SSH and RDP.  Disable it.

             . You may need to free disk space to install SP2, or allocate a partition.
                   du -sm C:/WINDOWS/SoftwareDistribution/Download
                 rm -rf C:/WINDOWS/SoftwareDistribution/Download/*
                   du -sm C:/WINDOWS/ServicePackFiles/i386
                 find C:/WINDOWS/ServicePackFiles/i386 -type f | xargs rm -f

      - Could be a good time to update Cygwin as well.  
        . Beware of stepping on the sshd.exe patches.

875
     . Uninstall the experimental net devices in Computer Management/Device Manager.
876
       (This was from our attempts to make a pc850/pc600 image.  Is it needed?)
877
         # Check which one is the control net interface.
878 879 880
         ipconfig /all
       Select a non-control net interface, hit delete, enter.
       Takes about 15 seconds per interface.
881

882
     . Run prepare to clear out experiment-specific state.
883 884
        rootpc $pc
          # Ignore complaints about all of the C:/Documents and Settings directories
885
          # that were never created because the users didn't log in...
886
          prepare
887

888
          exit
889

890 891
    . Add an entry at the beginning of xpimage-log.txt, and create the image
      descriptor if it's not an existing image.
892

893 894
    . Capture the image with imagezip.  
      You can specify the PC from which to grab the image when you create an image-id.
895
      Do it in red-dot mode so you can set the Reboot Waittime to 240 seconds.
896 897 898
      When updating existing images, I do it by hand in two stages, as below.
    
        # [On boss.]
899 900 901 902
        set pc=61 img=SP1 image=SP1_2005-08-22
        set pc=72 img=SP0 image=SP0_2005-08-22
        set pc=109 img=UPDATE image=UPDATE_2005-08-22

903 904
        df -m /proj/testbed/images /usr/testbed/images
          # Verify SSH working.
905 906
          rootpc $pc id

907
        # Boot into the MFS.  The serial console will show you when it's open for business.
908
        wap node_admin on pc$pc &
909
              # Should not be necessary if ssh from root@boss to the node is working.
910
              rootpc $pc /sbin/reboot
911
        # Wait until the node is in the MFS.
912 913 914

        # Make WINXP-TMP on /proj, then move it to /usr/testbed/images with the right name.
        rootpc $pc
915 916 917 918 919
            cd /proj/testbed/images
            df -m /proj/testbed/images
            ls -l WINXP*
          imagezip -o -I 2 -I 3 -I 4 /dev/ad0 /proj/testbed/images/WINXP-TMP.ndz
            ls -l WINXP*
920 921
          exit

922 923
        # Reboot the source node back into Windows.
        wap node_admin off pc$pc &
924 925 926 927 928 929 930 931 932 933

        # Move the image to /usr/testbed/images to avoid NFS reads, for faster swap-in.
          ls -l /{proj,usr}/testbed/images/WIN*
        ls -l /proj/testbed/images/WINXP-TMP.ndz /usr/testbed/images/WINXP-$image.ndz
        df -m /usr/testbed/images
        cp /{proj,usr}/testbed/images/WINXP-TMP.ndz
        # Check.
        ls -l /{proj,usr}/testbed/images/WINXP-TMP.ndz
        cksum /usr/testbed/images/WINXP-TMP.ndz & ssh ops cksum /proj/testbed/images/WINXP-TMP.ndz
        # Install with mv.  Frisbee might have the old inode still open.
934
          ls -l /usr/testbed/images/WINXP-{TMP,$image}.ndz
935 936 937 938 939 940 941 942
        mv /usr/testbed/images/WINXP-{TMP,$image}.ndz
        ls -l /usr/testbed/images/WINXP-$image.ndz
        df -m /usr/testbed/images
        # Clear the temp from /proj.
        rm -f /proj/testbed/images/WINXP-TMP.ndz
        df -m /proj/testbed/images
          ls -l /{proj,usr}/testbed/images/WIN*
        
943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959
        # Make symlinks without the date suffix, corresponding to the Image ID's.
        ls -l /usr/testbed/images/WINXP-$img*
          # -new images for testing, before we commit.
          rm /usr/testbed/images/WINXP-$img-new.ndz
          ln -s WINXP-$image.ndz /usr/testbed/images/WINXP-$img-new.ndz
        rm /usr/testbed/images/WINXP-$img.ndz
        ln -s WINXP-$image.ndz /usr/testbed/images/WINXP-$img.ndz

      - Explorer/Help/About Windows says this for SP2: 
               Version 5.1 (Build 2600.xpsp_sp2_gdr.050301-1519: Service Pack 2)
             vs. for SP1, partially updated to SP2:
               Version 5.1 (Build 2600.xpsp2.050301-1526: Service Pack 1)
             vs. for SP1:
               Version 5.1 (Build 2600.xpsp1.020828-1920: Service Pack 1)
             vs. for SP0 (no SP's):
               Version 5.1 (Build 2600.xpclient.010817-1148)

960
    . DEMOTING an image to /proj/testbed/images (edit the image descriptor.)
961 962 963 964 965 966 967 968 969 970
        ls -l /usr/testbed/images/WINXP-$image.ndz
        df -m /proj/testbed/images
        cp /usr/testbed/images/WINXP-$image.ndz /proj/testbed/images
        # Check.
        ls -l /{usr,proj}/testbed/images/WINXP-$image.ndz
        ssh ops cksum /proj/testbed/images/WINXP-$image.ndz & cksum /usr/testbed/images/WINXP-$image.ndz
        # Clear the old copy.
        df -m /usr/testbed/images
        rm -f /usr/testbed/images/WINXP-$image.ndz
        df -m /usr/testbed/images