• David Johnson's avatar
    Simplify the hadoop image creation and track modified conf files. · 06e1878b
    David Johnson authored
    Lots of little changes here (but the experiment configuration for the
    paper experiments is preserved)... now all the hadoop setup scripts and
    config files that are baked into the hadoop VM image are stored here in
    the source tree, and then are also placed in the overall capnet ext
    tarball that osp-capnet.py references.  Thus, no need for all the extra
    and conf tarballs.  Now we only download hadoop and a wordfile (for
    reproducibility of input) from www.emulab.net when we create the hadoop
    The hadoop config files included here are the ones that we need and are
    working.  During image creation, they get baked into a tarball in the
    image, and then extracted at VM runtime once the hadoop install scripts
    have unpacked the hadoop tarball.  We wait til runtime to unpack hadoop
    because it's huge.  But the conf dir we use is in the unpacked dir,
    hence the need to wait to unpack our overlay conf tarball.
    The hadoop config files here are slightly different than Unni's (but of
    course they are the ones we used for the paper); there
    are changes so that the slaves can contact the tracker on the master (I
    think that's what it was); and more imporantly JVM and hadoop memory
    limit adjustments to make the wordcount case work for our experiments.
    I don't know how well they'll work for others... I might have
    inadvertently required that VMs have 4096MB of memory minimum :(.  But
    that is ok for us.
osp-capnet.py 49.8 KB