• Leigh Stoller's avatar
    Quick fix for watchdog/backup interaction; use a script lock. · 72b4ba32
    Leigh Stoller authored
    From Slack:
    
    What I notice is that mysqldump is read locking all of the tables for a
    long time. This time gets longer and longer of course as the DB gets
    bigger. Last night enough stuff backed up (trying to get various write
    locks) that we hit the 500 thread limit. I only know this cause mysql
    prints "killing 501" threads at 2:03am. Which makes me wonder if our
    thread limit is too small (but seems like it would have to be much
    bigger) or if our backup strategy is inappropriate for how big the DB is
    and how busy the system is. But to be clear, I am not even sure if
    mysqld throws in the towel when it hits 500 threads, I am in the midst
    of reading obtuse mysql documentation. (edited) There a bunch of other
    error messages that I do not understand yet.
    
    I can reproduce this in my elabinelab with a 10 line perl script. Two
    problems; one is that we do not use the permission system, so we cannot
    use dynamic permissions, which means that the single thread that is left
    for just this case, can be used by anyone, and so the server is fully
    out of threads. And 2) then the Emulab mysql watchdog cannot perform its
    query, and so it thinks mysqld has gone catatonic and kills it, right in
    the middle of the backup. Yuck * 2. (edited)
    
    And if anyone is curious about a more typical approach: "If you want to
    do this for MyISAM or mixed tables without any downtime from locking the
    tables, you can set up a slave database, and take your snapshots from
    there. Setting up the slave database, unfortunately, causes some
    downtime to export the live database, but once it's running, you should
    be able to lock it's tables, and export using the methods others have
    described. When this is happening, it will lag behind the master, but
    won't stop the master from updating it's tables, and will catch up as
    soon as the backup is complete"
    72b4ba32
Name
Last commit
Last update
account Loading commit data...
apache Loading commit data...
apt Loading commit data...
assign Loading commit data...
autoconf Loading commit data...
autofs Loading commit data...
backend Loading commit data...
bugdb Loading commit data...
cdrom Loading commit data...
clientside Loading commit data...
collab Loading commit data...
daikon Loading commit data...
db Loading commit data...
delay Loading commit data...
dhcpd Loading commit data...
discvr Loading commit data...
doc Loading commit data...
event Loading commit data...
firewall Loading commit data...
flash Loading commit data...
fwrules Loading commit data...
hw_config Loading commit data...
hyperviewer Loading commit data...
image-test Loading commit data...
install Loading commit data...
ipod Loading commit data...
mobile Loading commit data...
mote Loading commit data...
named Loading commit data...
node_usage Loading commit data...
ntpd Loading commit data...
os Loading commit data...
patches Loading commit data...
pelab Loading commit data...
protogeni Loading commit data...
pxe Loading commit data...
rc.d Loading commit data...
robots Loading commit data...
rpms Loading commit data...
security Loading commit data...
sensors Loading commit data...
sql Loading commit data...
ssl Loading commit data...
sysadmin Loading commit data...
tbsetup Loading commit data...
testsuite Loading commit data...
tip Loading commit data...
tmcd Loading commit data...
tools Loading commit data...
utils Loading commit data...
vis Loading commit data...
wiki Loading commit data...
www Loading commit data...
xmlrpc Loading commit data...
.gitattributes Loading commit data...
.gitignore Loading commit data...
.gitmodules Loading commit data...
.loc-ignore Loading commit data...
AGPL-COPYING Loading commit data...
GNUmakefile.in Loading commit data...
GNUmakerules Loading commit data...
GPL-COPYING Loading commit data...
LGPL-COPYING Loading commit data...
MOVED-TO-WIKI Loading commit data...
Makeconf.in Loading commit data...
README Loading commit data...
TODO Loading commit data...
TODO.plab Loading commit data...
VERSION Loading commit data...
WEBtemplate.in Loading commit data...
config.h.in Loading commit data...
configure Loading commit data...
configure.ac Loading commit data...
defs-amaricq-emulab Loading commit data...
defs-apt Loading commit data...
defs-cloudlab-clemson Loading commit data...
defs-cloudlab-utah Loading commit data...
defs-cloudlab-wisc Loading commit data...
defs-default Loading commit data...
defs-duerig-emulab Loading commit data...
defs-elabinelab Loading commit data...
defs-example Loading commit data...
defs-gtw-apt Loading commit data...
defs-gtw-emulab Loading commit data...
defs-johnsond-emulab Loading commit data...
defs-kwebb-apt Loading commit data...
defs-kwebb-cloudlab Loading commit data...
defs-kwebb-emulab Loading commit data...
defs-mike-emulab Loading commit data...
defs-onelab Loading commit data...
defs-ricci-emulab Loading commit data...
defs-stoller-apt Loading commit data...
defs-stoller-emulab Loading commit data...
defs-stoller-home Loading commit data...
defs-stoller-lbsdb Loading commit data...
defs-uky Loading commit data...
defs-utahclient Loading commit data...
defs-wbsun-emulab Loading commit data...
defs-wide Loading commit data...
pnet-favicon.ico Loading commit data...