I woke up, thinking everything was fine and that Filespace had launched without a hitch. Read my e-mail, played around on IRC, chatted with Liz. At one point, she went to poke Filespace and found it wasn't working.
Several hours of furious troubleshooting later, I reconstructed the following:
1. At 6:08, someone tickled a bug in mod_perl (one of the pieces of software that runs the site). This caused mod_perl to repeatedly output messages to the server log, until it grew to Linux's maximum file size of 2GB. Two gigabytes of this cryptic message: "failed to get bucket brigade".
2. At 6:29, cron (a utility that runs programs at scheduled times, like Scheduled Tasks on Windows) ran the logrotate utility. logrotate closed Apache (the web server software), tried to move the error log to another file (which failed, probably due to the file's size), and tried to restart Apache.
3. Apache either detected that the log file was too big and refused to restart, or tried to write into the log file that it had successfully restarted and failed. Either way, it didn't start up again.
The fix was to delete the bloated log file, apply a patch that added about four lines of code to mod_perl, then recompile and install the new version of it. Fortunately, someone else ran across this bug about a week ago, so a Google turned up the patch pretty quickly.
That brought me to noon. The parents had been waiting all morning for me to fix this thing, because there was yard work to be done, so I quickly scarfed down some lunch and got to work.
We're redoing the backyard. Half of it is a little English garden (gravelled paths with flowers alongside); the other half is a big gravelled area with a couple trees where we'll put some patio furniture. Today's assignment was to move all the gravel from the driveway (in front of the house) to the backyard, and to put it all where it belonged.
We had six scoops of gravel to move. "Scoop" refers to the one found on a piece of construction machinery.
That's a lot of gravel.
Now, to do this, my brother and I had to use normal shovels to fill a wheelbarrow. Dad then wheeled it into the backyard and dumped it, where Mom and my sister would spread it and make it all prettylike.
Suffice it to say, it took four hours, with two or three five-minute breaks.
So, now it's four PM. I'm physically and mentally exhausted. And what am I thinking about?
How I can write a patch to Apache so it'll warn you when the log file's too big.