A How To: Roll Your Apache Logs Sensibly

|

(Originally submitted to AFP548.com, a website for Mac server admins and the perpetually curious.)

Hi, everybody!

If you haven’t noticed by now, and chances are you have, you have noticed that OSXS along with Apache can indeed roll and archive logs. What you’ve probably also noticed is that the way Apache rolls logs, and the way OSXS sets it up to roll logs, leave you with a log directory full of log files with seemingly-meaningless numbers appended to the log file names. Furthermore, you’ve also probably noticed is that the darned things don’t just disappear! They keep building up, day after day, week after week, until you go in and manually clean them out.

And that’s a pain.

So why doesn’t OSXS roll logs like it rolls the rest of them? I don’t know. But I particularly like the way that log files roll for other services. The previous log files have their numbers rolled and the current log gets cut and renamed. Sounds simple, right?

And it really is. One minor “gotcha’” here is that I use AWStats, so I would want AWStats to process the logs before I roll them. And there’s the “gotcha’” that Apache introduces by needing to be told to restart gracefully after renaming the current log because Apache keeps right on using the log regardless of what you do to rename it. Oh, and there’s launchd to contend with (and this was where most of my headache was, but mostly because I had some syntax errors in it).

Right! Here we go. First, the code of the log rolling script: [These files are available in an archive below. MT truncated a couple of lines.]

#!/bin/bash

# Update statistics for AWStats... delete this if you're not using AWStats...
cd /Library/WebServer/awstats/wwwroot/cgi-bin
for i in `ls *.conf | sed -e 's/awstats.//' -e 's/.conf//'` ; do
    /Library/WebServer/awstats/wwwroot/cgi-bin/awstats.pl -update -config="$i"
done
# or replace it with something useful to you.

# Now go rotate Apache2's logs. Change the path below to /var/log/httpd for Apache 1.x installs.
cd /var/log/apache2/

# Do your logs end in "_log"? If not, change it below to whatever they do end in.
for i in `ls *_log` ; do
    echo "$i";
    if [ -f "${i}" ]; then
        echo -n " ${i}"
        if [ -x /usr/bin/gzip ]; then gzext=".gz"; else gzext=""; fi
        if [ -f "${i}.6${gzext}" ]; then mv -f "${i}.6${gzext}" "${i}.7${gzext}"; fi
        if [ -f "${i}.5${gzext}" ]; then mv -f "${i}.5${gzext}" "${i}.6${gzext}"; fi
        if [ -f "${i}.4${gzext}" ]; then mv -f "${i}.4${gzext}" "${i}.5${gzext}"; fi
        if [ -f "${i}.3${gzext}" ]; then mv -f "${i}.3${gzext}" "${i}.4${gzext}"; fi
        if [ -f "${i}.2${gzext}" ]; then mv -f "${i}.2${gzext}" "${i}.3${gzext}"; fi
        if [ -f "${i}.1${gzext}" ]; then mv -f "${i}.1${gzext}" "${i}.2${gzext}"; fi
        if [ -f "${i}.0" ]; then mv -f "${i}.0" "${i}.1" && if [ -x /usr/bin/gzip ]; then gzip -9 "${i}.1"; fi; fi
        if [ -f "${i}" ]; then mv -f "${i}" "${i}.0"; fi
    fi

apachectl graceful

done

It’s really pretty simple in that all it does is look for AWStats configuration files and make AWStats update the statistics for each .conf file. Then it looks for files ending in “_log” in the apache2 log directory. In both cases, you may need to modify the paths to point to the AWStats script (or comment out the AWStats-related stuff if you don’t have it) and the correct Apache log file directory. After the logs are rolled, Apache is told to restart gracefully, thus cutting the logs and starting new files.

One of the shortcomings of this script is that there is some time between when AWStats updates its statistics and when Apache starts a new log file. Worst case, though, this is usually only a few seconds and, unless you’re running a particularly busy server where every hit is critical, you probably won’t notice at all.

You’ll notice that I don’t zip up the latest rolled log. That’s because I oftentimes find myself needing to go back and look at the last period’s log and I don’t want to go through the hassle of unzipping it. (And I’m not usually sitting at the machine, so using Console to do the job for me is impractical.)

Place this script somewhere convenient. I chose /usr/local/bin and called it rolllogs.sh. I also had to set its permissions to allow execution. (“chmod a+x rolllogs.sh”.)

The next challenge was to get this script to run once a week. I discovered that, with the right syntax errors, it is possible to get launchd to consume fully 100% of a CPU and make launching new tasks—like LoginWindow, ssh and other very, very important tasks—impossible. Fortunately, my errors only affected the admin account when it was logged in so I could SSH into the box after it was rebooted and delete the offending script. It was painful, to say the least.

Bottom line: make sure you use this next bit with respect. I take no responsibility for your use of either the script above or the plist below. In fact, I supply these items completely free of any and all warranty that they will do anything useful or predictable. Please use them at your own risk.

That having been said, here is the plist that you’ll need for launchd:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>Label</key>
        <string>net.eccles.rolllogs</string>
        <key>ProgramArguments</key>
        <array>
                <string>/usr/local/bin/rolllogs.sh</string>
        </array>
        <key>RunAtLoad</key>
        <true/>
        <key>StartCalendarInterval</key>
        <dict>
                <key>Hour</key>
                <integer>3</integer>
                <key>Minute</key>
                <integer>15</integer>
                <key>Weekday</key>
                <integer>0</integer>
        </dict>
</dict>
</plist>

I called this plist net.eccles.rolllogs.plist and placed it in /Library/LaunchDaemons. It doesn’t have to have any special permissions, but I did set the ownership to root:wheel. (I don’t remember if I had to do this manually or if it was done for me when I sudo’d to copy it into this directory. “sudo chown root:wheel net.eccles.rolllogs.plist” will do it, of course.)

This plist tells launchd to run rolllogs.sh every Sunday at 3:15am. You can change both the time and interval, though I don’t know how and, quite frankly, I’m not going to mess around with it now that I have it working. Experiment on your own, or ask someone else for help—I’m not much use on this one.

Last, but not least (hold your breath!), tell launchd to load this plist and execute it:

sudo launchctl load /Library/LaunchDaemons/net.eccles.rolllogs.plist

If all goes well, you won’t see anything happen except that your prompt will return. If you’d like to see if things went well, ask launchctl to tell you what it’s up to, but filter out the stuff that isn’t relevant:

sudo launchctl list | grep net

This will show you anything it’s loaded and is ready to use if your plist has “net” in its name. Change it to “com” or whatever if your plist is named differently.

Also, did you notice that the plist has “RunAtLoad” set to “true?” It sure does! So if all went well, you’ll be able to go look at your log directory to see if things rolled properly.

Did things go wrong? Well, unload the plist (if it was ever loaded):

sudo launchctl unload /Library/LaunchDaemons/net.eccles.rolllogs.plist

and test your script manually:

sudo ./usr/local/bin/rolllogs.sh

and watch to see that things worked correctly.

You’ll also note that the script itself has some debugging “echo” lines in it. When things are working correctly and launchd is running the script for you, you’ll see these show up in the system.log file at 3:15am on Sunday morning.

Well, that’s about it! Comments, suggestions, improvements, etc. are all appreciated. Let’s have ‘em!

By the way, I learned a lot of this stuff the hard way, by reading Apple’s documentation and other online sources. But a lot of help came from khiltd (of AFP548.com). Thanks, Nathan!

/Bill

www.bill.eccles.net/bills_words Tolland, CT

rollYourLogs.tar.gz

Recent Comments