We bumped into an intermittent DB table contention thing that occasionally meant that a customer's call to our SOAP service would take 140-some seconds to return data as opposed to the usual 6 seconds. Our service still gave them the data, but after they had stopped listening. So our logs had no errors, access_log.nnnnnn shows a 200 (i.e. OK), mod_jk.log.nnnnnn is all wonderful, but they experienced a timeout. This was exacerbated because when their service came along and queried again 15 minutes later, there was even more data to retrieve, which slightly increased the chance that another DB contention would slow 'em down.
The DBA is working on a fix. In the meantime, I was charged with tailing the mod_jk log nearly non-stop to see if the round-trip-time blipped high again. Our first clue shouldn't be the customer calling us and informing us "it happened again."
JkRequestLogFormat "%w %V %T"
I needed to watch that "%T" value scroll by. This got boring fast, and a ruined weekend was on the horizon.
Hyperic OpenSource (free) will watch logs, but again, this situation causes no "error". Besides, our log file names change every evening at 00:00 GMT, and the input for the name of the log file to watch appears to not accept "/my/path/to/logs/mod_jk.log.*" wildcard. Grr.
The free level of splunk> is fantastic for using a regex for the log filename(s), and picking out a metric to track from within the log data (also via regex), and making awesome graphs on the fly. But it won't email my blackberry (short of the Enterprise not-quite-as-free version).
I read lots of stuff in these forums, and the Hyperic Documentation, about how to take input from a shell script. The trickiest part was figuring out which of the advice was misleading and needed to be ignored. Case in point, I beat my head against this for hours because I believed I needed my script to do something like this...
$ ${HQ_HOME}/plugins/scripts/script.pl
Availability=0
Metric=8192
...but that failed.
What I needed my script to do, if run from the command line, is spit out a real number, and nothing else, or at least nothing else visible to the console. An exit code of 0 is real helpful.
Actual script and explanation to follow in a reply post.
The DBA is working on a fix. In the meantime, I was charged with tailing the mod_jk log nearly non-stop to see if the round-trip-time blipped high again. Our first clue shouldn't be the customer calling us and informing us "it happened again."
JkRequestLogFormat "%w %V %T"
I needed to watch that "%T" value scroll by. This got boring fast, and a ruined weekend was on the horizon.
Hyperic OpenSource (free) will watch logs, but again, this situation causes no "error". Besides, our log file names change every evening at 00:00 GMT, and the input for the name of the log file to watch appears to not accept "/my/path/to/logs/mod_jk.log.*" wildcard. Grr.
The free level of splunk> is fantastic for using a regex for the log filename(s), and picking out a metric to track from within the log data (also via regex), and making awesome graphs on the fly. But it won't email my blackberry (short of the Enterprise not-quite-as-free version).
I read lots of stuff in these forums, and the Hyperic Documentation, about how to take input from a shell script. The trickiest part was figuring out which of the advice was misleading and needed to be ignored. Case in point, I beat my head against this for hours because I believed I needed my script to do something like this...
$ ${HQ_HOME}/plugins/scripts/script.pl
Availability=0
Metric=8192
...but that failed.
What I needed my script to do, if run from the command line, is spit out a real number, and nothing else, or at least nothing else visible to the console. An exit code of 0 is real helpful.
Actual script and explanation to follow in a reply post.