NAME
pmdaweblog - performance metrics domain agent (PMDA) for
Web server logs
SYNOPSIS
$PCP_PMDAS_DIR/weblog/pmdaweblog [-Cp] [-d domain] [-h
helpfile] [-i port] [-l logfile] [-n idlesec] [-S num] [-t
delay] [-u socket] configfile
DESCRIPTION
pmdaweblog is a Performance Metrics Domain Agent (PMDA(3))
that scans Web server logs to extract metrics characteriz
ing Web server activity. These performance metrics are
then made available through the infrastructure of the Per
formance Co-Pilot (PCP).
The configfile specifies which Web servers are to be moni
tored, their associated access logs and error logs, and a
regular-expression based scheme for extracting detailed
information about each Web access. This file is main
tained as part of the PMDA installation and/or de-instal
lation by the scripts Install and Remove in the directory
$PCP_PMDAS_DIR/weblog. For more details, refer to the
section below covering installation.
Once started, pmdaweblog monitors a set of log files and
in response to a request for information, will process any
new information that has been appended to the log files,
similar to a tail(1). There is also periodic "catch up"
to process new information from all log files, and a
scheme to detect the rotation of log files.
Like all other PMDAs, pmdaweblog is launched by pmcd(1)
using command line options specified in $PCP_PMCDCONF_PATH
- the Install script will prompt for appropriate values
for the command line options, and update $PCP_PMCD
CONF_PATH.
A brief description of the pmdaweblog command line options
follows:
-C Check the configuration and exit.
-d domain
Specify the domain number. It is absolutely cru
cial that the performance metrics domain number
specified here is unique and consistent. That is,
domain should be different for every PMDA on the
one host, and the same domain number should be used
for the pmdaweblog PMDA on all hosts.
encapsulated in the file
$PCP_PMDAS_DIR/weblog/domain.h will suffice. For
alternate values, check $PCP_PMCDCONF_PATH for the
domain values already in use on this host, and the
file $PCP_VAR_DIR/pmns/stdpmid contains a reposi
tory of ``well known'' domain assignments that
probably should be avoided.
-h helpfile
Get the help text from the supplied helpfile rather
than from the default location.
-i port
Communicate with pmcd(1) on the specified Internet
port (which may be a number or a name).
-l logfile
Location of the log file. By default, a log file
named weblog.log is written in the current direc
tory of pmcd(1) when pmdaweblog is started, i.e.
$PCP_LOG_DIR/pmcd. If the log file cannot be cre
ated or is not writable, output is written to the
standard error instead.
-n idlesec
If a Web server log file has not been modified for
idlesec seconds, then the file will be closed and
re-opened. This is the only way pmdaweblog can
detect any asynchronous rotation of the logs by Web
server administrative scripts. The default period
is 20 seconds. This value may be changed dynami
cally using pmstore(1) to modify the value of the
performance metric web.config.check.
-p Communicate with pmcd(1) via a pipe.
-S num Specify the maximum number of Web servers per
sproc. It may be desirable (from a latency and
load balancing perspective) or necessary (due to
file descriptor limits) to delegate responsibility
for scanning the Web server log files to several
sprocs. pmdaweblog will ensure that each sproc
handles the log files for at most num Web servers.
The default value is 80 Web servers per sproc.
-t delay
To avoid the need to scan a lot of information from
the Web server logs in response to a single request
for performance metrics, all log files will be
checked at least once every delay seconds. The
default is 15 seconds. This value may by changed
dynamically using pmstore(1) to modify the value of
the performance metric web.config.catchup.
Communicate with pmcd(1) via the given Unix domain
socket.
INSTALLATION
The PCP framework allows metrics to be collected on one
host and monitored from another. These hosts are referred
to as collector and monitor hosts, respectively. A host
may be both a collector and a monitor.
Collector hosts require the installation of the agent,
while monitoring hosts require no agent installation at
all.
For collector hosts do the following as root:
# cd $PCP_PMDAS_DIR/weblog
# ./Install
The installation procedure prompts for a default or non-
default installation. A default installation will search
for known server configurations and automatically config
ure the PMDA for any server log files that are found. A
non-default installation will step through each server,
prompting the user for other server configurations and
arguments to pmdaweblog. The end result of a collector
installation is to build a configuration file that is
passed to pmdaweblog via the configfile argument.
If you want to undo the installation, do the following as
root:
# cd $PCP_PMDAS_DIR/weblog
# ./Remove
pmdaweblog is launched by pmcd(1) and should never be exe
cuted directly. The Install and Remove scripts notify
pmcd(1) when the agent is installed or removed.
CONFIGURATION
The configuration file for the weblog PMDA is an ASCII
file that can be easily modified. Empty lines and lines
beginning with '#' are ignored. All other lines must be
either a regular expression or server specification.
Regular expressions, which are used on both the access and
error log files, must be of the form:
regex regexName regexp
or
regex_posix regexName ordering regexp_posix
The regexName is a word which uniquely identifies the
server specification. The regexp for access logs is in
the format described for regcmp(3). The regexp_posix for
access logs is in the format described for regcomp(3).
Note that on IRIX post release 6.2, it is not recommended
that the POSIX compliant form be used for performance rea
sons. The argument ordering is explained below. The Posix
form should be available on all platforms.
The regular expression requires the specification of up to
four arguments to be extracted from each line of a Web
server access log, depending on the type of server. In the
most common case there are two arguments representing the
method and the size.
For the non- Posix version, argument $0 should contain the
method: GET, HEAD , POST or PUT. The method PUT is
treated as a synonym for POST, and anything else is cate
gorized as OTHER.
The second argument, $1, should contain the size of the
request. A size of ``-'' or `` '' is treated as unknown.
Argument $3 should contain the status code returned to the
client browser and argument $4 should contain the status
code returned to the server from a remote host. These
latter two arguments are used for caching servers and must
be specified as a pair (or $3 will be ignored). For fur
ther information on status codes, refer to the web site
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
Some legal non- Posix regex expression specifications for
monitoring an access log are:
# pattern for CERN, NCSA, Netscape etc Access Logs
regex CERN ] "([A-Za-z][-A-Za-z]+)$0 .*" [-0-9]+ ([-0-9]+)$1
# pattern for FTP Server access logs (normally in SYSLOG)
regex SYSLOG_FTP ftpd: ([gp][-A-Za-z]+)$0( )$1
There is 1 special types of access logs with the RegexName
SQUID. This formats extract 4 parameters but since the
Squid log file uses text-based status codes, it is handled
as a special case.
In the examples below, NS_PROXY parses the Netscape/W3C
Common Extended Log Format and SQUID parses the default
Squid Object Cache format log file.
# pattern for Netscape Proxy Server Extended Logs
regex NS_PROXY ] "([A-Za-z][-A-Za-z]+)$0 .*" ([-0-9]+)$2 \
([-0-9]+)$1 ([-0-9]+)$3
# pattern for Squid Cache logs
([_A-Z]+)$3([0-9]+)$2 ([0-9]+)$1 ([A-Z]+)$0
The regexp for the error logs does not require any argu
ments, only a match. Some legal expressions are:
# pattern for CERN, NCSA, Netscape etc Error Logs
regex CERN_err .
# pattern for FTP Server error logs (normally in SYSLOG)
regex SYSLOG_FTP_err FTP LOGIN FAILED
If POSIX compliant regular expressions are used, addi
tional information is required since the order of parame
ters cannot be specified in the regular expression. For
backwards compatibility, the common case of two parameters
the order may be specified as method,size or size,method
In the general case, the ordering is specified by one of
the following methods:
n1,n2,n3,n4
where nX is a digit between 1 and 4. Each comma-
seperated field represents (in order) the arument
number for method,size,client_status,server_status
- Used for cases like the error logs where the content
is ignored.
As for the non- Posix format, the SQUID RegexName is
treated as a special case to match the non-numerical sta
tus codes.
Some legal Posix regex expression specifications for moni
toring an access log are:
# pattern for CERN, NCSA, Netscape, Apache etc Access Logs
regex_posix CERN method,size ][ \]+"([A-Za-z][-A-Za-z]+) \
[^"]*" [-0-9]+ ([-0-9]+)
# pattern for CERN, NCSA, Netscape, Apache etc Access Logs
regex_posix CERN 1,2 ][ \]+"([A-Za-z][-A-Za-z]+) \
[^"]*" [-0-9]+ ([-0-9]+)
# pattern for FTP Server access logs (normally in SYSLOG)
regex_posix SYSLOG_FTP method,size ftpd: \
([gp][-A-Za-z]+)( )
# pattern for Netscape Proxy Server Extended Logs
regex_posix NS_PROXY 1,3,2,4 ][ ]+"([A-Za-z][-A-Za-z]+) \
[^"]*" ([-0-9]+) ([-0-9]+) ([-0-9]+)
# pattern for Squid Cache logs
regex_posix SQUID 4,3,2,1 [0-9]+.[0-9]+[ ]+[0-9]+ \
[a-zA-Z0-9.]+ ([_A-Z]+)([0-9]+) ([0-9]+) ([A-Z]+)
regex_posix CERN_err - .
# pattern for FTP Server error logs (normally in SYSLOG)
regex_posix SYSLOG_FTP_err - FTP LOGIN FAILED
A Web server can be specified using this syntax:
server serverName on|off accessRegex accessFile errorRegex errorFile
The serverName must be unique for each server, and is the
name given to the instance for the associated performance
metrics. See PMAPI(3) for a discussion of PCP instance
domains. The on or off flag indicates whether the server
is to be monitored when the PMDA is installed. This can
altered dynamically using pmstore(1) for the metric
web.perserver.watched, which has one instance for each Web
server named in configfile.
Two files are monitored for each Web server, the access
and the error log. Each file requires the name of a pre
viously declared regular expression, and a file name. The
log files specified for each server do not have to exist
when the weblog PMDA is installed. The PMDA will continue
to check for non-existent log files and open them when
possible. Some legal server specifications are:
# Netscape Server on Port 80 at IP address 127.55.555.555
server 127.55.555.555:80 on CERN /logs/access CERN_err /logs/errors
# FTP Server.
server ftpd on SYSLOG_FTP /var/adm/SYSLOG SYSLOG_FTP_err /var/adm/SYSLOG
CAVEATS
Specifying regular expressions with an incorrect number of
arguments, anything other than 2 for access logs, and none
for error logs, may cause the PMDA to behave incorrectly
and even crash. This is due to limitations in the inter
face of regex(3).
FILES
$PCP_PMDAS_DIR/weblog
installation directory for the weblog PMDA
$PCP_PMDAS_DIR/weblog/Install
installation script for the weblog PMDA
$PCP_PMDAS_DIR/weblog/Remove
de-installation script for the weblog PMDA
$PCP_LOG_DIR/pmcd/weblog.log
default log file for error reporting
pmcd configuration file that specifies the com
mand line options to be used when pmdaweblog is
launched
$PCP_LOG_DIR/NOTICES
log of PMDA installations and removals
$PCP_VAR_DIR/config/web/weblog.conf
likely location of the weblog PMDA configuration
file
$PCP_DOC_DIR/pcpweb/index.html
the online HTML documentation for PCPWEB
PCP ENVIRONMENT
Environment variables with the prefix PCP_ are used to
parameterize the file and directory names used by PCP. On
each installation, the file /etc/pcp.conf contains the
local values for these variables. The $PCP_CONF variable
may be used to specify an alternative configuration file,
as described in pcp.conf(4).
SEE ALSO
pmcd(1), pmchart(1), pmdawebping(1), pminfo(1),
pmstore(1), pmview(1), tail(1), weblogvis(1), webvis(1),
sproc(2), PMAPI(3), PMDA(3) and regcmp(3).
Man(1) output converted with
man2html