NAME
PCPIntro - introduction to the Performance Co-Pilot (PCP)
INTRODUCTION
Performance Co-Pilot (PCP) is an SGI product designed for
monitoring and managing system-level performance. These
services are distributed and scalable to accommodate the
most complex system configurations and performance prob
lems.
From the perspective of package installation managers
(such as inst(1) on IRIX and rpm(1) on Linux), PCP is com
posed of two products.
On Linux the products are:
PCP Collector
This is the part of PCP that collects and extracts
performance data from various sources, e.g. the
Linux /proc pseudo filesystem. It is available
under GPL/LPGL from oss.sgi.com/projects/pcp.
PCP Monitor
This is the part of PCP that displays data col
lected from hosts (or archives) that have the PCP
Collector installed.
On Irix the products are:
pcp_eoe For IRIX 6.5, this is included in the IRIX CD set.
For other IRIX versions, this comes from the PCP
product distribution.
pcp From the PCP product distribution
This manual entry describes the high-level features and
options common to the images within both products on each
platform.
OVERVIEW
The PCP architecture is distributed in the sense that any
PCP tool may be executing remotely. On the host (or
hosts) being monitored, each domain of performance met
rics, whether IRIX, a service layer, a database management
system, a web server, an application, etc. requires a
Performance Metrics Domain Agent (PMDA) which is responsi
ble for collecting performance measurements from that
domain. All PMDAs are controlled by the Performance Met
rics Collector Daemon (pmcd(1)) on the same host.
Client applications (the monitoring tools) connect to
pmcd(1), which acts as a router for requests, by
the responses to the clients. Clients may also access
performance data from a PCP archive (created using pmlog
ger(1)) for retrospective analysis.
The following performance monitoring applications may be
launched directly from the command line, or from the
PerfTools page of the IRIX Interactive Desktop (trademark)
Icon Catalog.
Each tool or command is documented completely in its own
reference page.
oview Displays a three-dimensional visualization of the
topology and performance of an Origin system.
pmkstat
Outputs an ASCII high-level summary of system per
formance.
pmie An inference engine that can evaluate predicate-
action rules to perform alarms and automate system
management tasks.
pmem ASCII report of memory usage by process.
pminfo Interrogate specific performance metrics and the
meta data that describes them.
pmlogger
Generates PCP archives of performance metrics suit
able for replay by most PCP tools.
pmval Simple periodic reporting for some or all instances
of a performance metric.
If the PCP product is installed (along with the associated
valid PCP licenses) then the following additional tools
are available.
pmchart
Displays trends over time of arbitrarily selected
performance metrics from one or more hosts.
osvis Displays a three-dimensional bar chart of high-
level CPU, disk, memory and network activity.
mpvis Displays a three-dimensional bar chart of multipro
cessor CPU utilization.
dkvis Displays a three-dimensional bar chart showing
activity in the disk subsystem.
pmgsys Displays a system-level visual monitor of a single
nfsvis Displays a three-dimensional bar chart of Network
File System (NFS) client and server activity.
pmdumptext
Produce ASCII reports for arbitrary combinations of
performance metrics.
COMMON COMMAND LINE ARGUMENTS
There is a set of common command line arguments that are
used consistently by most PCP tools.
-a archive
Performance metric information is retrospectively
retrieved from the Performance Co-Pilot (PCP)
archive, previously generated by pmlogger(1). The
-a and -h options are mutually exclusive.
-a archive[,archive,...]
An alternate form of -a for applications that are
able to handle multiple archives.
-h hostname
Unless directed to another host by the -h option,
or to an archive by the -a option, the source of
performance metrics will be the Performance Metrics
Collector Daemon (PMCD) on the local host. The -a
and -h options are mutually exclusive.
-n pmnsfile
Normally the distributed Performance Metrics Name
Space (PMNS) is used, however if the -n option is
specified an alternative local PMNS is loaded from
the file pmnsfile.
-s samples
The argument samples defines the number of samples
to be retrieved and reported. If samples is 0 or
-s is not specified, the application will sample
and report continuously (in real time mode) or
until the end of the PCP archive (in archive mode).
-z Change the reporting timezone to the local timezone
at the host that is the source of the performance
metrics, as identified via either the -h or -a
options.
-Z timezone
By default, applications report the time of day
according to the local timezone on the system where
the application is executed. The -Z option changes
the timezone to timezone in the format of the envi
ronment variable TZ as described in environ(5).
Most PCP tools operate with periodic sampling or report
ing, and the -t and -A options may be used to control the
duration of the sample interval and the alignment of the
sample times.
-t interval
Set the update or reporting interval.
The interval argument is specified as a sequence of
one or more elements of the form
number[units]
where number is an integer or floating point con
stant (parsed using strtod(3C)) and the optional
units is one of: seconds, second, secs, sec, s,
minutes, minute, mins, min, m, hours, hour, h,
days, day and d. If the unit is empty, second is
assumed.
In addition, the upper case (or mixed case) version
of any of the above is also acceptable.
Spaces anywhere in the interval are ignored, so 4
days 6 hours 30 minutes, 4day6hour30min, 4d6h30m
and 4d6.5h are all equivalent.
Multiple specifications are additive, e.g. ``1hour
15mins 30secs'' is interpreted as 3600+900+30 sec
onds.
-A align
By default samples are not necessarily aligned on
any natural unit of time. The -A option may be
used to force the initial sample to be aligned on
the boundary of a natural time unit. For example
-A 1sec, -A 30min and -A 1hour specify alignment on
whole seconds, half and whole hours respectively.
The align argument follows the syntax for an inter
val argument described above for the -t option.
Note that alignment occurs by advancing the time as
required, and that -A acts as a modifier to advance
both the start of the time window (see the next
section) and the origin time (if the -O option is
specified).
TIME WINDOW SPECIFICATION
Many PCP tools are designed to operate in some time window
of interest, e.g. to define a termination time for real-
time monitoring or to define a start and end time within a
PCP archive log.
In the absence of the -O and -A options to specify an
the PCP application will retrieve the first sample at the
start of the time window.
The following options may be used to specify a time window
of interest.
-S starttime
By default the time window commences immediately in
real-time mode, or coincides with time at the start
of the PCP archive log in archive mode. The -S
option may be used to specify a later time for the
start of the time window.
The starttime parameter may be given in one of
three forms (interval is the same as for the -t
option as described above, ctime is described
below):
interval
To specify an offset from the current time
(in real-time mode) or the beginning of a
PCP archive (in archive mode) simply specify
the interval of time as the argument. For
example -S 30min will set the start of the
time window to be exactly 30 minutes from
now in real-time mode, or exactly 30 minutes
from the start of a PCP archive.
-interval
To specify an offset from the end of a PCP
archive log, prefix the interval argument
with a minus sign. In this case, the start
of the time window precedes the time at the
end of archive by the given interval. For
example -S -1hour will set the start of the
time window to be exactly one hour before
the time of the last sample in a PCP archive
log.
@ctime To specify the calendar date and time (local
time in the reporting timezone) for the
start of the time window, use the ctime(3C)
syntax preceded by an at sign. For example
-S '@ Mon Mar 4 13:07:47 1996'
-T endtime
By default the end of the time window is unbounded
(in real-time mode) or aligned with the time at the
end of a PCP archive log (in archive mode). The -T
option may be used to specify an earlier time for
the end of the time window.
The endtime parameter may be given in one of three
described above, ctime is described below):
interval
To specify an offset from the start of the
time window simply use the interval of time
as the argument. For example -T 2h30m will
set the end of the time window to be 2 hours
and 30 minutes after the start of the time
window.
-interval
To specify an offset back from the time at
the end of a PCP archive log, prefix the
interval argument with a minus sign. For
example -T -90m will set the end of the time
window to be 90 minutes before the time of
the last sample in a PCP archive log.
@ctime To specify the calendar date and time (local
time in the reporting timezone) for the end
of the time window, use the ctime(3C) syntax
preceded by an at sign. For example -T '@
Mon Mar 4 13:07:47 1996'
-O origin
By default samples are fetched from the start of
the time window (see description of -S option) to
the end of the time window (see description of -T
option). The -O option allows the specification of
an origin within the time window to be used as the
initial sample time. This is useful for interac
tive use of a PCP tool with the pmtime(1) VCR
replay facility.
The origin argument accepted by -O conforms to the
same syntax and semantics as the starttime argument
for the -T option.
For example -O -0 specifies that the initial posi
tion should be at the end of the time window; this
is most useful when wishing to replay ``backwards''
within the time window.
The ctime argument for the -O, -S and -T options is based
upon the calendar date and time format of ctime(3C), but
may be a fully specified time string like Mon Mar 4
13:07:47 1996 or a partially specified time like Mar 4
1996, Mar 4, Mar, 13:07:50 or 13:08.
For any missing low order fields, the default value of 0
is assumed for hours, minutes and seconds, 1 for day of
the month and Jan for months. Hence, the following are
equivalent: -S '@ Mar 1996' and -S '@ Mar 1 00:00:00
If any high order fields are missing, they are filled in
by starting with the year, month and day from the current
time (real-time mode) or the time at the beginning of the
PCP archive log (archive mode) and advancing the time
until it matches the fields that are specified. So, for
example if the time window starts by default at ``Mon Mar
4 13:07:47 1996'', then -S @13:10 corresponds to 13:10:00
on Mon Mar 4, 1996, while -S @10:00 corresponds to
10:00:00 on Tue Mar 5, 1996 (note this is the following
day).
For greater precision than afforded by ctime(3C), the sec
onds component may be a floating point number.
Also the 12 hour clock (am/pm notation) is supported, so
for example 13:07 and 1:07 pm are equivalent.
PERFORMANCE METRICS - NAMES AND IDENTIFIERS
The number of performance metric names supported by PCP in
IRIX is of the order of a few thousand. There are fewer
metrics on Linux, but still a considerable number. The
PCP libraries and applications use an internal identifica
tion scheme that unambiguously associates a single integer
with each known performance metric. This integer is known
as the Performance Metric Identifier, or PMID. Although
not a requirement, PMIDs tend to have global consistency
across all systems, so a particular performance metric
usually has the same PMID.
For all users and most applications, direct use of the
PMIDs would be inappropriate (e.g. this would limit the
range of accessible metrics, make the code hard to main
tain, force the user interface to be particularly baroque,
etc.). Hence a Performance Metrics Name Space (PMNS) is
used to provide external names and a hierarchic classifi
cation for performance metrics. A PMNS is represented as
a tree, with each node having a label, a pointer to either
a PMID (for leaf nodes) or a set of descendent nodes in
the PMNS (for non-leaf nodes).
A node label must begin with an alphabetic character, fol
lowed by zero or more characters drawn from the alphabet
ics, the digits and character `_´ (underscore). For
alphabetic characters in a node label, upper and lower
case are distinguished.
By convention, the name of a performance metric is con
structed by concatenation of the node labels on a path
through the PMNS from the root node to a leaf node, with a
``.'' as a separator. The root node in the PMNS is unla
beled, so all names begin with the label associated with
one of the descendent nodes below the root node of the
this is not a requirement) there would be at most one name
for each PMID in a PMNS. For example kernel.all.cpu.idle
and disk.dev.read are the unique names for two distinct
performance metrics, each with a unique PMID.
Groups of related PMIDs may be named by naming a non-leaf
node in the PMNS tree, e.g. disk.
There may be PMIDs with no associated name in a PMNS; this
is most likely to occur when specific PMIDs are not avail
able in all systems, e.g. if ORACLE is not installed on a
system, there is no good reason to pollute the PMNS with
names for all of the ORACLE performance metrics.
Note also that there is no requirement for the PMNS to be
the same on all systems, however in practice most applica
tions would be developed against a stable PMNS that was
assumed to be a subset of the PMNS on all systems. Indeed
the PCP distribution includes a default local PMNS for
just this purpose.
The default local PMNS is located at
$PCP_VAR_DIR/pmns/root however the environment variable
PMNS_DEFAULT may be set to the full pathname of a differ
ent PMNS which will then be used as the default local
PMNS.
Most applications do not use the local PMNS, but rather
import parts of the PMNS as required from the same place
that performance metrics are fetched, i.e. from pmcd(1)
for live monitoring or from a PCP archive for retrospec
tive monitoring.
To explore the PMNS use pminfo(1), or if the PCP product
is installed the Metric Selection browser within
pmchart(1).
PERFORMANCE METRIC SPECIFICATIONS
In configuration files and (to a lesser extent) command
line options, metric specifications adhere to the follow
ing syntax rules.
If the source of performance metrics is real-time from
pmcd(1) then the accepted syntax is
host:metric[instance1,instance2,...]
If the source of performance metrics is a PCP archive log
then the accepted syntax is
archive/metric[instance1,instance2,...]
The host:, archive/ and [instance1,instance2,...] compo
nents are all optional.
replaced by white space.
Special characters in instance names may be escaped by
surrounding the name in double quotes or preceding the
character with a backslash.
White space is ignored everywhere except within a quoted
instance name.
An empty instance is silently ignored, and in particular
``[]'' is the same as no instance, while ``[one,,,two]''
is parsed as specifying just the two instances ``one'' and
``two''.
PMCD AND ARCHIVE VERSIONS
Since PCP version 2, version information has been associ
ated with pmcd(1) and PCP archives. The version number is
used in a number of ways, but most noticeably for the dis
tributed pmns(4). In PCP version 1, the client applica
tions would load the PMNS from the default PMNS file but
in PCP version 2, the client applications extract the PMNS
information from pmcd(1) or a PCP archive. Thus in PCP
version 2, the version number is used to determine if the
PMNS to use is from the default local file or from the
actual current source of the metrics.
ENVIRONMENT
In addition to the PCP run-time environment and configura
tion variables described in the PCP ENVIRONMENT section
below, the following environment variables apply to all
installations.
PCP_STDERR
Many PCP tools support the environment variable
PCP_STDERR, which can be used to control where
error messages are sent. When unset, the default
behavior is that ``usage'' messages and option
parsing errors are reported on standard error,
other messages after initial startup are sent to
the default destination for the tool, i.e. standard
error for ASCII tools, or a dialog for GUI tools.
If PCP_STDERR is set to the literal value DISPLAY
then all messages will be displayed in a dialog.
This is used for any tools launched from the IRIX
Interactive Desktop (trademark) or from the
PerfTools icon catalog page.
If PCP_STDERR is set to any other value, the value
is assumed to be a filename, and all messages will
be written there.
This environment variable, previously used by
pmlaunch(5), pmgsys(1), pmview(1) and the pmview(1)
front-end scripts (such as mpvis(1)), has been dep
recated from the PCP 2.0 release onward and
replaced by PCP_STDERR.
PMCD_CONNECT_TIMEOUT
When attempting to connect to a remote pmcd(1) on a
machine that is booting, the connection attempt
could potentially block for a long time until the
remote machine finishes its initialization. Most
PCP applications and some of the PCP library rou
tines will abort and return an error if the connec
tion has not been established after some specified
interval has elapsed. The default interval is 5
seconds. This may be modified by setting PMCD_CON
NECT_TIMEOUT in the environment to a real number of
seconds for the desired timeout. This is most use
ful in cases where the remote host is at the end of
a slow network, requiring longer latencies to
establish the connection correctly.
PMCD_RECONNECT_TIMEOUT
When a monitor or client application loses a con
nection to a pmcd(1), the connection may be re-
established by calling a service routine in the PCP
library. However, attempts to reconnect are con
trolled by a back-off strategy to avoid flooding
the network with reconnection requests. By
default, the back-off delays are 5, 10, 20, 40 and
80 seconds for consecutive reconnection requests
from a client (the last delay will be repeated for
any further attempts after the fifth). Setting the
environment variable PMCD_RECONNECT_TIMEOUT to a
comma separated list of positive integers will re-
define the back-off delays, e.g. setting
PMCD_RECONNECT_TIMEOUT to ``1,2'' will back-off for
1 second, then attempt another connection request
every 2 seconds thereafter.
PMCD_REQUEST_TIMEOUT
For monitor or client applications connected to
pmcd(1), there is a possibility of the application
"hanging" on a request for performance metrics or
metadata or help text. These delays may become
severe if the system running pmcd crashes, or the
network connection is lost. By setting the envi
ronment variable PMCD_REQUEST_TIMEOUT to a real
number of seconds, requests to pmcd will timeout
after this number of seconds. The default behavior
is to be willing to wait 10 seconds for a response
from every pmcd for all applications.
When pmcd(1) is started from $PCP_RC_DIR/pcp then
the primary instance of pmlogger(1) will be started
if the configuration flag pmlogger is chkconfig'ed
on, some key applications from the pcp.sw.base sub
system are installed and pmcd is running and
accepting connections.
The check on pmcd's readiness will wait up to
PMCD_WAIT_TIMEOUT seconds. If pmcd has a long
startup time (such as on a very large system), then
PMCD_WAIT_TIMEOUT can be set to provide a maximum
wait longer than the default 60 seconds.
PMNS_DEFAULT
If set, then interpreted as the the full pathname
to be used as the default local PMNS for pmLoad
NameSpace(3). Otherwise, the default local PMNS is
located at $PCP_VAR_DIR/pcp/pmns/root for base PCP
installations.
PCP_COUNTER_WRAP
Many of the performance metrics exported from PCP
agents have the semantics of counter meaning they
are expected to be monotonically increasing. Under
some circumstances, one value of these metrics may
smaller than the previously fetched value. This
can happen when a counter of finite precision over
flows, or when the PCP agent has been reset or
restarted, or when the PCP agent is exporting val
ues from some underlying instrumentation that is
subject to some asynchronous discontinuity.
The environment variable PCP_COUNTER_WRAP may be
set to indicate that all such cases of a decreasing
``counter'' should be treated as a counter over
flow, and hence the values are assumed to have
wrapped once in the interval between consecutive
samples. This ``wrapping'' behavior was the
default in earlier PCP versions, but by default has
been disabled in PCP release from version 1.3 on.
PCP_LICENCE_NOWARNING or PCP_LICENSE_NOWARNING
Many of the PCP client programs require that a
valid software license be present on the host on
which the client is running (the license is node-
locked). In the case that such a valid license is
present, but is due to expire within the next 30
days, a message or popup notifier appears informing
the user of this condition. These warnings can be
disabled by setting PCP_LICENCE_NOWARNING or
PCP_LICENSE_NOWARNING in the environment.
The PMDA_PATH environment variable may be used to
modify the search path used by pmcd(1) and pmNew
Context(3) (for PM_CONTEXT_LOCAL contexts) when
searching for a daemon or DSO PMDA. The syntax
follows that for PATH in sh(1), i.e. a colon sepa
rated list of directories, and the default search
path is ``/var/pcp/lib:/usr/pcp/lib''.
PMCD_PORT
The TPC/IP port used by pmcd(1) to create the
socket for incoming connections and requests, is by
default port number 4321. This may be over-ridden
by setting PMCD_PORT to a different port number.
If a non-default port is used when pmcd(1) is
started, then every monitoring application connect
ing to that pmcd(1) must also have PMCD_PORT set in
their environment before attempting a connection.
The following environment variables are relevant to
installations in which pmlogger(1), the PCP archive log
ger, is used.
PMLOGGER_PORT
The environment variable PMLOGGER_PORT may be used
to change the base TCP/IP port number used by
pmlogger(1) to create the socket to which pmlc(1)
instances will try and connect. The default base
port number is 4330. When used, PMLOGGER_PORT
should be set in the environment before pmlogger(1)
is executed.
If you have the PCP product installed, then the following
environment variables are relevant to the Performance Met
rics Domain Agents (PMDAs).
PMDA_LOCAL_PROC
If set, then a context established with the type of
PM_CONTEXT_LOCAL will have access to the ``proc''
PMDA to retrieve performance metrics about individ
ual processes.
PMDA_LOCAL_SAMPLE
If set, then a context established with the type of
PM_CONTEXT_LOCAL will have access to the ``sample''
PMDA if this optional PMDA has been installed
locally.
PMIECONF_PATH
If set, pmieconf(1) will form its pconf(5) specifi
cation (set of parameterized pmie(1) rules) using
all valid pconf files found below each subdirectory
in this colon-separated list of subdirectories. If
not set, the default is /var/pcp/pconf.
/etc/pcp.conf
Configuration file for the PCP runtime environ
ment, see pcp.conf(4).
$PCP_RC_DIR/pcp
Script for starting and stopping pmcd(1).
$PCP_PMCDCONF_PATH
Control file for pmcd(1).
$PCP_PMCDOPTIONS_PATH
Command line options passed to pmcd(1) when it
is started from $PCP_RC_DIR/pcp. All the com
mand line option lines should start with a
hyphen as the first character. This file can
also contain environment variable settings of
the form "VARIABLE=value".
$PCP_BINADM_DIR
Location of PCP utilities for collecting and
maintaining PCP archives, PMDA help text, PMNS
files etc.
$PCP_PMDAS_DIR
Parent directory of the installation directory
for Dynamic Shared Object (DSO) PMDAs.
$PCP_LOG_DIR/pmcd
Default location of log files for pmcd(1), cur
rent directory for running PMDAs. Archives gen
erated by pmlogger(1) are generally below
$PCP_LOG_DIR/pmlogger.
$PCP_LOG_DIR/pmcd/pmcd.log
Diagnostic and status log for the current run
ning pmcd(1) process. The first place to look
when there are problems associated with pmcd.
$PCP_LOG_DIR/pmcd/pmcd.log.prev
Diagnostic and status log for the previous
pmcd(1) instance.
$PCP_LOG_DIR/NOTICES
Log of pmcd(1) and PMDA starts, stops, additions
and removals.
$PCP_VAR_DIR/config
Contains directories of configuration files for
several PCP tools.
$PCP_VAR_DIR/config/pmcd/rc.local
Local script for controlling PCP boot, shutdown
and restart actions.
$PCP_VAR_DIR/pmns
Directory containing the set of PMNS files for
all installed PMDAs.
$PCP_VAR_DIR/pmns/root
The ASCII PMNS(4) exported by pmcd(1) by
default. This PMNS is be the super set of all
other PMNS files installed in $PCP_VAR_DIR/pmns.
In addition, if the PCP product is installed the following
files and directories are relevant.
$PCP_LOG_DIR/NOTICES
In addition to the pmcd(1) and PMDA activity, may
pmpost(1).
$PCP_VAR_DIR/config/pmlogger/control
Control file for pmlogger(1) instances launched
from $PCP_RC_DIR/pcp and/or managed by pmlog
ger_check(1) and pmlogger_daily(1) as part of a
production PCP archive collection setup.
$PCP_VAR_DIR/config/pmsnap/control
Control file for pmsnap(1) to produce GIF images of
recent performance as displayed by pmchart(1) from
PCP archives.
$PCP_DEMOS_DIR
Contains examples for using a variety of PCP tools
and the PCP online tutorial.
PCP ENVIRONMENT
Environment variables with the prefix PCP_ are used to
parameterize the file and directory names used by PCP. On
each installation, the file /etc/pcp.conf contains the
local values for these variables. The $PCP_CONF variable
may be used to specify an alternative configuration file,
as described in pcp.conf(4).
SEE ALSO
oview(1), pmcd(1), pmem(1), pmie(1), pminfo(1), pmk
stat(1), pmlogger(1), pmval(1), pcp.conf(4), pcp.env(4),
pmns(4) and pmlaunch(5).
If the Performance Co-Pilot product is installed, then the
following entries are also relevant:
pmlogger_daily(1), dkvis(1), mpvis(1), nfsvis(1),
osvis(1), pcp(1), pmchart(1), pmdumptext(1), pmgevctr(1)
and pmgsys(1).
Also refer to the Insight books Performance Co-Pilot
User's and Administrator's Guide and Performance Co-Pilot
Programmer's Guide.
If you have the PCP product, relevant information is also
available from the on-line PCP Tutorial. Provided the
pcp.man.tutorial subsystem from the PCP images has been
installed, access the URL file:$PCP_DOC_DIR/Tuto
rial/index.html from your web browser.
Man(1) output converted with
man2html