NAME
pmie - inference engine for performance metrics
SYNOPSIS
pmie [-bCdefVvWxz] [-A align] [-a archive] [-c filename]
[-h host] [-l logfile] [-n pmnsfile] [-O offset] [-S
starttime] [-T endtime] [-t interval] [-Z timezone] [file
name ...]
DESCRIPTION
pmie accepts a collection of arithmetic, logical, and rule
expressions to be evaluated at specified frequencies. The
base data for the expressions consists of performance met
rics values delivered in real-time from any host running
the Performance Metrics Collection Daemon (PMCD), or using
historical data from Performance Co-Pilot (PCP) archive
logs.
As well as computing arithmetic and logical values, pmie
can execute actions (popup alarms, write system log mes
sages, and launch programs) in response to specified con
ditions. Such actions are extremely useful in detecting,
monitoring and correcting performance related problems.
The expressions to be evaluated are read from configura
tion files specified by one or more filename arguments.
In the absence of any filename, expressions are read from
standard input.
A description of the command line options specific to pmie
follows:
-a archive is the base name of a PCP archive log written
by pmlogger(1). Multiple instances of the -a flag
may appear on the command line to specify a set of
archives. In this case, it is required that only one
archive be present for any one host. Also, any
explicit host names occurring in a pmie expression
must match the host name recorded in one of the
archive labels. In the case of multiple archives,
timestamps recorded in the archives are used to
ensure temporal consistency.
-b Output will be line buffered and standard output is
attached to standard error. This is most useful for
background execution in conjunction with the -l
option. The -b option is always used for pmie
instances launched from pmie_check(1).
-C Parse the configuration file(s) and exit before per
forming any evaluations. Any errors in the configu
ration file are reported.
the command line.
-d Normally pmie would be launched as a non-interactive
process to monitor and manage the performance of one
or more hosts. Given the -d flag however, execution
is interactive and the user is presented with a menu
of options. Interactive mode is useful mainly for
debugging new expressions.
-e When used with -V, -v or -W, this option forces
timestamps to be reported with each expression. The
timestamps are in ctime(3) format, enclosed in paren
thesis and appear after the expression name and
before the expression value, e.g.
expr_1 (Tue Feb 6 19:55:10 2001): 12
-f If the -l option is specified and there is no -a
option (ie. real-time monitoring) then pmie is run as
a daemon in the background (in all other cases fore
ground is the default). The -f option forces pmie to
be run in the foreground, independent of any other
options.
-h By default performance data is fetched from the local
host (in real-time mode) or the host for the first
named archive on the command line (in archive mode).
The host argument overrides this default. It does
not override hosts explicitly named in the expres
sions being evaluated.
-l Standard error is sent to logfile.
-n An alternative Performance Metrics Name Space (PMNS)
is loaded from the file pmnsfile.
-t The interval argument follows the syntax described in
PCPIntro(1), and in the simplest form may be an
unsigned integer (the implied units in this case are
seconds). The value is used to determine the sample
interval for expressions that do not explicitly set
their sample interval using the pmie variable delta
described below. The default is 10.0 seconds.
-v Unless one of the verbose options -V, -v or -W
appears on the command line, expressions are evalu
ated silently, the only output is as a result of any
actions being executed. In the verbose mode, speci
fied using the -v flag, the value of each expression
is printed as it is evaluated. The values are in
canonical units; bytes in the dimension of ``space'',
seconds in the dimension of ``time'' and events in
the dimension of ``count''. See pmLookupDesc(3) for
details of the supported dimension and scaling
is useful in monitoring the value of given expres
sions, evaluating derived performance metrics, pass
ing these values on to other tools for further pro
cessing and in debugging new expressions.
-V This option has the same effect as the -v option,
except that the name of the host and instance (if
applicable) are printed as well as expression values.
-W This option has the same effect as the -V option
described above, except that for boolean expressions,
only those names and values that make the expression
true are printed. These are the same names and val
ues accessible to rule actions as the %h, %i and %v
bindings, as described below.
-x Execute in domain agent mode. This mode is used
within the Performance Co-Pilot product to derive
values for summary metrics, see pmdasummary(1). Only
restricted functionality is available in this mode
(expressions with actions may not be used).
-Z Change the reporting timezone to timezone in the for
mat of the environment variable TZ as described in
environ(5).
-z Change the reporting timezone to the timezone of the
host that is the source of the performance metrics,
as identified via either the -h option or the first
named archive (as described above for the -a option).
The -S, -T, -O, and -A options may be used to define a
time window to restrict the samples retrieved, set an ini
tial origin within the time window, or specify a ``natu
ral'' alignment of the sample times; refer to PCPIntro(1)
for a complete description of these options.
Output from pmie is directed to standard output and stan
dard error as follows:
stdout
Expression values printed in the verbose -v mode and
the output of print actions.
stderr
Error and warning messages for any syntactic or
semantic problems during expression parsing, and any
semantic or performance metrics availability problems
during expression evaluation.
EXAMPLES
The following example expressions demonstrate some of the
capabilities of the inference engine.
other annotated examples of pmie expressions.
The variable delta controls expression evaluation fre
quency. Specify that subsequent expressions be evaluated
once a second, until further notice:
delta = 1 sec;
If total syscall rate exceeds 5000 per second per CPU,
then display an alarm notifier:
kernel.all.syscall / hinv.ncpu > 5000 count/sec
-> alarm "high syscall rate";
If the high syscall rate is sustained for 10 consecutive
samples, then launch top(1) in an xwsh(1G) window to moni
tor processes, but do this at most once every 5 minutes:
all_sample (
kernel.all.syscall @0..9 > 5000 count/sec * hinv.ncpu
) -> shell 5 min "xwsh -e 'top'";
The following rules are evaluated once every 20 seconds:
delta = 20 sec;
If any disk is performing more than 60 I/Os per second,
then print a message identifying the busy disk to standard
output and launch dkvis(1):
some_inst (
disk.dev.total > 60 count/sec
) -> print "disk %i busy " &
shell 5 min "dkvis";
Refine the preceding rule to apply only between the hours
of 9am and 5pm, and to require 3 of 4 consecutive samples
to exceed the threshold before executing the action:
$hour >= 9 && $hour <= 17 &&
some_inst (
75 %_sample (
disk.dev.total @0..3 > 60 count/sec
)
) -> print "disk %i busy ";
The following rules are evaluated once every 10 minutes:
delta = 10 min;
If either the / or the /usr filesystem is more than 95%
full, display an alarm popup, but not if it has already
been displayed during the last 4 hours:
filesys.capacity #'/dev/root' < 0.05
-> alarm 4 hour "root filesystem (almost) full";
filesys.free #'/dev/usr' /
filesys.capacity #'/dev/usr' < 0.05
-> alarm 4 hour "/usr filesystem (almost) full";
The following rule requires a machine that supports the
PCP environment metrics. If the machine environment tem
perature rises more than 2 degrees over a 10 minute inter
val, write an entry in the system log:
environ.temp @0 - environ.temp @1 > 2
-> alarm "temperature rising fast" &
syslog "machine room temperature rise alarm";
And last, something interesting if you have performance
problems with your Oracle database:
db = "oracle.ptg1";
host = ":moomba.melbourne.sgi.com";
lru = "#'cache buffers lru chain'";
gets = "$db.latch.gets $host $lru";
total = "$db.latch.gets $host $lru +
$db.latch.misses $host $lru +
$db.latch.immisses $host $lru";
$total > 100 && $gets / $total < 0.2
-> alarm "high lru latch contention";
QUICK START
The pmie specification language is powerful and large.
To expedite rapid development of pmie rules, the
pmieconf(1) tool provides a facility for generating a pmie
configuration file from a set of generalized pmie rules.
The supplied set of rules covers a wide range of perfor
mance scenarios.
The pmrules(1) tool provides a GUI-based facility for gen
erating pmie rules from parametrized templates. The sup
plied templates cover a wide range of performance scenar
ios.
The development efforts of the PCP engineering team are
focused on pmieconf rather than pmrules, and thus pmieconf
is the recommended tool for quickly deploying useful pmie
rules.
The Performance Co-Pilot User's and Administrator's Guide
provides a detailed tutorial-style chapter covering pmie.
The PCP Tutorial from the pcp.man.tutorial subsystem
file:$PCP_DOC_DIR/Tutorial/pmie.html and
file:$PCP_DOC_DIR/Tutorial/lab-pmie.html from your web
browser.
EXPRESSION SYNTAX
This description is terse and informal. For a more com
prehensive description see the Performance Co-Pilot User's
and Administrator's Guide.
A pmie specification is a sequence of semicolon terminated
expressions.
Basic operators are modeled on the arithmetic, relational
and Boolean operators of the C programming language.
Precedence rules are as expected, although the use of
parentheses is encouraged to enhance readability and
remove ambiguity.
Operands are performance metric names (see PMNS(4)) and
the normal literal constants.
Operands involving performance metrics may produce sets of
values, as a result of enumeration in the dimensions of
hosts, instances and time. Special qualifiers may appear
after a performance metric name to define the enumeration
in each dimension. For example,
kernel.percpu.cpu.user :foo :bar #cpu0 @0..2
defines 6 values corresponding to the time spent executing
in user mode on CPU 0 on the hosts ``foo'' and ``bar''
over the last 3 consecutive samples. The default inter
pretation in the absence of : (host), # (instance) and @
(time) qualifiers is all instances at the most recent sam
ple time for the default source of PCP performance met
rics.
Host and instance names that do not follow the rules for
variables in programming languages, ie. alphabetic option
ally followed by alphanumerics, should be enclosed in sin
gle quotes.
Expression evaluation follows the law of ``least sur
prises''. Where performance metrics have the semantics of
a counter, pmie will automatically convert to a rate based
upon consecutive samples and the time interval between
these samples. All expressions are evaluated in double
precision, and where appropriate, automatically scaled
into canonical units of ``bytes'', ``seconds'' and
``counts''.
A rule is a special form of expression that specifies a
condition or logical expression, a special operator (->)
be true.
The following table summarizes the basic pmie operators:
+----------------+--------------------------------------------+
| Operators | Explanation |
+----------------+--------------------------------------------+
|+ - * / | Arithmetic |
|< <= == >= > != | Relational (value comparison) |
|! && || | Boolean |
|-> | Rule |
|rising | Boolean, false to true transition |
|falling | Boolean, true to false transition |
|rate | Explicit rate conversion (rarely required) |
+----------------+--------------------------------------------+
Aggregate operators may be used to aggregate or summarize
along one dimension of a set-valued expression. The fol
lowing aggregate operators map from a logical expression
to a logical expression of lower dimension.
+-------------------------+-------------+--------------------------+
| Operators | Type | Explanation |
+-------------------------+-------------+--------------------------+
|some_inst | Existential | True if at least one set |
|some_host | | member is true in the |
|some_sample | | associated dimension |
+-------------------------+-------------+--------------------------+
|all_inst | Universal | True if all set members |
|all_host | | are true in the associ |
|all_sample | | ated dimension |
+-------------------------+-------------+--------------------------+
|N%_inst | Percentile | True if at least N per |
|N%_host | | cent of set members are |
|N%_sample | | true in the associated |
| | | dimension |
+-------------------------+-------------+--------------------------+
The following instantial operators may be used filter or
limit a a set-valued logical expression, based on regular
expression matching of instance names. The logical
expression must be a set involving the dimension of
instances, and the regular expression is of the form used
by egrep(1) or the Extended Regular Expressions of reg
comp(3G).
| Operators | Explanation |
+-------------+------------------------------------------+
|match_inst | For each value of the logical expression |
| | that is ``true'', the result is ``true'' |
| | if the associated instance name matches |
| | the regular expression. Otherwise the |
| | result is ``false''. |
+-------------+------------------------------------------+
|nomatch_inst | For each value of the logical expression |
| | that is ``true'', the result is ``true'' |
| | if the associated instance name does not |
| | match the regular expression. Otherwise |
| | the result is ``false''. |
+-------------+------------------------------------------+
For example, the expression below will be ``true'' for
disks attached to controllers 2 or 3 performing more than
20 operations per second:
match_inst "^dks[23]d" disk.dev.total > 20;
The following aggregate operators map from an arithmetic
expression to an arithmetic expression of lower dimension.
+-------------------------+-----------+--------------------------+
| Operators | Type | Explanation |
+-------------------------+-----------+--------------------------+
|min_inst | Extrema | Minimum value across all |
|min_host | | set members in the asso |
|min_sample | | ciated dimension |
+-------------------------+-----------+--------------------------+
|max_inst | Extrema | Maximum value across all |
|max_host | | set members in the asso |
|max_sample | | ciated dimension |
+-------------------------+-----------+--------------------------+
|sum_inst | Aggregate | Sum of values across all |
|sum_host | | set members in the asso |
|sum_sample | | ciated dimension |
+-------------------------+-----------+--------------------------+
|avg_inst | Aggregate | Average value across all |
|avg_host | | set members in the asso |
|avg_sample | | ciated dimension |
+-------------------------+-----------+--------------------------+
The aggregate operators count_inst, count_host and
count_sample map from a logical expression to an arith
metic expression of lower dimension by counting the number
of set members for which the expression is true in the
associated dimension.
For action rules, the following actions are defined:
|Operators | Explanation |
+----------+----------------------------------------+
|print | Display on standard output |
|shell | Execute with sh(1) |
|alarm | Raise a visible alarm with xconfirm(1) |
|syslog | Append to /var/adm/SYSLOG |
+----------+----------------------------------------+
Multiple actions may be separated by the & and | operators
to specify respectively sequential execution (both actions
are executed) and alternate execution (the second action
will only be executed if the execution of the first action
returns a non-zero error status.
Arguments to actions are an optional suppression time, and
then one or more expressions (a string is an expression in
this context). Strings appearing as arguments to an
action may include the following special selectors that
will be replaced at the time the action is executed.
%h Host(s) that make the left-most top-level expression
in the condition true.
%i Instance(s) that make the left-most top-level expres
sion in the condition true.
%v Values(s) from the left-most top-level expression in
the condition subject to the host and instance assign
ments that make the condition true.
Note that expansion of the special selectors is done by
repeating the whole argument once for each unique binding
to any of the qualifying special selectors. For example
if a rule were true for the host mumble with instances
grunt and snort, and for host fumble the instance puff
makes the rule true, then the action
...
-> shell myscript "Warning: %h-%i busy ";
will execute myscript with the argument string "Warning:
mumble-grunt busy Warning: mumble-snort busy Warning: fum
ble-puff busy".
By comparison, if the action
...
-> shell myscript "'Warning! busy:" " %i@%h" "'";
were executed under the same circumstances, then myscript
would be executed with the argument string '"Warning!
busy: grunt@mumble snort@mumble puff@fumble"'.
The semantics of the expansion of the special selectors
leads to a common usage, where one argument is a constant
(contains no special selectors) the second argument con
tains the desired special selectors with minimal separator
characters, and an optional third argument provides a
ing from the first argument). If necessary post-process
ing (eg. in myscript) can provide the necessary enumera
tion over each unique expansion of the string containing
just the special selectors.
For complex conditions, the bindings to these selectors is
not obvious. It is strongly recommended that pmie be used
in the debugging mode (specify the -W command line option
in particular) during rule development.
SCALE FACTORS
Scale factors may be appended to arithmetic expressions
and force linear scaling of the value to canonical units.
Simple scale factors are constructed from the keywords:
nanosecond, nanosec, nsec, microsecond, microsec, usec,
millisecond, millisec, msec, second, sec, minute, min,
hour, byte, Kbyte, Mbyte, Gbyte, Tbyte, count, Kcount,
Mcount, Gcount and Tcount, and the operator /, for example
``Kbytes / hour''.
MACROS
Macros are defined using expressions of the form:
name = constexpr;
Where name follows the normal rules for variables in pro
gramming languages, ie. alphabetic optionally followed by
alphanumerics. constexpr must be a constant expression,
either a string (enclosed in double quotes) or an arith
metic expression optionally followed by a scale factor.
Macros are expanded when their name, prefixed by a dollar
($) appears in an expression, and macros may be nested
within a constexpr string.
The following reserved macro names are understood.
minute Current minute of the hour.
hour Current hour of the day, in the range 0 to 23.
day Current day of the month, in the range 1 to 31.
month Current month of the year, in the range 0 (Jan
uary) to 11 (December).
year Current year.
day_of_week
Current day of the week, in the range 0 (Sunday)
to 6 (Saturday).
delta Sample interval in effect for this expression.
(see description of -Z and -z command line options above).
AUTOMATIC RESTART
It is often useful for pmie processes to be started and
stopped when the local host is booted or shutdown, or when
they have been detected as no longer running (when they
have unexpectedly exited for some reason). Refer to
pmie_check(1) for details on automating this process.
FILES
$PCP_DEMOS_DIR/pmie/*
annotated example rules
$PCP_VAR_DIR/pmns/*
default PMNS specification files
$PCP_TMP_DIR/pmie
pmie maintains files in this directory to iden
tify the running pmie instances and to export
runtime information about each instance - this
data forms the basis of the pmcd.pmie perfor
mance metrics
$PCP_VAR_DIR/config/pmie/control
the default set of pmie instances to start at
boot time - refer to pmie_check(1) for details
$PCP_VAR_DIR/config/pmie/*
the predefined alarm action scripts (email, log,
popup and syslog), the example action script
(sample)and the concurrent action control file
(control.master, see also pmrules(1)).
/usr/pcp/lib/pmie-common
common shell procedures for the predefined alarm
action scripts
/etc/config/pmie
chkconfig(1M) control flag, to control launching
of pmie from /etc/init.d/pmie - see also
pmie_check(1)
BUGS
The lexical scanner and parser will attempt to recover
after an error in the input expressions. Parsing resumes
after skipping input up to the next semi-colon (;), how
ever during this skipping process the scanner is ignorant
of comments and strings, so an embedded semi-colon may
cause parsing to resume at an unexpected place. This
behavior is largely benign, as until the initial syntax
error is corrected, pmie will not attempt any expression
evaluation.
PCP ENVIRONMENT
Environment variables with the prefix PCP_ are used to
parameterize the file and directory names used by PCP. On
each installation, the file /etc/pcp.conf contains the
local values for these variables. The $PCP_CONF variable
may be used to specify an alternative configuration file,
SEE ALSO
PCPIntro(1), pmcd(1), pmdumplog(1), pmieconf(1),
pmie_check(1), pminfo(1), pmlogger(1), pmval(1), PMAPI(3),
pcp.conf(4) and pcp.env(4).
TUTORIAL AND USER GUIDE
Relevant information is also available from the on-line
PCP Tutorial. Provided the pcp.man.tutorial subsystem
from the PCP images has been installed, access the URLs
file:$PCP_DOC_DIR/Tutorial/pmie.html and
file:$PCP_DOC_DIR/Tutorial/lab-pmie.html from your web
browser.
For a more complete description of the pmie language,
refer to the Performance Co-Pilot Users and Administrators
Guide. This is distributed in insight(1) format as part
of the pcp.books subsystem, or in HTML format from:
http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?\
db=bks&fname=/SGI_Admin/PCP_UAG/
CAVEAT
The example rules in $PCP_DEMOS_DIR/pmie are part of the
pcp.sw.demo subsystem and are not installed by default,
they must be explicitly installed using inst(1) or
swmgr(1).
Man(1) output converted with
man2html