NAME

       pmie - inference engine for performance metrics


SYNOPSIS

       pmie  [-bCdefVvWxz]  [-A align] [-a archive] [-c filename]
       [-h host] [-l  logfile]  [-n  pmnsfile]  [-O  offset]  [-S
       starttime] [-T endtime] [-t interval] [-Z timezone] [file­
       name ...]


DESCRIPTION

       pmie accepts a collection of arithmetic, logical, and rule
       expressions to be evaluated at specified frequencies.  The
       base data for the expressions consists of performance met­
       rics  values  delivered in real-time from any host running
       the Performance Metrics Collection Daemon (PMCD), or using
       historical  data  from  Performance Co-Pilot (PCP) archive
       logs.

       As well as computing arithmetic and logical  values,  pmie
       can  execute  actions (popup alarms, write system log mes­
       sages, and launch programs) in response to specified  con­
       ditions.   Such actions are extremely useful in detecting,
       monitoring and correcting performance related problems.

       The expressions to be evaluated are read  from  configura­
       tion  files  specified  by one or more filename arguments.
       In the absence of any filename, expressions are read  from
       standard input.

       A description of the command line options specific to pmie
       follows:

       -a   archive is the base name of a PCP archive log written
            by  pmlogger(1).   Multiple  instances of the -a flag
            may appear on the command line to specify  a  set  of
            archives.  In this case, it is required that only one
            archive be present  for  any  one  host.   Also,  any
            explicit  host  names  occurring in a pmie expression
            must match the host  name  recorded  in  one  of  the
            archive  labels.   In  the case of multiple archives,
            timestamps recorded  in  the  archives  are  used  to
            ensure temporal consistency.

       -b   Output  will  be line buffered and standard output is
            attached to standard error.  This is most useful  for
            background  execution  in  conjunction  with  the  -l
            option.  The  -b  option  is  always  used  for  pmie
            instances launched from pmie_check(1).

       -C   Parse  the configuration file(s) and exit before per­
            forming any evaluations.  Any errors in the  configu­
            ration file are reported.

            the command line.

       -d   Normally pmie would be launched as a  non-interactive
            process  to monitor and manage the performance of one
            or more hosts.  Given the -d flag however,  execution
            is  interactive and the user is presented with a menu
            of options.  Interactive mode is  useful  mainly  for
            debugging new expressions.

       -e   When  used  with  -V,  -v  or  -W, this option forces
            timestamps to be reported with each expression.   The
            timestamps are in ctime(3) format, enclosed in paren­
            thesis and  appear  after  the  expression  name  and
            before the expression value, e.g.
                 expr_1 (Tue Feb  6 19:55:10 2001): 12

       -f   If  the  -l  option  is  specified and there is no -a
            option (ie. real-time monitoring) then pmie is run as
            a  daemon in the background (in all other cases fore­
            ground is the default).  The -f option forces pmie to
            be  run  in  the foreground, independent of any other
            options.

       -h   By default performance data is fetched from the local
            host  (in  real-time  mode) or the host for the first
            named archive on the command line (in archive  mode).
            The  host  argument  overrides this default.  It does
            not override hosts explicitly named  in  the  expres­
            sions being evaluated.

       -l   Standard error is sent to logfile.

       -n   An  alternative Performance Metrics Name Space (PMNS)
            is loaded from the file pmnsfile.

       -t   The interval argument follows the syntax described in
            PCPIntro(1),  and  in  the  simplest  form  may be an
            unsigned integer (the implied units in this case  are
            seconds).   The value is used to determine the sample
            interval for expressions that do not  explicitly  set
            their  sample  interval using the pmie variable delta
            described below.  The default is 10.0 seconds.

       -v   Unless one of  the  verbose  options  -V,  -v  or  -W
            appears  on  the command line, expressions are evalu­
            ated silently, the only output is as a result of  any
            actions  being executed.  In the verbose mode, speci­
            fied using the -v flag, the value of each  expression
            is  printed  as  it  is evaluated.  The values are in
            canonical units; bytes in the dimension of ``space'',
            seconds  in  the  dimension of ``time'' and events in
            the dimension of ``count''.  See pmLookupDesc(3)  for
            details   of  the  supported  dimension  and  scaling

            is  useful  in  monitoring the value of given expres­
            sions, evaluating derived performance metrics,  pass­
            ing  these  values on to other tools for further pro­
            cessing and in debugging new expressions.

       -V   This option has the same effect  as  the  -v  option,
            except  that  the  name  of the host and instance (if
            applicable) are printed as well as expression values.

       -W   This  option  has  the  same  effect as the -V option
            described above, except that for boolean expressions,
            only  those names and values that make the expression
            true are printed.  These are the same names and  val­
            ues  accessible  to rule actions as the %h, %i and %v
            bindings, as described below.

       -x   Execute in domain agent  mode.   This  mode  is  used
            within  the  Performance  Co-Pilot  product to derive
            values for summary metrics, see pmdasummary(1).  Only
            restricted  functionality  is  available in this mode
            (expressions with actions may not be used).

       -Z   Change the reporting timezone to timezone in the for­
            mat  of  the  environment variable TZ as described in
            environ(5).

       -z   Change the reporting timezone to the timezone of  the
            host  that  is the source of the performance metrics,
            as identified via either the -h option or  the  first
            named archive (as described above for the -a option).

       The -S, -T, -O, and -A options may be  used  to  define  a
       time window to restrict the samples retrieved, set an ini­
       tial origin within the time window, or specify  a  ``natu­
       ral''  alignment of the sample times; refer to PCPIntro(1)
       for a complete description of these options.

       Output from pmie is directed to standard output and  stan­
       dard error as follows:

       stdout
            Expression  values printed in the verbose -v mode and
            the output of print actions.

       stderr
            Error and  warning  messages  for  any  syntactic  or
            semantic  problems during expression parsing, and any
            semantic or performance metrics availability problems
            during expression evaluation.


EXAMPLES

       The  following example expressions demonstrate some of the
       capabilities of the inference engine.

       other annotated examples of pmie expressions.

       The  variable  delta  controls  expression evaluation fre­
       quency.  Specify that subsequent expressions be  evaluated
       once a second, until further notice:

            delta = 1 sec;

       If  total  syscall  rate  exceeds 5000 per second per CPU,
       then display an alarm notifier:

            kernel.all.syscall / hinv.ncpu > 5000 count/sec
            -> alarm "high syscall rate";

       If the high syscall rate is sustained for  10  consecutive
       samples, then launch top(1) in an xwsh(1G) window to moni­
       tor processes, but do this at most once every 5 minutes:

            all_sample (
                kernel.all.syscall @0..9 > 5000 count/sec * hinv.ncpu
            ) -> shell 5 min "xwsh -e 'top'";

       The following rules are evaluated once every 20 seconds:

            delta = 20 sec;

       If any disk is performing more than 60  I/Os  per  second,
       then print a message identifying the busy disk to standard
       output and launch dkvis(1):

            some_inst (
                disk.dev.total > 60 count/sec
            ) -> print "disk %i busy " &
                 shell 5 min "dkvis";

       Refine the preceding rule to apply only between the  hours
       of  9am and 5pm, and to require 3 of 4 consecutive samples
       to exceed the threshold before executing the action:

            $hour >= 9 && $hour <= 17 &&
            some_inst (
              75 %_sample (
                disk.dev.total @0..3 > 60 count/sec
              )
            ) -> print "disk %i busy ";

       The following rules are evaluated once every 10 minutes:

            delta = 10 min;

       If either the / or the /usr filesystem is  more  than  95%
       full,  display  an  alarm popup, but not if it has already
       been displayed during the last 4 hours:

                filesys.capacity #'/dev/root' < 0.05
            -> alarm 4 hour "root filesystem (almost) full";

            filesys.free #'/dev/usr' /
                filesys.capacity #'/dev/usr' < 0.05
            -> alarm 4 hour "/usr filesystem (almost) full";

       The following rule requires a machine  that  supports  the
       PCP  environment metrics.  If the machine environment tem­
       perature rises more than 2 degrees over a 10 minute inter­
       val, write an entry in the system log:

            environ.temp @0 - environ.temp @1 > 2
            -> alarm "temperature rising fast" &
               syslog "machine room temperature rise alarm";

       And  last,  something  interesting if you have performance
       problems with your Oracle database:

            db = "oracle.ptg1";
            host = ":moomba.melbourne.sgi.com";
            lru = "#'cache buffers lru chain'";
            gets = "$db.latch.gets $host $lru";
            total = "$db.latch.gets $host $lru +
                     $db.latch.misses $host $lru +
                     $db.latch.immisses $host $lru";

            $total > 100 && $gets / $total < 0.2
            -> alarm "high lru latch contention";


QUICK START

       The pmie specification language is powerful and large.

       To  expedite  rapid  development  of   pmie   rules,   the
       pmieconf(1) tool provides a facility for generating a pmie
       configuration file from a set of generalized  pmie  rules.
       The  supplied  set of rules covers a wide range of perfor­
       mance scenarios.

       The pmrules(1) tool provides a GUI-based facility for gen­
       erating  pmie rules from parametrized templates.  The sup­
       plied templates cover a wide range of performance  scenar­
       ios.

       The  development  efforts  of the PCP engineering team are
       focused on pmieconf rather than pmrules, and thus pmieconf
       is  the recommended tool for quickly deploying useful pmie
       rules.

       The Performance Co-Pilot User's and Administrator's  Guide
       provides  a detailed tutorial-style chapter covering pmie.

       The  PCP  Tutorial  from  the  pcp.man.tutorial  subsystem

       file:$PCP_DOC_DIR/Tutorial/pmie.html                   and
       file:$PCP_DOC_DIR/Tutorial/lab-pmie.html   from  your  web
       browser.


EXPRESSION SYNTAX

       This description is terse and informal.  For a  more  com­
       prehensive description see the Performance Co-Pilot User's
       and Administrator's Guide.

       A pmie specification is a sequence of semicolon terminated
       expressions.

       Basic  operators are modeled on the arithmetic, relational
       and Boolean  operators  of  the  C  programming  language.
       Precedence  rules  are  as  expected,  although the use of
       parentheses  is  encouraged  to  enhance  readability  and
       remove ambiguity.

       Operands  are  performance  metric names (see PMNS(4)) and
       the normal literal constants.

       Operands involving performance metrics may produce sets of
       values,  as  a  result of enumeration in the dimensions of
       hosts, instances and time.  Special qualifiers may  appear
       after  a performance metric name to define the enumeration
       in each dimension.  For example,

           kernel.percpu.cpu.user :foo :bar #cpu0 @0..2

       defines 6 values corresponding to the time spent executing
       in  user  mode  on  CPU 0 on the hosts ``foo'' and ``bar''
       over the last 3 consecutive samples.  The  default  inter­
       pretation  in  the absence of : (host), # (instance) and @
       (time) qualifiers is all instances at the most recent sam­
       ple  time  for  the default source of PCP performance met­
       rics.

       Host and instance names that do not follow the  rules  for
       variables in programming languages, ie. alphabetic option­
       ally followed by alphanumerics, should be enclosed in sin­
       gle quotes.

       Expression  evaluation  follows  the  law  of ``least sur­
       prises''.  Where performance metrics have the semantics of
       a counter, pmie will automatically convert to a rate based
       upon consecutive samples and  the  time  interval  between
       these  samples.   All  expressions are evaluated in double
       precision, and  where  appropriate,  automatically  scaled
       into   canonical   units  of  ``bytes'',  ``seconds''  and
       ``counts''.

       A rule is a special form of expression  that  specifies  a
       condition  or  logical expression, a special operator (->)

       be true.

       The following table summarizes the basic pmie operators:

     +----------------+--------------------------------------------+
     |   Operators    |                Explanation                 |
     +----------------+--------------------------------------------+
     |+ - * /         | Arithmetic                                 |
     |< <= == >= > != | Relational (value comparison)              |
     |! && ||         | Boolean                                    |
     |->              | Rule                                       |
     |rising          | Boolean, false to true transition          |
     |falling         | Boolean, true to false transition          |
     |rate            | Explicit rate conversion (rarely required) |
     +----------------+--------------------------------------------+
       Aggregate  operators may be used to aggregate or summarize
       along one dimension of a set-valued expression.  The  fol­
       lowing  aggregate  operators map from a logical expression
       to a logical expression of lower dimension.

   +-------------------------+-------------+--------------------------+
   |       Operators         |    Type     |       Explanation        |
   +-------------------------+-------------+--------------------------+
   |some_inst                | Existential | True if at least one set |
   |some_host                |             | member is true in the    |
   |some_sample              |             | associated dimension     |
   +-------------------------+-------------+--------------------------+
   |all_inst                 | Universal   | True if all set members  |
   |all_host                 |             | are true in the associ­  |
   |all_sample               |             | ated dimension           |
   +-------------------------+-------------+--------------------------+
   |N%_inst                  | Percentile  | True if at least N per­  |
   |N%_host                  |             | cent of set members are  |
   |N%_sample                |             | true in the associated   |
   |                         |             | dimension                |
   +-------------------------+-------------+--------------------------+
       The  following  instantial operators may be used filter or
       limit a a set-valued logical expression, based on  regular
       expression   matching  of  instance  names.   The  logical
       expression must  be  a  set  involving  the  dimension  of
       instances,  and the regular expression is of the form used
       by egrep(1) or the Extended Regular  Expressions  of  reg­
       comp(3G).

       | Operators   |               Explanation                |
       +-------------+------------------------------------------+
       |match_inst   | For each value of the logical expression |
       |             | that is ``true'', the result is ``true'' |
       |             | if the associated instance name matches  |
       |             | the regular expression.  Otherwise the   |
       |             | result is ``false''.                     |
       +-------------+------------------------------------------+
       |nomatch_inst | For each value of the logical expression |
       |             | that is ``true'', the result is ``true'' |
       |             | if the associated instance name does not |
       |             | match the regular expression.  Otherwise |
       |             | the result is ``false''.                 |
       +-------------+------------------------------------------+
       For  example,  the  expression  below will be ``true'' for
       disks attached to controllers 2 or 3 performing more  than
       20 operations per second:
            match_inst "^dks[23]d" disk.dev.total > 20;

       The  following  aggregate operators map from an arithmetic
       expression to an arithmetic expression of lower dimension.

    +-------------------------+-----------+--------------------------+
    |       Operators         |   Type    |       Explanation        |
    +-------------------------+-----------+--------------------------+
    |min_inst                 | Extrema   | Minimum value across all |
    |min_host                 |           | set members in the asso­ |
    |min_sample               |           | ciated dimension         |
    +-------------------------+-----------+--------------------------+
    |max_inst                 | Extrema   | Maximum value across all |
    |max_host                 |           | set members in the asso­ |
    |max_sample               |           | ciated dimension         |
    +-------------------------+-----------+--------------------------+
    |sum_inst                 | Aggregate | Sum of values across all |
    |sum_host                 |           | set members in the asso­ |
    |sum_sample               |           | ciated dimension         |
    +-------------------------+-----------+--------------------------+
    |avg_inst                 | Aggregate | Average value across all |
    |avg_host                 |           | set members in the asso­ |
    |avg_sample               |           | ciated dimension         |
    +-------------------------+-----------+--------------------------+
       The   aggregate   operators   count_inst,  count_host  and
       count_sample map from a logical expression  to  an  arith­
       metic expression of lower dimension by counting the number
       of set members for which the expression  is  true  in  the
       associated dimension.

       For action rules, the following actions are defined:

          |Operators |              Explanation               |
          +----------+----------------------------------------+
          |print     | Display on standard output             |
          |shell     | Execute with sh(1)                     |
          |alarm     | Raise a visible alarm with xconfirm(1) |
          |syslog    | Append to /var/adm/SYSLOG              |
          +----------+----------------------------------------+
       Multiple actions may be separated by the & and | operators
       to specify respectively sequential execution (both actions
       are  executed)  and alternate execution (the second action
       will only be executed if the execution of the first action
       returns a non-zero error status.

       Arguments to actions are an optional suppression time, and
       then one or more expressions (a string is an expression in
       this  context).   Strings  appearing  as  arguments  to an
       action may include the following  special  selectors  that
       will be replaced at the time the action is executed.

       %h  Host(s)  that  make the left-most top-level expression
           in the condition true.

       %i  Instance(s) that make the left-most top-level  expres­
           sion in the condition true.

       %v  Values(s)  from  the left-most top-level expression in
           the condition subject to the host and instance assign­
           ments that make the condition true.

       Note  that  expansion  of the special selectors is done by
       repeating the whole argument once for each unique  binding
       to  any  of the qualifying special selectors.  For example
       if a rule were true for the  host  mumble  with  instances
       grunt  and  snort,  and  for host fumble the instance puff
       makes the rule true, then the action
            ...
            -> shell myscript "Warning: %h-%i busy ";
       will execute myscript with the argument  string  "Warning:
       mumble-grunt busy Warning: mumble-snort busy Warning: fum­
       ble-puff busy".

       By comparison, if the action
            ...
            -> shell myscript "'Warning! busy:" " %i@%h" "'";
       were executed under the same circumstances, then  myscript
       would  be  executed  with  the  argument string '"Warning!
       busy: grunt@mumble snort@mumble puff@fumble"'.

       The semantics of the expansion of  the  special  selectors
       leads  to a common usage, where one argument is a constant
       (contains no special selectors) the second  argument  con­
       tains the desired special selectors with minimal separator
       characters, and an  optional  third  argument  provides  a

       ing from the first argument).  If necessary  post-process­
       ing  (eg.  in myscript) can provide the necessary enumera­
       tion over each unique expansion of the  string  containing
       just the special selectors.

       For complex conditions, the bindings to these selectors is
       not obvious.  It is strongly recommended that pmie be used
       in  the debugging mode (specify the -W command line option
       in particular) during rule development.


SCALE FACTORS

       Scale factors may be appended  to  arithmetic  expressions
       and  force linear scaling of the value to canonical units.
       Simple scale factors are constructed  from  the  keywords:
       nanosecond,  nanosec,  nsec,  microsecond, microsec, usec,
       millisecond, millisec, msec,  second,  sec,  minute,  min,
       hour,  byte,  Kbyte,  Mbyte,  Gbyte, Tbyte, count, Kcount,
       Mcount, Gcount and Tcount, and the operator /, for example
       ``Kbytes / hour''.


MACROS

       Macros are defined using expressions of the form:

            name = constexpr;

       Where  name follows the normal rules for variables in pro­
       gramming languages, ie. alphabetic optionally followed  by
       alphanumerics.   constexpr  must be a constant expression,
       either a string (enclosed in double quotes) or  an  arith­
       metic expression optionally followed by a scale factor.

       Macros  are expanded when their name, prefixed by a dollar
       ($) appears in an expression, and  macros  may  be  nested
       within a constexpr string.

       The following reserved macro names are understood.

       minute    Current minute of the hour.

       hour      Current hour of the day, in the range 0 to 23.

       day       Current  day of the month, in the range 1 to 31.

       month     Current month of the year, in the range 0  (Jan­
                 uary) to 11 (December).

       year      Current year.

       day_of_week
                 Current day of the week, in the range 0 (Sunday)
                 to 6 (Saturday).

       delta     Sample interval in effect for this expression.

       (see description of -Z and -z command line options above).


AUTOMATIC RESTART

       It is often useful for pmie processes to  be  started  and
       stopped when the local host is booted or shutdown, or when
       they have been detected as no longer  running  (when  they
       have  unexpectedly  exited  for  some  reason).   Refer to
       pmie_check(1) for details on automating this process.


FILES

       $PCP_DEMOS_DIR/pmie/*
                 annotated example rules
       $PCP_VAR_DIR/pmns/*
                 default PMNS specification files
       $PCP_TMP_DIR/pmie
                 pmie maintains  files in this directory to iden­
                 tify  the  running  pmie instances and to export
                 runtime information about each instance  -  this
                 data  forms  the  basis of the pmcd.pmie perfor­
                 mance metrics
       $PCP_VAR_DIR/config/pmie/control
                 the default set of pmie instances  to  start  at
                 boot time - refer to pmie_check(1) for details
       $PCP_VAR_DIR/config/pmie/*
                 the predefined alarm action scripts (email, log,
                 popup and syslog),  the  example  action  script
                 (sample)and  the  concurrent action control file
                 (control.master, see also pmrules(1)).
       /usr/pcp/lib/pmie-common
                 common shell procedures for the predefined alarm
                 action scripts
       /etc/config/pmie
                 chkconfig(1M) control flag, to control launching
                 of  pmie  from  /etc/init.d/pmie  -   see   also
                 pmie_check(1)


BUGS

       The  lexical  scanner  and  parser will attempt to recover
       after an error in the input expressions.  Parsing  resumes
       after  skipping  input up to the next semi-colon (;), how­
       ever during this skipping process the scanner is  ignorant
       of  comments  and  strings,  so an embedded semi-colon may
       cause parsing to resume  at  an  unexpected  place.   This
       behavior  is  largely  benign, as until the initial syntax
       error is corrected, pmie will not attempt  any  expression
       evaluation.


PCP ENVIRONMENT

       Environment  variables  with  the  prefix PCP_ are used to
       parameterize the file and directory names used by PCP.  On
       each  installation,  the  file  /etc/pcp.conf contains the
       local values for these variables.  The $PCP_CONF  variable
       may  be used to specify an alternative configuration file,



SEE ALSO

       PCPIntro(1),    pmcd(1),    pmdumplog(1),     pmieconf(1),
       pmie_check(1), pminfo(1), pmlogger(1), pmval(1), PMAPI(3),
       pcp.conf(4) and pcp.env(4).


TUTORIAL AND USER GUIDE

       Relevant information is also available  from  the  on-line
       PCP  Tutorial.   Provided  the  pcp.man.tutorial subsystem
       from the PCP images has been installed,  access  the  URLs
       file:$PCP_DOC_DIR/Tutorial/pmie.html                   and
       file:$PCP_DOC_DIR/Tutorial/lab-pmie.html  from  your   web
       browser.

       For  a  more  complete  description  of the pmie language,
       refer to the Performance Co-Pilot Users and Administrators
       Guide.   This  is distributed in insight(1) format as part
       of the pcp.books subsystem, or in HTML format from:
           http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?\
               db=bks&fname=/SGI_Admin/PCP_UAG/


CAVEAT

       The example rules in $PCP_DEMOS_DIR/pmie are part  of  the
       pcp.sw.demo  subsystem  and  are not installed by default,
       they  must  be  explicitly  installed  using  inst(1)   or
       swmgr(1).


Man(1) output converted with man2html