array_services - overview of array services


DESCRIPTION

       Along  with the power and flexibility of clustered systems
       comes some additional complexity in the area  of  adminis­
       tering  and managing the array as a whole.  Array Services
       provides several services to  help  ease  this  situation.
       Some  of  these  services  revolve around the notion of an
       array session, which is a set of processes,  perhaps  run­
       ning on different nodes in an array, that are conceptually
       related as a single "job".  Additional services  are  pro­
       vided  by the array services daemon, which knows about the
       configuration of an array and is therefore able to provide
       functions for describing and administering it.

   ARRAY SESSIONS
       A principal use of an array system is to run jobs that are
       large enough to span two or more machines.  Unfortunately,
       the  mechanisms that are typically used to manage multiple
       related processes (for example: process  groups,  terminal
       sessions)  are limited in scope to a single machine.  As a
       result, mundane tasks such as killing a job or  accounting
       for  all  of  its resource usage can become very difficult
       when the job runs across several machines.  Some means  of
       correlating  related  processes  on  different machines is
       required.  Array Services provides this function with  the
       notion of an "array session".

       In  formal  terms,  an array session is a set of processes
       all related to each other by a single  unique  identifier,
       the  array session handle (ASH).  A child process ordinar­
       ily inherits the ASH of its parent  when  it  is  created,
       thus  becoming  a  member  of  its parent's array session.
       However, it is possible for a process to  leave  its  par­
       ent's  array  session  and start a new one.  This would be
       done by programs such as login(1) or rshd(1M) so that log­
       ging  in  to the system will effectively start a new array
       session.  This is also done by programs like cron(1M)  and
       su(1M) so that work done on behalf of another user will be
       done in its own array session.  When the last process with
       a  given ASH exits, a session accounting record containing
       accumulated statistics for all of the processes  that  ran
       in  the  array  session  is  written and the array session
       ceases to exist.

       The array session handle itself is  a  64-bit  value.   By
       default,  a unique, increasing value (similar to a process
       ID) is assigned to each new array session as  its  handle.
       This  type  of ASH is referred to as a local ASH: although
       it is guaranteed to be unique on the local machine, it may
       also  be  in use by a different session on another machine
       in the same array.  Because of this, a local  ASH  is  not
       appropriate  for identifying multi-machine jobs.  However,

       These  are  assigned  by  the  array  services daemon (see
       below) and are supposed to be  unique  across  the  entire
       array.  By arranging for the same global ASH to be associ­
       ated with each process in a job, it is possible  to  treat
       the  set of processes as a single entity, even though some
       of the processes may be running on different machines.

       The next trick is "arranging for the same global ASH to be
       associated  with  each  process  in a job".  This involves
       several steps.  First, each machine that is to run part of
       the  job  must  start  a  new array session to contain the
       related processes.  By default,  this  new  array  session
       will  only have a local ASH, so it must "upgrade" its han­
       dle to a global ASH.  If this is the first machine to  run
       part  of the job, it would need to obtain a new global ASH
       from the array services daemon (this is done with a single
       library  call,  asallocash(3X)).  Additional machines that
       are called into service for the job would need  to  get  a
       copy  of  the  first machine's global ASH; presumably this
       information would be passed along at the same time as  the
       rest  of  the information concerning the new job.  Once an
       appropriate global ASH has been settled upon, it can  then
       be assigned to the new array session, replacing the origi­
       nal local ASH.  The process on each machine  that  started
       the  new  array session is now free to fork off any number
       of children to do the required work.  These children  will
       all have the same ASH and can therefore be correlated with
       each other for administrative tasks such as job control or
       accounting.

   ARRAY SERVICES DAEMON
       Although being able to correlate related processes on dif­
       ferent machines in an array is necessary  for  the  stated
       goal  of administering an array in a reasonable way, it is
       not sufficient: something still needs to find all of those
       related  processes  and act upon them.  That is the job of
       the array services daemon.

       Each machine in an array should  have  an  array  services
       daemon  running on it.  The array services daemon (arrayd)
       performs several different tasks:

       - It allocates global array session handles

       - It knows the current array configuration and can provide
         that information to other commands and programs

       - It  can determine which processes belong to a particular
         array session and provide that information to other com­
         mands and programs

       - It  can  forward  commands  to all of the machines in an
         array

       As mentioned earlier, a global  array  session  handle  is
       important  for  keeping  track of jobs that run on several
       machines in an array.  Because the array  services  daemon
       knows  the  configuration of an array, it is better suited
       to providing a unique global ASH than  the  kernel,  which
       necessarily knows only about the local machine.

       When a program needs to allocate a global ASH it invokes a
       single library call, specifying (optionally)  which  array
       the  ASH  is to be allocated for.  The library call, which
       is part of libarray (see below), takes care of  the  prag­
       matic  issues  of  contacting  and  communicating with the
       local array services daemon.  The resulting global ASH can
       then  be  passed  to the setash(2) system call.  Note that
       while anybody can allocate a global ASH,  only  a  process
       with  root  privileges  can  actually change its ASH using
       setash.

       The global ASH itself is an "opaque" value:  it  does  not
       necessarily  have any specific information embedded in it,
       other than to distinguish it from a local ASH  (a  library
       function is provided to make this distinction).  Neverthe­
       less, the identity of the specific machine that creates  a
       global ASH and the array for which it is intended may play
       some role in the generation of the ASH value itself.  Sys­
       tem  administrators  may  specify  particular values to be
       used for this purpose if desired.

   Array Configuration Database
       Each array services daemon has knowledge about one or more
       arrays  and  the machines that make up each of them.  This
       information can be provided to other programs and commands
       with  straightforward library calls in libarray.  Ideally,
       this should make it unnecessary for  other  array-oriented
       programs  to  maintain their own separate array configura­
       tion data.

       An array services daemon obtains its configuration  infor­
       mation  from  a  configuration  file  located in its local
       filespace.  Each daemon has  its  own  configuration  file
       which  must  be  synchronized  by the system administrator
       with  configuration  files  on  other  machines   in   the
       array(s).

   Array Session Information
       To take advantage of array sessions, it is necessary to be
       able to enumerate the processes that are  contained  in  a
       given  array  session.   For certain applications (monitor
       programs for example) it may also be useful  to  enumerate
       ALL  of the known array sessions.  The array services dae­
       mon can obtain both types of information and provide it to
       other programs via libarray functions.

       Command  forwarding  pulls all of the other array services
       together: it allows a user on one machine to issue a  sin­
       gle command and have it executed on all of the machines in
       an array, perhaps only affecting a particular  array  ses­
       sion.   With the appropriate setup, this could be used for
       such tasks as killing a runaway job or  shutting  down  an
       entire array.

       Users  use  a  simple client program (array(1)) to specify
       the command they want to execute,  any  arguments  it  may
       require and the array they want to execute it on.  Such an
       array command might look like this:

            aarrrraayy --aa DDeevvAArrrraayy kkiillllaasshh 1133554433442233

       This example says "execute the command 'killash  13543423'
       on  the  machines  in  the array 'DevArray'".  The command
       "killash" is not necessarily  an  actual  program  on  any
       machine  in  the  array;  instead it refers to an entry in
       each machine's array configuration file.  The entry itself
       specifies which program to execute, which arguments should
       be passed to  it,  which  user/group/project  the  command
       should  be  executed under, etc.  This allows each machine
       in an array to handle a particular command differently, or
       not handle it at all.

       The  "array"  program  itself  is  fairly basic: it simply
       passes the user's command to the local array services dae­
       mon.  The local array services daemon forwards the command
       to each machine in the specified array, then  gathers  the
       results  which are then passed back to the "array" program
       and then the user.


THE ARRAY SERVICES LIBRARY

       In general, users should never have any direct interaction
       with  the array services daemon.  Instead, all interaction
       with the array services daemon is done through  the  array
       services  library,  libarray.  libarray provides functions
       for dealing with  global  ASH's,  describing  the  current
       array configuration, and executing array commands.

       There are a number of libarray functions, all of which are
       documented in chapter 3X man pages.  Some of the  libarray
       functions include:

           ASH Functions
               aassaallllooccaasshh       - Allocates a global ASH
               aassaasshhiissgglloobbaall    - Indicates whether an ASH is global or local
               aasslliissttaasshhss_aarrrraayy - Returns all global ASH's in specified array

           Configuration Functions
               aasslliissttaarrrraayyss     - Returns info on all known arrays
               aasslliissttmmaacchhiinneess   - Returns info on all machines in specified array

               aassccoommmmaanndd        - Execute an array command


SEE ALSO

       array(1), arrayd(1M), newsess(1), asallocash(3X), asashis­
       global(3X),   ascommand(3X),   aslistarrays(3X),    aslis­
       tashs_array(3X),      aslistashs_server(3X),     aslistma­
       chines(3X), arrayd.conf(4), array_sessions(5).


Man(1) output converted with man2html