storage.conf - configuration file for storage manager


DESCRIPTION

       The storage manager is a unified interface between INN and
       a variety of different storage method, allowing  the  news
       administrator  to  choose between different storage method
       with different tradeoffs (or even use several at the  same
       time  for  different  newsgroups  or articles of different
       sizes).  The rest of INN need not care what type of  stor­
       age  method was used for a given article; the storage man­
       ager will figure this out automatically when that  article
       is retrieved via the storage API.

       <pathetc in inn.conf>/storage.conf file contains the rules
       to be used in  assigning  articles  to  different  storage
       methods.

       The  file  consists of a series of storage method entries.
       Blank lines and lines beginning with a number sign (``#'')
       are ignored.  The maximum number of character in each line
       is 255.  The order of entries in this file is important.

       Each entry specifies a storage method and a set of  rules.
       Articles  that  match all of the rules of a storage method
       entry will be stored using that  storage  method.   If  an
       article matches multiple storage method entries, the first
       one will be used.  Each entry is formatted as follows:

              method <methodname> {
                   class: <storage_class>
                   newsgroups: <wildmat>
                   size: <minsize>[,<maxsize>]
                   expires: <mintime>[,<maxtime>]
                   options: <options>
              }

       If spaces or tabs are included in a value, that value must
       be  quoted with ``"''.  If either ``#'' or ``"'' are meant
       to be included verbatim in a value, they should be escaped
       with ``\''.

       <methodname>  is  the  name of a storage method to use for
       articles that match the rules of  this  entry.   The  cur­
       rently  available storage methods are ``timecaf'', ``time­
       hash'', ``cnfs'', ``tradspool'' and  ``trash''.   See  the
       STORAGE METHODS section below for more details.

       The meanings of the keys in each entry are as follows:

       class  An   identifier  for  this  storage  method  entry.
              <storage_class> should be a number  and  should  be
              unique  across  all  of  the  entries in this file.
              It's used mainly by  expire.ctl(5)  for  specifying


       newsgroups
              What  newsgroups  are  stored  using  this  storage
              method.  <wildmat> is a wildmat(3) pattern that  is
              matched against the newsgroups an article is posted
              to.  If ``storeonxref'' in  inn.conf  is  ``true'',
              this  pattern will be matched against the newsgroup
              names in the ``Xref'' header; otherwise, it will be
              matched  against  newsgroup  names  in  the ``News­
              groups''  header.    Poison   wildmat   expressions
              (expressions  starting  with ``@'') are allowed and
              can be used  to  exclude  certain  group  patterns.
              ``!''  cannot be used, however.  The <wildmat> pat­
              tern is matched in  order.   There  is  no  default
              newsgroups  pattern;  if  an entry should match all
              newsgroups, use an explicit ``newsgroups: *''.

       size   A range of article sizes (in bytes) that should  be
              stored  using this storage method.  If <maxsize> is
              ``0'' or not given, the upper size of  articles  is
              limited only by ``maxartsize'' in inn.conf(5).  The
              ``size'' field  is  optional  and  may  be  omitted
              entirely  if  you  want  articles of any size (that
              otherwise fulfill the requirements of this  storage
              method  entry) to be stored in this storage method.

       expires
              A range of article expiration times that should  be
              stored using this storage method.  Be careful; this
              is less useful than it may appear at  first.   This
              is  based  only  on  the  ``Expires'' header of the
              article, not on any local  expiration  policies  or
              anything  in  expire.ctl(5)!   If <mintime> is non-
              zero, then this entry will not  match  any  article
              without  an ``Expires'' header.  This key is there­
              fore only really useful for assigning articles with
              requested longer expire times to a separate storage
              method.  <mintime> and <maxtime> are boundaries  on
              the  amount of time into the future the ``Expires''
              header of  the  article  requests  that  it  remain
              around,  and  are  formatted 0d0h0m0s (days, hours,
              minutes, and seconds into the  future).   If  <max­
              time>  is  ``0s''  or is not specified, there is no
              upper bound on expire times falling into this entry
              (note that this key has no effect on when the arti­
              cle will actually be expired, only  on  whether  or
              not  the  article will be stored using this storage
              method).  This field is also optional  and  may  be
              omitted entirely if all articles with or without an
              ``Expires''  header  (that  otherwise  fulfill  the
              requirements  of  this storage method entry) should
              be stored according to it.

              This key is for passing special options to  storage
              methods   that   require   them   (currently   only
              ``cnfs'').  See the STORAGE METHODS  section  below
              for a description of its use.

       If  an article matches all of the constraints of an entry,
       it is stored via that storage  method  and  is  associated
       with  that <storage_class>.  This file is scanned in order
       and the first matching entry is used to store the article.

       If  an  article  doesn't  match any entry, either by being
       posted to a newsgroup that doesn't match any of the <wild­
       mat>  patterns  or  by  being outside the size and expires
       ranges of all entries whose  newsgroups  pattern  it  does
       match,  the  article  is  not  stored  and  is rejected by
       innd(8).  When this happens, the error message

              cant store article: no matching entry in storage.conf

       is logged to syslog.  If you want to silently  drop  arti­
       cles  matching  certain  newsgroup  patterns  or  size  or
       expires ranges,  assign  them  to  the  ``trash''  storage
       method  rather  than  having  them  not  match any storage
       method entry.


STORAGE METHODS

       Currently, there are four storage methods available.  Each
       method  has  its  characteristics.   You can choose any of
       them to be suitable for your environment.  Note that  each
       method  has  an  attribute  ``EXPENSIVESTAT''  which means
       whether checking existense of article is expensive or not.
       This is used to run expireover(8).

       cnfs   The  ``cnfs''  storage  method  stores  articles in
              large cyclic buffers (CNFS stands for  Cyclic  News
              File System).  It's by far the fastest of all stor­
              age methods (except for ``trash''), since it elimi­
              nates  the  overhead  of dealing with a file system
              and creating new files.   Articles  are  stored  in
              CNFS  buffers in arrival order, and when the buffer
              fills, it wraps around to the beginning and  stores
              new articles over top of the oldest articles in the
              buffer.  The expire time of articles stored in CNFS
              buffers  is  therefore  entirely  determined by how
              long it takes the  buffer  to  wrap  around,  which
              depends  on  how  much  data is being stored in it.
              (This method is therefore said to have  self-expire
              functionality.)  ``EXPENSIVESTAT'' is ``FALSE'' for
              this method.  CNFS has its own configuration  file,
              cycbuff.conf(5).   Storage  method  entries for the
              ``cnfs'' storage method must  have  an  ``options''
              field  specifying  the metacycbuff into which arti­
              cles matching that entry  should  be  stored.   See


       timecaf
              This  method  stores multiple articles in one file,
              whose name is based on the article's  arrival  time
              and  the  storage  class.   The  file  name will be
              <patharticles in inn.conf>/timecaf-nn/bb/aacc.CF,
              where  ``nn''  is  the  hexadecimal value of <stor­
              age_class> and ``bb'' and ``aacc'' are  hexadecimal
              components  of the arrival time.  (The arrival time
              in seconds since epoch  is  converted  to  hex  and
              interpreted as 0xaabbccFF, with ``aa'', ``bb'', and
              ``cc'' used to build the path.)  This  method  does
              not   have   self-expire   functionality   (meaning
              expire(8) has to run  periodically  to  delete  old
              articles).  ``EXPENSIVESTAT'' is ``FALSE'' for this
              method.

       timehash
              This method is very similar to  ``timecaf''  except
              that  each  article  is  stored in a separate file.
              The name of the file for a given  article  will  be
              <patharticles in inn.conf>/time-nn/bb/cc/yyyy-aadd,
              where ``nn'' is the  hexadecimal  value  of  <stor­
              age_class>, ``yyyy'' is a hexadecimal sequence num­
              ber, and ``bb'', ``cc'', and  ``aadd''  are  compo­
              nents  of  the  arrival  time  in  hexadecimal (the
              arrival time is converted to hex and interpreted as
              0xaabbccdd).  This method does not have self-expire
              functionality.  ``EXPENSIVESTAT'' is  ``TRUE''  for
              this method.

       tradspool
              Traditional  spool, or ``tradspool'', is the tradi­
              tional news article storage format.   Each  article
              is      stored      in      a      file      named:
              <patharticles in inn.conf>/news/group/name/nnnnn,
              where  ``news/group/name'' is the name of the news­
              group to which the article  was  posted  with  each
              period  changed  to  a  slash, and ``nnnnn'' is the
              sequence number of the article in  that  newsgroup.
              For  crossposted  articles,  the  article is linked
              into each newsgroup  to  which  it  is  crossposted
              (using either hard or symbolic links).  This is the
              way all versions of INN prior  to  2.0  stored  all
              articles, as well as being the article storage for­
              mat used by C News and earlier news systems.   This
              method  does  not  have  self-expire functionality.
              ``EXPENSIVESTAT'' is ``TRUE'' for this method.

       trash  This method silently discards all  articles  stored
              in  it.  Its only real uses are for testing and for
              silently discarding articles matching a  particular
              storage   method   entry   (for  whatever  reason).

              space  and  can  never be retrieved, so this method
              has self-expire functionality of a sort.   ``EXPEN­
              SIVESTAT'' is ``FALSE'' for this method.


EXAMPLE

       The  following  sample  storage.conf  file would store all
       articles posted to alt.binaries.* in the ``BINARIES'' CNFS
       metacycbuff,  all  articles over roughly 50KB in any other
       hierarchy in the ``LARGE''  CNFS  metacycbuff,  all  other
       articles  in  alt.*  in  one timehash class, and all other
       articles in any newsgroups in  a  second  timehash  class,
       except  for  the  internal.*  hierarchy which is stored in
       traditional spool format.

              method tradspool {
                  class: 1
                  newsgroups: internal.*
              }

              method cnfs {
                  class: 2
                  newsgroups: alt.binaries.*
                  options: BINARIES
              }

              method cnfs {
                  class: 3
                  newsgroups: *
                  size: 50000
                  options: LARGE
              }

              method timehash {
                  class: 4
                  newsgroups: alt.*
              }

              method timehash {
                  class: 5
                  newsgroups: *
              }

       Notice that the  last  storage  method  entry  will  catch
       everything.   This  is a good habit to get into; make sure
       that you have at least one catch-all entry  just  in  case
       something  you  didn't  expect  falls  through the cracks.
       Notice also that the special rule for the internal.* hier­
       archy is first, so it will catch even articles crossposted
       to alt.binaries.* or over 50KB in size.


HISTORY

       Written by Katsuhiro Kondou <kondou@nec.co.jp> for  Inter­
       NetNews.  This is revision 1.7.2.2, dated 2001/04/17.

       cycbuff.conf(5),   expire.ctl(5),   inn.conf(5),  innd(8),
       newsfeeds(5), wildmat(3).


Man(1) output converted with man2html