innfeed.conf - configuration file for innfeed


DESCRIPTION

       This man page describes the configuration file for version
       1.0 of innfeed. This format has changed dramatically since
       version 0.9.3.

       The  file  innfeed.conf  is used to control the innfeed(1)
       program. It is a fairly free-format file that consists  of
       three  types  of entries: key/value, peer and group.  Com­
       ments are taken from the hash character ``#'' to  the  end
       of the line.

       Key/value entries are a keyword and a value separated by a
       colon (which can itself be surrounded by whitespace).  For
       example:

              max-connections: 10

       A  legal  key  starts with a letter and contains only let­
       ters, numbers and ``_'', ``-''.

       There are 5 different type of values: integers,  floating-
       point  numbers, characters, booleans, and strings. Integer
       and floating point numbers are as to  be  expected  except
       that  exponents  in  floating  point  numbers are not sup­
       ported. A boolean value is either  ``true''  or  ``false''
       (case  is not significant). A character value is a single-
       quoted character as defined by the  C-language.  A  string
       value  is  any other sequence of characters. If the string
       needs to contain whitespace, then it must be  quoted  with
       double quotes, and uses the same format for embedding non-
       printing characters as normal C-language string.

       Peer entries look like:

              peer <name> {
                      # body ...
              }

       The word ``peer'' is required. The ``<name>'' is the  same
       as  the  site  name in INN's newsfeeds file. The body of a
       peer  entry  contains  some  number  (possibly  zero)   of
       key/value entries.

       Group entries look like:

              group <name> {
                   # body
              }

       The  word  ``group''  is  required.  The ``<name>'' is any
       string valid as a key. The body of a group entry  contains

       pairs can be defined inside a  group,  and  peers  can  be
       nested  inside  a  group,  and  other groups can be nested
       inside a group.

       Key/value entries that are defined outside of all peer and
       group  entries  are  said to be at ``global scope''. There
       are global key/value entries that apply to the process  as
       a  whole  (for  example  the  location of the backlog file
       directory), and there are global  key/value  entries  that
       act  as  defaults for peers. When innfeed looks for a spe­
       cific value in a peer entry (for example, the maximum num­
       ber of connections to set up), if the value is not defined
       in the peer entry, then the enclosing groups are  examined
       for  the  entry (starting at the closest enclosing group).
       If there are no enclosing groups, or the enclosing  groups
       don't define the key/value, then the value at global scope
       is used.

       A small example could be:

              # Global value applied to all peers that have
              # no value of their own.
              max-connections: 5

              # A peer definition. ``uunet'' is the name used by innd in
              # the newsfeeds file.
              peer uunet {
                   ip-name: usenet1.uu.net
              }

              peer vixie {
                      ip-name: gw.home.vix.com
                      max-connections: 10      # override global value.
              }

              # A group of two peers who can handle more connections
              # than normal
              group fast-sites {
                   max-connections: 15

                   # Another peer. The ``max-connections'' value from the
                   # ``fast-sites'' group scope is used. The ``ip-name'' value
                   # defaults to the peer's name.
                   peer data.ramona.vix.com {
                   }

                   peer bb.home.vix.com {
                        max-connections: 20 # he can really cook.
                   }
              }

       Given the above  configuration  file,  the  defined  peers
       would   have   the   following   values   for  the  ``max-


              uunet                  5
              vixie                 10
              data.ramona.vix.com   15
              bb.home.vix.com       20

       Innfeed ignores key/value pairs it is not  interested  in.
       Any  config  file value that can be set via a command line
       option, is not used if the command-line option is given.

       Config files can be included in other config files via the
       syntax:
              $INCLUDE filename
       There is a maximum nesting depth of 10.

       For  a  fuller  example config file, see the supplied inn­
       feed.conf.


GLOBAL VALUES

       The following listing show all the keys that apply to  the
       process  as  whole.  These  are  not required (compiled-in
       defaults are used where needed).

       news-spool
              This key requires a pathname  value.  It  specifies
              where  the top of the article spool is. This corre­
              sponds to the ``-a'' command-line option.

       input-file
              This key requires a pathname  value.  It  specifies
              the  pathname  (relative  to the backlog-directory)
              that should be read in funnel-file mode. This  cor­
              responds to giving a filename as an argument on the
              command-line (i.e.  its presence also implies  that
              funnel-file mode should be used).

       pid-file
              This  key  requires  a pathname value. It specifies
              the pathname (relative  to  the  backlog-directory)
              where  the  pid  of  the  innfeed process should be
              stored. This corresponds to the ``-p'' command-line
              option.

       debug-level
              This key defines the debug level for the process. A
              non-zero number generates  a  lot  of  messages  to
              stderr,  or  to  the  config-defined  ``log-file''.
              This corresponds to the ``-d'' command-line option.

       use-mmap
              This  key  requires  a  boolean value. It specifies
              whether mmaping should be used if innfeed has  been
              built with mmap support. If article data on disk is

              line),  then after mmaping the article is read into
              memory and fixed up, so  mmaping  has  no  positive
              effect (and possibly some negative effect depending
              on your system), and so in such a case  this  value
              should  be  false.  This  corresponds to the ``-M''
              command-line option.

       log-file
              This key requires a pathname  value.  It  specifies
              where  any  logging  messages that couldn't be sent
              via syslog(3) should go (such  as  those  generated
              when  a  positive  value  for  ``debug-value'',  is
              used). This corresponds to the ``-l''  command-line
              option.  A  relative  pathname  is  relative to the
              ``backlog-directory'' value.

       backlog-directory
              This key requires a pathname  value.  It  specifies
              where  the  current  innfeed  process  should store
              backlog files. This corresponds to the ``-b''  com­
              mand-line option.

       backlog-highwater
              This  key  requires  a  positive  integer value. It
              specifies how many articles should be kept  on  the
              backlog  file  queue  before  starting to write new
              entries to disk.

       backlog-ckpt-period
              This key requires  a  positive  integer  value.  It
              specifies  how  many seconds between checkpoints of
              the input backlog file. Too  small  a  number  will
              mean  frequent  disk  accesses,  too large a number
              will mean after a crash innfeed will re-offer  more
              already-processed articles than necessary.

       backlog-newfile-period
              This  key  requires  a  positive  integer value. It
              specifies  how  many  seconds  between  checks  for
              externally  generated  backlog files that are to be
              picked up and processed.

       dns-retry
              The key  requires  a  positive  integer  value.  It
              defines  the  number of seconds between attempts to
              re-lookup host information that previous failed  to
              be resolved.

       dns-expire
              The  key  requires  a  positive  integer  value. It
              defines the number of seconds between refreshes  of
              name  to  address  DNS translation. This is so long
              running processes don't get stuck with stale  data,


       close-period
              The  key  requires  a positive integer value. It is
              the maximum number of seconds a  connection  should
              be  kept  open.  Some  NNTP servers don't deal well
              with connections being held open for long  periods.

       gen-html
              This  key  requires  a  boolean value. It specifies
              whether the status-file should be HTML-ified.

       status-file
              This key requires a pathname  value.  It  specifies
              the  pathname  (relative  to the backlog-directory)
              where the periodic status of  the  innfeed  process
              should  be  stored.  This corresponds to the ``-S''
              command-line option.

       connection-stats
              This key requires a boolean value. If the value  is
              true, then whenever the transmission statistics for
              a peer are logged, then each active connection logs
              its  own statistics. This corresponds to the ``-z''
              command-line option.

       host-queue-highwater
              This key requires  a  positive  integer  value.  It
              defines  how  many articles will be held internally
              for a peer before new arrivals cause article infor­
              mation to be spooled to the backlog file.

       stats-period
              This  key  requires  a  positive  integer value. It
              defines how many seconds innfeed waits between gen­
              erating statistics on transfer rates.

       stats-reset
              This  key  requires  a  positive  integer value. It
              defines  how  many  seconds  innfeed  waits  before
              resetting  all  internal  transfer counters back to
              zero (after logging one final time). This is  so  a
              innfeed-process running more than a day will gener­
              ate ``final'' stats that will be picked up by  log­
              file processing scripts.

       initial-reconnect-time
              This  key  requires  a  positive  integer value. It
              defines how  many  seconds  to  first  wait  before
              retrying  to  reconnect after a connection failure.
              If the next attempt fails too, then  the  reconnect
              time  is approximately doubled until the connection
              succeeds, or max-reconnection-time is reached.

              This key requires an integer value. It defines  the
              maximum  number  of seconds to wait between attempt
              to reconnect to  a  peer.  The  initial  value  for
              reconnection  attempts is defined by initial-recon­
              nect-time, and it is doubled after each failure, up
              to this value.

       stdio-fdmax
              This  key requires a non-negative integer value. If
              the value is greater than  zero,  then  whenever  a
              network  socket  file  descriptor is created and it
              has a value less than  this,  the  file  descriptor
              will  be  dup'ed to bring the value up greater than
              this. This is to leave lower numbered file descrip­
              tors free for stdio. Certain systems, Sun's in par­
              ticular, require this. SunOS 4.1.x usually requires
              a value of 128 and Solaris requires a value of 256.
              The default if this is not specified, is 0.

       bindaddress
              Which interface IP address innfeed should bind  the
              local  end  of its connections to.  Must be in dot­
              ted-quad format  (nnn.nnn.nnn.nnn).   If  not  set,
              innfeed  defaults to letting the kernel choose this
              address.  The default value is unset.


GLOBAL PEER DEFAULTS

       All the key/value pairs mentioned in this section must  be
       specified  at  global  scope.  They  may also be specified
       inside a group or peer definition. Note  that  when  peers
       are added dynamically (i.e. when innfeed receives an arti­
       cle for an unspecified peer), it will add  the  peer  site
       using the parameters specified at global scope.

       article-timeout
              This  key requires a non-negative integer value. If
              no articles need to be sent to the  peer  for  this
              many  seconds, then the peer is considered idle and
              all its active connections are torn down.

       response-timeout
              This key requires a non-negative integer value.  It
              defines  the  maximum  amount of time to wait for a
              response from the peer after issuing a command.

       initial-connections
              This key requires a non-negative integer value.  It
              defines  the  number  of  connections  to be opened
              immediately when setting up a peer binding. A value
              of  0 means no connections will be created until an
              article needs to be sent.

              This  key  requires  positive  integer  value.   It
              defines the maximum number of connections to run in
              parallel to the peer. A value of zero specifies  an
              unlimited number of maximum connections. In general
              use of an unlimited number of  maximum  connections
              is not recommended. Do not ever set max-connections
              to zero with dynamic-method 0  set,  as  this  will
              saturate  peer  hosts with connections. [ Note that
              in previous versions of innfeed, a value of 1 had a
              special  meaning.  This  is  no  longer the case, 1
              means a maximum of 1 connection ].

       dynamic-method
              This key requires a value between 0 and 3. It  con­
              trols  how  connections (up to max-connections) are
              opened up to the maximum specified  by  max-connec­
              tions.  In general (and specifically, with dynamic-
              method 0) a new connection is opened when the  cur­
              rent  number  of  connections  is below max-connec­
              tions, and an article is to be sent whilst no  cur­
              rent   connections   are   idle.   Without  further
              restraint (i.e. using dynamic-method 0),  in  prac­
              tice  this  means  that max-connections connections
              are established whilst articles are being sent. Use
              of  other dynamic-method settings imposes a further
              limit on the amount  of  connections  opened  below
              that  specified  by  max-connections. This limit is
              calculated in  different  ways,  depending  of  the
              value  of  dynamic-method.   Users should note that
              adding additional connections is not always produc­
              tive  -  just because opening twice as many connec­
              tions results in a  small  percentage  increase  of
              articles  accepted  by the remote peer, this may be
              at considerable resource cost both locally  and  at
              the remote site, whereas the remote site might well
              have received the extra articles sent from  another
              peer  a  fraction  of a second later. Opening large
              numbers of connections  is  considered  antisocial.
              The meanings of the various settings are:

              0 no method
                     Increase  of  connections  up to max-connec­
                     tions is unrestrained.

              1 maximize articles per second
                     Connections are increased (up to max-connec­
                     tions)  and  decreased so as to maximize the
                     number of articles per second  sent,  whilst
                     using the fewest connections to do this.

              2 set target queue length
                     Connections are increased (up to max-connec­
                     tions) and decreased so as to keep the queue

                     by dynamic-backlog-low and  dynamic-backlog-
                     high, whilst using the minimum resource pos­
                     sible.  As the queue will tend  to  fill  if
                     the  site  is  not  keeping  up, this method
                     ensures that the maximum number of  articles
                     are  offered  to  the  peer whilst using the
                     minimum number  of  connections  to  achieve
                     this.

              3 combination
                     This  method uses a combination of methods 1
                     and 2 above. For  sites  accepting  a  large
                     percentage  of  articles,  method  2 will be
                     used to ensure these sites  are  offered  as
                     complete  a  feed  as  possible.  For  sites
                     accepting a small  percentage  of  articles,
                     method   1   is  used,  to  minimize  remote
                     resource usage. For intermediate  sites,  an
                     appropriate combination is used.

       dynamic-backlog-low
              This key takes a value between 0 and 100 and repre­
              sents (as a percentage) the low water mark for  the
              host  queue.  When  the host queue falls below this
              level, when using dynamic-method 2 or 3,  if  2  or
              more  connections are open, innfeed will attempt to
              drop connections to the  host.  An  IIR  filter  is
              applied  to  the  value  to prevent connection flap
              (see dynamic-filter). A value  of  25.0  is  recom­
              mended.  This  value  must be smaller than dynamic-
              backlog-high.

       dynamic-backlog-high
              This key takes a value between 0 and 100 and repre­
              sents (as a percentage) the high water mark for the
              host queue. When the host queue  rises  above  this
              level,  when  using  dynamic-method 2 or 3, if less
              than max-connections are open to the host,  innfeed
              will  attempt  to  open  further connections to the
              host. An IIR filter is applied to the value to pre­
              vent  connection flap (see dynamic-filter). A value
              of 50.0 is recommended.  This value must be  larger
              than dynamic-backlog-low.

       dynamic-backlog-filter
              This key takes a floating-point value between 0 and
              1 which represents the filter coefficient  used  by
              the  IIR  filter used to implement dynamic-method 2
              and 3.  The recommended value  of  this  filter  is
              0.7,  giving a time constant of 1/(1-0.7) articles.
              Higher values will result  in  slower  response  to
              queue  fullness  changes,  lower values with faster
              response.

              This key requires  a  positive  integer  value.  It
              defines  the  maximum number of articles to process
              at one time when using streaming to transmit  to  a
              peer.  Larger  numbers mean more memory consumed as
              articles usually get pulled into  memory  (see  the
              description of use-mmap).

       streaming
              This  key  requires  a  boolean  value.  It defines
              whether streaming commands  are  used  to  transmit
              articles to the peers.

       no-check-high
              This  key  requires  a  floating-point number which
              must be in the range  [0.0,  100.0].  When  running
              transmitting  with  the streaming commands, innfeed
              attempts an optimization called ``no-CHECK''  mode.
              This  involves  not asking the peer if it wants the
              article, but just  sending  it.  This  optimization
              occurs when the percentage of the articles the peer
              has accepted gets larger than this number. If  this
              value  is set to 100.0, then this effectively turns
              off no-CHECK mode, as the percentage can never  get
              above  100.0.  If this value is too small, then the
              number of articles the peer rejects will get bigger
              (and  your  bandwidth  will  be wasted). A value of
              95.0 is usually pretty good. NOTE: In innfeed 0.9.3
              and earlier this value was in the range [0.0, 9.0].

       no-check-low:
              This key requires  a  floating-point  number  which
              must  be  in the range [0.0, 100.0), and it must be
              smaller that the value for no-check-high. When run­
              ning  in  no-CHECK mode, as described above, if the
              percentage of articles  the  remote  accepts  drops
              below  this  number, then the no-CHECK optimization
              is turned off until the percentage gets  above  the
              no-check-high  value  again. If there is small dif­
              ference between this and  the  no-check-high  value
              (less  than about 5.0), then innfeed may frequently
              go in and out of no-CHECK mode. If  the  difference
              is  too big, then it will make it harder to get out
              of no-CHECK  mode  when  necessary  (wasting  band­
              width).  Keeping  this to between 5.0 and 10.0 less
              than no-check-high is usually pretty good.

       no-check-filter
              This is a floating  point  value  representing  the
              time  constant, in articles, over which the CHECK /
              no-CHECK calculations  are  done.  The  recommended
              value is 50.0 which will implement an IIR filter of
              time constant 50. This roughly equates to making  a
              decision  about  the  mode  over  the  previous  50

              response   to   changing  percentages  of  articles
              accepted; a lower number will result  in  a  faster
              response.

       port-number
              This  key  requires  a  positive  integer value. It
              defines the tcp/ip port number to use when connect­
              ing to the remote.

       drop-deferred
              This key requires a boolean value. By default it is
              set to false. When set to true, and a peer  replies
              with  code  431  or 436 (try again later) just drop
              the article and don't try to re-send  it.  This  is
              useful  for some peers that keep on deferring arti­
              cles for a long time to prevent innfeed from trying
              to offer the same article over and over again.

       min-queue-connection
              This key requires a boolean value. By default it is
              set to  false.  When  set  to  true,  innfeed  will
              attempt  to  use  a connection with the least queue
              size (or the first empty connection).  If this  key
              is  set  to  true,  it is recommended that dynamic-
              method be set to 0. This allows for article  propa­
              gation with the least delay.

       no-backlog
              This  key  requires  a  boolean value. It specifies
              whether spooling  should  be  enabled  (false,  the
              default)  or  disabled  (true).  Note that when no-
              backlog is set, articles reported as "spooled"  are
              actually silently discarded.

       backlog-limit
              This  key requires a non-negative integer value. If
              the number is 0 then backlog files are  allowed  to
              grown without bound when the peer is unable to keep
              up with the article flow. If this number is greater
              than  0  then  it specifies the size (in bytes) the
              backlog file should get truncated to when the back­
              log file reaches a certain limit. The limit depends
              on whether backlog-factor or backlog-limit-high  is
              used.

       backlog-factor
              This  key  requires  a  floating point value, which
              must be larger than 1.0. It is used in  conjunction
              with  the  peer key backlog-limit. If backlog-limit
              has a value greater than zero, then when the  back­
              log file gets larger than the value backlog-limit *
              backlog-factor, then the backlog file will be trun­
              cated  to  the  size  backlog-limit. For example if

              factor  has  a value of 2.0, then when the backlog­
              file gets to be larger than 2000000 bytes in  size,
              it  will  be truncated to 1000000 bytes.  The front
              portion of the file is removed,  and  the  trimming
              happens  on  line boundaries, so the final size may
              be a bit less than this number.  If  backlog-limit-
              highwater is defined too, then backlog-factor takes
              precedence.

       backlog-limit-highwater
              This key requires a  positive  integer  value  that
              must be larger than the value for backlog-limit. If
              the size of the backlog file gets larger than  this
              value  (in  bytes),  then  the backlog file will be
              shrunk down to the size of backlog-limit.  If  both
              backlog-factor   and   backlog-limit-highwater  are
              defined, then the value of backlog-factor is  used.


PEER VALUES

       As  previously explained, the peer definitions can contain
       redefinitions of any of the key/value pairs  described  in
       the  GLOBAL  PEER  DEFAULTS  section  above.  There is one
       key/value pair that is specific to a peer definition.

       ip-name
              This key requires a word value.  The  word  is  the
              host's FQDN, or the dotted quad ip-address. If this
              value is not specified then the name of the peer is
              taken  to  also  be  its ip-name. See the entry for
              data.ramona.vix.com in the example below.


RELOADING

       If innfeed gets a SIGHUP signal, then it will  reread  the
       config file. All values at global scope except for ``back­
       log-directory'' can be changed. Any new  peers  are  added
       and any missing peers have their connections closed.


EXAMPLE

       Below is the sample innfeed.conf file.
              #
              # innfeed.conf file. See the comment block at the
              # end for a fuller description.
              #

              ##
              ## Global values. Not specific to any peer. These
              ## are optional, but if used will override the
              ## compiled in values. Command-line options used
              ## will override these values.
              ##

              pid-file:               innfeed.pid
              debug-level:            0

              log-file:               innfeed.log
              stdio-fdmax:            0

              backlog-directory:      innfeed
              backlog-rotate-period:  60
              backlog-ckpt-period:    30
              backlog-newfile-period: 600

              dns-retry:              900
              dns-expire:             86400
              close-period:           3600
              gen-html:               false
              status-file:            innfeed.status
              connection-stats:       false
              host-queue-highwater:   200
              stats-period:           600
              stats-reset:            43200

              max-reconnect-time:     3600
              initial-reconnect-time: 30

              ##
              ## Defaults for all peers. These must all exist at
              ## global scope. Any of them can be redefined
              ## inside a peer or group definition.
              ##

              article-timeout:        600
              response-timeout:       300
              initial-connections:    1
              max-connections:        5
              max-queue-size:         25
              streaming:              true
              no-check-high:          95.0
              no-check-low:           90.0
              no-check-filter:        50.0
              port-number:            119
              backlog-limit:          0
              backlog-factor:         1.10
              backlog-limit-highwater:0
              dynamic-method:         3
              dynamic-backlog-filter: 0.7
              dynamic-backlog-low:    25.0
              dynamic-backlog-high:   50.0
              no-backlog:             false

              ##
              ## Peers.
              ##
              peer decwrl {
                      ip-name:                news1.pa.dec.com
              }

                      ip-name:                news.uunet.uu.net
                      max-connections:        10
              }

              peer data.ramona.vix.com {
                      # ip-name defaults to data.ramona.vix.com
                      streaming:              false
              }

              peer bb.home.vix.com {
                      ip-name:        192.5.5.33
              }

              # Blank lines are ignored. Everything after a '#'
              # is ignored too.
              #
              # Format is:
              #               key : value
              #
              # See innfeed.conf(5) for a description of
              # necessary & useful keys. Unknown keys and their
              # values are ignored.
              #
              # Values may be a integer, floating-point, c-style
              # single-quoted characters, boolean, and strings.
              #
              # If a string value contains whitespace, or
              # embedded quotes, or the comment character
              # (``#''), then the whole string must be quoted
              # with double quotes.  Inside the quotes, you may
              # use the standard c-escape sequence
              # (\t,\n,\r,\f,\v,\",\').
              #
              # Examples:
              #       eg-string:      "NewConfigfile0
              #       eg-long-string: "A long string that goes
              #                       over multiple lines. The
              #                       newline is kept in the
              #                       string except when quoted
              #                       with a backslash #                       as here."
              #       eg-simple-string: A-no-quote-string
              #       eg-integer:     10
              #       eg-boolean:     true
              #       eg-char:        'a'
              #       eg-ctrl-g:      ' 07'


HISTORY

       Written  by  James Brister <brister@vix.com> for InterNet­
       News.  This is revision 1.9.2.1, dated 2001/03/11.

       innfeed(1), newsfeeds(5)


Man(1) output converted with man2html