NAME

       ::htmlparse - Procedures to parse HTML string


SYNOPSIS

       package require htmlparse ?1.0?

       ::htmlparse::parse  ?-cmd  cmd?  ?-vroot  tag?  ?-split n?
       ?-incvar var? ?-queue q? html

       ::htmlparse::debugCallback ?clientdata?  tag  slash  param
       textBehindTheTag

       ::htmlparse::mapEscapes html

       ::htmlparse::2tree html tree

       ::htmlparse::removeVisualFluff tree

       ::htmlparse::removeFormDefs tree

_________________________________________________________________


DESCRIPTION

       The  ::htmlparse  package  provides  commands  that  allow
       libraries and applications to parse HTML in a string  into
       a representation of their choice.

       The following commands are available:

       ::htmlparse::parse  ?-cmd  cmd?  ?-vroot tag? ?-split n?
              ?-incvar var? ?-queue q? html

              This command is the basic parser for HTML. It takes
              a HTML string, parses it and invokes a command pre­
              fix  for every tag encountered. It is not necessary
              for the HTML to be valid for this parser  to  func­
              tion.  It  is  the  responsibility  of  the command
              invoked  for  every  tag  to  check  this.  Another
              responsibility  of  the invoked command is the han­
              dling of  tag  attributes  and  character  entities
              (escaped  characters).  The parser provides the un-
              interpreted tag attributes to the  invoked  command
              to aid in the former, and the package at large pro­
              vides a helper command, ::htmlparse::mapEscapes, to
              aid  in the handling of the latter. The parser does
              ignore leading DOCTYPE declarations and  all  valid
              HTML comments it encounters.

              All  information  beyond  the HTML string itself is
              specified via options, these are explained below.

              To  help  understanding  the  options   some   more


              It is capable to detect incomplete tags in the HTML
              string given to it. Under normal circumstances this
              will cause the parser to throw an error, but if the
              option -incvar is used  to  specify  a  global  (or
              namespace)  variable  the  parser  will  store  the
              incomplete part of the  input  into  this  variable
              instead.  This  will aid greatly in the handling of
              incrementally arriving HTML as the parser will han­
              dle  whatever  he can and defer the handling of the
              incomplete part until more data has arrived.

              Another feature of the parser are its two  possible
              modes of operation. The normal mode is activated if
              the option -queue is not  present  on  the  command
              line  invoking  the  parser.  If  it is present the
              parser will go into the incremental mode instead.

              The main difference is that a parser in normal mode
              will immediately invoke the command prefix for each
              tag it encounters. In incremental mode however  the
              parser  will  generate  a  number  of scripts which
              invoke the command prefix for groups of tags in the
              HTML  string  and  then  store these scripts in the
              specified queue. It is then the  responsibility  of
              the caller of the parser to ensure the execution of
              the scripts in the queue.

              Note: The queue object given to the parser  has  to
              provide  the same interface as the queue defined in
              tcllib -> struct. This does for example  mean  that
              all  queues  created via that part of tcllib can be
              immediately used here.  Still,  the  queue  doesn't
              have  to  come from tcllib -> struct as long as the
              same interface is provided.

              In both modes  the  parser  will  return  an  empty
              string to the caller.

              To  a  parser in incremental mode the option -split
              can be given and  will  specify  the  size  of  the
              groups  he  creates. In other words, -split 5 means
              that each of the generated scripts will invoke  the
              command  prefix  for 5 consecutive tags in the HTML
              string. A parser in normal mode  will  ignore  this
              option and its value.

              The  option  -vroot specifies a virtual root tag. A
              parser in normal mode will invoke the command  pre­
              fix  for  it  immediately  before and after he pro­
              cesses the tags in the HTML, thus  simulating  that
              the  HTML  string is enclosed in a <vroot> </vroot>
              combination. In incremental mode however the parser

              never knows when the input  is  complete.  In  this
              case  the first script generated by each invocation
              of the parser will contain  an  invocation  of  the
              command  prefix  for  the virtual root as its first
              command.

              The following options are available:

              -cmd cmd
                     The command prefix to invoke for  every  tag
                     in  the  HTML  string.  Defaults  to ::html­
                     parse::debugCallback.

              -vroot tag
                     The virtual root tag to add around the  HTML
                     in  normal  mode.  In incremental mode it is
                     the first tag in each chunk processed by the
                     parser,  but  there will be no closing tags.
                     Defaults to hmstart.

              -split n
                     The size of the groups produced by an incre­
                     mental  mode  parser. Ignored when in normal
                     mode. Defaults to 10. Values <=  0  are  not
                     allowed.

              -incvar var
                     The  name of the variable where to store any
                     incomplete HTML into. This makes most  sense
                     for  the  incremental  mode. The parser will
                     throw an error if he  sees  incomplete  HTML
                     and  has no place to store it to. This makes
                     sense for the normal mode.  Only  incomplete
                     tags   are   detected,   not  missing  tags.
                     Optional, defaults to 'no variable'.

              Interface to the command prefix
                     In normal mode the parser  will  invoke  the
                     command  prefix with for arguments appended.
                     See   ::htmlparse::debugCallback    for    a
                     description.

                     In  incremental  mode  however the generated
                     scripts will invoke the command prefix  with
                     five  arguments  appended.  The last four of
                     these are  the  same  which  were  mentioned
                     above.  The  first  however is a placeholder
                     string (\win\) for a clientdata value to  be
                     supplied  later  during the actual execution
                     of the generated scripts. This could be a tk
                     window  path,  for  example. This allows the
                     user of  this  package  to  preprocess  HTML
                     strings   without   committing   them  to  a

                     parsing.  This connection can be made later.
                     This also means that it is possible to cache
                     preprocessed  HTML.  Of course, nothing pre­
                     vents the user of the parser to replace  the
                     placeholder with an empty string.

       ::htmlparse::debugCallback  ?clientdata?  tag slash param
              textBe­ hindTheTag
              This  command  is the standard callback used by the
              parser in ::htmlparse::parse if none was  specified
              by  the user. It simply dumps its arguments to std­
              out.  This callback can be used for both normal and
              incremental  mode  of  the calling parser. In other
              words, it accepts four or five arguments. The  last
              four  arguments  are  described below. The optional
              fifth argument contains the clientdata value  given
              to  the  callback  by a parser in incremental mode.
              All callbacks have to follow the signature of  this
              command  in  the last four arguments, and callbacks
              used in incremental parsing  have  to  follow  this
              signature in the last five arguments.

              The  first  argument,  clientdata,  is optional and
              present only if this command is invoked by a parser
              in  incremental mode. It contains whatever the user
              of this package wishes.

              The second argument, tag, contains the name of  the
              tag which is currently processed by the parser.

              The  third argument, slash, is either empty or con­
              tains a slash character. It allows the callback  to
              distinguish  between  opening  (slash is empty) and
              closing tags (slash contains a slash character).

              The fourth argument, param, contains the  un-inter­
              preted list of parameters to the tag.

              The fifth and last argument, textBehindTheTag, con­
              tains the text found by the parser behind  the  tag
              named in tag.

       ::htmlparse::mapEscapes html
              This  command  takes a HTML string, substitutes all
              escape sequences with their actual  characters  and
              then  returns  the  resulting string.  HTML strings
              which do not contain escape sequences are  returned
              unchanged.

       ::htmlparse::2tree html tree

              This command is a wrapper around ::htmlparse::parse
              which takes a HTML string (in html) and converts it

              parsed document. The name of the tree is  given  to
              the command as its second argument (tree). The com­
              mand does not  generate  the  tree  by  itself  but
              expects  that the caller provided it with an exist­
              ing and empty tree. It also expects that the speci­
              fied  tree object follows the same interface as the
              tree object in tcllib -> struct. It doesn't have to
              be from tcllib -> struct, but it has to provide the
              same interface.

              The internal callback does some basic  checking  of
              HTML  validity  and  tries to recover from the most
              basic errors. The command returns the  contents  of
              its  second argument. Side effects are the creation
              and manipulation of a tree object.

       ::htmlparse::removeVisualFluff tree
              This command walks a tree as generated  by  ::html­
              parse::2tree and removes all the nodes which repre­
              sent visual tags and not structural ones. The  pur­
              pose  of  the command is to make the tree easier to
              navigate without  getting  bogged  down  in  visual
              information  not  relevant  to the search. Its only
              argument is the name of the tree to cut down.

       ::htmlparse::removeFormDefs tree
              Like ::htmlparse::removeVisualFluff this command is
              here  to cut down on the size of the tree as gener­
              ated by ::htmlparse::2tree. It  removes  all  nodes
              representing  forms  and  form  elements.  Its only
              argument is the name of the tree to cut down.


SEE ALSO


KEYWORDS

       html, parsing


Man(1) output converted with man2html