NAME
::htmlparse - Procedures to parse HTML string
SYNOPSIS
package require htmlparse ?1.0?
::htmlparse::parse ?-cmd cmd? ?-vroot tag? ?-split n?
?-incvar var? ?-queue q? html
::htmlparse::debugCallback ?clientdata? tag slash param
textBehindTheTag
::htmlparse::mapEscapes html
::htmlparse::2tree html tree
::htmlparse::removeVisualFluff tree
::htmlparse::removeFormDefs tree
_________________________________________________________________
DESCRIPTION
The ::htmlparse package provides commands that allow
libraries and applications to parse HTML in a string into
a representation of their choice.
The following commands are available:
::htmlparse::parse ?-cmd cmd? ?-vroot tag? ?-split n?
?-incvar var? ?-queue q? html
This command is the basic parser for HTML. It takes
a HTML string, parses it and invokes a command pre
fix for every tag encountered. It is not necessary
for the HTML to be valid for this parser to func
tion. It is the responsibility of the command
invoked for every tag to check this. Another
responsibility of the invoked command is the han
dling of tag attributes and character entities
(escaped characters). The parser provides the un-
interpreted tag attributes to the invoked command
to aid in the former, and the package at large pro
vides a helper command, ::htmlparse::mapEscapes, to
aid in the handling of the latter. The parser does
ignore leading DOCTYPE declarations and all valid
HTML comments it encounters.
All information beyond the HTML string itself is
specified via options, these are explained below.
To help understanding the options some more
It is capable to detect incomplete tags in the HTML
string given to it. Under normal circumstances this
will cause the parser to throw an error, but if the
option -incvar is used to specify a global (or
namespace) variable the parser will store the
incomplete part of the input into this variable
instead. This will aid greatly in the handling of
incrementally arriving HTML as the parser will han
dle whatever he can and defer the handling of the
incomplete part until more data has arrived.
Another feature of the parser are its two possible
modes of operation. The normal mode is activated if
the option -queue is not present on the command
line invoking the parser. If it is present the
parser will go into the incremental mode instead.
The main difference is that a parser in normal mode
will immediately invoke the command prefix for each
tag it encounters. In incremental mode however the
parser will generate a number of scripts which
invoke the command prefix for groups of tags in the
HTML string and then store these scripts in the
specified queue. It is then the responsibility of
the caller of the parser to ensure the execution of
the scripts in the queue.
Note: The queue object given to the parser has to
provide the same interface as the queue defined in
tcllib -> struct. This does for example mean that
all queues created via that part of tcllib can be
immediately used here. Still, the queue doesn't
have to come from tcllib -> struct as long as the
same interface is provided.
In both modes the parser will return an empty
string to the caller.
To a parser in incremental mode the option -split
can be given and will specify the size of the
groups he creates. In other words, -split 5 means
that each of the generated scripts will invoke the
command prefix for 5 consecutive tags in the HTML
string. A parser in normal mode will ignore this
option and its value.
The option -vroot specifies a virtual root tag. A
parser in normal mode will invoke the command pre
fix for it immediately before and after he pro
cesses the tags in the HTML, thus simulating that
the HTML string is enclosed in a <vroot> </vroot>
combination. In incremental mode however the parser
never knows when the input is complete. In this
case the first script generated by each invocation
of the parser will contain an invocation of the
command prefix for the virtual root as its first
command.
The following options are available:
-cmd cmd
The command prefix to invoke for every tag
in the HTML string. Defaults to ::html
parse::debugCallback.
-vroot tag
The virtual root tag to add around the HTML
in normal mode. In incremental mode it is
the first tag in each chunk processed by the
parser, but there will be no closing tags.
Defaults to hmstart.
-split n
The size of the groups produced by an incre
mental mode parser. Ignored when in normal
mode. Defaults to 10. Values <= 0 are not
allowed.
-incvar var
The name of the variable where to store any
incomplete HTML into. This makes most sense
for the incremental mode. The parser will
throw an error if he sees incomplete HTML
and has no place to store it to. This makes
sense for the normal mode. Only incomplete
tags are detected, not missing tags.
Optional, defaults to 'no variable'.
Interface to the command prefix
In normal mode the parser will invoke the
command prefix with for arguments appended.
See ::htmlparse::debugCallback for a
description.
In incremental mode however the generated
scripts will invoke the command prefix with
five arguments appended. The last four of
these are the same which were mentioned
above. The first however is a placeholder
string (\win\) for a clientdata value to be
supplied later during the actual execution
of the generated scripts. This could be a tk
window path, for example. This allows the
user of this package to preprocess HTML
strings without committing them to a
parsing. This connection can be made later.
This also means that it is possible to cache
preprocessed HTML. Of course, nothing pre
vents the user of the parser to replace the
placeholder with an empty string.
::htmlparse::debugCallback ?clientdata? tag slash param
textBe hindTheTag
This command is the standard callback used by the
parser in ::htmlparse::parse if none was specified
by the user. It simply dumps its arguments to std
out. This callback can be used for both normal and
incremental mode of the calling parser. In other
words, it accepts four or five arguments. The last
four arguments are described below. The optional
fifth argument contains the clientdata value given
to the callback by a parser in incremental mode.
All callbacks have to follow the signature of this
command in the last four arguments, and callbacks
used in incremental parsing have to follow this
signature in the last five arguments.
The first argument, clientdata, is optional and
present only if this command is invoked by a parser
in incremental mode. It contains whatever the user
of this package wishes.
The second argument, tag, contains the name of the
tag which is currently processed by the parser.
The third argument, slash, is either empty or con
tains a slash character. It allows the callback to
distinguish between opening (slash is empty) and
closing tags (slash contains a slash character).
The fourth argument, param, contains the un-inter
preted list of parameters to the tag.
The fifth and last argument, textBehindTheTag, con
tains the text found by the parser behind the tag
named in tag.
::htmlparse::mapEscapes html
This command takes a HTML string, substitutes all
escape sequences with their actual characters and
then returns the resulting string. HTML strings
which do not contain escape sequences are returned
unchanged.
::htmlparse::2tree html tree
This command is a wrapper around ::htmlparse::parse
which takes a HTML string (in html) and converts it
parsed document. The name of the tree is given to
the command as its second argument (tree). The com
mand does not generate the tree by itself but
expects that the caller provided it with an exist
ing and empty tree. It also expects that the speci
fied tree object follows the same interface as the
tree object in tcllib -> struct. It doesn't have to
be from tcllib -> struct, but it has to provide the
same interface.
The internal callback does some basic checking of
HTML validity and tries to recover from the most
basic errors. The command returns the contents of
its second argument. Side effects are the creation
and manipulation of a tree object.
::htmlparse::removeVisualFluff tree
This command walks a tree as generated by ::html
parse::2tree and removes all the nodes which repre
sent visual tags and not structural ones. The pur
pose of the command is to make the tree easier to
navigate without getting bogged down in visual
information not relevant to the search. Its only
argument is the name of the tree to cut down.
::htmlparse::removeFormDefs tree
Like ::htmlparse::removeVisualFluff this command is
here to cut down on the size of the tree as gener
ated by ::htmlparse::2tree. It removes all nodes
representing forms and form elements. Its only
argument is the name of the tree to cut down.
SEE ALSO
KEYWORDS
html, parsing
Man(1) output converted with
man2html