User Guide for Data Descriptor by Example
Leonard Berman
Note, this userguide has been removed from the distribution of DDbE. All links that
refer to other files in the distribution are broken. DDbE can be downloaded from IBM's alphaworks
site.
Abstract:
Data Descriptor by Example (DDbE) is a java component library which constructs
a data descriptor, e.g. a DTD, from a set of well-formed XML documents. The resulting
descriptor can be used to validate the original input documents or any other documents
which follow the syntactic rules inferred from the original examples.
The API documentation is at DDbE/Docs/apiDocs/index.html.
If you are reading this, you probably know where and how to get DDbE; however, just in
case, DDbE can be downloaded from IBM's alphaworks
site.
(You may also want to check out some of
the other interesting
technologies
which are also available from a text
based site).
- Requirements
- DDbE requires xml4j v 2.0. It has been most thoroughly tested with
xml4j v2.0.11. If you are having problems, try downloading the newest
xml4j.jar. (Even if this fixes a
problem, we would still appreciate hearing about it.)
- Installation
-
A command line application which constructs DTDs from
collections of well formed xml examples is contained in
DDbECommandLine.java..
Scripts for building and running the application as well as input files (with extension
.input) and files applying attribute constraints (extension .p) are in that
directory. The remainder of this file describes the parameters accepted by
DDbECommandLine and the use of the DDbE.jar library. Some additional example input
files are contained in
Samples/examples.
In this directory tree, files whose names end with '.input' are suitable for use with
the -s flag described below. Files ending with '.p' contain examples of the
attribute and content model parameter flags. These flags are included by the '.input'
files.
The directory rooted at
Samples/examples
contains
two other example applications written with the DDbE library.
input files which illustrate the use of DDbE with a variety of parameters.
There are four main steps in the construction of a DTD from a set of examples documents.
- 1.
- Input of documents. The documents may be stored in many ways, for example as
files on the local system, as a String, an
org.w3c.dom.Document, or in a
URI. The
FileStringRW
class enables DDbE input to be furnished either as names of files on containing xml
documents, URIs or as Strings. If examples are available as org.w3c.dom.Document the
instances, the user may wish to implement
ExampleGeneratorIF
directly.
- 2.
- Generation of example instances. This is encapsulated in the
ExampleGeneratorIF and ExampleIF interfaces. ExampleIFs
are trivial extensions of
the org.w3c.dom.Node interface wrapped to simplify use in this application. The DDbE
library impelments ExampleGeneratorIF in the ExampleGenerator
class whose construction
requries an instance of FileStringRW.
- 3.
- Processing of examples. The
DDModel
implements the
DDModelIF
interface. DDModel has two constructors, DDModel(java.lang.String[]
args)
which expects a String[] containing command line arguments and
DDModel(Parameter
args)
which expects an object from the class
Parameter.
The examples, encapsulated in the ExampleGeneratorIF, are
furnished to DDModel using the setExampleGenerator
method.
- 4.
- Constructing and emitting the DTD. DDModelIF has two methods for accessing the DTD constructed from the examples:
- (a)
- DDModelIF.saveDTD
method writes the resulting DTD to either a file or to stdout.
- (b)
- The DTD can be retrieved as a String using the DDModel.getDTD
method.
The DDbE Command line java application is a very simple program that illustrates how
the DDbE library can be used. It reads a set of well-formed xml documents and
constructs a DTD from them.
To build the DDbE command line application, go to the DDbE/Samples/CommandLine
directory in the distribution. This directory contains:
- The DDbECommandLine.java
file, which contains the code for the application, and the
DDbECommandLine.class file.
- Shell scripts, ddbe and build, and .bat files, ddbe.bat and build.bat, which can be modified to build and run the command line application.
- Input files:
TeXML.input,
personal.input,
and par.p
which illustrate how the
command line application is used.
DDbECommandLine can be compiled directly from the command line using one of the commands
below. (Of course, any java compiler can be substituted for IBM's jikes compiler.)
- On linux:
jikes
-classpath .:/usr/local/java/classes.zip:../../DDbE.jar:../../xml4j.jar DDbECommandLine.java
- On windows:
jikes
-classpath
.;D:\Jav1.1.8\lib\classes.zip;..\..\DDbE.jar;..\..\xml4j.jar
DDbECommandLine.java
In both cases, the classpath will depend on your particular environment.
Now, to generate a DTD from a set of XML source files, simply run java with the DDbECommandLine
class as argument.
The class file can be run using IBM's jdk or any java VM. The IBM jdk is available for
linux,
and
windows.
On linux, you will need a 2.2 kernel to use the IBM JVM since it uses native threads.
In any case, it is run using the command:
java -classpath
<
classpath >
DDbECommandLine file0 file1 ... fileN ( -w
dtdOutputFile )?
Where
< classpath > is the same string that was used to compile the java file,
{ filei } are example the xml files, and dtdOutputFile is the base name of the
desired output file. In the distribution, ../examples/personal.xml is a valid example
file and DDbECommandLine can be run of this file using:
java -classpath
< classpath > DDbECommandLine ../examples/personal.xml -w dtdOutputFile
Controlling the output of DDbE
DDbE constructs a data descriptor from a set of example xml documents in a three step
process.
- 1.
- For each example document, the xml is read and a dom tree is constructed. The
resulting dom element nodes are grouped according to element type.
- 2.
- For each element type, a set of strings, representing all instances of the
element type in the examples, is constructed. Finally, a content model, which contains all
of these examples, is constructed and associated with the element type.
- 3.
- For each attribute, the validity constraints on attribute
types and the actual values which occurred in
the example documents are used to determine the types and defaults that the attribute is permitted in a
valid document. Each attribute is declared to have one of these permitted types and and
defaults.
Associated with each stage of this process, there is a set of parameters which guide
program behavior during that stage.
DDbE permits control, on a global or element by element basis, of properties which
effect the construction of content models and the
declaration of attributes.
This section describes how each stage of the process is controlled when using either the
DDbE library or the command line interface.
All command line arguments can be constructed and passed to the library as strings;
however, for most there are methods which permit all parameters to be set directly.
Command line arguments may repeat. If they do, the final flag, ie. the flag which occurs farthest to
the right, sets the final value of the parameter.
Parameters controlling I/O
Syntactically, command line I/O arguments fall into four classes:
- 1.
- A ``#'' makes the rest of the line a comment.
- 2.
- The markupdecl flag (-m) which requests that the output matches the
markupdecl
production
production of the xml spec.
- 3.
- A parameter flag, i.e. a ``-'' followed by a single character and a string. Spaces
between the single character and string are optional.
- 4.
- Everything else is interpreted as a file name or URL containing a well-formed xml example.
The meaning and syntax of the command line arguments, other than the comment symbol #, is given below.
Parameters controlling content model construction
Content models are regular expressions.
The following example will illustrate the terminology used to describe the effect of the parameters.
Let
{ex[i]| 0
i < n}, be a set of examples of the content of some element
presented as a comma separated sequence of element names and assume that the set can
be rewritten
wL,(x[0] + ... + x[p]), wR + (ex[p + 1] + ... + ex[n])
where ' + ' indicates set union, ', ' indicates concatenation and for
k = 0..., p : wL, x[k], wR = ex[k]. If p < n, we say that the depth
of factorization is 1, that the examples,
ex[p + 1] + ... + ex[n], are at level 0,
and that the factored examples,
x[0] + ... + x[p], are at level 1. If p = n, we say
the depth of factorization is 0 and that
x[0] + ... + x[p], are at
level 0.
Content model parameter strings are introduced by a '-e' flag and obey the
following syntax:
elementName? '%' (l|c#|d#|o#)*
If the elementName string is present, the constrain applies to the content model of the named element,
otherwise it applies to all content models (Recall that if a flag occurs more than once
the rightmost occurrence is used.) The meaning of the individual fields is
- Name of element effected if present, otherwise parameters apply to all models.
- Maximum number of consecutive identical tokens not replaced by a list. Default (1) replaces all
sequences of two or more with a positive list. That is,
a, a = > a+.
- Maximum depth of factorization. If negative, depth is unlimited. Default = 2.
- Should a list be used at maximum factorization depth. If present, the l flag
directs that, at maximum depth of factorization (specified by the d# flag), the
remaining terms are replaced by a list of the
tokens which occur in the terms.
- Effects the content model constructed for elements whose examples contain repeated
elements. Valid values are 0 - 3 with 3 resulting most likely to result in a ``list
model'' for repeating elements.
Examples of effect of command line arguments on content
model.
The sample program, DDbEContentModel,
permits the user to experiment with the effect of different parameters on the
construction of content models.
For example, The file
ex0.input
contains the three strings a ; a,a,a ; a,b,c. Varying the 'c' parameter yields:
- With the default:
(a+,(bc)?).
- With -e%c2:
(a,(a|(b, c))?)
Parameters controlling attibute declarations
Attribute model parameter strings permit the
user to constrain declarations and defaults of attributes. They are
introduced by a '-a' flag and obey the
following syntax:
( '!'|elementName)? '%' attributeName?
'%' (('!')? ('i' | 'e' | 'c' | (('r' | 'n' | 'x'| 'y')('s')?)))* +
('F' | (('I' | 'R' )attVal))?
Spaces are permitted between the flag,'a', and the parameter string; however, no
spaces are allowed in the parameter string itself. The '%' is a field separator. The
first two fields identify the attribute. The flags of the final field occur in two
groups and indicate how the attribute type and default declarations are constrained.
If elementName appears, an
element with the given name and attributes will occur in the resulting data descriptor. If
elementName does not appear, the constraints apply to all attributes with the given name,
or to all attributes if no attribute name is given. (For example, the elementName and attributeName
might be omitted if the user wanted attributes declared as NMTOKEN when possible.)
If the parameter string is starts with a '!,' we require that the attribute name appear in the
parameter string. In this case, an attribute with the given name will be declared
for all elements.
The first group of flags in the final field consists of those flags which precede the
optional 'F.'
These flags constrain type declarations and consist of an optional '!', a single
character, and possibly an 's'. Each of the single character refers to a different
type of attribute declaration:
- i ID.
- e ENUMERATION.
- c CDATA.
- r IDREF.
- n NMTOKEN.
- x ENTITY.
- y NOTATION.
If the '!' is present, the flag insists that the the attribute not be declared with the
given type, otherwise it requests that the attribute be declared with the given type.
If the 's' is present, it indicates that the attribute should be declared in the plural
form even if all occurrences of the attribute contain only a single item.
We note that there is a natural partial order on attribute types induced by lexical
restrictions. In this three level partial order, ID, IDREF, ENTITY, and NOTATION are maximal, CDATA is minimal, and ENUMERATION and NMTOKEN
are intermediate. If more than one attribute type is requested and legal, a maximal
requested type is assigned if possible. If no requested type is legal ( e.g. if the type
requested was IDREF but this was not a valid assignment, perhaps because an occurring value
was a Nmtoken but not a Name, or because an occurring value did not occur as the value of
an IDREF in the same document) NMTOKEN(S) is assigned if legal. In this case, a
message is included in the output dtd.
The final group of flags mirrors the default declarations of the xml specification,
i.e. using a F flag will cause the attribute type to be declared as FIXED if
possible while a R flag will cause the attribute type to be declared as REQUIRED
if possible.
From the library, attributes are
constrained by constructing the appropriate constraint String, using the constructor
AttributeConstraint(String), and using AttributeModelIF.setAttributeConstraint(AttributeConstraint).
In Samples/CommandLine, the file par.p requests that all attributes be declared as
IDREF, the file req.p that all attributes be declared as REQUIRED.
- The pre-release version accepted any xml for which IBM's xml4j parser would build a dom
tree. In this version, only well-formed xml is accepted.
- The pre-release documentation regarding content models and attribute declarations was incorrect. This has
been replaced with valid documentation.
- API documentation is included.
- Both content model and attribute parameters are furnished in a new way.
- The 'm' flag has been added.
The translation was initiated by on 1999-08-11
1999-08-11