next up previous

User Guide for Data Descriptor by Example

Leonard Berman

Note, this userguide has been removed from the distribution of DDbE. All links that refer to other files in the distribution are broken. DDbE can be downloaded from IBM's alphaworks site.

Abstract:

Data Descriptor by Example (DDbE) is a java component library which constructs a data descriptor, e.g. a DTD, from a set of well-formed XML documents. The resulting descriptor can be used to validate the original input documents or any other documents which follow the syntactic rules inferred from the original examples.

The API documentation is at DDbE/Docs/apiDocs/index.html.

Getting and Using DDbE

If you are reading this, you probably know where and how to get DDbE; however, just in case, DDbE can be downloaded from IBM's alphaworks site. (You may also want to check out some of the other interesting technologies which are also available from a text based site).

Requirements
DDbE requires xml4j v 2.0. It has been most thoroughly tested with xml4j v2.0.11. If you are having problems, try downloading the newest xml4j.jar. (Even if this fixes a problem, we would still appreciate hearing about it.)
Installation

Quick Start.

A command line application which constructs DTDs from collections of well formed xml examples is contained in DDbECommandLine.java.. Scripts for building and running the application as well as input files (with extension .input) and files applying attribute constraints (extension .p) are in that directory. The remainder of this file describes the parameters accepted by DDbECommandLine and the use of the DDbE.jar library. Some additional example input files are contained in Samples/examples. In this directory tree, files whose names end with '.input' are suitable for use with the -s flag described below. Files ending with '.p' contain examples of the attribute and content model parameter flags. These flags are included by the '.input' files.

The directory rooted at Samples/examples contains two other example applications written with the DDbE library.

input files which illustrate the use of DDbE with a variety of parameters.

Using the DDbE library

There are four main steps in the construction of a DTD from a set of examples documents.
1.
Input of documents. The documents may be stored in many ways, for example as files on the local system, as a String, an org.w3c.dom.Document, or in a URI. The FileStringRW class enables DDbE input to be furnished either as names of files on containing xml documents, URIs or as Strings. If examples are available as org.w3c.dom.Document the instances, the user may wish to implement ExampleGeneratorIF directly.
2.
Generation of example instances. This is encapsulated in the ExampleGeneratorIF and ExampleIF interfaces. ExampleIFs are trivial extensions of the org.w3c.dom.Node interface wrapped to simplify use in this application. The DDbE library impelments ExampleGeneratorIF in the ExampleGenerator class whose construction requries an instance of FileStringRW.
3.
Processing of examples. The DDModel implements the DDModelIF interface. DDModel has two constructors, DDModel(java.lang.String[] args) which expects a String[] containing command line arguments and DDModel(Parameter args) which expects an object from the class Parameter. The examples, encapsulated in the ExampleGeneratorIF, are furnished to DDModel using the setExampleGenerator method.
4.
Constructing and emitting the DTD. DDModelIF has two methods for accessing the DTD constructed from the examples:
(a)
DDModelIF.saveDTD method writes the resulting DTD to either a file or to stdout.
(b)
The DTD can be retrieved as a String using the DDModel.getDTD method.

Building a simple application with the library

The DDbE Command line java application is a very simple program that illustrates how the DDbE library can be used. It reads a set of well-formed xml documents and constructs a DTD from them.

To build the DDbE command line application, go to the DDbE/Samples/CommandLine directory in the distribution. This directory contains:

DDbECommandLine can be compiled directly from the command line using one of the commands below. (Of course, any java compiler can be substituted for IBM's jikes compiler.)

In both cases, the classpath will depend on your particular environment.

Now, to generate a DTD from a set of XML source files, simply run java with the DDbECommandLine class as argument. The class file can be run using IBM's jdk or any java VM. The IBM jdk is available for linux, and windows. On linux, you will need a 2.2 kernel to use the IBM JVM since it uses native threads.

In any case, it is run using the command:

 java -classpath < classpath > DDbECommandLine file0 file1 ... fileN ( -w dtdOutputFile )?

Where < classpath > is the same string that was used to compile the java file, { filei } are example the xml files, and dtdOutputFile is the base name of the desired output file. In the distribution, ../examples/personal.xml is a valid example file and DDbECommandLine can be run of this file using:

java -classpath < classpath > DDbECommandLine ../examples/personal.xml -w dtdOutputFile

  
Controlling the output of DDbE

DDbE constructs a data descriptor from a set of example xml documents in a three step process.
1.
For each example document, the xml is read and a dom tree is constructed. The resulting dom element nodes are grouped according to element type.
2.
For each element type, a set of strings, representing all instances of the element type in the examples, is constructed. Finally, a content model, which contains all of these examples, is constructed and associated with the element type.
3.
For each attribute, the validity constraints on attribute types and the actual values which occurred in the example documents are used to determine the types and defaults that the attribute is permitted in a valid document. Each attribute is declared to have one of these permitted types and and defaults.

Associated with each stage of this process, there is a set of parameters which guide program behavior during that stage. DDbE permits control, on a global or element by element basis, of properties which effect the construction of content models and the declaration of attributes.

This section describes how each stage of the process is controlled when using either the DDbE library or the command line interface.

All command line arguments can be constructed and passed to the library as strings; however, for most there are methods which permit all parameters to be set directly.

Command line arguments may repeat. If they do, the final flag, ie. the flag which occurs farthest to the right, sets the final value of the parameter.

  
Parameters controlling I/O

Syntactically, command line I/O arguments fall into four classes:
1.
A ``#'' makes the rest of the line a comment.
2.
The markupdecl flag (-m) which requests that the output matches the markupdecl production production of the xml spec.
3.
A parameter flag, i.e. a ``-'' followed by a single character and a string. Spaces between the single character and string are optional.
4.
Everything else is interpreted as a file name or URL containing a well-formed xml example.
The meaning and syntax of the command line arguments, other than the comment symbol #, is given below.

  
Parameters controlling content model construction

Content models are regular expressions. The following example will illustrate the terminology used to describe the effect of the parameters.

Let {ex[i]| 0 $ \leq$ i < n}, be a set of examples of the content of some element presented as a comma separated sequence of element names and assume that the set can be rewritten

wL,(x[0] + ... + x[p]), wR + (ex[p + 1] + ... + ex[n])

where ' + ' indicates set union, ', ' indicates concatenation and for k = 0..., p : wL, x[k], wR = ex[k]. If p < n, we say that the depth of factorization is 1, that the examples, ex[p + 1] + ... + ex[n], are at level 0, and that the factored examples, x[0] + ... + x[p], are at level 1. If p = n, we say the depth of factorization is 0 and that x[0] + ... + x[p], are at level 0.

Syntax and meaning of content model parameters

Content model parameter strings are introduced by a '-e' flag and obey the following syntax:

elementName? '%' (l|c#|d#|o#)*

If the elementName string is present, the constrain applies to the content model of the named element, otherwise it applies to all content models (Recall that if a flag occurs more than once the rightmost occurrence is used.) The meaning of the individual fields is

  
Examples of effect of command line arguments on content model.

The sample program, DDbEContentModel, permits the user to experiment with the effect of different parameters on the construction of content models.

For example, The file ex0.input contains the three strings a ; a,a,a ; a,b,c. Varying the 'c' parameter yields:

  
Parameters controlling attibute declarations

Attribute model parameter strings permit the user to constrain declarations and defaults of attributes. They are introduced by a '-a' flag and obey the following syntax:

( '!'|elementName)? '%' attributeName? '%' (('!')? ('i' | 'e' | 'c' | (('r' | 'n' | 'x'| 'y')('s')?)))* + ('F' | (('I' | 'R' )attVal))?

Spaces are permitted between the flag,'a', and the parameter string; however, no spaces are allowed in the parameter string itself. The '%' is a field separator. The first two fields identify the attribute. The flags of the final field occur in two groups and indicate how the attribute type and default declarations are constrained.

Identifying the attribute

If elementName appears, an element with the given name and attributes will occur in the resulting data descriptor. If elementName does not appear, the constraints apply to all attributes with the given name, or to all attributes if no attribute name is given. (For example, the elementName and attributeName might be omitted if the user wanted attributes declared as NMTOKEN when possible.) If the parameter string is starts with a '!,' we require that the attribute name appear in the parameter string. In this case, an attribute with the given name will be declared for all elements.

Constraining the attribute type.

The first group of flags in the final field consists of those flags which precede the optional 'F.' These flags constrain type declarations and consist of an optional '!', a single character, and possibly an 's'. Each of the single character refers to a different type of attribute declaration: If the '!' is present, the flag insists that the the attribute not be declared with the given type, otherwise it requests that the attribute be declared with the given type. If the 's' is present, it indicates that the attribute should be declared in the plural form even if all occurrences of the attribute contain only a single item.

We note that there is a natural partial order on attribute types induced by lexical restrictions. In this three level partial order, ID, IDREF, ENTITY, and NOTATION are maximal, CDATA is minimal, and ENUMERATION and NMTOKEN are intermediate. If more than one attribute type is requested and legal, a maximal requested type is assigned if possible. If no requested type is legal ( e.g. if the type requested was IDREF but this was not a valid assignment, perhaps because an occurring value was a Nmtoken but not a Name, or because an occurring value did not occur as the value of an IDREF in the same document) NMTOKEN(S) is assigned if legal. In this case, a message is included in the output dtd.

Constraining the attribute default.

The final group of flags mirrors the default declarations of the xml specification, i.e. using a F flag will cause the attribute type to be declared as FIXED if possible while a R flag will cause the attribute type to be declared as REQUIRED if possible.

Constraining attributes from the library

From the library, attributes are constrained by constructing the appropriate constraint String, using the constructor AttributeConstraint(String), and using AttributeModelIF.setAttributeConstraint(AttributeConstraint).

Examples of effect of command line arguments on attribute declarations.

In Samples/CommandLine, the file par.p requests that all attributes be declared as IDREF, the file req.p that all attributes be declared as REQUIRED.

Changes

Changes from the pre-release version

The translation was initiated by on 1999-08-11


next up previous

1999-08-11