The Text Encoding Initiative's Guidelines for electronic text encoding and interchange ([2]) provide exhaustive recommendations for the encoding of key features in literary and linguistic textual materials. These recommendations, instantiated by a modular XML-based architecture in which DTD fragments and documentation are combined according to user-specified requirements, are very effective and are widely adopted in digital library, language engineering, and many other projects (http://www.tei-c.org/Applications/).

One of the projects (http://www.tei-c.org/Activities/META/) of the TEI's Technical Council is to rewrite the Guidelines so that underlying metalanguage is independent of SGML or XML DTD language, allowing for automatic generation of schemas, DTDs, or any future constraint languages. The first stage of this work resulted in a set of RelaxNG (http://wwww.relaxng.org/) grammar files automatically generated from the Guidelines (available from http://www.tei-c.org/Schemas/RelaxNG/P4X/). This work was described at XML Europe 2002 ([1]), so we will only provide a summary description here, but we will have to recap some of the explanation.

Manipulation of the TEI is possible because the TEI is not maintained as DTD files, but in a literate programming (cf [4]) system which documents and describes elements in a largely abstract manner, and describes their interdependence using an independently-documented set of element classes. This is probably best demonstrated by an example. The <persName> element is specified by the following markup:

<tagDoc id="PERSNAME" usage="opt"> 
 <gi>persName</gi>
 <name>personal name</name> 
 <desc>contains a proper noun or proper-noun
phrase referring to a person, possibly including any or all of the
person's forenames, surnames, honorifics, added names, etc.</desc>
  <attList>
    <attDef usage="mwa">
      <attName>type</attName>
      <desc>describes the personal name more fully using an open-ended
list  of words or phrases which help to indicate the function, e.g.
<q>married name</q>, <q>maiden name</q>,
<q>pen name</q>, <q>religious name</q>, etc.</desc>
      <datatype>CDATA</datatype>
      <valDesc>Any string of characters.</valDesc>
      <default>#IMPLIED</default>
    </attDef>
  </attList>
  <exemplum>...</exemplum>
  <remarks/>
  <part type="top" name="ND"/>
  <classes names="DEMOG NAMES DATA"/>
  <elemDecl> %om.RR; ( #PCDATA | %m.personPart;
                   | %m.phrase; | %m.Incl; )* </elemDecl>
  <ptr target="NDPER"/>
</tagDoc>

The key features here are

This information allows a processor to construct a DTD fragment for the element as follows:

!ELEMENT persName ( #PCDATA | %m.personPart;
                   | %m.phrase; | %m.Incl; )* > 
<!ATTLIST %n.persName;
      %a.global;
      %a.names;
      type CDATA #IMPLIED
      TEIform CDATA 'persName'  >

Note here the addition of more attributes, from the classes of which this element is a member.

The problem with the system described above is the dependence on explicit DTD content models, which are not amenable to processing using standard XML tools. We therefore replace the <elemDecl> with the following:

<elemDecl>
  <rng:zeroOrMore xmlns:rng="http://relaxng.org/ns/structure/1.0">
    <rng:choice>
      <rng:text/>
      <rng:ref name="m.personPart"/>
      <rng:ref name="m.phrase"/>
      <rng:ref name="m.Incl"/>
    </rng:choice>
  </rng:zeroOrMore>
</elemDecl>

This is much easier to analyze, and is (reasonably!) easy to turn back into DTD markup if needed. A processor can now assemble all the information needed to construct a complete RelaxNG grammar.

The translation of the TEI Guidelines to use RelaxNG markup to encode content models is fairly stable, and the challenge now is to find ways of making use of the extra power provided by schemas.

In addition to the class system for maintaining relationships between elements, the TEI also works on the basis of a set of mutually exclusive^[1] basic tag sets. The choice is between:

A normal TEI document will start with one of these scenarios, and then add modules from the following list:

It is important to understand that a user must make sort of choice—there is no one TEI DTD or schema which is the default. In addition, the TEI has a clear system for extending the tagset, which again utilises the class system by allowing new elements to be added to classes, and to refer to existing classes.^[2]

How does a casual user make sense of this complexity? It requires a good understanding of DTD or Schema languages to manipulate the right parameter entities or pattern definitions, so the TEI offers an interface for building customized views of the system. In the DTD-only release of the TEI, this is done using a web form and a utility called carthago; the job of this program was to ‘compile’ DTDs, expanding all parameter entity references and removing references elements which were not available.^[3]

The web application is known as the TEI Pizza Chef, because it allows the customer to choose what toppings they want for a particular base. However, it has to leave most of the work to the user, by creating a pair of skeleton DTD extension files which the user downloads, edits, and uploads again. Editing these DTD files by hand is error-prone, fairly forbidding, and cannot be used to modify schemas. A revised system has therefore been built which attempts to keep all the knowledge or DTD or schema in the application itself, and simply ask the user to select options on web forms. This is fancifully known as the TEI Sushi Bar, following the model of an endless choice of clean, distinct, options continually being presented to the user, rather than a rather oily mound of congealing cheese and tomato. More precisely, the Sushi Bar is a web application running scripts known generically as roma.

Roma starts by asking the user to choose which base tagsets and extra modules they require. There are two interfaces, one verbose (Figure Figure 1) and one for the expert (Figure Figure 2). There are also two important choices to make:

The choices here affect the next stage. Firstly, if a DTD is requested, the user is allowed to choose some ISO entity sets to include (Figure Figure 3). Secondly, if element configuration is requested, all the elements in the chosen tagsets are listed, with radio buttons which allow the element to be included in the result, or excluded (Figure Figure 4). The links in this table are to the documentation of each element on the TEI web site. At this stage, the user can rename elements; the example shown in Figure Figure 5 has <figure> being renamed to <graphic>, and <figDesc> to <caption>, while <table>, <row>, <cell>, and <formula> are declared as unwanted. We will see shortly how this is implemented.

In the second stage of Roma, there is also a set of general options which can be turned on and off:

After all these choices are made, the Submit button prompts the user to download the resulting DTD or schema.

The look of the result depends on whether or not a compiled form has been selected. Given a simple set of choices, a RelaxNG grammar could result as follows:

<grammar 
  xmlns="http://relaxng.org/ns/structure/1.0"
  xmlns:a="http://relaxng.org/ns/compatibility/annotations/1.0" 
  datatypeLibrary="http://www.w3.org/2001/XMLSchema-datatypes">
<include href="http://www.tei-c.org/Schemas/RelaxNG/P4X/tei2.dtd.rng">

 <define name="TEI.prose"><ref name="INCLUDE"/></define>
 <define name="TEI.figures"><ref name="INCLUDE"/></define>

 <define name="formula"><notAllowed/></define>
 <define name="table"><notAllowed/></define>
 <define name="figDesc">
  <element name="caption">
   <ref name="c.figDesc"/>
  </element>
 </define>
 <define name="row"><notAllowed/></define>
 <define name="figure">
  <element name="graphic">
   <ref name="c.figure"/>
  </element>
 </define>
 <define name="cell"><notAllowed/></define>

<!-- overrides to make ISOdate a formal datatype -->
  <define name="ISO-date">
      <data type="date" 
          datatypeLibrary="http://www.w3.org/2001/XMLSchema-datatypes"/>
 </define>
</include>
</grammar>

There are some important points to note here:

The last point deserves some more explanation. The original definition for <figure> is like this:

<define name="figure">
  <element name="figure">
   <ref name="c.figure"/>
  </element>
 </define>

that is, a pattern is defined called figure, which defines an element called <figure>, with a content model given in the pattern c.figure. By redefining figure as follows:

<define name="figure">
  <element name="graphic">
   <ref name="c.figure"/>
  </element>
 </define>

we define an element called <graphic>, which has the same content model as the old <figure>, and is inside athe pattern called figure. This is what other definitions will refer to; so anything which wants to include the ‘figure’ element will say <ref name="figure"/>, and it will not matter that the actual element is renamed. The original name of the element is preserved by an attribute called TEIform, defined as <attribute name="TEIform" a:defaultValue="figure"> <text/> </attribute>, so it is easy to relate this changed setup to the basic TEI. The renaming feature may be extended in future to allow complete translations of the TEI element names to predefined language sets, allowing the user to simply request "all elements in Spanish, please".

If a compiled output is requested, then the skeleton DTD or Schema will be put through a flattening process to remove redundant elements and references to external files. This has the advantage that a single file is produced, which considerably aids portability, and the removal of unused elements can make it much smaller.

DTD flattening is performed by the existing carthago application, and schema flattening is performed by an XSLT transform of a RelaxNG grammar. The other outputs (compact RelaxNG and W3C Schema) are done by calls to James Clark's trang program (http://www.thaiopensource.com/trang/).

MathML and SVG inclusion are managed by simplying <include>ing the relevant RelaxNG grammars, each in their own namespace.

We have so far seen examples of simply choosing subsets of the TEI, or adding standard new features. What if we want to add some elements? This may be for one of two reasons:

If the user chooses to add elements, they are asked to decide which of these two situations they want to address, and to give the element a name and description. In Figure Figure 6 we show the addition of the <homeurl> element, in the addrPart class. Of course, this assumes some familiarity with the TEI class system, (see section Section 4 for a summary of the TEI classes) and the interface is not yet friendly enough for someone completely new to the TEI. The list of elements and classes are derived, of course, dynamically from the TEI Guidelines.

There are three further facilities which Roma does not yet provide:

Here is a list of the currently defined classes of the TEI system:

The increasing power provided by schemas, and the stress on modularity, argue in favour of moving towards (conceptual) two stage validation. In the first phase, the important check is that the document uses the right vocabulary, in our case meaning the 441 elements currently described by the TEI. The structure here can be quite loose. In the second phase, which can depend on individual projects, validation can be a lot more precise, with detailed datatyping and inter-dependency validation. For example, the basic rule may say that an <text> must have a <author>, <title> and <date>, but be agnostic about their order. A particular project may wish to enforce a rule that they must occur in a fixed order; or it may wish to more limited than the base schema, and say that <date> is not permitted at all. Thus a typical document may be checked once to ensure that it uses TEI vocabulary and broad grammatical structure, and then checked again to make sure it talks the right dialect.

The relevance of this work is that it shows a way forward for XML users which does not involve low-level interaction with DTDs or Schemas. Unlike the graphic direct manipulation tools in eg XML Spy, the Roma tool works at the level of the TEI class system. Together with the support for other namespaces via schemas, these tools take the TEI one step further on the road to a universal markup language.