XML Schema languages

Which one?

1. What is a XML schema language?

In plain "common" English, a schema is: An outline or image universally applicable to a general conception, under which it is likely to be presented to the mind; as, five dots in a line are a schema of the number five; a preceding and succeeding event are a schema of cause and effect (Websters dictionary).

Looking at the so-called XML Schema languages, which I will present today, it seems obvious that a XML Schema language isn't a language which describes a schema in XML: most of the schema languages are most complex to "present to the mind" or even to a program than the instance documents which it describes and, more important and less subjective, they often focus on defining validation rules more than on modeling a class of documents.

The common point between these languages is that they are in fact defining (or schematizing) transformations to apply on a class of instance documents and XML schemas should be considered as transformations!

These transformations take instance documents as input and produce a mandatory "validation report" which includes at least a return code saying if the document is valid or not and an optional "Post Schema Validation Infoset" ( Post Schema Validation Infoset (PSVI)) updating the original XML Infoset with additional information (default values, datatypes, ...).

2. XML Schemas considered harmful?

Before we dive into the features of these schema languages, I'd like us to step back and look at the downsides of the usage of any schema language.

One of the key strengths of XML (sometimes called late "binding") is the decoupling between the writer and the reader of a XML document which gives to the reader the ability to have its own interpretation and understanding of the document.

By being more directive on the way to interpret the document XML Schemas (and their "adjunct frameworks") can not only reduce the possibility of erroneous interpretations, but also the ability to add value to the documents by unexpected interpretations.

Furthermore, modeling a XML tree is very complex and the schema languages often do some expectations about "good" and "bad" practices to limit their complexity, validation processing times but also the set of possibilities offered to the XML designers.

Reducing the set of possibilities offered by a technology still relatively young is a risk since these "good" or "bad" practices are still blur and rapidly evolving.

The many advantages of using and widely distributed XML schemas should be balanced with the risk of narrowing the flexibility and eXtensibility of XML which is however the meaning of the first letter of its acronym.

3. What kind of validation?

We can differentiate at least three levels of validation:

• The validation of the markup is controlling the structure of a document.

• The validation of the content of individual leaf nodes (datatyping)

• The validation of the integrity, i.e. of the links between nodes within a document or between documents.

• Any other tests (often called "business rules").

The validation of the markup and the datatypes are the most "dangerous" since they often imply a modeling limiting the diversity of the markup and datatypes.

Link validation, especially between document is poorly covered by the current schema languages.

4. A short history of XML Schema languages

The list of schema languages is long and would need to include languages developed for Simple Generalized Markup Language (SGML), before XML was born to be complete and the list which I propose below is far from being exhaustive and include only the major proposals which have influenced the schema languages covered in this presentation.

4.1. The DTD family

Mandatory for any SGML application, a simplified version of the SGML DTDs has been introduced in the XML 1.0 recommendation [XML]. Even though a DTD is not mandatory for an application to read and understand a XML document, many developers highly recommend writing DTDs for your XML applications.

4.2. The W3C XML Schema family

The World Wide Web Consortium (W3C) XML Schema Working Group has received many proposals contributed as notes:

1. XML-Data, submitted as a note [XML-Data] in January 98 by people from Microsoft, DataChannel, Arbortext, Inso Corporation and University of Edinburgh included most of the basic concepts developed by W3C XML Schema. Although the details were not fully developed, the note was covering a lot of grounds which have been kept out of W3C XML Schema, such as internal and external entity definitions and the mapping with Resource Description Framework (RDF) and object oriented structures.

2. XML-Data-Reduced (XML-Data-Reduced (XDR)), submitted in July 98 [XDR] by editors from Microsoft and University of Edinburgh was presented to "refine and subset those ideas down to a more manageable size in order to allow faster progress toward adopting a new schema language for XML" (mappings were one of the concepts that were left out). XDR has been implemented by Microsoft and used by the Biztalk framework.

3. Document Content Description (DCD) (Document Content Description for XML), also submitted in July 98 [DCD] by editors from Textuality, Microsoft and IBM was a "subset of the XML-Data Submission [XML-Data] and expresses it in a way which is consistent with the ongoing W3C RDF (Resource Description Framework) [RDF] effort". Mapping considerations were left out, but the language was taking care to be consistent with RDF through features such as "Interchangeability of Elements and Attributes".

4. Schema for Object-Oriented XML (SOX) (Schema for Object-Oriented XML) was developed by Veo Systems/ Commerce One and submitted as a note in September 98 (a second version has been submitted in July 99 [SOX]) as "informed by the XML 1.0 [XML] specification as well as the XML-Data submission [XML-Data], the Document Content Description submission [DCD] and the EXPRESS language reference manual [ISO-10303-11]". SOX was very influenced by object oriented design and included concepts of interface and implementation but also by the DTDs and also included a support for "parameters". SOX has been widely used by Commerce One.

5. Document Definition Markup Language (DDML) (or XSchema) was the "result of contributions from a large number of people on the XML-Dev mailing list [XML-DEV], coordinated by a smaller group of editors" (Ronald Bourret , John Cowan, Ingo Macherius and Simon St. Laurent) submitted as a note in January 99 [DDML]and its purpose was to "encode the logical (as opposed to physical) content of DTDs in an XML document". Great attention had been paid to the definition of the back and forward conversions back between DTDs and DDML and the document also included an "experimental" chapter proposing "Inline DDML elements". DDML made a clear distinction between structures and data and decided to left datatypes apart.

6. W3C XML Schema, published as a Recommendation in May 2001 [XMLS0] [XMLS1] [XMLS2] acknowledges the influence of DCD, DDML, SOX, XML-Data and XDR in its list of references and appears to have picked pieces from each of these proposals but also to be a compromise between them. The main sponsors of the two languages still actively used and developed (Microsoft for XDR and Commerce One for SOX) have both announced that they would support W3C XML Schema for their new developments and W3C XML Schema should become the only living member of this family.

4.3. The RELAX NG family

1. First published in March 2000 as a Japanese ISO Standard (JIS) (Japanese ISO Standard) Technical Report (TR) written by MURATA Makoto, REgular LAnguage description for XML (RELAX) Core (REgular LAnguage description for XML) [RLX] is both simple ("Tired of complicated specifications? You just RELAX ! ") and built on a solid mathematical ground (the adaptation of the hedge automaton theory by MURATA Mako to XML trees). It has been approved as an ISO/IEC Technical Report in May 2001.

2. transduce (XDuce) [XDUCE]has been first announced in March 2000: "XDuce ('transduce') is a typed programming language that is specifically designed for processing XML data. One can read an XML document as an XDuce value, extract information from it or convert it to another format, and write out the result value as an XML document". Although its purpose is not to be a schema language, its typing system has influence the schema languages.

3. Published by James Clark in January 2001, Tree Regular Expressions for XML (TREX) (Tree Regular Expressions for XML) [TREX] is "basically the type system of XDuce with an XML syntax and with a bunch of additional features". The name and content model of the elements used to define the tree patterns of a TREX schema have been carefully chosen and TREX schemas are usually as easy to read as a plain text description. The simplicity of the structure of the language also allows to resurrect a consistent treatment between elements and attributes reminding a feature lost since DCD.

4. Announced in May 2001, RELAX New Generation (RELAX NG) (RELAX New Generation) is the result of the merge between RELAX and TREX developed by an Organization for the Advancement of Structured Information Standards (OASIS) Technical Comitee (TC) [RNG] and coedited by James Clark and MURATA Makoto: "The key features of RELAX NG are that it is simple, easy to learn, uses XML syntax, does not change the information set of an XML document, supports XML namespaces, treats attributes uniformly with elements so far as possible, has unrestricted support for unordered content, has unrestricted support for mixed content, has a solid theoretical basis, and can partner with a separate datatyping language (such W3C XML Schema Datatypes)". RELAX NG is expected to become an OASIS specification and probably an ISO/IEC TR.

4.4. Schematron

Nontypical, Schematron [SCH] has been first proposed in September 1999 by Rick Jelliffe of the Academia Sinica Computing Centre and defines validation rules using XPath expressions.

4.5. Examplotron

Starting from the observation that instance documents are usually much easier to understand than the schemas which are describing them and that the schema languages often need to give examples of instance documents to help human readers to understand their syntax, Examplotron [EG] has been proposed in March 2001 by the author of this presentation to define "schemas by example" using sample instance documents as actual schemas.

5. Our sample application

All over the presentation, I will be using the following (simple) sample application:

<?xml version="1.0" encoding="utf-8"?>
<library>
  <book id="_0836217462">
    <isbn>0836217462</isbn>
    <title>Being a Dog Is a Full-Time Job</title>
    <author-ref id="Charles-M.-Schulz"/>
    <character-ref id="Peppermint-Patty"/>
    <character-ref id="Snoopy"/>
    <character-ref id="Schroeder"/>
    <character-ref id="Lucy"/>
  </book>
  <book id="_0805033106">
    <isbn>0805033106</isbn>
    <title>Peanuts Every Sunday </title>
    <author-ref id="Charles-M.-Schulz"/>
    <character-ref id="Sally-Brown"/>
    <character-ref id="Snoopy"/>
    <character-ref id="Linus"/>
    <character-ref id="Snoopy"/>
  </book>
  <author id="Charles-M.-Schulz">
    <name>Charles M. Schulz</name>
    <nickName>SPARKY</nickName>
    <born>November 26, 1922</born>
    <dead>February 12, 2000</dead>
  </author>
  <character id="Peppermint-Patty">
    <name>Peppermint Patty</name>
    <since>Aug. 22, 1966</since>
    <qualification>bold, brash and tomboyish</qualification>
  </character>
  <character id="Snoopy">
    <name>Snoopy</name>
    <since>October 4, 1950</since>
    <qualification>extroverted beagle</qualification>
  </character>
  <character id="Schroeder">
    <name>Schroeder</name>
    <since>May 30, 1951</since>
    <qualification>
      brought classical music to the Peanuts strip
    </qualification>
  </character>
  <character id="Lucy">
    <name>Lucy</name>
    <since>March 3, 1952</since>
    <qualification>bossy, crabby and selfish</qualification>
  </character>
  <character id="Sally-Brown">
    <name>Sally Brown</name>
    <since>Aug, 22, 1960</since>
    <qualification>always looks for the easy way out</qualification>
  </character>
  <character id="Linus">
    <name>Linus</name>
    <since>Sept. 19, 1952</since>
    <qualification>the intellectual of the gang</qualification>
  </character>
</library>

6. DTDs

Facts sheet:

Author:	W3C
Status:	Recommendation ("embedded" in XML 1.0)
Location:	http://www.w3.org/TR/REC-xml
PSVI:	Yes
Structures:	Yes
Datatypes:	Yes (weak)
Integrity:	Yes (internal through ID/IDREF/IDREFS attributes)
Rules:	No
Vendor support:	Excellent
Miscellaneous:	Non XML Syntax; no support for namespaces. Schema definition is only one of the features of DTDs.

Inherited from SGML, the XML DTDs are the "natural" and most widely mean of defining XML schemas. Defined in the XML 1.0 recommendation, they do not support namespaces which have been specified after them and this is, with the fact that their datatype system is weak and does only apply to attribute, one of the main motivation of the W3C to develop a new schema language.

The DTD for our sample could be:

<?xml version="1.0" encoding="UTF-8"?>
<!ELEMENT author (name, nickName, born, dead)>
<!ATTLIST author
  id ID #REQUIRED
>
<!ELEMENT author-ref EMPTY>
<!ATTLIST author-ref
  id IDREF #REQUIRED
>
<!ELEMENT book (isbn, title, author-ref*, character-ref*)>
<!ATTLIST book
  id ID #REQUIRED
>
<!ELEMENT born (#PCDATA)>
<!ELEMENT character (name, since, qualification)>
<!ATTLIST character
  id ID #REQUIRED
>
<!ELEMENT character-ref EMPTY>
<!ATTLIST character-ref
  id IDREF #REQUIRED
>
<!ELEMENT dead (#PCDATA)>
<!ELEMENT isbn (#PCDATA)>
<!ELEMENT library (book+, author*, character*)>
<!ELEMENT name (#PCDATA)>
<!ELEMENT nickName (#PCDATA)>
<!ELEMENT qualification (#PCDATA)>
<!ELEMENT since (#PCDATA)>
<!ELEMENT title (#PCDATA)>

7. W3C XML Schema

Facts sheet:

Author:	W3C
Status:	Recommendation
Location:	http://www.w3.org/TR/xmlschema-0/
PSVI:	Yes
Structures:	Yes
Datatypes:	Yes
Integrity:	Yes (internal through ID/IDREF/IDREFS and xs:unique/xs:key/xs:keyref)
Rules:	No
Vendor support:	Potentially excellent but still often unmatured.
Miscellaneous:	Borrows many ideas from object oriented design; considered complex; paranoiac about determinism; part of the foundation brick of XML in the vision of the W3C.

W3C XML Schema has been published by the W3C to provide an alternative to the schema feature of the DTDs which supports the namespaces and facilitate the design of open and extensible vocabularies and meet the requirement of "data oriented" applications for a richer datatyping system.

It does so borrowing many features from object oriented languages and the fit with the tree structure of XML documents is sometimes difficult to make. It is generally considered as complex, partly because of the number of its features and partly because of the style of the recommendation which describes the validation process more than the modeling features.

W3C XML Schema is a strongly typed schema language which has chosen to eliminate any non-deterministic design from the described markup to insure that there is no ambiguity in the determination of the datatypes and that the validation van be made by a finite state machine.

A W3C XML Schema schema for our sample could be:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="library">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="book" maxOccurs="unbounded">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="isbn" type="xs:string"/>
              <xs:element name="title" type="xs:string"/>
              <xs:element name="author-ref" minOccurs="0" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:attribute name="id" type="xs:IDREF" use="required"/>
                </xs:complexType>
              </xs:element>
              <xs:element name="character-ref" minOccurs="0" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:attribute name="id" type="xs:IDREF" use="required"/>
                </xs:complexType>
              </xs:element>
            </xs:sequence>
            <xs:attribute name="id" type="xs:ID" use="required"/>
          </xs:complexType>
        </xs:element>
        <xs:element name="author" minOccurs="0" maxOccurs="unbounded">
          <xs:complexType>
            <xs:sequence>
              <xs:element ref="name"/>
              <xs:element name="nickName" type="xs:string"/>
              <xs:element name="born" type="xs:string"/>
              <xs:element name="dead" type="xs:string"/>
            </xs:sequence>
            <xs:attribute name="id" type="xs:ID" use="required"/>
          </xs:complexType>
        </xs:element>
        <xs:element name="character" minOccurs="0" maxOccurs="unbounded">
          <xs:complexType>
            <xs:sequence>
              <xs:element ref="name"/>
              <xs:element name="since" type="xs:string"/>
              <xs:element name="qualification" type="xs:string"/>
            </xs:sequence>
            <xs:attribute name="id" type="xs:ID" use="required"/>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
  <xs:element name="name" type="xs:string"/>
</xs:schema>

8. RELAX NG

Facts sheet:

Author:	OASIS and possibly ISO
Status:	Working Draft
Location:	http://relaxng.org/
PSVI:	No
Structures:	Yes
Datatypes:	No, but a modular mechanism has been defined to plug datatype systems (W3C XML Schema part2 and others if needed).
Integrity:	No (except through ID/IDREF/IDREFS features of a datatype system)
Rules:	No
Vendor support:	To be seen.
Miscellaneous:	Result of the merge between RELAX and TREX, might become an ISO TR. Strong mathematical ground. Alternate non XML syntax proposed by James Clark.

Its editors (James Clark and MURATA Makoto) define RELAX NG as "the next generation schema language for XML: clean, simple and powerful". RELAX NG appears to be closer to a description of the instance documents in "plain English" and simpler than W3C XML Schema for which it might become a serious alternative.

Many constraints, especially those which are on the fringe to non-deterministic models, can be expressed by RELAX NG and not by W3C XML Schema and some combinations in document structures which are forbidden by W3C XML Schema can be described by RELAX NG.

Even though RELAX NG seems to be technically superior to W3C XML Schema, the support by the software vendors and XML developers is uncertain now that W3C XML Schema is a Recommendation.

A RELAX NG schema for our sample could be:

<?xml version="1.0" encoding="UTF-8"?>
<grammar 
   datatypeLibrary="http://www.w3.org/2001/XMLSchema-datatypes" 
   xmlns="http://relaxng.org/ns/structure/0.9">
  <start>
    <choice>
      <ref name="library"/>
    </choice>
  </start>
  <define name="library">
    <element name="library">
      <oneOrMore>
        <ref name="book"/>
      </oneOrMore>
      <zeroOrMore>
        <ref name="author"/>
      </zeroOrMore>
      <zeroOrMore>
        <ref name="character"/>
      </zeroOrMore>
    </element>
  </define>
  <define name="author">
    <element name="author">
      <attribute name="id">
        <data type="ID"/>
      </attribute>
      <element name="name">
        <text/>
      </element>
      <element name="nickName">
        <text/>
      </element>
      <element name="born">
        <text/>
      </element>
      <element name="dead">
        <text/>
      </element>
    </element>
  </define>
  <define name="book">
    <element name="book">
      <ref name="id-attribute"/>
      <ref name="isbn"/>
      <ref name="title"/>
      <zeroOrMore>
        <element name="author-ref">
          <attribute name="id">
            <data type="IDREF"/>
          </attribute>
          <empty/>
        </element>
      </zeroOrMore>
      <zeroOrMore>
        <element name="character-ref">
          <attribute name="id">
            <data type="IDREF"/>
          </attribute>
          <empty/>
        </element>
      </zeroOrMore>
    </element>
  </define>
  <define name="id-attribute" >
    <attribute name="id">
      <data type="ID"/>
    </attribute>
  </define>
  <define name="character">
    <element name="character">
      <ref name="id-attribute"/>
      <ref name="name"/>
      <ref name="since"/>
      <ref name="qualification"/>
    </element>
  </define>
  <define name="isbn">
    <element name="isbn">
      <text/>
    </element>
  </define>
  <define name="name">
    <element name="name">
      <text/>
    </element>
  </define>
  <define name="nickName">
    <element name="nickName">
      <text/>
    </element>
  </define>
  <define name="qualification">
    <element name="qualification">
      <text/>
    </element>
  </define>
  <define name="since">
    <element name="since">
      <data type="date"/>
    </element>
  </define>
  <define name="title">
    <element name="title">
      <text/>
    </element>
  </define>
</grammar>

9. Schematron

Facts sheet:

Author:	Rick Jelliffe and other contributors.
Status:	Unofficial
Location:	http://www.ascc.net/xml/schematron/
PSVI:	No (not directly)
Structures:	No (not directly)
Datatypes:	No (not directly)
Integrity:	No (not directly)
Rules:	Yes, through XPath expressions
Vendor support:	Low
Miscellaneous:	Pure rule expression.

Schematron is a XPath/XSLT based language to define context dependent rules. Schematron doesn't directly support structure nor datatype validation, but a schema author may write rules which implement structure and datatype checks. To write a full schema with Schematron, the author needs to take care to include all the rules needed to qualify the structure of the document.

A partial Schematron schema for our sample could be:

<?xml version="1.0" encoding="UTF-8"?>
<sch:schema xmlns:sch="http://www.ascc.net/xml/schematron">
  <sch:title>Schematron Schema for library</sch:title>
  <sch:pattern>
    <sch:rule context="/">
      <sch:assert test="library">
	The document element should be "library".
	</sch:assert>
    </sch:rule>
    <sch:rule context="/library">
      <sch:assert test="book">
	There should be at least a book!
	</sch:assert>
      <sch:assert test="not(@*)">
	No attribute for library, please!
	</sch:assert>
    </sch:rule>
    <sch:rule context="/library/book">
      <sch:assert test="not(following-sibling::book/@id=@id)">
	Duplicated ID for this book.
	</sch:assert>
      <sch:assert test="@id=concat('_', isbn)">
	The id should be derived from the ISBN.
	</sch:assert>
    </sch:rule>
    <sch:rule context="/library/*">
      <sch:assert test="name()='book' or name()='author' or name()='character'">
	This element shouldn't be here...
	</sch:assert>
    </sch:rule>
  </sch:pattern>
</sch:schema>

10. Examplotron

Facts sheet:

Author:	Eric van der Vlist
Status:	Unofficial
Location:	http://examplotron.org/
PSVI:	No(t yet)
Structures:	Yes
Datatypes:	No (not directly)
Integrity:	No (not directly)
Rules:	Yes, through XPath expressions
Vendor support:	Null
Miscellaneous:	Schema by example (a sample document is a schema) with rules checking (syntax borrowed to Schematron).

Examplotron is an experiment to define a schema language based on sample trees which is not unlike early proposals for XPath.

An Examplotron schema for our sample could be:

<?xml version="1.0" encoding="UTF-8"?>
<library xmlns:eg="http://examplotron.org/0/">
  <book id="_0836217462" 
     eg:occurs="+" 
     eg:assert="not(following-sibling::book/@id=@id) and @id=concat('_', isbn)">
    <isbn>0836217462</isbn>
    <title>Being a Dog Is a Full-Time Job</title>
    <author-ref id="Charles-M.-Schulz"  eg:occurs="*"/>
    <character-ref id="Peppermint-Patty"  eg:occurs="*"/>
  </book>
  <author id="Charles-M.-Schulz" eg:occurs="*">
    <name>Charles M. Schulz</name>
    <nickName>SPARKY</nickName>
    <born>November 26, 1922</born>
    <dead>February 12, 2000</dead>
  </author>
  <character id="Peppermint-Patty" eg:occurs="*">
    <name>Peppermint Patty</name>
    <since>Aug. 22, 1966</since>
    <qualification>bold, brash and tomboyish</qualification>
  </character>
</library>

11. Embedded languages

We have seen that the features of some of those languages are more complementary than overlapping and there is room for interesting combinations, especially with Schematron and the structure and datatype based languages.

Some early implementations are available which support the embedding of Schematron rules in xs:annotation/xs:appinfo W3C XML Schema elements. The combination of W3C XML Schema and Schematron is enabling to use each language for it has been designed: structure and datatype validation for W3C XML Schema and rules for Schematron. The power of the rules expressed with Schematron can also compensate some of the weaknesses of W3C XML Schema.

Discussions are also under way to embed Schematron rules in RELAX NG schemas. This would then lead to a combination between RELAX NG for the structure, W3C XML Schema part 2 for the datatypes and Schematron for the rules which would demonstrate the extensibility of XML applications!

12. Comparisons

To wrapup, we can try to summarize the pros and cons of each language in a set of classifications:

Tool support (as of today):

1. Best: DTD

2. Most promising: W3C XML Schema

3. Challenger: RELAX NG

4. Niche: Schematron and Examplotron

Features

1. Structures: DTD, W3C XML Schema, RELAX NG, Examplotron.

2. Datatype: W3C XML Schema

3. Integrity: W3C XML Schema, Schematron, Examplotron

4. Rules: Schematron, Examplotron

Flexibility (i.e. ability to describe a wide range of structures):

1. Most flexible: Schematron (but everything needs to be defined "by hand").

2. Most flexible structure based language: RELAX NG

3. Integrity: W3C XML Schema, Schematron, Examplotron

4. Challenger: Examplotron

5. Behind: W3C XML Schema

6. Less flexible: DTD (lack of namespace support)

13. So what?

There is (currently), no perfect XML Schema languages... fortunately, there are a number of good choices, each of them with its strengths and weaknesses and these choices can be combined. Just pick the right combination for your application!

Design and Development by deepX Ltd.