Streaming Transformations for XML (STX) is a transformation language designed to work on a stream of SAX events. It was born from the insight that XSL Transformations (XSLT) may not be the best choice for some kinds of transformation tasks. XSLT is relatively easy to learn and doesn't require programming skills for simple transformations. So, XSLT can be seen as a scripting language in XML for XML. Most if not all of XSLT's popularity stems from these advantages.

Since XSLT is based on XPath, it operates on a tree representation of an XML document. An XSLT-based transformation requires parsing the whole document, building an in-memory tree representation, performing the transformation process, building a result tree, and serializing this tree as XML. The first and the last step may be left out, depending on your application and the data it requires, but in the end the tree transformation is the heart of XSLT. That's why memory requirements proportional to the size of the XML data belongs to XSLT like angle brackets to XML.

So, how to process large XML files? If you're able to write a program in your favorite programming language then you can choose Simple API for XML (SAX). SAX simply delivers events to the application and doesn't store any data by itself. That's the job of the application that uses SAX. The drawback of this approach is the effort you have to spend for storing relevant data yourself in custom data structures and keeping track of the XML element hierarchy. An alternative on a programming level offers W3C Document Object Model (DOM). DOM provides a tree structure to the application where all data from the XML document are stored in correspondent nodes. Of course DOM is not the only Application Programming Interface (API) for that purpose, but it is a typical representative.

So, on the programming level you have the choice between tree oriented (DOM) and event oriented (SAX) processing of XML data. For a script-like transforming language there is (well, has been) only XSLT, which is tree oriented. STX fills the gap by providing an event based transformation language on the top of SAX. As we will see in an instant, STX reuses many of XSLT's concepts and language constructs.

The STX initiative began in February 2002 with a post from Petr Cimprich to the xml-dev mailing list. Shortly after a special mailing list and a project on sourceforge.net have been established, hosting the current specifications, see [STX]. At the moment there are two open source implementations for STX available: the Perl-based processor XML::STX (http://stx.gingerall.cz/stx/xml-stx/) by Petr Cimprich and the Java-based processor Joost (http://joost.sourceforge.net/) by Oliver Becker, the author of this article. Everybody who is interested is invited to join the discussion and to bring the STX development forward. An article introducing the basic concepts of STX has been published by xml.com, see [Intro].

The transformation of the Resource Description Framework (RDF) dump of the Open Directory (see http://rdf.dmoz.org/) seems to be a perfect task for STX.^[1] The content dump is about 1 GByte large and looks like this:

<?xml version='1.0' encoding='UTF-8'?>
<RDF xmlns:r="http://www.w3.org/TR/RDF/"
     xmlns:d="http://purl.org/dc/elements/1.0/"
     xmlns="http://dmoz.org/rdf">

<Topic r:id="Top">
  <catid>1</catid>
</Topic>
...
<Topic r:id="Top/Shopping">
  <catid>13<catid>
  <link r:resource="http://www.esmarts.com/"/>
  <link r:resource="http://www.bdscodak.com"/>
  <link r:resource="http://www.choicemall.com/"/>
</Topic>

<ExternalPage about="http://www.esmarts.com/">
  <d:Title>eSmarts</d:Title>
  <d:Description>
    eSmarts helps consumers find the lowest possible prices on the
    web.  They compare prices at different Internet stores, list
    coupons (including many $10 off coupons), discuss sales and
    share great shopping tips.
  </d:Description>
</ExternalPage>

<ExternalPage about="http://www.bdscodak.com">
  <d:Title>
    BD Scodak - personalized children's books for your child's education
  </d:Title>
  <d:Description>
    BD Scodak is your source for personalized children's books
    customized with your child's information right next to popular
    cartoon, religious, sports, and tv characters and themes.
  </d:Description>
</ExternalPage>

<ExternalPage about="http://www.choicemall.com/">
  <d:Title>Choice World</d:Title>
  <d:Description>
    Choice Mall - The #1 global marketplace on the Internet.
    Thousands of stores offer quality, unique products and services,
    art and entertainment, books and music, gifts, food, real estate,
    health, sports, and fitness -- all under one roof!
  </d:Description>
</ExternalPage>

<Topic r:id="Top/Society">
  <catid>14</catid>
  <link r:resource="http://www.yforum.com/"/>
</Topic>

<ExternalPage about="http://www.yforum.com/">
  <d:Title>Y? The National Forum on People's Differences</d:Title>
  <d:Description>
    The nation's only forum allowing people to ask and receive answers
    to the uncomfortable and even embarrassing questions they've
    always wanted to ask people who are different from themselves.
    All questions and answers are acceptable, as long as they promote
    dialogue and are not asked or answered out of hate.
  </d:Description> 
</ExternalPage>

<!-- more <Topic>s and <ExternalPage>s -->
...
</RDF>

The hierarchical structure of the directory is represented as a flat list of Topic elements. Each of them uses the r:id attribute to identify the category within the hierarchy and a catid child for its unique identifier. The further content of a Topic is a list of resources for this category given in link elements. Each resource in turn is described by means of an ExternalPage element following the Topic.

The task for this kind of data is to find the resources belonging to a given category specified by its id (catid) and to output these as HTML. While this task is not particularly complicated to solve in XSLT, it is the size of the XML input that makes it impossible for XSLT. The STX transformation sheet for extracting the requested information is given here:

<?xml version="1.0"?>
<stx:transform xmlns:stx="http://stx.sourceforge.net/2002/ns"
               xmlns:r="http://www.w3.org/TR/RDF/"
               xmlns:d="http://purl.org/dc/elements/1.0/"  
               xmlns:od="http://dmoz.org/rdf"
               xmlns="http://www.w3.org/1999/xhtml"
               version="1.0">

  <!-- External parameter identifying the requested category -->
  <stx:param name="catid" />

  <stx:template match="od:RDF">
    <html>
      <body>
        <stx:process-children />
      </body>
    </html>
  </stx:template>

  <stx:variable name="resources" />

  <!-- Group for Topic elements -->
  <stx:group>
    <stx:variable name="found" select="false()" />

    <stx:template match="od:Topic" visibility="public">
      <stx:assign name="resources" select="()" />
      <stx:process-children />
      <stx:if test="$found and $resources">
        <!-- We found the category and there are resources -->
        <h3>Resources in <stx:value-of select="@r:id" /></h3>
        <dl>
          <stx:process-siblings while="od:ExternalPage|text()"
                                group="ep" />
        </dl>
      </stx:if>
    </stx:template>

    <stx:template match="od:catid">
      <stx:assign name="found" select=". = $catid" />
    </stx:template>

    <stx:template match="od:link">
      <stx:assign name="resources"
                  select="($resources, @r:resource)" />
    </stx:template>
  </stx:group>

  <!-- Group for ExternalPage elements -->
  <stx:group name="ep">
    <stx:template match="od:ExternalPage">
      <!-- Is this page among the resources? -->
      <stx:if test="@about = $resources">
        <stx:process-children />
      </stx:if>
    </stx:template>

    <!-- Output Title and Description -->
    <stx:template match="d:Title">
      <dt><a href="{../@about}"><stx:value-of select="." /></a></dt>
    </stx:template>

    <stx:template match="d:Description">
      <dd><stx:value-of select="." /></dd>
    </stx:template>
  </stx:group>

</stx:transform>
               

Looking at this transformation you will notice a lot of similarities with XSLT.

The document element stx:transform encloses a set of templates which describe transformation rules for special kinds of nodes. A template must have a match attribute with a pattern that determines the nodes it is responsible for. This is just like in XSLT. The set of allowed patterns is almost the same. The conflict resolution with the help of priorities works in the usual way.

The first evident difference is the existence of the stx:process-children instruction. If you recall that STX processes its input as a stream of SAX events, then it should be clear that the transformation process itself must run in document order. stx:process-children is the counterpart to XSLT's xsl:apply-templates, whereas the possibility to select arbitrary nodes by means of an additional select attribute can't be part of the STX language.

The second apparent difference (well, actually I don't know in which order you will detect the differences ...) is the new element stx:group. A group encloses a set of templates and other groups. Only the templates of the current group are visible when processing a new node. Thus groups replace the mode concept in XSLT. The rules for entering a group are a little bit complex and will be covered by section Section 3.3.

Finally STX has changeable variables, a well-known concept in procedural programming languages. While XSLT is functional and stateless, STX maintains a state and works in a more imperative way. The order of instructions in STX is determined by the input (especially the document order of the input nodes), whereas an XSLT transformation may process a set of nodes from the input tree in parallel.

Many STX instructions were just adopted from XSLT, for example stx:param, stx:value-of, stx:if, stx:choose, stx:element, stx:attribute, and others. Users familiar with XSLT shouldn't have too many problems to get STX working.

After becoming acquainted with the STX language let me answer the question, for what kind of transformation tasks STX may provide a useful solution. From the characterization of STX it should be clear that transformations that require an overall view and random access to the data in the XML document cannot be practicably executed.

A general rule of thumb is: a transformation that keeps the order of the incoming data is a good candidate for STX. Thus typical tasks are:

For scenarios that don't clearly belong to one of the categories "pure sequential" and "full random access" a hybrid procedure appears to be useful: the STX processor collects preprocessed XML data chunks into a buffer and passes the resulting fragment to an independent XML processor, for example an XSLT engine. This way XML data that is too big for an XSLT engine but too complicated to transform solely in STX will become manageable.

The last aspect I want to address is the use of an STX transformation engine within a Java application. The simplest way is the treatment of such an engine as a SAX XMLFilter object that may be chained into your processing flow. This approach is a kind of low-level approach and requires intimate knowledge of the used engine and its API. Fortunately there is a vendor-independent API for XML transformations.

The Transformation API for XML (also known as TrAX)^[2] became a part of Sun's Java API for XML processing (JAXP) with the 1.1 version of JAXP. One of the design goals for TrAX was to introduce a level of abstraction which allows the usage of general XML transformations within a Java application without the need to struggle with special XSLT transformer APIs. Moreover, even the usage of XSLT as transformation language wasn't hard-wired into the TrAX API. So it's not astonishing to enable STX transformations via TrAX. In the end this can even be seen as a proof of concept that TrAX is indeed able to handle different transformation languages.

The following refers to the STX processor implementation Joost, written in Java (http://joost.sourceforge.net/). To create a factory for STX transformations it is necessary to use the value "net.sf.joost.trax.TransformerFactoryImpl" for the system property "javax.xml.transform.TransformerFactory". After that you can simply create Transformer or Templates objects with an STX transformation sheet as source. The calls to invoke the transformation process itself remain unchanged. That means: switching from XSLT to STX within a Java application requires at most two lines of code to be changed:

// 1. Set the property for the new factory
System.setProperty("javax.xml.transform.TransformerFactory",
                   "net.sf.joost.trax.TransformerFactoryImpl");
...
// 2. Use an STX transformation sheet instead of an XSLT stylesheet
Transformer transformer =
   factory.newTransformer(new StreamSource("sheet.stx"));

Moreover, it is no problem to switch dynamically between several transformation methods just by changing the system property noted above and the source for the transformation at runtime. The integration of STX based transformations into existing Java applications which do use already the TrAX interfaces is indeed very simple.

You may wonder what happens to the DOM versions of Source and Result, since STX requires SAX events to be processed. Well, a specific TrAX implementation can of course handle these special types. Additional helper objects can accept a DOMSource by traversing a DOM tree and emitting SAX events from it or supply a DOMResult by building a DOM tree during the consumption of SAX events. It should be clear that using DOM in this case will again foil STX's advantage of being more memory sparing than XSLT.