Hello World!

For many of us, these words have become synonymous with illustrating key concepts and processes with the intent of improving your understanding and making your efforts more effective than they otherwise would be.

It is the intent of this paper to illustrate and pass on to you an understanding of some new key concepts and processes that may indeed change the way you think about and develop XML, DTDs, stylesheets, and related systems.

In some ways, this paper has been many years in the making. Many years ago the author of this paper was brought in to help develop a hyper-document link-aware SGML-based content management system. Led by notable industry figures, Eliot Kimber (XML, HyTime, XTM, XSL:FO) and John Heintz (a skilled OO architect, database and versioning expert), this project developed a reference implementation of what I still believe to be the most advanced and complete implementation of versioning semantics and cross-document hyperlinking to date [SnapCM]. It was an intense learning experience developed on a tight time-line. The end result was a code-base that was well architected, robust, and upon which large-scale re-implementations and code changes could be performed without hesitation.

Aside from the commitment and technical expertise of the development team, one of the keys to making this large system so easy to change was the development methodology we followed: Extreme Programming (XP)[XP]. This paper is not about Extreme Programming. However, it is about one of the fundamental aspects of XP that transcends the development philosophy: automated test frameworks. Over his subsequent markup standards-centric career, the author of this paper has learned that the benefits of driving your development processes with an automated 'test first' mindset can extend far beyond its origins with object oriented programming languages.

This is also a paper of the risks inherent in XML-based systems. Several of these fundamental risks will be identified and examined. An approach to applying test-driven development to mitigate each risk, enabling higher quality and rapid change will be presented. When applicable, current open source tools and technologies for that type of testing will be discussed. The strengths and weaknesses will be analyzed. Discussion of the potential cost savings of using that approach will also be discussed.

This set of risk mitigation approaches represent several of the author's experiences with applying test-driven processes to XML-based systems. It is the hope of the author that these processes will become accepted and eventually mainstream to development of XML systems, thus improving the quality, consistency, profitability, and user satisfaction of tools and systems within the XML family of standard specifications.

For those readers who are unfamiliar with automated unit testing, please refer to the Appendix to learn:

Fundamentally, XML documents consist of hierarchically-organized text data encoded in a standard encoding, such as Unicode Transformation Format 8-Bit encoding (UTF-8) [UTF]. The backbone of an XML system is the Document Type Declaration (DTD) [DTD]or Schema (W3C[W3C Schema] or RelaxNG[RelaxNG]). The DTD or Schema(s) provide the structural constraints against which an arbitrary document of that type can be validated.

Validation of XML documents against a corresponding DTD/Schema is a critical aspect of nearly any XML-related product or process. When XML documents are loaded by authoring tools, they are validated. When XML documents are imported or exported from a database, the data should be validated. Before a user-defined XML configuration document is applied to define and set web application server features, it should be validated.

Typically the DTD/Schema is used when defining the Stylesheet transformations used to convert, render for view, or otherwise output an XML document. Often times the DTD/Schema is also involved in how the underlying data repository stores and retrieves XML documents and their content. DTDs/Schemas are indeed the cornerstone of an XML solution.

Large scale systems can contain dozens of DTDs/Schemas and tens to hundreds of millions of documents. The side effects of change in systems of this scale can be traumatic and catastrophic. The net effect of changes to the DTDs and Schemas ripple changes throughout all of the mentioned system dependencies and more. Every time the DTD/Schema changes, the tens of millions of documents depending on them need to be updated for validity.

How do you mitigate risk of document-structural changes in systems of this scale? It has been this author's experience that you can help reduce the impact of these changes by setting up an XML-based automated testing framework. When the DTDs/Schemas are initially authored, they should be authored alongside a set of sample XML test documents. These test documents should minimally exercise the significant structural paths defined by the corresponding structures. An automated testing framework can be setup to individually test validation of each of these sample documents. Any time the DTD or Schema changes, this test suite can be executed providing detailed information on the location and nature of the changes. To take this one step further, a document conversion agent could be created along with an XSLT[XSLT] stylesheet to automate the transformation necessary to update the data with the corresponding DTD changes. The agent can be tested against the sample documents. Once the kinks are ironed out, systematic updates to the millions of corresponding XML documents can be performed.

The following graphic illustrates the expected sequence of operations for a 'test-first' DTD-authoring XML validation test suite when no DTD changes have been made. In this situation, running the test suite should result in a visual indication of all tests passing.

The following graphic illustrates the expected sequence of operations for the same 'test-first' DTD-authoring XML validation test suite. When the DTD is updated and/or changed, the sample documents should be updated first. The DTD changes are then made. The test suite is executed, visually indicating the success and failures of the individual validation tests. The test samples, DTD, and tests themselves are updated and retested until all of the tests pass, confirming that you have an automated way to confirm the DTD and samples are up to date. As previously mentioned, a stylesheet could be written and tested during this process to help automate the application of the content changes for the actual production data.

The end result (of this sequence) is updated sample content, successful DTD changes in a demonstrable way, and a start on a stylesheet that can be used to automate updates to the production information. Any time the DTD changes in the future, this suite of tests can be run to detect and repair content side-effects of those changes. Conceptually the process is simple and straightforward. In practice, the stylesheet for automating changes should be developed before the remediation of failed validation tests takes place. Thus, your sample data should help you find flaws in your stylesheet as well.

What open source tools are available to help? Any programming language or a build tool (such as Apache Ant[Ant]) that has programmatic access to a decent XML parser and stylesheet transformation engine can be used to automate the validation and processes mentioned above. Most popular programming languages also have an open source unit testing framework (such as JUnit[JUnit]) available that can be leveraged. Automated validation testing should be integrated into most XML-related projects. Nearly any unit testing programming language tool can be used for the task. Include a validation test for every sample and test XML document that you use. Separate each validation into a unit of testing, so they are quickly identifiable.

The cost of implementing this approach is the cost of adequately developing and maintaining comprehensive sample XML documents and corresponding validation tests for your DTD. Each time your DTD changes, the cost of maintaining your test suite is nearly linear. While the cost of implementing this approach is estimable in man-hours, the total cost savings is a more complex equation.

The total cost savings include rapid detection and reduction in the rippling side-effects from each change. Thus decreasing the 'mystery' time of hunting down and determining the source of failures. This 'mystery tracking' time is typically greater than the linear time spent to simply update a test for a particular change. The time necessary to enact or react to content or structural changes will be minimized. The time spent to setup a test suite process that helps enable the automated update of content is much lower than the time and cost of manual conversion of an arbitrarily large number of production XML documents. The more frequently your DTDs change combined with the more complex of a system in terms of DTD-based dependencies, the greater the total time and cost savings should be.

The most frequent counter-arguments to this approach revolve around the misconception that the effort necessary to properly automate a test-first suite is unnecessary effort or that it adds time to your development schedule. When you calculate the total cost savings, not just those immediately apparent, the savings over the total life-cycle of a project can be significant.

Extensible Stylesheet Language Transformations (XSLT)[XSLT] is quite interesting from a technology perspective. Who'd have thought that a recursive, templatized processing language whose limitations are compensated for with what are essentially XPath[XPath] 'goto's, would be the chosen technology format for performing complex XML to XML transformations. Despite these interesting design choices, XSLT is an immensely useful language that has been widely accepted and supported as the 'standard' way for performing XML transformations to XML or another markup language.

These rather unique design decisions have created, likewise, unique technical risks, not the least of which is how to adequately guarantee the output quality of large, complicated sets of transforms to a set of output formats. The logical answer to this would seem to be through 'adequate testing'.

So how do you 'adequately test' XSLT output? One set of testing options is common to transformations whose output is to a more human-friendly viewable format than XML, such as an XSLT transform to HTML or similarly an XSL Formatting Object (XSL:FO)[XSL:FO] transform that eventually becomes PostScript (PS) or Portable Document Format (PDF). In these situations, it is possible to run the transformation on a set of test documents and visually check every single detail of the output. Every time the transformation templates change, you have to visually check the output again. If the change rippled side-effects, you might not know unless you visually check every detail of the output again. Tools are available to help with this type of testing, however the fundamental flaws and time consumption caused by repeated extensive manual visual examination (not to mention human error) remain significant.

Those lucky individuals who are writing transformations to XML or less-visually appealing output formats have an even harder time doing this visual output checking. While visual HTML/PDF output may be an egregious (yet acceptable) testing method in some cases, a better solution must exist to support this other category of transformation. Further complicating the matter is the wide range of persons capable of implementing XSLTs or XSL:FOs on a regular basis. Due to the aforementioned design decisions, expert XSLT implementers don't need a Computer Science degree and don't have to be familiar with good programming practices in order to be proficient in writing transforms. This increases the acceptance, adoption, and number of candidates capable of using XSLT/FO. However, as we will see below, it can also have some negative side-effects.

A couple of open source tools exist to support automated unit testing of XSLT stylesheets. Perhaps the most notable and widespread is XSLTunit[XSLTunit] testing framework. The premise behind XSLTunit is simple and admirable. Write more XSLT to test XSLT. XSLTunit provides templates and the vocabulary in the XSLTunit namespace to perform assertions and other testing on the results of a transformation. There are a couple of potential drawbacks to using XSLTunit. First, the verbosity required to author your tests in XSLT seems excessive. It may be an acceptable format for apply testing in clever ways by experienced software engineers or computer scientists. However, this leads us back to the wide potential user base for XSLT authors. Many of the experienced stylesheet authors this author knows are Information Systems majors and are not versed in fundamental computer science programming concepts or best practices. This may not be a universal truth, but these authors, while experts at XSLT development, would not have the time or patience to implement more XSLT to test their stylesheets. That is the unfortunate truth.

Another notable open source tool to assist with XSLT unit-based testing is XMLUnit[XMLUnit]. XMLUnit is a Java programming language (also available for .NET) implemented toolkit for unit testing of XML documents. It includes some unique features, such as the ability to perform assertions on output of transformations and XML-based differencing (diff) functionality. The toolkit supports use of the Transformation API for XML (TrAX[TrAX]) and should be usable with any conformant DocumentBuilder, TransformerFactory, etc... The toolkit ships tested and configured for use with Apache Xerces-J[Xerces]/Xalan-J[Xalan]. If you try to configure other TrAX-compliant combinations, you can get some strange results. For instance, the AElfred[AElfred] parser used with the Saxon[Saxon] transformation engine isn't validating. Thus some features (including the tests that ship with XMLUnit) will not work when configured with the typical Saxon installation. For those users whose stylesheets require use of extension functions from other transformation engines, XMLUnit's transformation support is of little use. That constraint aside, the toolkit has some very helpful features.

Back to the problem at hand... How do you adequately test an XSL transformation? This author recommends a home-brewed approach to stylesheet testing.

Fundamentally XSLT is composed primarily of templates that match for certain elements and perform operations and specify output based upon context. 'Goto'-like navigation to siblings or other arbitrary document locations is possible with XPath expressions. There are many more advanced features and functions, but typically a sample XML document is processed by a stylesheet matching its elements in a recursively descending fashion. Thus templates are logical points for units of testing. The context of the match within a sample document, along with the matched element's subtree provide the test inputs. The various output paths for a given template and its expected recursively matched subtree (as defined/expected by examination of the DTD/Schema) provide the expected output for the input. Therefore a test suite can consist of a sequence of these template subtree tests that are combined to exercise the various significant combinations of subtree input. The tests themselves can have output comparisons against the sets of expected outputs for those input variations.

For large scale systems with frequently-changing stylesheets that contain complex, multilevel, recursive sorting of multi-language documents with context-specific output conditions and other various ugliness due to either poor structural (DTD/Schema) design choices (such as semantic tables) or restrictive business rules, this type of low-level unit testing solution can be very intensive and manually labor intensive. This is less a side-effect of the solution as of the overall complexity of the problem, going all the way back to the aforementioned interesting design decisions.

The compromise is to define your unit-based template tests to significant subtrees. Using the 80-20 rule as a guide [Pareto's Principle], focus on thoroughly testing the 20% of the templates that contain the 80% (majority) of the complexity. This takes a good deal of experience and discretion. However, the benefit of automatically being able to do regression testing on a set of stylesheets and confirming that the majority of the complexity is still providing the expected output is a huge benefit over the alternatives. Calculating the cost to setup and maintain an automated, comprehensive stylesheet testing framework versus recovery time upon system failure, perpetual stylesheet 'fixes', and eventual customer dissatisfaction is left as an exercise for the reader. Do you really need a calculation?

Good decision-making while defining the boundaries of problems to be solved in extension functions versus within the stylesheet itself can further simplify the problems. There are many problems you can encounter during transformation for which XSLT is not necessarily the ideal implementation language [Things XSLT can't do]. Advanced mathematical calculations and non-trivial string manipulations are candidate situations where extension functions may be a better choice for the solution. Thus the types of processing that aren't good within the constraints of a recursive process can be implemented in a real programming language with more typical unit-based testing.

Additional benefits can be leveraged from utilizing all of the tools at your disposal. Perhaps the output can more appropriately be tested in slices, via XPath. Perhaps a robust test against an in-memory model DOM will decrease the number and complexity of your test documents. Perhaps writing XSLT to test your XSLT with XSLTUnit is right for your situation. The next section looks at these additional areas of test application with common XML processing models and open source tools.

The XML family of standards has opened up the potential to realize many dimensions of implementation, that before were just ideas. Thanks in part to its inheritance from SGML and the efforts, analysis, and lessons learned by XML's forefathers, advanced systems composed of subsets of features including single-source, multi-output, multi-language publications taking advantage of cross-document linking and reuse with time-based version resolution are now a reality that is much more accessible than ever before.

With few exceptions, regardless of the XML system or solution, this author stands behind the belief that a test-driven mindset and automated unit testing processes can provide some benefits. It won't always be the ideal testing solution. However, it can be effectively applied to a far wider context than has been discussed in this paper.

This section describes the usage of other XML technologies to assist with automated testing. SAX[SAX] and DOM[DOM] are frequently used within XML processing scenarios and tools. DOM-based comparisons can provide your unit tests with comprehensive in-memory document models for detailed testing. XPaths can be integrated into unit tests to evaluate path expression results against expected node sets. Using a series of XPath statements separated into several unit test cases that take actual output as an argument and compare it to the evaluation of the test set of corresponding 'expected' XPath expressions with complementary code to support result comparison, you have a quick way to minimally test small sets of transformations.

Various implementations of XML processors and transformation engines also have additional features, extensions, and configurations that, while proprietary, can still be leveraged for testing. With all of the available choices for testing, pairing the correct testing metaphor and processes with the correct problem scenario can be a daunting task. However, experience is a great teacher and (in this author's opinion) some testing is always better than none!

This paper has taken a detailed look at several applications of XML-based automated unit testing for validation and transformations as well as DOM/SAX testing. However, large scale customer and tool integration requirements can often dictate harsh conceptual and implementation implications. Unit testing can be harnessed to help minimize the risk associated with many of these larger issues as well. Many more complex issues exist that can complicate your ability to unit test. Research, common sense, ingenuity, and a determination to automate your testing can help you find solutions to many, but not all of these problems.

Automated test-driven development processes can provide many key benefits to your XML systems. This paper has provided guidance on where and how those processes can be utilized to provide time, cost, and maintainability benefits. Unit testing can be leveraged for automated validation for any number of document types and related documents. Unit testing can provide granular and accurate confirmation that many, if not all, of the XSL transformations in your system are working properly. XPath can be leveraged in unit tests to quickly test slices of document content. Likewise, DOM can be leveraged as an in-memory model for granular, detailed document comparisons. Open source testing frameworks, such as JUnit, XSLTUnit, or XMLUnit can be utilized to help as needed. Open source XML parsers and XSLT processors, along with their specialized features and enhancements, can and should be used to support your testing efforts.

Automated unit testing can and should be used on most XML-related projects regardless of other development process details. Key to the successful integration of test-driven development in projects is development of the appropriate developer mindset: Test everything and you will become empowered. Hello World!