Using XML Technologies in Multi-Tier Applications

Application Overview

The application we will discuss in this paper can be described as a planning front-end to a manufacturing execution system for the semiconductor industry. It supports creating, modifying, versioning and searching of process plans and related business objects. A process plan describes the various process steps and their relations that are part of the manufacturing process for a particular semiconductor product. The application covers the whole lifecycle and workflow of various business objects. When completed, the business objects are transferred to the manufacturing execution system.

One of the challenges in the development of the application was the huge amount of data that is associated with a process plan. As an example, a process plan can consist of 500 or more single steps and can have up to 1500 objects attached. The typical initial load of a database for a single fab consists of about two millions objects, with about 170000 objects being modified and about 100000 objects being created throughout a year. Initial deployment of the application covers two fabs, one of these being a newly opened one with a smaller initial database load. There are also some strict performance requirements. As an example, opening and presenting a process plan with 500 steps in the graphical view over a WAN connection should not take longer than 5 seconds on average.

The Low-level Approach: DOM and XPath

The first attempt in the implementation of the client targeted a clear separation between graphical user interface implementation and actual business objects. The presentation was described in XML, using XPath (see ) to reference the business object's relationships and attributes. Business objects were represented by DOM documents. Additionally, for every business object class there was an XML Schema (see ) document describing the business object's attributes and relationships. This schema information was used to build the user interface controls (e.g. text fields, drop-down lists, etc.) for editing a business object's attributes.

An example for a presentation description is shown below.

<sheet>
    <page title="Properties" presentation="form">
        <attributes xpathContent="/ProjectFolder">
            <attribute title="Project Title" xpathContent="/ProjectTitle" xpathType="//complexType [@name='ProjectFolder']/complexContent/extension/sequence/element [@name='ProjectTitle']" nameType="ProjectFolder"/>
            <attribute title="Description" xpathContent="/Description" xpathType="//complexType [@name='Object']/sequence/element [@name='Description']" nameType="Object"/>
            <attribute title="Project Leader" xpathContent="/ProjectLeader" xpathType="//complexType [@name='ProjectFolder']/complexContent/extension/sequence/element [@name='ProjectLeader']" nameType="ProjectFolder"/>
            ...
        </attributes>
    </page>
    ...
</sheet>

The corresponding business object looks as follows.

...
<ProjectFolder id="ProjectFolder/DRS-87578379/1">
    <Name>DRS-87578379</Name>
    <Revision>1</Revision>
    <ProjectTitle>...</ProjectTitle>
    <Description>...</Description>
    <Owner>...</Owner>
    <Location>...</Location>
    <State>In Work</State>
    <LastModifiedAt>11-08-2001</LastModifiedAt>
    <LastModifiedBy>...</LastModifiedBy>
    <ProjectLeader>Frank Nobody</ProjectLeader>
    <SecurityLevel>For Internal Use Only</SecurityLevel>
    <Source href="MasterProcessPlan/DRS-00012345/1"/>
    <Source href="ProcessTarget/DRS-04564536/2"/>
    <Source href="SingleProcessSpecification/DRS-85490459/6"/>
    <Target href="ProcessEquipmentConnection/DRS-08690345/1"/>
    ...
</ProjectFolder>
...

However, it was clear too soon that it was impossible to meet the specified performance requirements using this approach. The DOM documents were too large and XPath is rather slow for nagivating through complex DOM documents. Additionally, working with abstract DOM objects instead of business objects made the development of the application a tiresome and error-prone task. Another factor was that business objects had a lot of references to other business objects, either in the same or in other DOM documents, which had yet to be fetched from the server. In many editing views in the user interface, business objects must be presented together with the objects they reference, e.g. in a table or tree view.

Finally, the low-level approach resulted in a lot of functionality being coded more than once by different developers who needed the functionality in ways not covered by the core components.

Introducing an XML Business Objects Engine

After the problems with the initial approach became severe enough to seriously threaten the project's success, a better solution was strongly demanded. It was decided to introduce an XML-based business objects engine that should fix most of the problems we experienced with the first approach.

fig03 shows the core classes of the XML business objects engine. The business objects engine provides the developer with an object oriented interface to a business object. A business object is seen as a collection of attributes and relationships to other business objects. The relationships can be of the part-of or references type. Additionally, the developer can query the meta data for a given business object.

Among the important things that the business objects engine does is to parse the XML Schema documents describing the various business object classes and to build an efficient object-based representation of them. One of the major contributors to the bad performance of the first approach was the querying of XML Schema documents with XPath expressions. The second important thing the business objects engine does is to build some kind of index over a DOM document. This allows for quickly finding the DOM element associated with a specific business object attribute or relationship. Last but not least the business objects engine employs an efficient two-level caching (memory and file system) and object preloading strategy. Once a business object has been transferred from the server, all business objects referenced by that object are also transferred from the server in a background thread, unless they are already in the client's cache.

While the business objects engine did not solve all of the performance problems, it was still a huge improvement to the overall application design. Besides the performance improvements it brought to the application, the most important contribution to the project was that the business objects engine freed the developers from working with abstract DOM documents, XML Schemas and cryptic XPath expressions. Instead it presented them a familiar object oriented view of things.

A prerequisite that must not be underestimated for project of this kind is the availability of a fast and efficient DOM implementation for C++. CenterPoint's own DOM implementation (freely available from ) underwent several performance improvements during the development of this project. Especially, pooling of element and attribute names in a DOM document is essential, otherwise memory requirements become unfeasible.

Lessons Learned so far

XPath is rather slow for nagivating through large and complex documents without the aid of some indexing machinery like the ones provided by XML databases. Having to deal with DOM documents containing a lot of references to other documents does not make things faster. Working with abstract DOM objects instead of business objects makes application development a tiresome and error-prone task. It will also result in a lot of functionality being coded more than once by different developers facing the same or similar problems. XML Schema was not meant to be parsed with XPath, at least if performance is a concern. Schema documents should be parsed only once and an efficient, maybe application specific representation of the schema should be build for further use. Sequentially searching large DOM documents is slow, so some kind of element indexing is needed. Finally, the DOM implementation should be fast and memory efficient.

Describing the User Interface with XML

As was described above, the application's user interface is for the most part described using XML. Separating the user interface description from the application code makes making modifications to the visual appearance of business objects easier. It also allows non-developers to make adjustments to the user interface. Furthermore it saves a lot of work early in the development process when minor changes to the database schema are likely.

We considered using one of the existing XML applications for describing user interfaces. XUL (see ), which was developed by the Mozilla project, and XForms (see ), the designated successor to HTML forms. But soon we learned that our requirements were quite different from the ones targeted by XUL and XForms. For one, we're less concerned about what color any given text has, but rather where the text to be displayed comes from. Also, the whole user interface is based on property pages containing different views. Mostly generic grid-like or tree-like views are used, with a few custom views like the graphical process plan editor in addition. Based on these observations we decided to create our own XML vocabulary for describing the user interface, customized to our specific needs.

The basic idea behind our user interface description is as follows. For every business object, there exists what we call a sheet. Each sheet consists of at least one page and every page visualizes a different aspect of the business object. In the user interface, a sheet is represented by a property sheet with the pages being represented by the property sheet's tabs. fig02 shows a window containing two sheets. An example for such a sheet description can be seen below.

<sheet>
    <page title="Properties" presentation="form">
        <attributes titleWidth="100" valueWidth="600">
            <attribute title="Project Title" name="ProjectTitle" color="blue" static="true"/>
            <attribute title="Description" name="Description"/>
            <attribute title="Project Leader" name="ProjectLeader"/>
            ...
        </attributes>
    </page>

    <page title="Content" presentation="ObjectTree">
        <table xpath="Target|Source|TargetDone|SourceDone">
            <query>
                <select>
                    <attr>Responsibles</attr>
                    <attr>state</attr>
                    <attr>location</attr>
                </select>
                <from>
                    <expand relationship="Target Source TargetDone SourceDone" direction="from">
                        <from><object>$SOURCEID</object></from>
                    </expand>
                </from>
            </query>

            <attribute title="Object" xpath="@href|@id" width="125"/>
            <attribute title="State" xpath="@state" width="145"/>
            <attribute title="Responsible" xpath="Responsibles" width="105" />
            <attribute title="Final State Source" xpath="SourceMarkedAs" class="Source" width="115"/>
            <attribute title="Final State Target" xpath="TargetMarkedAs" class="Target" width="145"/>
            ...
        </table>
    </page>
    
    <page name="Check" presentation="Signature">
        <attributes>
            <attribute title="Group" name="Checkers" width="25"/>
            <attribute title="Claim" name="CheckersClaim" width="8"/>
            <attribute title="Approve" name="CheckersCheck" width="8"/>
            <attribute title="Reject" name="CheckersReject" width="8"/>            
            ...
        </attributes>
    </page>
</sheet>

The above example shows three kinds of pages. The first page describes a simple form that shows various attributes of a business object. The second page describes a table showing the result of a database query. The last page describes a custom view. A special feature the form description above shows is the possibility to embed a database query into a page description. Throughout many places within the application, table-like and tree-like views of business objects and their relationships are needed. However, for those views only a few of an object's attributes need to be displayed. Fetching every object in the list from the database would be too slow, so a query is executed that returns an XML document containing just the required attributes for all objects in the list.

More Lessons Learned

Just describing the look and feel of the user interface is not enough. A really generic user interface description also needs a mechanism to describe where to get the data from. Only marrying XML syntax describing the user interface with XPath describing the data source is not sufficient, especially if performance is a concern. Mixing the user interface description with XPath and a generic query language showed the best results for us.

Describing Database Queries

All database queries within the application are described in an XML syntax. The result of a database query is an XML document. Queries are used in forms, like the one shown above, they are used internally by the application logic and they are used for user-defined queries. However, users do not write their queries in XML. Users can either formulate their queries in a Query-by-example style or using a simplified query language. The user queries are then transformed into XML and sent to the server. On the server, the XML query syntax is not executed directly, but transformed into the underlying system's native query language, which in our case is MQL (Matrix Query Language). Below is the document type definition for the query language.

<!ELEMENT query (select, from, where?)>
<!ELEMENT select (attr+)>
<!ELEMENT attr (#PCDATA)>
<!ELEMENT from (expand | object | class)>
<!ELEMENT expand (from, where?)>
<!ATTLIST expand 
    relationship NMTOKENS #REQUIRED
    direction (from | to) "from">
<!ELEMENT object (#PCDATA)>
<!ELEMENT class  (#PCDATA)>
<!ENTITY % op "and | or | equal | notequal | gt | lt | gtequal | ltequal | match">
<!ENTITY % binop "left  CDATA #REQUIRED
                  right CDATA #REQUIRED">
<!ELEMENT where (%op;)>
<!ELEMENT and (%op;)+>
<!ELEMENT or (%op;)+>
<!ELEMENT equal EMPTY>
<!ATTLIST equal %binop;>
<!ELEMENT notequal EMPTY>
<!ATTLIST notequal %binop;>
<!ELEMENT gt EMPTY>
<!ATTLIST gt %binop;>
<!ELEMENT lt EMPTY>
<!ATTLIST lt %binop;>
<!ELEMENT gtequal EMPTY>
<!ATTLIST gtequal %binop;>
<!ELEMENT ltequal EMPTY>
<!ATTLIST ltequal %binop;>
<!ELEMENT match EMPTY>
<!ATTLIST match %binop;>

The query language is based on the object-oriented database paradigm.

Client-Server Communication with SOAP

Client and server talk to each other by sending SOAP messages over HTTP (or HTTPS) connections. Sending large XML documents in plain text is quite inefficient, so we decided not to use pure SOAP messages. Instead, we chose to implement a submission to the W3C entitled SOAP Messages with Attachments (see ). This allows us to send the SOAP request as plain text, but the actual data in compressed form. An example for such a message, a response to a getObject request returning a business object, is shown below. The HTTP headers have been omitted.

--MIME_boundary
Content-Type: text/xml

<SOAP-ENV:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
    xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" 
    xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" 
    encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
    <SOAP-ENV:Body>
        <m:getObjectResponse xmlns:m="urn:xbo-getObject">
            <object href="ProjectFolder/DRS-87578379/1"/>
        </m:getObjectResponse>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

--MIME-boundry
Content-Type: text/xml
Content-Encoding: deflate
Content-ID: ProjectFolder/DRS-87578379/1

<<serialized object in deflated XML>>
--MIME-boundry--

Printing and Report Generation with XSLT

The fact that all the application's data is available in XML leads to the obvious decision to use XSLT for printing and report generation. Unfortunately, at the time of the writing of this paper no functionality in this regard has been implemented yet. What we plan as the first step is to use Microsoft Internet Explorer's XML and XSLT capabilities to transform our business objects into some kind of HTML output suitable for viewing in a browser window and for printing. Our customer also expressed some interest in creating PDF documents from the database contents, so this will present an opportunity to take a look at XSL-FO .