The W3C workshop on Binary Interchange of XML Information Item Sets was held at Sun Microsystem's campus in Santa Clara CA between the 24th and 26th of September, 2003 [W3C Workshop on Binary Interchange of XML Information Item Sets]. In what follows, we will refer to "binary interchange of XML information item sets" simply as binary XML ^[1].

A requirement to attend the workshop was to submit a position paper. A total of 36 papers from different organizations and individuals were accepted. The majority of the papers submitted included answers to the following questions:

This paper will elaborate some of the answers to questions 2 and 3 ---the reader is referred to [Report on the W3C Workshop] for details on the position of each organization with respect to question 1.

The first day of the workshop consisted of presentations from a number of companies including Adobe, BEA, Expway, IBM, Microsoft, and Sun, among others. Each organization presented their position with respect to question 1 and, in most cases, also discussed some of their use cases and solutions. During the second day, break-out groups were formed in order to collect requirements as well as to give an opportunity to those companies that had not presented their paper to do so. The third day (which was really a half day) was dedicated to discussions on further work within the W3C in the area of binary XML.

The paper is organized as follows. In Section 1.1, “Introduction”, we discuss a list of requirements for an ideal binary XML format. In Section 1.3, “Data Compression”, we address the always controversial topic of data compression for XML. Section 1.4, “Degrees of "Infosetness"” defines a way to classify binary XML encodings and outlines a taxonomy for the different solutions presented in the position papers. Finally, Section 1.6, “Conclusions” lists conclusions and explains future work items.

We divided the requirements collected at the workshop into three categories: format, transmission and processing. The format requirements serve as the basis for the requirements in the other two categories. They are the requirements that most of us think of first when discussing binary XML. However, they are not completely independent: requirements in the transmission and processing categories will influence how the binary format is defined (i.e., support for fragments may rely on features added to the underlying format).

A condensed version of the requirements collected at the W3C workshop, listed in the three categories outlined above, is shown below ^[2]. The complete set of requirements is available online [Report on the W3C Workshop].

Perhaps the most obvious requirement for a binary XML format is that it has to be efficient, both in document size and encoding/decoding time. The general consensus was that, unless the efficiency of a solution can be proven, there is little value in using it as an alternative to the widely successful textual format. How much efficiency? There was no consensus on this topic, but some suggested that in order for XML (and, more importantly, the benefit of using XML standards and tools) to be used in certain industries, an order-of-magnitude improvement is needed. It was agreed that the impact on applications and developers should be minimal, and that support for standard XML APIs such as SAX, DOM and Pull (StAX) was desirable. For many, preserving the self-descriptive aspect of XML was a must; for others, this was a commodity that could be traded in cases where processing power and memory are scarce.

When it comes to transmission of binary XML documents, especially in the context of Web services, there was a natural comparison against existing technologies such as CORBA, RMI and DCOM. Some recent work [Fast Web Services]shows that the performance of Web services is significantly worse than RMI and even RMI/IIOP. Re-gaining some of this performance was regarded as an important goal of a binary XML format. Having more than one wire format for Web services may create interoperability problems; consequently, the consensus was that binary XML should be used as an alternative to textual XML, and that support for the latter was necessary in order to have a fall-back mechanism. Finally, and still within the area of transmission, schema drift detection and schema evolution were deemed necessary for those solutions that optimize the wire format based on schema knowledge.

Other requirements included support for XML fragments and the ability to send deltas (i.e., versioned fragments). It was pointed out that existing technologies [MPEG-7]already provide support for these features in way that is much easier to use when compared to XML. Random access, i.e. the ability to access a portion of a document in sub-linear performance with respect to its size, is another requirement that was brought up several times [Adobe] [CCSDS Packaging Working Group]. It can be regarded as a requirement on the format, but it is listed in the processing category as the unit of random access was loosely defined.

Support for XML security was also mentioned. One of the challenges in supporting XML security is that it relies (perhaps too much?) on the lexical representation of datatype values. For example, a binary XML format that uses schema information may turn the character information items "001" into the binary number 1 whenever its type is xs:int. Even though the value space of the datatype is preserved, its lexical space is not (the use of xs:pattern in an XML schema poses similar challenges). Of course, always preserving the lexical space defeats the purpose of using binary XML in some cases; notably, large XML documents carrying scientific data [Cubewerx]. A possible solution for this problem is outlined in [Schema Centric XML Canonicalization] where a schema centric canonicalization is defined.

Since one of the goals of a binary XML format is to reduce message size, data compression is often mentioned as a possible solution. The question people ask is: "Why don't you use gzip?". Indeed, for some applications, using data compression is a solution. However, in practice only a small percentage of applications can benefit from using these techniques. There are two major problems in using data compression:

Data compression is useful when two high-speed power-unconstrained CPUs exchange messages over a low-bandwidth line [Mitre] [Software AG]. Successful use of gzip has also been reported for large SVG files [Nokia]. For other cases involving the exchange of messages, data compression is not a viable solution. Even for storage of XML documents a binary XML representation may be preferred, especially if it supports a form of random access [Adobe].

Conceptually, there is nothing that prevents the use of data compression in conjunction with a binary XML format. Many binary formats store string data in textual form (e.g., ASN.1 supports UTF8), thus the compression of all or part of a binary message may still be desirable.

A convenient way to define a taxonomy for the different binary XML formats is to classify them based on how much of the original XML infoset they preserve. At one extreme, we have solutions that attempt to preserve the entire XML infoset as defined in [XML Information Set]. Atthe other, we have solutions that, aided by external information, attempt to minimize the amount of data by omitting certain infoset items.

In practice, the degree of infosetness can be controlled by how much schema information is employed. The more schema information used, the more compact the resulting format, but the more infoset items tend to be omitted. Perhaps not surprisingly, empirical data presented at the W3C workshop indicates that the more schema information used the more performance gained [Sun] [Expway] [Mitre] [KDDI] [OSS Nokalva].

A pure infoset-based solution will still perform better than textual XML as, in a binary format, it is possible to do clever sharing of items and item properties to avoid the redundancy that characterizes XML documents. Simple examples of sharable parts are element and attribute names, namespace prefixes and namespaces URIs. Conversely, a pure schema-based solution will omit all element and attribute names (as these can be recovered from the schema) and will use datatype information to efficiently encode leaf nodes (i.e., character items) in order to get as close as possible to an Information Theory minimum.

There are also hybrid solutions that take advantage of some schema information whenever it is available. For example, it is possible to use datatype information from a schema to encode numeric data while at the same time preserving all the self-describing information (i.e., element names, attribute names, etc.). In some hybrid solutions, it is even possible to add typing information to the binary stream to aid the decoding process in the absence of the schema [Cubewerx].

Table 1lists solutions presented in the position papers accepted to the workshop. Table 2lists other technologies mentioned in the position papers as well as standards on which solutions fromTable 1 are based. For each row in a table, a "Y" indicates that the feature is supported, an "N" that it is not and a "?" that not enough information was available to determine either way ^[4].

The number of organizations that attended the workshop indicates that there is great interest in binary XML. It was clear from the discussions that this is not a solution without problem. Some see the adoption of a binary XML format as a way to optimize their existing systems; others as the only possible way by which their systems can take advantage of existing XML technologies and tools.

A significant percentage of the attendees were skeptical that a single binary XML format will be suitable for all purposes. The more optimistic participants indicated that a mixed solution that combines infoset-based and schema-based encodings may successfully hit the 80:20 point, especially if redundancy-based compression was (optionally) supported as well. It was clear from the discussions that further work is needed to ascertain which of these positions will prevail.

One of the outcomes of the workshop was the decision to (i) create a forum for further discussion and (ii) prepare a draft for a possible WG at the W3C to further evaluate the need for a standardized binary format. This WG will not be formed for the purpose of producing a specification for a binary format, but to refine the list of requirements and use cases, and to evaluate functionality and performance (via a normalized method) of existing solutions.

The volunteers for drafting a WG charter are: Mark Nottingham (BEA Systems), Robin Berjon (Expway), Stephen Williams (High Performance Technologies), Santiago Pericas-Geertsen (Sun Microsystems), Selim Balcisoy (Nokia), Kimmo Raatikainen (University of Helsinki), John Schneider (Agile Delta), Alex Danilo (CISCO), Don Brutzman (Web3D Consortium) and Mike Cokus (Mitre).