XTech 2005: XML, the Web and beyond.
Michael Brauer is a Technical Architect on the StarOffice/OpenOffice.org development team at Sun Microsystems, focusing on XML technologies. He joined the StarOffice/OpenOffice.org development team in 1995, and is the lead of the OpenOffice.org XML Project that developed the OpenOffice.org XML file format since its formation in 2000. He is also the chair of the OASIS Open Document Format Technical Committee that created and maintains “OpenDocument”, the standardized file format for office productivity applications like word processors, presentation and spreadsheet programs.
The development of the OASIS Open Document Format for Office Applications OpenDocument-Spec goes back to early 1999. At that time, the office productivity suite StarOffice was owned by a company called Star Division, that had its development department in Hamburg, Germany. The OpenOffice.org project OOo, to which Sun contributed the StarOffice source code a year later, didn't exist at that time.
In 1999, the current release of StarOffice was StarOffice 5.0. The file format used by StarOffice 5.0 was a binary file format, as was common in those days. But because interoperability on the file format level was becoming more and more important, the StarOffice development department wished to have a file format that was more interoperable than a binary file format could ever be.
At the same time, it was decided to add Unicode Unicode support to StarOffice. Because this required an incompatible change of the file format as well, the timing was good to look for a new technology that would solve the interoperability issue. Actually, the search for a new technology was easy. XML XML had been standardized only a year before, and it was crystal clear that using any other technology than this emerging standard would be a step in the wrong direction and especially would not solve customer requirements in the near future. Even better, XML supported Unicode already without any extra effort. So, the StarOffice development department decided to develop an XML file format for office applications.
The newly founded StarOffice XML development team did not only have the ambition to develop an XML file format for StarOffice in particular. The ambition was to develop an XML file format that ensured real interoperability by abstracting from the data structures that StarOffice uses internally, and could therefore be used by other office productivity applications as well. Thus, the charter that the OpenOffice.org XML project OOo-XML set up a year later for the development of the OpenOffice.org XML file format was already in place before OpenOffice.org was founded:
Core Requirements (these items are absolutely required)
- The file format must be capable of being used as an office program's native file format. The format must be "non-lossy" and must support (at least) the full capability of a StarOffice/OpenOffice.org document. The format is likely to be used for document interchange but that use alone is not enough.
- Structured content should make use of XML's structuring capabilities and be represented in terms of XML elements and attributes.
- The file format must be fully documented and have no "secret" features.
- OpenOffice.org must be the reference implementation for this file format.
Core Goals (these items are highly desired)
- The file format should be developed in such a way that it will be accepted by the community and can be placed under community control for future development and format evolution.
- The file formats should be suitable for all office types: text processing, spreadsheet, presentation, drawing, charting, and math.
- The file formats should reuse portions of each other as much as possible (so for example a spreadsheet table definition can work also as a text processing table definition).
Within the same year, StarOffice was acquired by Sun Microsystems. Due to the important role that Sun plays in developing and using open standards, specifically in the XML area, this obviously was an enormous boost for the development of the XML file format. The idea to submit the format to OASIS OASIS was already born at that time, even though it took a few years before this actually became reality.
A year later, in 2000, the OpenOffice.org project OOo was founded. The basis of that project was the StarOffice source code that Sun contributed to OpenOffice.org. For the development of the XML file format, the OpenOffice.org XML project OOo-XML was founded. Its charter has been mentioned already.
From then on, the format was called the OpenOffice.org XML file format, and its development took place within an open source project. The OpenOffice.org XML file format specification drafts OOo-Spec were (and still are) publicly available at the OpenOffice.org web site. The first specification draft was released in 2000. A year later, in 2001, OpenOffice.org 1.0 and Sun's StarOffice 6.0, that was based on it, were released. Both use the new OpenOffice.org XML file format as native file format.
It's worth mentioning that the development of the StarOffice binary format was discontinued after StarOffice 5.2 had been released early in 2000. Therefore, there was no alternative binary file format for OpenOffice.org 1.0 and StarOffice 6.0. All documents saved by these applications and later OpenOffice.org/StarOffice versions are XML documents.
Though an open and documented specification for an XML file format for office applications now existed, the development of the OpenOffice.org XML file format did not stop. In 2002 definitions for CJK and complex text layout languages were added. In the same year, the OpenOffice.org and KOffice KOffice projects started a dialog regarding the file format, and agreed to extend the package format used by the OpenOffice.org XML file format so that it allows Unix's file magic to properly recognize OpenOffice.org XML documents.
Again in 2002, the preparations began for the foundation of an OASIS technical committee (TC) that would standardize the open XML file format for office applications based on the OpenOffice.org file format. On December 16th, the technical committee started its work by having its first conference call. Among the founding members of the TC were Boeing, Stellent, Arbortext, Sun Microsystems, the National Archive of Australia, and the Society of Biblical Literature. Since the OpenOffice.org project itself is not a company, it could not join the TC as “OpenOffice.org”, but among the founding TC members was also a member representing the OpenOffice.org project. Only two months later, the KOffice project joined the TC on the same basis.
It was a considered decision to start the standardization effort at OASIS. The OASIS membership model allows open source projects to participate in the standardization process, and the OASIS rules allow it to use a specification that has already proven its usefulness as a basis for standardization, and therefore makes it possible to develop a 700-page specification in a reasonable time frame.
Like the OpenOffice.org XML file format charter, the charter of the OASIS Open Office TC OpenDocument-Charter focuses on interoperability:
Statement of Purpose
The purpose of this TC is to create an open, XML-based file format specification for office applications.
The resulting file format must meet the following requirements:
- it must be suitable for office documents containing text, spreadsheets, charts, and graphical documents,
- it must be compatible with the W3C Extensible Markup Language (XML) v1.0 and W3C Namespaces in XML v1.0 specifications,
- it must retain high-level information suitable for editing the document,
- it must be friendly to transformations using XSLT or similar XML-based languages or tools,
- it should keep the document's content and layout information separate such that they can be processed independently of each other, and
- it should 'borrow' from similar, existing standards wherever possible and permitted.
During 2003, the OASIS Open Office TC worked hard to adopt the OpenOffice.org specification to standard needs, but also to recent developments in the XML and office application area. The first committee draft was approved by the TC in March 2004. The major changes that have been made to the OpenOffice.org specification are:
In December 2004, a 2nd committee draft was approved. The title of this draft was changed from “OASIS Open Office specification” to “OASIS Open Document Format for Office Applications (OpenDocument)”. This was for two reasons. First, the TC is of the opinion that this name better reflects the fact that the the file format defined be the specification is not only suitable for documents created by office applications and is not limited to documents used within office suites. Second, the TC wants to emphasize that the format is not a format for the OpenOffice.org application, but also suitable for other applications by giving it a name that is neutral and not only related to the OpenOffice.org project. In January 2005, the TC renamed itself to “OASIS Open Document Format for Office Applications (OpenDocument)”, too.
Beside the name change, the 2nd committee draft addresses inconsistencies and errors found in the first committee draft, and again contains enhancements to keep the specification up to date with the latest releases of office productivity suites. This 2nd draft went through a 4-week public review phase. The feedback received during the public review was integrated into a 3rd committee draft OpenDocument-Spec, that was approved in February 2005. That draft is also the basis for the OASIS standard voting that is taking place as this paper is being written (April 2005).
Right now (April 2005), the voting for an OASIS OpenDocument standard is taking place within the OASIS membership. If it is successful, there will be an open document format for office applications developed by a vendor-independent standardization body for the first time in the 25-year history of office applications. There will be a document format for office applications that has not been developed by a single vendor, who also owns the format. The format will be the default format of OpenOffice.org 2.0 and the large family of office suites based on the OpenOffice.org source code, such as Sun's StarOffice 8. OpenDocument will be also supported by KOffice.
Not surprisingly, the work of the OpenDocument TC has been recognized by the public sector. One of the groups that very closely watches the work of the OpenDocument TC is the European Commission's Telematics between Administrations Committee (TAC). In May 2004, the TAC published some recommendations TAC-Recommendations regarding the use of open document formats by the public sector. These recommendations include:
The OpenDocument TC's members welcome the idea of submitting the OpenDocument specification to ISO. Therefore, the TC will ask OASIS to submit the format to ISO as soon as OpenDocument 1.0 has been approved as an OASIS standard, and will support OASIS in the ISO standardization effort.
In addition to this, the OpenDocument TC will continue to enhance the specification. Among the areas where enhancements are planned are custom-schema support as well as security features, like digital signatures and encryption. The TC will also continue to maintain the specification, discussing all enhancements to keep the specification up to date with developments in office applications.
The following illustration shows the most important milestones in the development of OpenDocument:
The OASIS OpenDocument format OpenDocument-Spec is an XML file format for office applications. What does that mean? That means that it is an XML file format tailored to office applications. That also means that the format is feature-complete: All information an office application needs to store can be stored in OpenDocument files. The OpenDocument format can therefore be used as native file format for office applications, saving documents in this format without losing features. It can fully replace the traditional binary formats that have been used and very often still are used by office applications as native file formats. The purpose of an open, standardized office application file format like OpenDocument is to maximize interoperability and to make sure the data in office application documents is owned 100% by the document authors and 0% by the application vendors.
This contrasts to another XML feature that some office applications support: so called custom schema support. Custom schema support means that an application is able to mix office content with content stored within arbitrary XML schemas. These schemas may be either defined by a standardization body, or may be defined by a customer. They are not tailored to office applications and in particular are not able to keep all office content. This means that either a huge amount of office content is lost when saving a document in a custom schema or that the content of the custom schema has to be embedded within some other application-specific file format. The application-specific file format can be an XML file format, but it can also be a binary format. This means that the ability to process the custom schema information contained in a document still depends on the office application file format itself, and it explains why an application that has custom schema support still requires a standardized and open XML file format. In other words: Interoperable custom schema support can be implemented on top of a standardized and open office document format, but it cannot replace it.
Technically, as well as from the user interface perspective, there are several ways to implement custom schema support. One possibility is to mix the custom schema with an office document schema, and to provide some XML editor capabilities within the office application's user interface. A completely different approach is to use W3C XForms XForms to embed the custom schema instance into a document and to provide XForms based form functionality within the user interface. The later approach is used by Sun's StarOffice application. The implementation of this feature has been contributed to OpenOffice.org, and the required OpenDocument extensions have been integrated into the OpenDocument specification already.
Somewhere in the middle between custom schemas and office application schemas are schemas like DocBook or XHTML, that can be used to store documents that are similar to office documents. Usually, these schemas are not mixed with office document content, but rather office document content is transformed to or from these formats. Applications that have a native XML file format can implement these transformations using standard XML technologies like XSLT XSLT.
This section lists the requirements that an open document format for office applications must meet to deserve that name. It also shows how the OpenDocument format meets these requirements. Most of the requirements are technical ones, but there are also non-technical requirements that are at least as important as the technical ones. Some of the requirements depend on others, while others could be viewed independently. All these requirements are important. Not fulfilling even just one of these requirements may have a severe impact on the openness of the format.
As discussed above, one of the key requirements for an open office application file format is completeness. Completeness here does not only mean feature completeness within a certain document type (like text documents, spreadsheets, etc.), but also that the various document types that modern office applications support are covered. That is, an open office application file format should not only include the classic text and spreadsheet documents, but also business charts and presentations. Because it is common practice for documents of one type to contain embedded documents of another type, the requirement to support various document types is even important in cases where only one document type is primarily used. An example for this could be text documents that very often contain formulas or business charts. When saving such a document in a format that supports text documents but not formulas or business charts would either mean that some information would get lost or would be saved in a binary format.
Completeness is required for the long term preservation of all office documents, but completeness is also a key requirement for interoperability and obviously a precondition for being usable as the default file format for office applications.
The OpenDocument specification supports text documents, spreadsheets, presentations, business charts, formulas as well as graphics and images.
Extensibility is important for several reasons. First of all, it fosters the format's development and improvement. But extensibility also allows building formats on top of a standard. Finally, extensibility simplifies the task of backward compatibility during the development of a file format.
As an XML format, OpenDocument has all the extensibility features that XML specifies. In addition, the OpenDocument format contains precise extensibility rules where the demand for extensibility seems to be highest: for meta information, and formatting properties.
Another source of extensibility is OpenDocument's package concept. A package is a zip file ZIP that on the one hand allows it to split a single OpenDocument document into several streams, where each stream contains a certain kind of information, but on the other hand also allows it to add arbitrary streams that contain information not defined by OpenDocument itself. That is, an OpenDocument package file may contain streams that are not specified by the OpenDocument specification, and therefore may contain extensions to the OpenDocument format. The streams specified by OpenDocument are “meta.xml” for meta information, “styles.xml” for styles, “content.xml” for the document content, and “settings.xml” for settings, like the printer that has been used to print the document.
Other benefits of OpenDocument packages are that they support compression and also allow images to be saved in their original binary format, which, given the number of tools that support these formats, should be seen as an advantage rather than a disadvantage.
The following illustration shows the layout of an OpenDocument package:
The use of packages is optional. OpenDocument defines documents that are stored within a package as well as documents that are stored as a plain or flat XML file. However, for documents stored by end users, the packaged OpenDocument variant will be used in almost all cases. The purpose of the flat variant is the exchange of documents between XML based components, for instance within XML pipelines. Consequently, most office application user interfaces will only offer loading and saving of the packed OpenDocument files and will support flat XML files only as part of their programming API.
The OpenDocument format is also extensible on a completely different, non technical level. Because the format is developed by an OASIS technical committee, it can be influenced by OASIS members. That is, if an OASIS member, regardless whether it is a company or an individual, requires an enhancement of the format, she or he may join the technical committee to get official approval of the enhancement. Obviously, that's an option that does not exist for vendor controlled formats. This non technical extensibility is extremely important for office application vendors that don't want to develop their own proprietary file formats, but also don't want their own products to depend on the capabilities of a file format defined by some other vendor.
Interoperability is another important requirement for an open document format for office applications. Interoperability here has two aspects. One is the interoperability between office applications, the other one is the interoperability between office applications and other types of programs.
Interoperability is important for several reasons. First of all, it supersedes the need for proprietary document formats. But interoperability also ensures that all information in a document can really be accessed. Another reason why interoperability is important is that it helps to reduce costs. The development of tools that operate on an interoperable file format is cheaper than the development of tools that operate on non-interoperable file formats. The same applies to training costs.
Regarding file formats, it is very important that interoperability is a design principle. That means that the file format has to be developed with interoperability in mind from the beginning. It's nearly impossible to turn an existing, non-interoperable file format into an interoperable one later on.
As already explained in the history section, interoperability has been a design goal for both, OpenOffice.org and the OpenDocument format from the beginning. The interoperability of the OpenDocument format is achieved in different ways. First of all, the OpenDocument format is not a copy of OpenOffice.org internal data structures or a copy of the data structures of any other application, but application-independent data structures are used. In addition to this, OpenDocument is first class XML. Wherever feasible, information is encoded in XML elements and attributes rather than being encoded in, for instance, integer attribute values or BASE64 encoded binary data. Furthermore, OpenDocument makes use of existing standards like HTML HTML4, SVG SVG, XSL XSL, SMIL SMIL20, XLink XLink, XForms XForms, MathML MathML or Dublin Core DCMI wherever possible. And finally, OpenDocument reuses its own document structures wherever possible. For instance, there is only one representation of paragraphs or tables, and this representation is used within text documents as well as within spreadsheet and presentations.
The adoption of the format by KOffice is a proof of concept for OpenDocument's interoperability design.
It has been said already that an open office application file format should be usable as default file format. But why is this so important? When a document is saved today, one very often does not know which applications will be used in ten or more years to read the document. Even more important, one very often does not know for which unknown future tasks the information contained in the document may be required. Even if it does not seem to be necessary to store a specific document in an open and standardized XML file format today, a requirement may exist in the near or far future to process the information stored in exactly that document. If the document is stored in a standardized XML file format today, no problem will arise in the future. But if the document is stored in a binary format or some other proprietary format, expensive document conversion may be required. This can easily be avoided if an office application stores all documents in a standardized XML file format like OpenDocument.
Very much related to the requirement of being usable as a default file format is the requirement for a reasonable file size. Office applications are multi-purpose applications. They are used at home or in small offices where they are used to write 1-page letters or 100-cell budget plans, but also in enterprises where they are used to create 800-page design specifications or million-cell business plans. All these documents must have a reasonable file size.
Moreover, an open office document format has to compete with the traditional binary formats regarding the user's expectation for file size. That is, many users will not accept increased file sizes even if they know that their documents are stored in XML now. Unfortunately, the verbosity of XML, that has many advantages, also has the disadvantage that documents never get as compact as they would if they were stored in a binary document. The differences can be dramatic. A spreadsheet document that has a size of 10MB if saved in a binary format easily may increase to 100MB if saved in an XML format. Therefore, saving documents in pure XML without any additional compression technologies may be the best choice from an information workers point of view, but it definitively is not compatible with end user expectations and requirement.
OpenDocument solves this issue with its package concept. As explained above, a package is a zip file, and therefore support compression.
An important non-technical requirement for an open document format is vendor independence. Whenever a format is not vendor-independent, there is a high risk for users of the format to become dependent of a certain vendor. Furthermore, vendor dependence can hinder standardization efforts and may also lead to monopolistic markets. Therefore, an open document format should be controlled by a standardization body rather than by a single vendor.
For the OpenDocument format, this standardization body is OASIS. That means that the format is developed by a vendor-independent technical committee according to the very open and transparent OASIS process.
Another important non-technical requirement for an open document format is the license under which the format is developed. Licenses and license changes may impact business models and may cause unforeseeable costs. For this reason, the license of an open document format must be easy to understand, safe to use, and in particular must not prevent certain user or developer groups from using the format.
This paper neither specifies what an “open license” is nor will it give any legal advice, but only alerts the reader not to underestimate the impact licenses can have on document formats.
The OASIS OpenDocument format has been developed using the OASIS license and IPR policies. In addition to this, the OpenDocument charter OpenDocument-Charter contains an “openness” standing rule.
Until now, it seems that we only took a look at requirements, but not at the benefits of the OpenDocument format. This first impression is wrong. The OpenDocument format meets all the requirements listed above, and that is in fact a very large benefit for all kinds of office application and office document format users. Document managers will especially like the extensibility of the format, while archivists may especially like its completeness. For home users, it might be most important that they get the benefits of XML without having to invest in additional disk space. But for all kinds of users, it is a great benefit that the OpenDocument format is developed by a vendor-independent standardization body under an open license.
Actually, the overall benefit of OpenDocument meeting all the requirements above could be turned into a list of individual benefits by looking at the requirements from a different perspective:
This section gives a short introduction into the features of OpenOffice.org that are based on OpenDocument.
It has been said several times already that OpenOffice.org 2.0 uses OpenDocument as default and native file format. All OpenOffice.org documents are saved by default in the OpenDocument format, and the OpenDocument format is the only file format that is capable of storing all OpenOffice.org 2.0 features. In particular, there is no native OpenOffice.org binary format. This also applies to the older OpenOffice.org versions 1.0 and 1.1 that use the OpenOffice.org XML file format and do not have a native binary format as well.
Within the user interface, OpenOffice.org supports only packaged OpenDocument files. Components that process flat OpenDocument files are available within the OpenOffice.org API. The flat OpenDocument format is also used internally, for instance by the XSLT-based filters described in the next section.
OpenOffice.org supports so-called XSLT-based filters. A filter in general is a component that supports loading or saving of documents in other formats than OpenDocument, for instance “.doc” or HTML. To differentiate these formats from the OpenDocument format, they are called “foreign” formats here. Filters that load documents are called import filters, while filters that save documents are called export filters.
An XSLT-based filter is a filter component that is based on an XSLT stylesheet XSLT. An XSLT-based import filter uses a stylesheet that transforms a foreign format into OpenDocument. An XSLT-based export filter uses a stylesheet that transforms OpenDocument into a foreign format. For an import filter, the foreign format must be an XML format. For an export filter, the foreign format can be an XML format, but also any other text format that can be created using XSLT.
OpenOffice.org has a user interface for the registration and maintenance of XSLT-based filters. To implement such a filter, nothing other than XSLT stylesheets are required. As soon as an XSLT-based filter has been registered, it appears like any other filter in the filter list of the “File - Open” and “File - Save” dialogs. This means that the (XML) format implemented by the filter can be loaded and saved as easy as any other format that OpenOffice.org supports. In particular, OpenOffice.org users never have to select an XSLT stylesheet to load or save a document.
XSLT-based filters are available for all document types, like text documents, spreadsheets, etc., and the use cases for XSLT-based filters are not limited to the traditional loading and saving of documents. XSLT-based import filters also can be used to load documents into OpenOffice.org for the purpose of printing them, or to publish them into PDF. The input format here can be similar to office document formats, like DocBook, but can also be entirely unrelated to office document formats, like UBL. In the later case, the purpose of the XSLT-based filter is to create a view of the document readable for humans. The target application for an UBL filter could be a word processor or a spreadsheet.
XSLT-based export filters can also be used for publishing documents into XML formats like XHTML.
The following illustration shows the data flow for an XSLT-based UBL import filter and for an XSLT-based HTML export filter.
Another OpenOffice.org feature that is related to OpenDocument is OpenOffice.org's XForms XForms support. Starting with OpenOffice.org 2.0, a text document may contain an XForms model instance, and form controls can be bound to the XForms instance using XForms bindings. This means that an arbitrary custom schema instance can be included into a document, and can be manipulated using controls. The controls used here are the same as the ones that can be bound to databases. That is, OpenOffice.org uses the XForms model but does not use the XForms UI. The XForms model implemented in OpenOffice.org supports data types, validations and calculations. This means that especially validations and calculations within forms can be implemented using XForms expressions. Neither StarBasic nor any other programming language is required for this purpose. The XForms instance can be saved separately, but it is also possible to submit it to a server.
OpenDocument can be used as host language for XForms. It already contains the definitions required for XForms, therefore OpenOffice.org documents that contains an XForms instance can be saved as OpenDocument files. Actually, this means that custom schema instances can be included into OpenDocument files. To use XForms for this purpose has the advantage that the custom schema instance can be included in the XML document at a distinct location as a whole, so that the custom schema markup is not mixed with the OpenDocument markup. Both markups can easily be processed and validated independent of each other. The following illustration shows the layout of an OpenDocument file that contains an XForms instance:
Due to its XForms support, OpenOffice.org can be used as form entry tool for XForms-based forms. But the XForms support can also be used to edit custom schema instances contained in a document, or to simply include such instances. What makes OpenOffice.org's XForms support so powerful is its combination with OpenDocument. While the data and logic of a form can be expressed using XForms, the visual representation of the form can be expressed by OpenDocument. This means that the same XSLT stylesheets that are for instance used to print a custom schema instance with OpenOffice.org can be used to create a visual representation of a data entry form.