XTech 2005: XML, the Web and beyond.
Since the rise of Wikipedia, the collaboratively authored encyplopedia, more and more people are turning a curious and sometimes skeptical eye towards "wikis". According to Wikipedia itself, “A Wiki or wiki (pronounced "wicky", "weekee" or "veekee") is a website (or other hypertext document collection) that allows a user to add content, as on an Internet forum, but also allows that content to be edited by anyone.”(emphasis added)
The definition suggests that the defining characteristics of wikis are hypertext and collaborative authoring. We feel that there is a third key ingredient that is not widely remarked upon: context. Wikis allow collaborative authoring in the context of a hypertext document set. At Blast Radius we believe that this issue of context is actually key to the usability of a collaboration environment. In fact, we view wikis as key examples of a document-contextual collaboration paradigm that we call DocumentSpaces.
Nevertheless, wikis are by no means perfect. They tend to have a very basic structure of HTML files joined by simple one-way links. More advanced structured markup concepts are often missing. Examples include documents composed of other documents, bidirectional links, typed links and annotation overlays. We propose that by adding XML concepts to wikis they can become substantially more powerful and ubiquitous. This paper describes how to add XML structure to wikis in order to more fully realize the DocumentSpaces vision.
Wikis were invented by Ward Cunningham, a man associated with many "big ideas" including object oriented programming, software patterns and extreme programming. Cunningham invented the wiki as a place for the community of object oriented programmers to collaborate on the development of patterns for object orientation programming. But he was unsure how to organize the content. Rather than define a top-down organization as is typical on most sites, he just created a site where anybody could create pages and where any person could modify any page. He claims he did this because he did not know how to organize the information himself and preferred to let the community of content creators define the organization through trial and error. This bottom-up approach has since been tested in a variety of Internet-based projects and been found to create surprisingly rich, robust and logical organizations of documents.
The Wikipedia project has more than half a million articles ranging from the ordinary ("Icebergs") to the bizarre ("the Spork")
In order to achieve widespread usage of the wiki, Cunningham had to keep it simple. The first wiki was simple in every way. It had few lines of code. It had a very sparse, simple user interface. It had few functions. To employ a favourite phrase of Cunningham's, it was the simplest collaborative hypertext authoring environment that could possibly work. Despite the fact that wikis must evolve to become mainstream, they can never become so complicated that the user interface becomes a barrier to contribution. As we extend wikis with structured authoring, simplicity becomes even more imperative.
Wikis are all about Hypertext. Hypertext is, of course, the dominant form of content on the Web. The defining characteristic of hypertext is its non-linearity. Where the very form of a book imposes a requirement to order things, the online medium has no such constraint. Furthermore, those on computers seldom wish to read novels or treatises. They almost always want to find or contribute a tiny bit of information. For our purposes we can call this atom of information a topic (XML/DITA terminology) or page (wiki terminology).
Links are the glue that tie together the pages in a wiki. Although they are rendered as standard HTML links they are typically authored using a syntax that is specific to the particular instance of the wiki. Each page has a name, and other pages link to them through their names, not their URLs. The main benefit of this is that names can be selected so that they are short and mnemonic. Rather than spend time cutting and pasting URLs, users typically form links using some textual convention such as CombiningMultipleWordsWithCapitalLetters (called WikiWords) or [putting a phrase in square brackets]. In either case, the visible text in the page is the name of the link target. This makes linking a one step-process, whereas most hypertext systems require the author to stop typing to copy and paste the link URL into a dialog box. If you attempt to link to a page that does not exist, the wiki creates it. This is another example of the wiki accelerating important actions and encouraging contribution.
People often say that wikis allow "anyone" to edit pages. Sometimes "anyone" means "anyone with access to the wiki's URL" and sometimes it means "anyone with proper authorization." The key is that pages do not have traditional owners. Readers are encouraged to also become contributors. According to Ward Cunningham:
Someone not familiar with authoring may have an idea, and the idea is a paragraph's worth of idea. They would write an editorial for a magazine, except a paragraph is too small. To write for a magazine, they would have to establish the context, say something important, say it in a way that a wide variety of people can understand it, and then bring it to a close. That's more than most people want to invest. But if you're reading somebody else's work, and you think, "Yeah, but there's another point," being able to drop in a paragraph that says, "Well, yeah, but there's actually this." There's an awful lot of counterpoint, the "Yeah, but..." kind of thought, on wiki. Discussion groups do the same thing, but with discussion groups it all gets lost.
Just as a reader might ask a question or clarify a point at a meeting, they can do so also on the wiki. Unlike a question asked of an author of a traditional book or web page, the question can be placed in the context of the topic itself.
Although the strength of wikis lies in their capacity to elicit feedback, it is also this capacity that gives rise to their sometimes chaotic nature. It is often difficult in a wiki to determine who wrote what, when they wrote it and how the wiki looked when they started writing. Wikis typically use a simplistic form of markup called WikiText. WikiText is roughly equivalent in expressive power to early versions of HTML.
In other words, Wikis tend to lack rich, semantic content structure. This is of course an area where XML could help. We'll get to that below.
Wikis are the most widespread instance of a collaboration model that Blast Radius calls DocumentSpaces. We can see the clear difference between a document format that enables collaboration (such as PDF) and a DocumentSpace (such as a wiki) by comparing the workflow of someone working in each paradigm.
In the traditional collaboration model, we treat a document as if it were a physical object. A workflow system routes it from place to place as a conveyor belt might route a manufactured product. Individuals perform their tasks and pass it along as if they were assembly line workers. Alternately, copies of the document are routed around in an attempt to parallelize the process. Although the copies are bit for bit the same, they still exhibit the property of being completely separate objects. They live in each person's file system or email program until they go back to the originator as a mess of versions with names like proposal-v3-pp.xml.
In the DocumentSpaces model, the document has a location (typically a URL). The document is treated as a place. People go to the place to collaborate. Everyone can see everyone else's collaboration in real-time. If the previous model was like an automobile assembly line, this is more like a garage where expert mechanics gather around a valuable vintage or concept car. The garage is the appropriate meeting room because the car is the reason and inspiration for collaboration. Rather than describing a problem with the design, team members can just point at it. Other team members can more easily understand comments because they are being made in context.
Wikis are DocumentSpaces. They are places you go to collaborate on the creation of documents that capture knowledge. Our XMetaL Reviewer product is a similarly rich collaborative space specialized to the needs of teams of reviewers. We are working with various customers to apply these insights to product development teams who wish to combine the collaborative benefits of wikis with the structure and semantic richess offered by XML.
We believe that new product development (NPD) processes will be a fruitful application area for wikis and other DocumentSpaces. This becomes apparent upon considering the pressures on modern product development teams.
Given increasing levels of global competition, it is absolutely incumbent on product companies to generate new products ever more rapidly. On the one end of the spectrum we have a company like Google that constantly rolls out new features to its service. Financial services companies are similar. At the other end of the spectrum are auto or aircraft manufacturers who simply cannot release new products as often but are still under pressure to minimize the product development time in order to beat their competitors to market. This means that product teams cannot work in a very serialized waterfall model. Engineering needs to prototype even as marketing builds a business case. Technical writers need to start writing documentation while engineers are still developing the product.
At the same time, it is not sufficient to release poor products faster. Products must be innovative, carefully researched, cost effective and appropriately timed. This requires collaboration across the development team. In order to do their jobs, technical writers must know how their readers think. Therefore their input should inform even the earliest product development decisions. Similarly, support technicians learn through painful repetition which product features cause customers the most problems.
Let's use an idealized high tech product development workflow as a basis for a comparison between current and proposed ways of working. In our perfectly collaborative world, content would flow through the product development process seamlessly.
Before the product is even conceived, knowledge about the market would be captured in the content/knowledge repository. Sales people would record product requests from customers. Marketing folks would keep tabs on what the competition is doing. Engineers would make notes about relevant advances in technology. Technical writers and support technicians would share their thoughts about issues with comparable products in the product line. At some point, a product manager ties together all of these disparate threads into a product development proposal. At that point the team refocuses on the task of defining the product's features. The end result of this next round of collaboration is a requirements document.
The process of creating the requirements document is inherently collaborative. Sales and marketing people must verify that it is likely to be of interest to customers. Engineers must verify that it is implementable. In a sense, the product manager is just a note taker, augmenting the group's collective knowledge with strategic vision. The technical writer should also keep careful track of the requirements, planning for the writing strategies that will be required to create excellent documentation.
After the requirements document is created, focus shifts to engineering in order to create design documents. Once again, there is likely to be quite a bit of collaborative give and take both within the department and throughout the wider organization. The impact on the product documentation becomes even more explicit, to the extent that parts of the engineering specification may be used as source content for the final end-user documentation.
In an old-fashioned, serialized enterprise, the technical writers may not get involved until the product is almost done. But in today's competitive environment, it is not acceptable for a product's release to await documentation. Writing should be well under way by the time other aspects of product development are complete. This leaves writers with the challenge of getting appropriate input from the engineering team while it is still engaged in actual engineering. There are huge systemic and sociological issues that make this difficult. Technical issues can compound these to the point where collaboration simply does not happen.
Every organization has a variety of bureaucratic hurdles preventing collaboration. Software cannot remove them, but it can be designed so that it facilitates collaboration rather than serving as yet another barrier. Unfortunately, the patchwork software environment deployed at most organizations does discourage collaboration.
The first barrier that is likely to arise in cross-departmental content collaboration is the actual sharing of files. There is a tendency at many organizations to revert to email as a least-common-denominator file exchange protocol. It has, after all, been universally deployed since the mid-1990s. Unfortunately, email has problems that are both legion and obvious:
On top of all of this, individuals are increasingly afraid to look in their email inboxes every morning for fear of what may have come in during the night!
"File shares" have a different set of problems. At the heart of them is the fact that only one person can edit a particular file at a particular time. This means that if reviews are to be contextual (e.g. using Word change-tracking or Adobe annotation marks) and done in parallel, then each team member must review his or her own copy of the document. The result is many variants of the same document that must be consolidated by some poor peon.
The only solution is to have some sort of content management system that is licensed for use by the whole company. Enterprise Content Management systems can meet this need if they are appropriately licensed. Wikis can also do so. Because they are generally free or licensed on a per-server basis, a wiki implemented in one department can be easily accessed from another. Because wikis allow bottom-up ontologies and navigational models to evolve, one department can actually adopt the use of another department's wiki without conflict.
The next barrier to inter-departmental collaboration is often file format conflicts. The product manager most likely uses Microsoft Word, the knowledge worker's default content creation application. Depending on how disciplined the organization is, the engineers may prefer to create documents in any format from raw text to FrameMaker -- with LaTeX, HTML and reStructured Text as intermediate points on the spectrum. If anyone on the team has adopted standards-based writing (typically XML) it is likely to be the technical writer. But even they are much more likely to use a mix of proprietary formats like FrameMaker and RoboHelp.
All of these different file formats introduce friction into the collaboration process. At best, the team wastes time converting files from one format to another. At worst, it lacks even that capability and must resort to a lowest common denominator such as raw text or (depending on the input document type) PDF.
XML is an obvious solution. In particular, the up and coming DITA specification is designed precisely to allow each department to maintain its own specialization of XML and yet exchange information across the enterprise. With smart, auto-configuring XML authoring applications, it is possible to combine the benefits of standardization and specialization.
Once it becomes possible for every team member to access and review the content, the next logical question is whether they can actually collaborate on it. In particular, the types of collaboration that are of interest are as follows:
Unfortunately, these collaborative capabilities are quite rare in general and nearly non-existent in XML-aware tools in particular. This leads people to use non-contextual models for reviewing such as emailed lists of suggestions, marked up printouts or (worst of all) emailed Excel spreadsheets.
These types of non-contextual collaboration are highly inefficient and can lead to miscommunications and lost knowledge. Furthermore, they raise huge barriers between reviewing and authoring.
In an ideal workflow, a team member should be able to go to a particular location in a document, insert a cursor and create a new section that he or she believes the document needs. Another reviewer should be able to comment on the suggestion in-context. The owner of the document should be able to accept that new content into the document with one or two mouse clicks. All of these operations should be logged and authenticated. This is easier in an XML universe because the document structure is well-known and contributions can be managed as annotations or entities.
Most product development teams use very primitive collaboration technologies. When they are co-located, this is sometimes acceptable. But when they are geographically distributed, things can get ugly. On good days they email around bulky copies of files. Then there are bad days where they use faxes and FedEx. In many environments, content management systems help with the problem. But content management systems are, unsurprisingly, designed for managing content, not for enabling collaboration and communication. Over time content management systems are adding collaboration features (e.g. eRoom features added to Documentum), but most are still quite primitive in this regard.
Of the three big technical problems preventing cross-departmental collaboration, XML solves one directly (file format incompatibility). A properly deployed wiki or ECM system solves the second (balkanized repositories), and the intersection of wiki, XML and contextual collaboration technologies can help with the third.
As you can see above, wikis have quite a bit to offer collaborative teams willing to make the shift. That said, they are not perfect. As you will recall, wikis were designed to be "the simplest thing that could possibly work". A variety of innovators have taken wikis into many different directions and now there are dozens of variants on the base idea. Only a few are focused on giving wikis more internal content structure.
We believe that Wikis can benefit greatly from using XML technologies such as DITA, XML Schemas, XQuery and XSLT and XML tools such as graphical XML authoring and reviewing applications.
We feel that there are several DITA features that are relevant to wikis.
Although there are wikis based on Docbook (including some we have helped to develop ourselves) we feel that DITA is a more solid basis for future integrations of XML with wikis.
XML Schemas can be used to ensure that wiki topics adhere to a structure that will make them useful for other processes such as the XQuery and XSLT rendering processes described below. Schemas can also be used to automatically bind forms interfaces to topic metadata fields. This process is not at all trivial because naive schema-driven forms tend to be very difficult to use.
More advanced wikis are starting to differentiate between the content repository underneath the wiki and the wiki user interface on top. XQuery would be a great language to use to communicate from the user interface to the underlying repository.
Many wikis support full-text search. Because few support metadata or more sophisticated content structure, few support advanced search. For those that do support advanced search, it would make sense to implement this feature based upon XQuery. Not only would this ease the implementation of the wiki, it would allow the creation of applications that run on top of the wiki and generate reports and navigation interfaces through queries.
No software applicaion is an end in itself. Wikis are often used for knowledge management wherein the content is only ever meant to be accessed through the wiki interface. But they can also be used for collaborative authoring of documents that will be delivered to customers or internal stakeholders in traditional publishing formats such as PDF or HTML. XSLT can be used to apply a print or Web-oriented stylesheet to topic sets for more formal publication.
Because of XML's rich structure and linking, it is easy to annotate a document with collaborative discussion or metadata without changing the content of the document itself.
It is also possible to make highly granular annotations on the document content. Our XMetaL Reviewer product allows revisions over any phrase or element in the document. These revisions are stored separately from the content of the document and can be considered as annotative overlays. This has profound implications for wikis because it becomes very easy to see which wiki content is "approved' and which is merely "suggested". The approval process may vary from situation to situation but the the concept is nearly universally applied in businesses.
Most of today's wikis have very text-oriented user interfaces optimized for the software developers who invented the concept. A few use richer HTML editors, but of course these are very limited in terms of the semantics that they can express. In contrast, we have embedded XMAX, our full-powered, graphical embedded XML authoring application into wikis. This allows for very powerful semantic markup, forms layout, table editing and multimedia insertion (e.g. SVG and Flash).
Every wiki document is inherently an addressable, unified collaborative space: a DocumentSpace. The wiki itself is a collection of these spaces. Therefore a wiki is a natural repository for the rich collaborative spaces that Blast Radius is incorporating into its products like XMetaL Author and XMetaL Reviewer. We are currently working with a few visionary customers and partners to integrate our collaboration technologies into their wiki platforms of choice.
We believe that by combining rich XML (DITA) structure, collaborative DocumentSpaces and wikis, we can help organizations break down the barriers that prevent them from achieving cross-departmental collaboration. We hope that this will allow these organizations to deliver more interesting products to market more quickly and efficiently.
Paul Prescod
Group Program Manager , Blast Radius XMetaL
Paul Prescod is the Group Program Manager for the XMetaL product. Paul is an experienced implementor of XML-based systems, co-author of the XML Handbook, author of numerous articles on XML and contributor to open source XML tools.