Major Experiment Fails. Before the ink was dry on ISO 8879, DND had launched its first "SGML" project. The scope was breathtaking: an integrated content management and publishing system that provided an advanced collaborative working environment for a departmental publishing group. Even when compared to modern content management systems, this application offered an impressive array of features. The initiative, as would be expected, encompassed a major data conversion effort, an effort that ultimately became the birthplace of the OmniMark language. The resulting mainframe-based environment may well be the first "content management system" ever constructed. Despite its avande guard qualities, or perhaps because of them, this initiative ultimately ground to a halt. The reasons for its failure are difficult to determine precisely, after so much time, but the following doubtless contributed to its downfall:

Outcome. The system was eventually decommissioned and, quietly, the rapidly aging mainframe hardware infrastructure was dismantled and the storage units destroyed. A $17 Million dollar investment and a large volume of converted data simply vanished.

Recollection. Returning to the Headquarters in 1989, the author of this paper entered the informatics division during the persecution of the guilty and the celebration of the uninvolved phase of this project. An entire generation of technology managers within the department had, at a turn, become declared enemies of SGML and document technologies in general.

A Time for Reckoning. At the time the repercussions of this initial disaster were being digested, the Department was prompted to look at the problem of technical information management a second time and this time in the context of a "flagship" weapon system. After the CF18 multi-role fighter aircraft fleet had been forced to convert over 1 Million pages of technical documentation twice in as many years, questions started to be raised. In this particular case, an initial conversion effort in the late 1980s had to be almost immediately repeated when the vendor for the initial target publishing format declared bankruptcy. At a higher level, a major review of the entire equipment program revealed, to the horror of many managers, that documentation maintenance had become the single largest line item in the annual budget for sustaining this fleet. Questions began to be asked.

Detailed Review Undertaken. Responding to the growing concerns over the annual cost of information maintenance, the department launched a new initiative but this time the focus fell on gathering concrete information upon which management decisions could be based. This review would start with a massive metrics gathering effort to determine how much content was really being managed and how much this was really costing.

The Numbers. For the first time within the Department, and perhaps one of the first times anywhere, a project team was assembled specifically for the purpose of compiling detailed metrics on enterprise information holdings and the change processes operating upon them. This project involved junior staff foraging through massive documentation warehouses, ones reminiscent of Raiders of the Lost Ark, pulling randomly selected technical manuals and documenting page counts, distribution lists, change histories, and general contents. Based on these concrete facts, extrapolations were conducted to determine the enterprise-wide profile. So cautious was the team that the sampling approach and extrapolations were independently reviewed by a statistics team at Harvard University. These results, presented in Table 1, were to be presented to the highest levels of the organization and given the bitter memories of investments in document technologies, there was no room for error or for questions of credibility.

Observations. The costing factors used in this study underwent an unusual level of scrutiny. The cost of each change page was the subject of an independent time and motion study. Parallel research was undertaken in other government departments, including the Royal Canadian Mounted Police, that essentially corroberated the findings. The high-level results of this research is presented in Table 2. Each change page entailed effort in the identification and verification of the change requirement, the drafting of the change (an activity that frequently entailed collaboration amongst contributors), research activities involving frequently inaccessible sources and occasionally direct recourse to the equipment item, and then a tortuous process of review, validation, translation, re-validation, formatting and dissemination.

Exclusions. Due to the magnitude of the numbers being unearthed, the project team elected not to attempt to incorporate collateral research that could be used to further characterize the status quo. Over the course of this study interesting metrics were established for the rate at which change pages were "mis-filed" during the distribution process (12%). It was determined that there were over 100,000 filing cabinets deployed within those units of headquarters that dealt with technical information. Meanwhile, parallel studies within the US military were identifying the high cost of "false replacements" on equipment components due to inaccurate or out-of-date documentation, on the costs associated with unnecessary re-testing of components by equipment manufacturers due to errors or uncertainties in the maintenance process, and on the overall equipment availability profile given the time requirements for routine maintenance and minor problem resolution. Perhaps most striking of all were the metrics gathered on "turn-around times" on publication change requests as the average was determined to be 18 months. All of this information, although highly relevant, and especially so to the operational units, was deemed to be too tenuous for the purposes of a strict calculation of quantifiable payback. Fortunately, the projected payback on modernization investments for the management and publishing of technical information were so large that the "operational benefits" were allowed to stand as "qualitative" benefits.

Outcomes. This business case, which was presented to and accepted by the highest levels of departmental management, identified an opportunity to realize a financial savings in the order of $100 Million a year. The investment required at the time, however, was daunting. The study team documented and costed the requirements for an upgraded departmental network infrastructure, the acquisition and deployment of computing hardware to maintenance and operational units, and the development of a department protocol for technical information management, publishing and exchange. Taken together, the projected investments approached $1 Billion and, somewhat surprisingly and on a distinctly incremental basis, these investments were in fact undertaken.

Second Experiment Fails. Emboldened by the results of this quantitative research, the headquarters unit that had historically played the role of managing and publishing technical documentation set out to implement the solution to rule all solutions. And once again, the first investment fell upon the identification, acquisition and implementation of a state-of-the-art content management environment. The combination of ornate technology and a "service bureau" mentality could only have been undertaken with reference to a payback model that offered some quick returns - the preverbial low-hanging fruit. This initiative sought to capture the savings associated with the automation of formatting and the streamlining in the administration of change pages. But as could have been easily forecast, the mixture of high early technology investments with a centrist service model led to a second failure. The equipment management teams simply withdrew their support and, in a number of high-profile cases, elected to outsource the publications management activities to industry rather than rely upon the headquarters unit.

Outlook in 1993. Given the downfall of two major initiatives that sought to bring under control the now well documented costs of the department's technical information holdings, the prospect for any new initiative in this area did not look promising. Undeterred, however, the department mobilized for a third assault.

A Review of the Numbers. A detailed review was undertaken of the research conducted in the 1991/1992 timeframe. The cost and potential benefits calculations were given particular attention and new findings, largely imported from the experiences in the first generation of DOD CALS implementations, were added into the mix. The results of this review were made interesting by virtue of the fact that the CALS mandate was such that the scope of analysis was broadened to include all aspects of materiel management including logistics planning, engineering management and training (in addition to publications management). One outcome of the review saw the costs that had been attributed to publications management largely validated. A second outcome of the review was more significant in that it illustrated two important facts:

The departmental understanding of the technical information management problem was forced to make a serious adjustment. The annual cost of maintaining the technical information holdings was in reality higher than had been previously acknowledged and the centers of those costs were now clearly visible. In essence, annual equipment information costs for the department stood as follows:

A number of considerations arose during this review. Among the most obvious was the limitation of the concept of "page" as measure of information volume. It was found however that despite its limitations, the "page" could not be easily replaced as a measurement unit. Instead, the "page" or "change page" came to be understood as a physical instantiation of a complex matrix of information sources. Consequently, the "change page" became a distillation or representation of a much more elaborate process and therefore came to be the embodiment of a greater set of costs. Whatever weaknesses this approach may have in specific contexts, it remained a useful measure when applied at the equipment program or the departmental levels.

The Real Problem. It had become clear, by this point, that the real problem lay in the fundamental disconnect in the way that information for equipment fleets was created and managed. On one side, equipment information was maintained within the context of one or more breakdown structures associated with one or more equipment components. As was typically the case, this was complicated by the fact that a variety of suppliers were responsible for the creation of the information for different equipment components. The scope of this information was typically restricted to engineering and logistics planning data which was maintained in a mixture of databases and engineering tools. On the other side, operation, maintenance and training information was produced and maintained as documentation items and these documentation items were not generally produced or maintained with any close fidelity to the structure of the logistics or engineering information that in many ways served as its source. On a grand scale, the problem was recognized as one of content silos. The problem is illustrated in Figure 1. A solution to this problem would need to introduce a method for managing documentation items in such a fashion as would enable the integration of all content into a single management framework upon which automated publishing and dissemination processes could run.

The Real Solution. The missing ingredient upon which any real solution would rely was a mechanism for establishing an integrated information architecture as a physical, as opposed to conceptual, component within an integrated solution. The governing information architecture needed to subsume the two domains conceptually and then provide a single physical instantiation that would allow all information resources to exist, if only for a time, as a single integrated information set. This information set would need to be portable amongst potential suppliers and it would need to permit the orderly decomposition of the content into a separately managed components. On one level the solution would demand a high degree of "abstraction" so that one solution framework could be applied in an infinite number of different circumstances. This would avoid the proliferation of different solutions to the same basic problem. On another, more physical, level, the solution had to provide a simple, relatively low-tech, approach to implementing the essential components of the solution so that expense and application sophistication would not thwart progress as it had in the past. The essential solution is illustrated in Figure 2. The solution depended, on both levels, upon the introduction of Standard Generalized Markup Language (SGML).

One Structure to Rule Them All. Through a long series of iterations, a core Assembly Information Architecture was finalized and validated. Initial versions suffered from many of the weaknesses associated with first generation SGML implementations: inefficient data structuring models, out-of-control semantic complexity, impractical application processing scenarios and grossly inadequate documentation. An aggressive program of pilot projects, each seeking to field an operational information product instead of completing yet another technical prototype, soon exercised those ghosts. Pilot projects included implementations in each service element (Army, Navy and Airforce) and one part of this program was aimed at winning back the stakeholder community. Although the impracticalities of the initial attempts caused a brief stumble in this process, it was not long before a clear and simple information architecture was stabilized. Figure 3 illustrates the general structure of the Assembly Information Architecture.

Architectural Features. A number of features are worth pointing out in this architecture. Firstly, the highest level of the structure specifies that the information architecture encompasses more than the information model. It also includes the documentation that must accompany any information model for it to be genuinely useful. The information architecture developed under the DND CALS Office was supported by over 1,000 pages of documentation including demonstration scenarios that illustrate exactly how the model was to be tailored to meet specific requirements. A complete set of "sample data" was supplied which, in being integrated with the demonstration scenarios ensured that the documentation could be more than just words - it could be seen. The solution architecture also included detailed definitions of the main processing scenarios that would be associated with implementing the solution. Again, these processing scenarios were supported with application components that could be used to run complete end-to-end demonstrations that would actually produce the types of information products needed. Going still further, these application components were far from simple "demo code", rather they provided a substantial amount of functionality so that individual equipment programs could tailor the applications to meet their requirements in a remarkably short period of time.

Distinguishing Qualities. It had been learned, largely within the DOD CALS initiative, that the specification of a standard effectively solves nothing. Conversely, as DND had so gloriously proven, large scale technology investments in all encompassing solutions will be just as ineffective and frequently far more disruptive. The middle ground was found to be occupied by a "solution template" that set forth a suite of protocols governing information creation, management and exchange, that provided a substantial array of start-up capability, and that could be adapted effectively and inexpensively to solve problems for actual equipment fleets. The Assembly Information Architecture is therefore closer to a "solution template" than to a recipe for implementation.

Adaptability. This solution template also foregrounded one more critical quality of the information architecture - adaptability. The Assembly Information Architecture assumed that adaptability would be a defining quality of any effective solution so accordingly the architecture made critical distinctions between the structures in which information is created, the contexts within which it will be managed and the organizing structures that will be used to plan and execute processing scenarios. All organizing structures, including the governing Assembly structure, are supported by a library of information building blocks which could be used to constitute new organizing structures. By a similar token, the building block library could also support the modification of individual building blocks to produce approved, equipment-specific variants. It could even support, and openly encouraged, the development of new building blocks to support specialised needs. The intention throughout was to avoid the circumstance in which many organizations found themselves wherein slight differences in requirements gave rise to multiple, often competing, solutions. Instead, it was argued, all equipment teams would be enabled to work within the framework and to leverage what was common without sacrificing their ability to address their oun requirements.

Information Model. Given the scope set out for the architecture, that of supporting the definition, creation, management, publication and exchange of all technical information associated with any equipment system, it comes as something of a surprise to learn that the Assembly Information Model, as a Document Type Definition (DTD), contains less tags than the HTML DTD. This terseness stems from the fact that the concept behind the information model is fundamentally simple. Assemblies represent the universal buidling block for modelling the structure of any equipment system, including its various physical, functional and abstract decomposition schemes. Assemblies also permit information objects to be associated with the description, operation, or servicing of any assembly. Using this simple replicating structure, the assembly hierarchy can be used to represent equipment systems, or aggregations of equipment systems, of bewildering complexity. The availability of delivery structures then allows this model to activate support for a wide range of different information products including IETMs and various online renditions, as well as printed manuals that conform to multiple specifications. The intent of the overall architecture, from the perspective of processing, was to stage a sequence of simplifications in each case so that processing applications were required to deal with as little complexity as possible for the task at hand. In terms of capturing content for different equipment systems and in implementing graceful support for a massive range of output information products, this information model has been proven to be remarkably robust. When it is considered that, in the five years since the underlying DTDs were baselined, fewer than 10 modifications have been made, and that at this same time dozens of implementations were successfully completed, the effectiveness of the solution cannot be questioned. Figure 4 illustrates the general deployment framework for the DND CALS DTD.

Validation Projects. From the period of 1995 through to 1997, a series of validation projects were undertaken as part of the finalization of the overall Assembly Information Architecture. In several cases, specific technology integrations were undertaken to refine the information model and associated application components within the context of interaction with mainstream information management tools such as Lotus Notes, PC DOCs, and Documentum. In other cases, information migrations to SGML were undertaken for different equipment systems so as to determine whether or not different equipment types would give rise to new requirements for the information model. Under this guise, implementations of integrated information management systems were completed within the context of tactical command and control systems, aircraft, naval anti-aircraft and anti-missile defenses, transport and combat vehicles, protective clothing, computing platforms, network environments, and software applications. It is unlikely that any SGML-enabled solution ever underwent the breadth and depth of validation undertaken for this integrated solution.

Shared Applications. Once the validation period was concluded and a final baseline was established for the Assembly Information Architecture, a period of frentic investment commenced on a suite of shared applications. This investment, tellingly, was not sponsored by the DND CALS Office at all, but rather by a community of equipment programs that had determined that their common interest lay in coordinating their requirements and sharing a jointly funded suite of advanced management and publishing tools. These tools effectively elevated the "demonstration capability" associated with the baselined Assembly Information Architecture to a production status. The tools encompassed the implementation of full support for bilingual loose-leaf printed technical manuals, a number of IETM products, the ability to publish content to an intranet or extranet environment in accordance with the government online standards, and even a fully functional web-enabled content management platform. Proving the value of an effective standard, these different equipment systems have been able to build and share application components continuously since 1997.

Implementation Scope. The Assembly Information Architecture and the associated solution components have been implemented within the following main equipment systems:

Taken together the information holdings migrated into the target Assembly Information Architecture represented the following in terms of quantities:

Information Rationalization. By far the most important consequence of the implementation of the Assembly Information Architecture, from the perspective of benefits realization, was the elimination of information redundancy both within the publications and between the different content silos that had historically existed. On average, the elimination of information redundancy, which was sometimes termed "Information Rationalization", reduced the physical size of the information holdings associated with an equipment item by over 55%. When it is considered that associated with a major equipment system will be several hundred thousand "pages" of technical information, the magnitude of the impact becomes clear. The effect of implementing the Assembly Information Architecture was the reduction by over half of the amount of information being modified, managed, translated, reviewed, and published. It is almost equally unbelievable that, prior to this "information rationalization", these equipment programs were literally expending funds modifying and translating the same content often dozens of times a year. Apart from the costs, the ill consequences of information redundancy on matters pertaining to the health and safety of the equipment operators and their passengers should have raised serious concerns.

Streamlined Change Processing. Through the introduction of enhanced automation in publishing and improved information management practices, the Assembly Information Architecture reduces the cost of each change "page" by over $150. These savings were realized by enabling a collaborative change identification, authoring, review and translation process and by fully automating the formatting of change packages.

Benefits Realized To Date. When the benefits realized for the main equipments systems are aggregated, a picture emerges of a long sought after return on investment.

Taken together, the quantifiable benefits associated with the rationalization of information holdings and the incremental streamlining of the information change process equate to an annual savings of $100,775,000. Should any quantification be applied whatsoever to the qualitative or operational benefits of the Assembly Information Architecture then this figure would grow substantially. It should also be noted that during 2003 a number of new equipment programs have adopted the departmental solution thereby increasing the adoption momentum and enhancing the benefits calculation at the departmental level.

Lessons Learned. Stepping back from all of the details, a participant can become an observer and from the perspective afforded a number of lessons emerge: