Although a common aim of XML standards is to allow process interoperability between companies, many financial enterprises are spending more effort on deploying XML standards internally, with proprietary extensions. This paper discusses some of the reasons why, and discusses practical issues that are involved in taking external standards and making them work as an internal solution.

Why standards? This may seem to be an odd question to pose at an XML conference, but since this paper discusses how and when to use standards to fast track enterprise development, it is important to understand the business cases for standards in the enterprise technology stack. Standards are sometimes promoted with almost religious fervour by technologists, but the "standards versus proprietary" argument is never a simple choice for the business decision makers who ultimately have to sign the cheques.

For business decision makers, the question is always one of what value any technology, standards-based or proprietary, brings to the business equation. Does it deliver immediate cost savings? Does it provide genuine leverage in negotiating deals with vendors? Some technologies offer theoretical leverage via a "levelling of the playing field", but that only works if there really are two or more vendors that the enterprise is prepared to deal with. Business stability and continuity are as important in judging vendors, and sometimes more important, than whether a vendor produces standards-based products or not. If there is only one vendor that satisfies an enterprise's non-technical selection criteria, then the question of "standards versus proprietary" becomes moot, because there is no such choice.

That said, for the many cases where there is a choice, where does the business value come from standards? Maximum value comes when you can install a new piece of hardware or software, and it just runs in concert with your existing system, without any integration effort. Even better if it also requires zero training effort for your staff, because it follows user interface standards that your staff are already familiar with. Note that this (idealised) scenario does not distinguish between open/public standards and proprietary "vendor" standards. The bottom line is that enterprises need to manage their total expenditure, and that includes total expended effort. Some freedom of vendor choice may be a nominal business requirement, but that cannot be at the expense of too much integration effort. There is a tendency now for enterprises to develop strategic partnerships with particular vendors in order to manage total cost of ownership at a direct business level, rather than placing faith in an expectation that standards and vendor competition together will guarantee lowest (actual) cost of ownership.

For data standards, e.g. XML standards, the strongest business case is when a standard enables the use of off-the-shelf applications at a cost that is lower than the cost of developing internal equivalents (and remember that internal applications may not require the full functionality of the off-the-shelf applications). You also want those applications to be maintained and supported by the producing companies (or perhaps by 3^rd party support specialists), rather than by your own support staff.

XML itself is well supported. Related specifications like SAX, DOM, XSLT, W3C XML Schema, and SOAP are also well supported. XQuery and XForms should also be well supported when they become full W3C recommendations. So in terms of generic "horizontal" XML, there are enough off-the-shelf applications to justify enterprise use of XML.

Some specifications serve a vertical market, but a large and broad vertical market. This is the case with electronic business using extensible markup language (ebXML) and universal business language (UBL) serving the needs of general commerce. Although commerce is a vertical market, these specifications address horizontal needs of the commerce community, and so they are likely to receive good application support, particularly given their UN/CEFACT backing.

Finance is a vertical market which overlaps with commerce. It comprises all of the activities that companies and governments undertake to raise investment capital. This includes business loans, government and corporate bonds, and financial markets. Finance is an area that spends billions of dollars/euros/pounds on IT each year. There are a number of financial XML specifications, including:

The curious thing is that to date, none of these XML standards has the level of off-the-shelf application support required to automatically make the value case for its use. Why is that? To some extent, it is historical. Finance enterprises are using XML standards internally, but still building their own systems for doing so. Finance has a history (from the 1980s and 1990s) of creating its own IT infrastructure — some argue that banks are just large IT firms that also have branches and automatic tellers — in spite of the cost. That came about because margins were good, and so the cost could be sustained. It is hard to say whether the cost was ever justified, but as banking and finance are service industries, it can be difficult to quantify a finance organisation's differentiators, its unique selling points. This has led to a culture of secrecy, where many finance organisations feel they can justify building their own IT systems on the grounds that it helps stop their competitors from copying the service(s) they provide.

However, the financial markets entered a downturn in mid-2000, and this was compounded by the tragedy of the World Trade Center in 2001. The consequent reduction in profits has forced many companies to rethink their IT approach, and look for ways of reducing recurrent costs. What is happening so far is not a complete change of philosophy, but a reduction in the scale of internal IT development.

The timing of the downturn was unfortunate, because 2000 was also the year when the finance community really started to take an interest in XML, and numerous groups started work on XML specifications to cover particular specialities within finance. So by the time the specifications were ready for implementation, the money available to develop applications had dried up. Companies were busy planning survival strategies, and application development with new XML data formats was out of the question.

The situation has improved in 2003, but it is still obvious that major players in the finance area are not all jumping to support XML standards in the products they supply to customers. The reasons for this are of a business/cultural nature, rather than of a technical nature. The biggest player (the incumbent) in any market typically has nothing to gain from standards (applied to their product delivery), and a lot to lose. The number two player also may not have a lot to gain, particularly if the other players are much smaller. So it is usually number three or four players who will be the first to implement a standard in a product offering, on the basis that what they lose to other players of the same size (or smaller) should be less that what they gain from the larger players. The larger players move only after their larger customers move (or threaten to move) to their smaller but standards-based competitors.

It is important to realise that big players in a market won't always be forced to apply standards to their products. If they can supply sufficient valuable functionality and reliability, at a suitable cost point, that can be enough for their clients to decide that alternative vendors are not an immediate priority, and hence that standards are also not an immediate priority.

The upshot is that neither off-the-shelf applications, nor financial product offerings, are driving the use of XML in finance at present. If these are not drivers, what is? A large part of the (perceived) value of a finance organisation is its knowledge of the financial markets. Such knowledge is a differentiator, because there is no strict agreement on how financial markets behave, and hence no agreement on what the best investment strategies are. Finance organisations maintain large data models which encode their particular view of the markets and how they work — what the data is, what the relationships between data items are, what the processes are which underly the lifecycles of financial transactions. These models are expensive to maintain, perhaps too expensive given the downturn and lower profit margins in the finance industry. Further, there are reasonable grounds to believe that the finance industry is not seeing a hiccup, but a seachange. Margins are unlikely to creep back up to where they once were, so companies instead have to find new strategies to allow them to live within their means, and still show decent profitability.

It is in the creation and maintenance of these data models, and the applications which implement them, that XML standards are attractive. Although companies have distinct proprietary aspects to their data models, they are modelling the same financial markets as their competitors are, so the majority of items in the competing models are more-or-less the same. Public standards, to which a company and/or its competitors have contributed, will tend to contain these majority common items, as those are items that the participants will have been able to agree upon. So the definitions contained within a public XML specification can be a good starting point onto which to build a proprietary enterprise model.

Do companies need proprietary models, in this age of standards? Yes. Companies have certain data that they need to store and/or model, and for large enterprises, it is unlikely that any set of standards will adequately cover the full range and depth of data. Companies cannot afford to limit their ambitions just because of the lack of suitable external data standards. So the need for proprietary models will always be with us.

What can you take from an external XML standard? Firstly, you can take the vocabulary (element/attribute names and associated datatypes). Just having a set of names is an important starting point, because naming is still the most difficult problem in IT, and it consumes an enormous amount of (expensive) meeting time. Starting with a large, consistent, and agreed-upon set of names gives an enormous head start compared to starting from scratch. Datatypes are also very important, because transforming element/attribute names is easy, but the same is not true for the values between the start and end tags, so having the right datatypes is also a big advantage. From there, you can take some or all of the structures as well. The closer the structures are to the needs of your company's data model, the more you save. Knowing your requirements for vocabulary and structures beforehand allows you to then review the available standards and find the content that you need to seed your data model.

Financial XML standards tend to focus on a particular niche. If they tried to cover the whole of finance in one hit, they would never produce anything within a reasonable amount of time. This means that no one standard is likely to cover your enterprise needs. So what the strategies you can use?

Integration of XML formats is easier than integration of textual and/or binary formats, because of the commonality that XML imposes. However, that doesn't make integration trivial, just easier. In particular, XML formats that use different schema languages (most likely DTD & W3C XML Schema) can only be integrated if you convert to a common schema language and then integrate. Even when using the same schema language, the formats can vary greatly in the way they use the schema language, as demonstrated by the financial MLs mentioned earlier (Section 4, “Financial XML Standards”).

As an aside, arguably ISO's document schema definition languages (DSDL) would allow integration without merging of the schema languages, but it doesn't yet have a prime-time slot in enterprise production environments. Maybe it will one day, if Apache implements it in Xerces at some point, and if developer tools then support it as well. That said, context-sensitive editors and forms generators for multi-schema DSDL documents have not yet appeared on the market, and may never, and that may be reason enough not to take the DSDL approach.

If you do your integration of financial schemas by hand, then when the next version of one of the schemas is released, you could have a large maintenance and re-integration job, assuming the updated version is worth moving to. The sheer cost might lead you to avoid updating, even if there are some worthwhile improvements. For schema conversion, if you use a tool to do the work (e.g. a DTD to W3C XML Schema converter), that helps a lot. You still have the re-integration effort, but at least the schema conversion task is reduced to a zero-cost push-button item.

As a practical example, in a project where the FpML 3.0 DTD was used to provide the starting point for a proprietary W3C XML Schema, Tibco's TurboXML was used to do the DTD to Schema conversion. It did a good job, converting the DTD into a consistently structured Schema. However, in the FpML DTD, datatypes were included as attribute values of the elements, and hence were converted into attributes in the Schema. Luckily, because the generated Schema was consistent in structure, it was a short exercise to write an XSLT script that, when applied to the generated Schema, set each datatype correctly from the value of the datatype attribute, and then deleted the datatype attribute. These are the kinds of tasks you need to do when integrating XML standards. XSLT is a very useful tool for manipulating W3C XML Schemas (particularly if their structure is consistent). DTDs are harder to manipulate, unless you are skilled with Perl. For finance, W3C XML Schema is the normal choice of schema language, because of the datatypes it provides.

The thing about integrating two or more formats is that you can do it, but with each new version that you want to upgrade to, you have to redo the integration. Although you can do it by hand once, and it may seem worth the effort, you might not be so happy to repeat that same effort every time an improved version of either/any format is released. So, once you have a grip on how you want the integration to work, and what you want the result to look like (this usually implies starting at least some of the integration by hand), there is benefit in automating the integration process. You won't see that benefit the first time you integrate (you may only see an extra cost), but the effort can pay for itself from the first time you have an update to any of the component standards.

Version management is easy to ignore the first time you do an integration, and it is an item that project managers are all too likely to cut out of the project plan (to reduce the inevitable overrun). It gets cut because it doesn't effect the delivery of the first integration deliverables. Unfortunately, it isn't necessarily possible to include proper version management for later deliverables, because the project manager then would have to explain why something wasn't done about it during the first delivery cycle. This can be a vicious circle. There isn't a simple or easy solution, either. You just have to be aware that this issue exists, and ignoring it during your first integration may not save you money in the long term. Still, modern businesses often don't aim to save money in the long term, as that is too speculative, and those gains might never be realised if the company goes out of business first. So businesses save money in the short or medium term of the current delivery cycle. This is a valid business choice, but be aware of the limitations of delivering the first cycle as cheaply as possible.

Even once you settle on a single schema language, there will be integration issues based on the differing "styles" of the schemas. Every XML format seems to have its own style, and there always seems to be a good reason why each new format must have its own style.

When integrating multiple XML standards/specifications, one option is to integrate the formats without integrating the styles. This requires the least effort, but can make the final format hard to use, because users (particularly developers writing generators or readers for the format) will constantly have to refer to documentation in order to know what the convention is for each different part of the integrated format. So, this approach saves money in early development, but at a continuing cost to users (e.g. application developers) thereafter.

If one format covers a significantly larger proportion of your needs than any of the others, it can be useful to copy the style of that majority part for the whole integrated format. Integration then involves reworking the minority formats into the style of the majority format. This is more work in integration, but the pay-off comes for users, who get a consistent and predictable format to work with.

For example, there is no agreed standard on how to construct XML element and attribute names. This is at least in part because the W3C itself has never been able to agree on such naming rules. Some specifications have "element-names-delimited-by-hyphens", some have "elementNamesInLowerCamelCase", some have "ElementNamesInUpperCamelCase", etc. Although this sounds like a small problem, naming remains the hardest problem in IT, and integrating specifications whose element/attribute names have different case conventions can be very tedious. If you don't align the cases to a single style, the integrated specification is difficult to use, and doesn't project any sense of quality. Writing algorithmic rules to modify case conventions is often difficult or impossible, as picking the word boundaries 100% correctly can be unreliable. The best you can do is to set up a manually maintained translation table (e.g. in XML), either for all names or as an override for names that are not correctly converted using an automatic algorithm, and then drive your element/attribute renaming code from that table.

Another major issue is global vs. local element scope. Some XML specifications make full use of W3C XML Schema's support for locally scoped element names, so that elements names can be re-used in different contexts with different content models. Other specifications continue to use the DTD-style approach of having globally unique element names. If you have to integrate a locally-named Schema so that the result is globally-named, then you again will need to have a renaming table.

Another major stylistic choice in schemas is whether to use element content versus attribute content. FpML and MDDL both have a policy of using element content almost exclusively, with attribute content reserved for attributes that are globally applicable to elements. An alternative design policy is to use element content for content that can be sensibly displayed to (at least some) humans, and attribute content for content that is only of interest for automatic processing. However, the "no or little attribute content" rule seems to be winning for data-oriented financial XML, and is not a difficult XSLT programming task to convert attributes in an XML Schema into equivalent child elements.

You also may not require certain features for your enterprise format. For example, you may only want data dictionary definitions from an XBRL taxonomy Schema, and not need any of the XLink information. You may want to re-use market data definitions from MDDL, but not need the shorthands like MDDL inheritance which can reduce the size of instance documents, but at the expense of complicating the Schema. In that case, the best thing is come up with a process (manual or automatic) for stripping out the features you do not want before you perform your integration. For an enterprise format, there is little point in including features that have no enterprise buy-in. The designers of XML standards are in a different position, they have to think about the requirements of all user firms. An enterprise format does not need to be inclusive, and unused features can be removed.

Some financial W3C XML Schemas are now making use of substitution groups, as an alternative to DTD-style choice groups. The biggest practical problem with substitution groups is that many application developers have made themselves dependent on Schema "compilers", particularly Sun's JAXB, and substitution groups are one of the W3C XML Schema features that is not supported by JAXB. There is a historical reason for this. Originally, Java provided a binary serialisation format for JavaBeans. This was intended for short term serialisation, e.g. for network transmission of objects. However, people started using it for long term serialisation as well, storing their serialised JavaBean as binary large objects (BLOBs) in relational databases. The problem was that there was no provision for version management, so if you updated your JavaBean class, you would break serialised instances of that class. The problem was so bad that one of Sun's senior JavaBean developers once apologised to me for Java serialisation. He also apologised for applets, but that is another story.

Sun's Swing team flirted with using XML for serialisation, but some of the Java folks in Sun realised that there would always be problems in serialising complex object graphs no matter what format you choose. Instead, they realised that if you reverse the process, and create Java classes from one or more XML schemas, you end up with a set of classes which are inherently serialisable. The upshot is that JAXB compiles XML schemas, but only enough to support Sun's object serialisation requirements. Some W3C XML Schema features are not supported, and may never be supported. This has become a support headache for anyone who dares to produce an XML Schema that JAXB cannot compile (even if it was created before JAXB existed), as developers complain about the lack of JAXB compatibility, without understanding what the broader issues are, nor why their lowest-common-denominator tool is now causing as many problems as it solves.

The practical implication of this is that it is now common for enterprises to replace substitution groups with choice groups when integrating external W3C XML Schemas into their proprietary Schemas, to make it possible to use JAXB and similar tools. It is not an insignificant change to make, nor to keep updated, but it is a task that can be automated, and taking this particularly pain during schema integration will make life easier for some application developers.

This integration approach was used to create a proprietary messaging format. Approximately half of the content came from FpML 3.0 (and this is currently being upgraded to 4.0). The DTD was converted into W3C XML Schema using Tibco's TurboXML and a custom XSLT post-processing script. A quarter of the content was manually translated from ISO 15022 version 1 into XML Schema. The remaining quarter of the content was proprietary.

For the project, a set of messages was required. Although one approach was to have a single Schema covering all messages, a better approach is to have a separate Schema for each message, so incorrectly delivered messages are rejected at validation time. This led to question of how to create the subset Schema corresponding to each message. Cutting and pasting from the main integrated Schema was tried initially, but it was too manually expensive, and too error-prone.

The solution was to once again use XSLT. The process was as follows:

This approach of using one XSLT to build up a picture of what needs to be done, before using another XSLT script to perform the required edits, works very well in manipulating data-oriented W3C XML Schemas. Trying to do everything in a single XSLT script is rarely helpful, and just leads to complicated and unmaintainable scripts. That may be less true of XSLT 2.0 than of XSLT 1.0, but since Schema manipulation of this kind is done as an off-line process, not a real-time process, there is nothing to be gained from trying to do everything in a single pass.

Although this process is workable, it requires a continuing in-house XSLT expertise, as some modifications to the integration process will be required whenever updated schemas introduce new issues. For the project in question, which did not have any permanent XSLT developers, 3^rd party data modelling tools are now being evaluated as an alternative for maintaining the proprietary messaging format. Nonetheless, the initial XSLT scripts were invaluable for quickly establishing what could be done, and for understanding what 3^rd party tools would need to do to genuinely provide a long term solution.

An alternative to generating your enterprise XML format from external XML formats is instead to generate it from a data model. Such models are available commercially for finance, and are interesting because they can be used as the basis for generating database schemas and application code as well as XML schemas. Particularly interesting will be when the ISO 15022 version 2 repository has a useful amount of content, as the UML model will be available as well as the Schemas (although there is expected to be a charge for the UML model). This may make it attractive for enterprises to use ISO 15022 as the basis for a consistent data model that spans the messaging schemas through to the application objects through to the database schemas.

Starting with an external standard or two is a good way to kick start development of internal XML data or messaging formats. However, it doesn't come for free. It has to be managed properly, if you want your internal format to be able to benefit from new versions of your donor standards. The difficulties of integration and management, however, are far less than the difficulties of building a winning enterprise data/messaging format from scratch. There is often a compromise between standards and proprietary development that gives the best return for your enterprise development dollar.