The FBI work had to be based on the latest paper forms for conventional submission. Developing XML representations of these documents required the generation of many XML Schemas, some of which were to be interdependent. However, these were only a small part of the documentation that was needed for the project.

Historically, one of the biggest burdens on the Inland Revenue has been the transcription of data from paper forms into the organisation's back-office systems. In recent years, it has encouraged the development of third-party software systems to automate this process, first for magnetic media and EDI submissions, and now for the FBI project. For FBI, the Inland Revenue wanted to provide a full development package for software vendors, consisting of schemas, documentation and data samples, and crucially, an online test service for the new software systems.

The FBI development process was to be a complex one requiring:

These processes required the involvement of a wide variety of geographically separate parties, many of them outside the Inland Revenue.

For example, business rules for validation of the submitted data had to be made explicit. The rules had to be drawn from a number of sources: systems staff familiar with EDI submissions; domain experts in the Inland Revenue and the National Insurance Contributions Office (NICO); the Electronic Business Unit (EBU) staff responsible for implementing Filing By Internet (FBI); and legal staff charged with updating tax collection practices. Once the business rules had been determined, XML structures could be designed to reflect business requirements. At this stage, a package consisting of XML Schema, validation requirements and sample data could be produced for third-party vendors as initial specifications for their submission applications. The schema package would be subject to review externally by vendors, and internally by domain and legal experts. The review process might be repeated several times while submission and processing mechanisms were clarified. Once a final schema had been agreed, a range of systems would be built:

The timetable for the FBI project was constrained by the need to fit in with existing form submission dates. In the time between the March 2000 Budget, which effectively laid down the requirements for the 2001 employer submissions, and the filing dates in Spring 2001, the FBI mechanisms had to be specified and implemented in full. Given the complexity of the development process, meeting this deadline necessitated a radical new design approach.

The Inland Revenue realised that the extension of their submission mechanisms from manual to magnetic media, to EDI, and now to XML, would put great strains on systems and business departments. A cumbersome review mechanism was used to bring the Inland Revenue's "corporate memory" to bear on each new structure and change. This was clearly unsustainable in the long run. So DecisionSoft decided to centralise forms data definitions in a "common data set", or metadata repository, which would provide a single source of format information and business rules for a given data element. A piece of data, such as a personal tax code, might be used in a number of different forms, each with variant formats and differing validation rules, with the potential for different versions for each tax year. In addition, each piece of data could have different formats in its paper, magnetic media, EDIFACT, and XML versions. Rather than attempt to impose uniformity on a series of complicated, mission-critical, systems, the Common Data Set would centralise the Inland Revenue's "corporate memory" and provide visibility for inconsistencies between implementations and uses of the same underlying datatypes.

Given the complexity of metadata and the number of disparate parties involved in the development process, a controlled publication process was essential. All documentation had to be managed and communicated across the community as the standards themselves went through multiple revisions. To achieve this a series of plugins were written to generate any requisite output from the metadata repository. These outputs were made available in a controlled fashion via a web page.

The Inland Revenue had two main requirements for data validation. All data passed to the IR from the Government Gateway had to be validated fully before being passed to its back-office systems; and most of the validation mechanisms had to be made available as public services to support third party developers to test their submission software.

In order of increasing complexity there are, effectively, five different stages of validating an XML document. These are:

Data being submitted to the Inland Revenue clearly had to pass all these tests. As a starting point, incoming data was expected to be schema-compliant, that is, each data element had to meet the requirements specified in the XML Schema (and therefore pass the first three stages above). At the next level, internal business rules specified the relationship between data elements. These rules might include consistency checking for sub-total values, or rules which require certain elements to have specified values depending on the value of other elements. Lastly, submissions which were internally consistent had to be validated against actual data already held in the IR's sytems.

For the Inland Revenue, the validation was further complicated by the need to support test data (which would not be expected to pass stage five) and the requirement to include the possibility of additional rules at stage four for live document validation (which would not be applied to test data).

For the FBI project, the Inland Revenue decided on implementing a sequence of five different validation process levels, each one dependent in turn on passing the previous level. They were:

For the public developer test service, the first two levels were provided over the Web, with further access to the same service through the Government Gateway to allow developers to reach level three.

For the validation of live submissions, all five levels were tested in turn, with failure at any level resulting in immediate rejection of the submission.

DecisionSoft's X-Meta product suite was used to provide a Common Data Set for the Inland Revenue. It consisted of:

The metadata database provided a single point of input for the definition of datatypes and business rules for - potentially - all forms used by the Inland Revenue. Individual datatypes were versioned by format, tax year, and form. These could be compiled into data structures for each of the formats commonly used by the Inland Revenue (EDIFACT, XML, GFF).

The generators matched the datatype definitions with validation instructions, documentation and usage information. Formal structural descriptions would be generated automatically in the appropriate format (XSDL or XDR schemas for XML, MIG documents for EDI).

Over the period of the FBI project a number of different "stakeholders" have been identified, each with different documentation requirements. Business process owners within the Inland Revenue needed a formal description of business rules; government interoperability requirements mandated a full set of XSDL schemas; external developers required fully indexed specifications; internal developers needed extensive test data. Over time, new generators have been added to provide a muutually-consistent range of outputs including:

This architecture was developed in order to speed up the creation of schemas and other documentation, by replacing the conventional process of manually-updated Microsoft Word files and repeated proof-reading. Automating the production of documentation allowed the design-review cycle to be dramatically shortened while still including external stakeholder forums.

By adding plugins for generating validation code and regression test data, deployment of test services could be properly sychronised with publication of documentation, and the deployment cycle itself cut from a matter of months to less than an hour. This was critical for the Inland Revenue, whose legislation and tax-year driven annual cycle effectively guarantees that all its submission mechanisms will undergo revision at least once a year.

The metadata repository was thus used to store not only the full structural definition of each schema and instance, but also the validation expressions required to validate instances at each level, together with English descriptions of the validation rules. Storing validation expressions in X-Meta's repository provided centralised management of the coding, documentation and testing of the submission validation mechanism. This separated the creation of instance-specific rules from the development of the validation engine, so that validation expressions could be updated and tested quickly to take schema changes into account. In addition to reducing the time taken to establish the completed validation system, this separation of function improved the accuracy of validation code and reduced the likelihood of gaps in implementation and testing.

The key to the validation services was to control all validation functionality through code and properties held in the metadata database. For the Inland Revenue, the first validation level (schema validation) was implicit in the metadata which described the schemas. Thus, a datatype defined in the metadata repository might have a "pattern" property which would in due course be published as part of the schema which referenced that datatype. But the next level of validation, public co-constraint validation, could not be defined within an XSDL schema. For instance, a simple data constraint such as "if gender is male then statutory maternity pay must be zero" had to refer to both the gender and statutory maternity pay nodes within the structural definition. This was handled by the creation of a boolean expression, stored in the metadata repository and linked to the relevant structural node or datatype. This would be expressed in Java, XSLT or XPath - or any other language appropriate to the downstream implementation of the rule.

The boolean expression would be a simplified version of the code which would be required to validate the specific co-constraint. For instance, the node references would be relative rather than absolute - in other words, the expression would assume that the context of the rule - the location in the structure from where the code was being run - was already known.

Secondly, there would be no attempt made to handle the navigation between one co-constraint and the next, as the validator moved through the list of validatable rules: this would be handled independently, by generic navigation code which would crawl over the structure of the instance document, firing validation rules as it reached the appropriate node.

Within the metadata repository, then, a given data constraint (or "business rule") was tied to the appropriate part of the document structure and associated with documentation rubric and other information (such as whether to be fired for test or live services). As part of the process of generating the full definition of a given interface, these data constraints would be exported to the outputs API as a list of validation expressions. At the time of export each expression would be associated with its contextual information and from that information validation code would be generated. This validation code would provide for the navigation through the target instance document and the firing of each individual validation expression at the appropriate point in the traversal.

The validation code would then be passed to an architectural wrapper layer, where the validation object would be created in the format required by the relevant target architecture, as for instance, a Java class, COM object or DLL. The resulting validation objects could then be registered and made available as appropriate to the validation server for test or live services.

The Inland Revenue's main objective in developing their "Common Data Set" was to support the needs of the third party private sector software developers whose investment in the development and annual update of submission systems was critical to the success of Filing By Internet.

The repository-driven architecture allowed third parties to be supported without reference to developers and business experts within the Inland Revenue, and ensured that each interface definition was:

For a given software vendor, initial definitions of a service could be obtained from the public document repository, in the form of sample instance documents, PDF specifications and sets of schemas. Product development for the vendor could, if wished, incorporate authoritative validation DLLs drawn from the same source and drawn, like the documentation, from the same version of the underlying metadata. As development continued, beta systems could be tested against the Inland Revenue's Third Party Validation Service (TPVS), driven by validation objects generated from the metadata repository. The same validation objects could also be made available behind the Vendor Single Integrated Proving Service (VSIPS) mechanism for product accreditation, as well as the live validation service used on live data being submitted through the Government Gateway.

Because X-Meta stores structures for XML and non-XML formats, it can be used to describe the relationship between incoming (XML) data and target (back-office) formats. As the Inland Revenue implementation continues, we expect to have to generate code to convert incoming submissions into the formats mandated by the back-office systems, cutting out the need to write interface specifications and translate those into code. This code would be run by a processing engine in a load-balanced environment designed to handle multiple transactions per second.

I mention automated validation of EDIFACT submissions here only because of the light it shines, by contrast, on XML practices. In the XML world, machine-readable schemas and readily-available, open source, validating parsers provide a level of interoperability which other data formats can only envy. As we have seen above, even in the XML world there is a shortfall between the requirement for full validation of co-constraints and the ability of XSDL as currently constituted to validate more than structural and datatype-based parameters.

In the EDIFACT world, data specifications are encapsulated in the Message Implementation Guidelines (MIG) which are, effectively, negotiated between two transacting parties and expressed in human-readable - but not machine-readable - language. Compared with the ability to express XML data constraints - at all levels - as machine-readable metadata, this imposes tremendous costs both for maintenance of standards and updating of interface and processing code.

In what we believe is one of the first examples of XML technology leaking back to legacy technologies, we are now working with the Inland Revenue to add to their EDIFACT operations the ability to create metadata-based data definitions and automated validation mechanisms.

The latest phase of the FBI project is developing a service to allow the online submission of Corporation Tax returns. A strategic decision has been taken by the Inland Revenue to use Extensible Business Reporting Language (XBRL) to encode these documents. XBRL taxonomies are defined within the metadata repository and a new generator plugin has been added to cope with the specific requirements of XBRL definitions. In due course, the project is expected to be expanded to support XBRL data constraints based on the Inland Revenue's internal business rules.

Validating XBRL, however, is considerably more complex than validating XML documents, and a full treatment would be outside the scope of this paper. At its simplest, XBRL's use of linkbases introduces an extra level of consistency requirements between taxonomy and instance. None of these types of validation are currently available in either the commercial or open source markets, and we are now developing a set of XBRL tools to provide generic XBRL validation. For further detail, please see http://xbrl.decisionsoft.com.