ebXML Conformance

1. Introduction

The Organization for the Advancement of Structured Information Standards (OASIS ) ConformanceTechnical Committee ( TC ) defines and develops conformance requirements, guidelines, white papers, and methodologies to enable the development of testable specifications and conforming implementations. The TC serves as a central source of information about conformance and conformance related topics. It provides guidance and assistance to ongoing conformance efforts and collaborates among those developing specifications, conformance test suites and conformance testing programs. To accomplish its goals, the TC cooperates closely with other OASIS TCs, W3C, and the ebXML working groups of UN/CEFACT.

The ebXML specification, like all standards, is not an end in itself but a means to an end. Standards are worthless if not implemented and meaningless if not implemented in a consistent and correct manner. The goal is to obtain implementations of the standard that correctly perform the functionality specified in the standard. Conformance testing helps to achieve correct implementation of the specification. It provides a way to determine whether or not an implementation conforms to the standards in question. Determining whether an implementation faithfully implements a specification is essential to creating interoperable solutions. Conformance testing performed by implementers early-on in the development process can help them to find and correct errors before the software reaches the marketplace. Moreover, conformance testing provides software developers and users assurance and confidence that the conforming product behaves as expected, performs functions in a known manner, and/or possesses a prescribed interface or format.

Validation is the process of testing implementations for conformance. The validation process consists of the steps necessary to perform the conformance testing by using an official test suite in a prescribed manner. Certification is the acknowledgment that a validation has been completed and the criteria established by the certifying organization has been met, culminating with the issuance of a certificate (or brand). It is important to note that certification cannot exist without validation, but validation can exist without certification. Similarly, validation cannot exist without conformance testing (i.e., a test suite), but conformance testing can be performed without validation.

2. Components of a testing program

There are several essential ingredients needed for conformance testing. In its purest sense, conformance testing involves two components:

1. Standard or Specification

2. Conformance Test Suite

In addition to the above, validation and certification also involves these four components:

1. Testing Laboratory

2. Certification Authority

3. Control Board

4. Testing Policies and Procedures

2.1. Standard

A standard and its conformance clause define the requirements that must be met in determining conformance. A conformance clause defines applicability (i.e., what must conform and/or be tested) and identifies the conditions that must be met in order to claim conformance

2.2. Conformance Test Suite

A test suite, which is the combination of test cases and test documentation, is used to check whether an implementation satisfies the requirements in the standard. The test cases, consisting of a test tool or a set of files (i.e., data, programs, scripts, or instructions for manual action) exercises each requirement in the specification to determine whether the results produced by the implementation match the expected results, as defined by the specification. Each test case includes:

• a description of the test purpose (i.e., what is being tested - the conditions, requirements, or capabilities which are to be addressed by a particular test),

• a pass/fail criteria,

• a reference to the requirement or section in the standard from which the test case is derived (i.e., traceability back to the specification).

The test documentation describes how the testing is to be done and the directions for the tester to follow. Additionally, the documentation should be detailed enough so that testing of a given implementation can be repeated with no change in test results.

Conformance testing is considered black box testing, that is, the functionality of an implementation is tested against the requirements in a specification. This means that the internal structure or the source code of a candidate implementation is not available to the tester. The test suite should be platform independent, non-biased and objective. Generally a conformance test suite is a collection of combinations of legal and illegal inputs to the implementation being tested, together with a corresponding collection of expected results. Only the requirements specified in the standard are tested. A test suite should not check any implementation properties that are not described by the standard or set of standards. A test suite cannot require features that are optional in a standard, but if such features are present, a test suite could include tests for those features. A test suite does not assess the performance of an implementation unless performance requirements are specified in the specification.

The results of conformance testing apply only to the implementation and environment for which the tests are run. Test suites may be provided as a web-based system executed on a remote server, downloadable files for local installation and execution, or a combination of remote and local access and execution. The method for providing and delivering the test suite is dependent on what is being tested as well as the objective for test suite use - that is, providing self-test capability vs. formal certification testing.

2.3. Testing Laboratory

A Testing Laboratory tests an implementation using the official conformance test suite. The testing laboratory is often called a third-party testing organization. A Testing Laboratory can be an organization or an individual and may be accredited by a formal accreditation organization. Alternatively, a Laboratory may not be accredited, but be recognized by the consumer, implementer, and/or Certification Authority as qualified to perform the testing. There may be several Testing Laboratories accredited or recognized to perform testing for a given validation program. A Testing Laboratory produces a Test Report, documenting the results of the conformance testing.

The test report should provide enough information that, if necessary, the testing effort could be repeatable and comparable across different Laboratories. The report should detail the pass/fail results of each individual test as well as including:

• a complete description of the implementation under test,

• the name of the testing laboratory,

• the signature of a testing laboratory official,

• the date that the testing was completed,

• the name and version number of the test suite,

• an unambiguous statement indicating the overall results (e.g., passed all the tests).

2.4. Certification Authority

The major responsibility of the Certification Authority is to issue certificates for validated products. The Certification Authority may be a trade association, consortium, standards group, government agency, or private sector company. The decision to issue a Certificate of Validation, sometimes referred to as a Brand, is based on the testing results and established criteria for issuing the certificates. The criteria include the percentage of tests that an implementation must pass (e.g., 100%, 90%, 0%) and/or whether specific tests must be passed in order to receive a certificate. The certificate can designate a specific duration indicated by an expiration date on the certificate.

For example, a certificate could be issued only if no errors are found. On the other hand, a certificate might allow any error, state that an implementation was tested to completion, and provide a list of the errors found. It would then be up to a purchaser to decide the criteria (how many or what kinds of errors) it wishes to use to make implementations eligible for purchase. Often a hybrid of these is appropriate - i.e., issuing a certificate for a longer period of time (e.g. 2 years) if no errors are found and for a shorter period (e.g., 1 year) if there are errors. At the end of the 1-year period, the implementer would have to correct the errors in order to retest and renew the certificate. The rationale is to be able to acknowledge all the implementations tested, but ‘reward’ those implementations that pass all the tests. The rationale for an expiration date is that the technology, test suite, and/or specification may change and newer versions of the implementation may be issued, thus requiring the implementation to be retested.

2.5. Control Board

A Control Board is a neutral body put in place to resolve technical questions or disputes related to the testing process. A question or dispute can be initiated by the user of the test suite, implementer, or a Testing Laboratory, and may pertain to the test suite, procedures, or test results. The Control Board can also serve as an advisor to the Certification Authority. Examples of this advisory role include: assisting the Certification Authority in developing criteria for recognizing Test Laboratories, assessing Test Laboratories according to that criteria, and making recommendations for changes to the certification process and/or policy. The Control Board is typically comprised of members from the Testing Laboratories, Certification Authority, and standards committees, as well as experts in the field.

2.6. Testing Policy and Procedures

Testing Policy and Procedures define the administrative and operational testing process. The purpose is to document the core ingredients for a test program, define the responsibilities of all parties involved, provide directions and requirements for those establishing and operating testing services, and inform those wishing to be tested of the requirements, rules by which they agree to abide, and conditions and criteria for certification if offered. It also serves to inform the client (i.e., those wishing to be tested) of the requirements, expectations, and resources available to them and required by them, the procedures they must follow to obtain certification, and the rules by which they agree to abide.

The policy and procedures should address at least the following:

• responsibilities of the Certification Authority, Testing Laboratory, Client, and Control Board,

• test laboratory recognition criteria and use of Test Laboratories,

• handling of queries and disputes, withdrawn tests, and maintenance,

• security and confidentiality, and publication of validation results,

• complete definition of the certification of conformance (i.e., criteria for obtaining certification).

3. Validation and Certification Process

The validation and certification process is initiated when a client contracts with a Testing Laboratory to have an implementation tested for conformance. The client and Testing Laboratory negotiate the scope of testing, the cost of testing, and the schedule for testing. The implementation under test (IUT) is tested in a specific environment that includes the hardware and software required to support the implementation. The actual testing consists of submitting test suite files (i.e., tests) to the IUT and comparing the IUT’s output to the output required by the standard. Upon completion of the testing, the results are documented in a Test Report. The Test Report also contains information about the client, validation test environment, test suite version, and any other information gathered during the validation process. If the IUT successfully completes all the tests and meets the criteria for issuing certificates, the Certification Authority issues a Certificate of Validation to the client. Typically, the Certification Authority maintains and makes available to the public a register containing a listing of products that have received Certificates of Validation. This list is often called the Validated Products List.

Once an implementation has been formally validated in at least one environment, the Certification Authority may offer Validation by Registration as an alternative for additional validations. Validation by Registration allows the client (implementer) to self-test additional environments. It provides a low cost method for testing these additional environments and registering the results in the Validated Products List. The self-tested environment is compared with the formally validated implementation. Generally, conformance is demonstrated if the output from the self-tested environment is identical to the output from the formally validated implementation. Additionally, some certification systems provide for extending the certified status of tested implementations by performing accepted maintenance procedures. It is the purview of the Certification Authority to permit Validation by Registration and to define the conditions under which registration may occur.

Many implementations of publicly available specifications or standards are marketed worldwide. For validation to be effective and efficient, implementations that are tested in one country should not have to be re-tested in other countries. International harmonization of the validation process is accomplished through mutual recognition of Test Reports. This process is accomplished by having Testing Laboratories in different countries sign a mutual recognition arrangement. This arrangement requires Testing Laboratories to recognize the Test Report of any other Laboratory that has signed this agreement. In this way, Certification Authorities in different countries base the issuance of a Certificate of Validation on a common Test Report, thus obviating the need for implementations to be tested more than once. Note that mutual recognition relies on a common Test Report rather than a common Certificate of Validation. This process recognizes that Certification Authorities in various countries may have different criteria for issuing certificates

4. Degrees of Formality

It is not always necessary to establish a formal validation and certification program. Providing a test suite and encouraging developers and users to self-test their implementations is useful without validation or certification. Generally, validation and certification is performed for critical applications, to assess security, to achieve interoperability with other applications, or for procurement purposes. The level and formality of the validation testing is determined by the buyer, an organization acting on behalf of a community of buyers, or by regulation. For example, some programs may require a very formal testing and certification approach consisting of independent (i.e., third party), nationally accredited testing laboratories while others may allow self-declaration and demonstration testing. Typically, the decision to establish a certification program is based on weighing the benefits achieved by obtaining conformance versus the costs of creating and running a certification program.

Traditionally, validation and certification have been a formal testing process sponsored by organizations such as government agencies or trade associations that culminate in the award of a certificate. A Testing Laboratory usually conducts the validation after the client (i.e., implementer) performs self-tests on the implementation in preparation for the validation.

More recently, with the advent of the Internet and rapidly changing information technology, the notion of conformance testing has become less formal, allowing implementers and users to assess for themselves the validity or correctness of the software or data with respect to its relevant specification. They can then make any necessary corrections to allow the implementation to conform. Often, this self-testing makes use of publicly available test suites or tools and is not associated with any formal validation or certification program.

The most common example of self-testing is checking the syntax of a program by submitting the program to a syntax checker or validating parser. For example, HTML documents or Cascading Style Sheets (CSS) can be automatically checked using validating tools and XML processors can validate the XML data stream prior to processing the data. Often these validating tools are embedded in application software (e.g., authoring tools) or are on-line, readily available, free of charge, and provide immediate results. Users of these tools need only access the appropriate web page, provide the requested input, and receive the results.

5. ebXML Testing Model

Conformance testing is important to the success of ebXML. Not only are conformance clauses incorporated into the specifications, but also test suites and tools as well as demonstrations are being initiated by several OASIS ebXML Working Groups. A conformance testing and certification program for ebXML may be a combination of various approaches, ranging from self-declaration and demonstration to formal certification. Regardless of the testing program, it is necessary to ensure that all testing is available to everyone, in an objective and fair manner.

In order to establish a cohesive ebXML testing program, it is necessary to develop policies and procedures that govern the overall program approach, including the implementation, administration, operation, and management of the program. The policies and procedures document the components of a test program, define responsibilities of all involved parties, define requirements for those establishing and operating testing services, and informs those wishing to be tested of the requirements, rules by which they agree to abide, and conditions and criteria for certification if offered. The policies and procedures should apply to all ebXML conformance testing efforts, although they may need to be supplemented for the various component specification testing efforts. Areas that should be documented include:

1. confidentiality of information,

2. disputed and withdrwan tests process,

3. caveats,

4. minimum test report contents,

5. release of testing results and publication.

If Certification will be conducted, then the following additional areas should be addressed:

1. Will Testing Laboratories be formally accredited or recognized?

   • Criteria for Testing Laboratory accreditation/recognition.

   • Responsibilities of the Testing Laboratory.

2. Will there be a Validated Products List published to list implementations that have been successfully validated?

3. What information is required in a Test Report and, if issued, on a certificate?

4. How will administrative processes be handled?

5. What is the adjudication process?

   • Control Board membership and criteria.

   • Handling of queries and disputes.

   • Disposition of disputed and withdrawn tests.

6. How will maintenance of the test suite and/or tools be addressed?

   • New versions of the test suite/tool.

   • Frequency of new version releases

   • Configuration/control of test suite, test tools, and associated documentation.

Key to developing an ebXML policy is determining what will be tested for each ebXML specification, how it will be tested, and whether or not certificates will be issued upon completion of the testing process. For each ebXML specification, it may be necessary to extend or tailor the overall ebXML policy to include information applicable to the specific specification. In particular, questions to address include:

1. Will there be levels of conformance?

2. What is used to determine conformance (e.g., test suite, test tool, use cases)?

3. How will the test suite/tool be delivered (e.g., downloadable file, on-line interactive test harness)?

4. How will testing be accomplished (e.g., self-test with self-declaration, formal validation)?

5. What information will be reported in the Test Report?

6. OASIS/ebXML Registry Conformance Testing

A registry/repository is a mechanism used to discover and retrieve documents, templates, and software (i.e., objects and resources) over the Internet. A registry is the mechanism used to discover the object. The registry provides information about the object, including the location of the object. A repository is where the object resides. A user retrieves an object from a repository. The OASIS/ebXML Registry TC is pursuing the development of conformance tests for its two specifications, the OASIS/ebXML Registry Information Model and Registry Services Specification. The goals of these tests are twofold: to provide evidence of ambiguities and inconsistencies in these specifications so that the Registry TC can make appropriate changes to the specifications and to provide a tool that implementers can use to help ensure that their implementations are correct with respect to the specifications. Obviously the work of specification development and conformance test development are not done sequentially. Rather, they are done in a circular fashion where the specification is always being ‘tested’ through the conformance test development process, and lessons-learned from the implementers using the tests are fed back into the specification development process and so on.

The Registry specifications contain conformance clauses that define the requirements for implementations claiming conformance to the Registry Information Model and/or the Registry Services. For the registry services, the conformance clause distinguishes between ebXML Registry implementations and ebXML Registry Client implementations.

Testing the registry implementations would consist of a set of conformance test suites consisting of test software, scripts (e.g., use cases) and expected test results. Specifically the test suites would consist of ebXML Registry Client software and a set of scripts that choreograph the submission, updates, deletions and queries of ebXML objects. The correct implementation of the semantics of the Registry Information Model would be exercised through the choreographed scripts. The correct implementation of the Registry Services would be exercised through the use of the implemented interfaces. It is anticipated that the software and scripts would be downloadable, allowing developers and users to self-test their implementations.

Design and Development by deepX Ltd.