Table of contents Author City Company Country State/Province Term Interchange  

A Generated XML Interface to An Abstract Marketdata Service

Brazile, Jason ,   Netcetera AG ,    Switzerland 

Biography

Vckovski, Andrej ,   Netcetera AG ,    Switzerland 

Biography

Abstract

A large Swiss bank wanted to develop a web-based internet quotes and trading information service for end users. At the same time, it wanted to provide an interface to this service general enough to migrate internal banking applications to this service away from the multiple, similar but separately maintained services already being provided in several areas of the bank. In addition to consolidating development and maintenance costs, this would also ensure that external customers and internal customers share a consistent view of the data. For ease of migration, such a consolidated service would need to accommodate multiple interface requirements (e.g. CORBA, MQSeries, XML, etc) because of conflicting bank standards developed in different areas of the bank, or even by previously separate banking groups that had been acquired over time.

We describe how we designed and built a transport framework which allowed us to develop the business logic of the server entirely in an embedded scripting language, yet provide equivalent interfaces to this service in multiple ``flavors''. By limiting ourselves to a common functional subset of the required interfaces (e.g. no subscribe/notify), we were able to provide equivalent CORBA and XML interfaces that satisfied existing middleware standards requirements in an automated way. By decoupling the ``transport'' interface from the abstract interface, we allow the development of new middleware interfaces such as SOAP to be developed independently from the core service itself. We further describe how this framework allows a small development team to rapidly implement changes and enhancements to the server and its abstract interface, while automatically generating the corresponding changes that are needed for the various (e.g. CORBA and XML) ``transport'' interfaces. Additionally, we show how we were able to automatically generate code to extend a scripting language in such a way that makes use of the same, generated, marshalling routines in order to develop a scriptable client which can be used to regression test the server, when full end-to-end testing is needed.



Requirements

In 1998, as the ``new internet economy'' started to look less like a passing phase, large, ``traditional'' companies were beginning to get nervous if they didn't have an ``internet strategy''. This was also true of large banks.

One large bank decided that it would be worthwhile to fund a small, business-directed, externally developed IT project to try as quickly as possible to develop a web-based quotes and on-line trading information service for end users. This was unusual in that previously, such business-directed requests for new IT services would be addressed directly to the bank's internal IT development division. This request would begin a long and well-defined process of analysis, design, specification, development, testing, acceptance and finally deployment.

However, the business group decided they needed something fast, and that they might be willing to accept a process of fewer checks and balances in exchange for development speed - especially if it could be done within the (smaller) budget of an experimental pilot project.

In addition to the direct needs of the project, it was felt that there was a higher chance of project success if this new project could be developed in a way that made it usable by multiple internal bank divisions (e.g. financial advisors, branch tellers, etc) as well as internet end users. This led to roughly the following list of requirements:

    Business Requirements (B)

  1. Must do what site X does, but ``better''

  2. Must use data delivered by sources Y and Z

  3. Must be generic (e.g. back-end usable by other divisions)

  4. Must use blessed IT platform (e.g. specific hardware/OS/CORBA)

  5. Must deploy in 6 months

The business group had foregone their internal IT division's process of checks and balances in favor of our promise of speed of development. However, based on our previous experience with banking and finance projects, we decided we needed to add our own requirements to cover unstated risks.

If multiple disparate groups were expected to eventually migrate toward our application, we should expect each group would want to query the data in (different) ways that are similar to what they are used to (e.g. request by constraint vs. by ID list, paged vs batched results, etc). We should be aware that different groups use products developed and deployed in different environments (e.g. OS, transport protocol, etc.). Business logic requirements can change with each new fad or with each new feature just launched by a competitor (e.g. related news, IPO announcements, research recommendations, etc.). Business relationships with 3rd party data vendors run hot and cold (e.g. vendor X was more influenceable and affordable when they were less well-known, etc). And most importantly, even though they are willing to give up something for development speed, they will not be happy with incorrect data or an unstable application. These issues led to roughly the following list of additional requirements:

    Implicit Requirements (I)

  1. Expect frequent and drastic business logic changes [B1]

  2. Expect frequent client-requested changes to the interface [B3]

  3. Should not rely on (data) vendor specific features [B2]

  4. Should be easy to ``plug in'' new data sources [B2, experience]

  5. Should provide for automated testing [B5]

  6. Core should be in scripting language [B5,I1,I2,I3,I4,I5]

  7. Should be ``transport agnostic'' [B3,B4,I6]

Upon noting CORBA as a requirement, many software architects might automatically dismiss the idea of designing the core application functionality in a scripting language such as Tcl, even though it is not without precedent. However, with all the requirements focusing on rapid deployment and feature flexibility, we felt it was important not to be distracted by the strict platform requirements imposed (e.g. CORBA).

Scripting

Since we use scripting concepts so pervasively in our architecture, it makes sense to describe our motivation and strategy.

We consider a language to be a scripting language not only because it is interpreted, but among other things also because it is extendible and embeddable. In other words a scripting language can be embedded in a system to provide configuration or scriptability to otherwise static applications. A scripting language can also be extended by adding new primitives or new native commands that provide functionality typically only provided in a staticly compiled environment. In our Marketdata service, we wanted touse scripting in the following ways:

In order to use scripting in this manner, we needed to define a list of tasks needed to implement our strategy and divide the work among our team to refine and develop these tasks:

Architecture

After deciding on a strategy, we were able to more easily envision the architecture:

Architecture

We had only 2 full-time persons available for development of the project, so after defining the tasks, doing some initial prototype and investigation together in pair programming sessions, the logical division of labor was for one person to work on the core implementation of the Marketdata service (in Tcl) and the other to work on the XML and CORBA wrapper/conversion infrastructure.

The Marketdata Service

At this point, it is worth mentioning that we were only developing the back-end service, and not the front-end web interface. That was to be done in parallel, outsourced to another external consulting company. So it was important to have a clear idea of the interface that would result with as few surprises as possible during later integration.

We built a prototype service in our scripting language using native language features to represent the data and exceptions. We only needed to ensure that this native representation could be directly mapped to and from all transport interfaces - in this case CORBA and XML.

Example
  % set requestContext {$sessionId $appId ENGLISH {DELAYED} $userId}
  % set dataSelector {BYCONSTRAINTS {{} {"USDollars"} {"SwissExchange"}}}
  % set returnFields {BYFIELDNAME {ID CURRENT_PRICE HIGH_PRICE}}
  #
  # this can throw an exception and would be surrounded with a ``catch''
  # 
  % getMarketData $requestContext $dataSelector $returnFields results 
  % puts $results
  {ID 324598234 CURRENT_PRICE 45.5 HI_PRICE 48.25} {ID 43098234 ...}

And the XML representation is: 

 
  <?xml version="1.0" encoding="UTF-8" ?>
  <DOCTYPE MADISRequest SYSTEM "madis.dtd">

  <MADISRequest>
    <getMarketDataRequest>
      <RequestContext>
        <sessionId>1</sessionId>
        <appId>web-quotes</appId>  
        <lang>ENGLISH<lang>
        <Profile><Role>DELAYED</Role></Profile>
        <user>guest</user> 
      </RequestContext>
      <MarketDataSelector>
        <SelectType>BYCONSTRAINTS</SelectType> 
        <Constraints>
          <Searchwords></Searchwords>
          <Currencies>USDollars</Currencies>
          <TradingPlaces>SwissExchange</TradingPlaces>
        </Constraints>
      </MarketDataSelector>
      <returnFields>
        <ProjectType>BYFIELDLIST</ProjectType>
        <FieldList>
          <Field>ID</Field>
          <Field>CURRENT_PRICE</Field>
          <Field>HIGH_PRICE</Field>
        </FieldList>
      </returnFields>
    </getMarketDataRequest>
  </MADISRequest>

Data Flow

The flow of data through the system is shown below. One of the most important features of the architecture is that the implementation of the service and the development of test scripts for the service could be done in a short-circuit manner - bypassing any CORBA or XML infrastructure.

Flow

This allowed development of the transport infrastructure in parallel with the core service. But more importantly, along with providing a native abstraction of the interface to the data sources, it allowed the development of the core service to be done entirely in a minimal scripting language development environment (e.g. laptop on a train), instead of the typically heavyweight development environments required for CORBA development and satellite feed data collection.

Another important benefit is that the same tests written in the scripting language during development can be used for end-to-end testing by using the extended Tcl clients which use the same client/server transport mechanism that any other XML or CORBA client would use.

Code Generation

One of the largest risks we took in developing our strategy was not knowing ahead of time whether or not there would be easy direct mappings to and from our scripting language model and a CORBA and/or XML model.

Since the first version of the Marketdata service needed only request/response access, we found a set of mappings during an early prototyping and discovery phase, that could easily be supported by all the transport mechanisms we would consider using.

There were still a few tricky issues such as how to map and handle exceptions, but for all of these issues we were able to define an acceptable solution.

It was also not clear how we should specify the interface - using a DTD, an IDL, or some other mechanism. However, since the bank was very focused on an IDL definition, and since such a definition was rich enough to define all the concepts that our model would implement, we decided to use IDL as our definition language.

Here is an example mapping of our Request Context structure which is passed along with each request to define default language and session and authorization information. 

  typedef <string> Role;
  typedef sequence <Role> Profile;
  struct RequestContext {
    int sessionId,
    string appId,
    string lang,
    Profile profile,
    string user,
  };

The above IDL definition has a natural mapping to Tcl lists: 

  % set reqContext {-1 web-quotes ENGLISH {SWISS_REALTIME NYSE_DELAYED} guest}

and the DTD would show: 

  <!ELEMENT Role (#PCDATA)>
  <!ELEMENT Profile (Role*)>
  <!ELEMENT RequestContext (sessionId, appId, lang, Profile, user)>

So in the end, with our architecture, scripting and code generation strategy in place, here is a summary of the process involved when needing to implement a new method to the Marketdata Service:

    Process for adding a new method to our service

  1. Add method/data structure definitions to the IDL file

  2. Write a Tcl script which implements the new method

  3. Write a Tcl test script to test the implementation

  4. Use "make" to generate ``nicely packaged'' kernel.so from kernel.tcl

  5. Use "make" to generate new ``CORBA wrapper'' server

  6. Use "make" to generate new ``XML wrapper'' server and DTD

  7. Use "make" to generate new CORBA and XML Tcl clients

  8. install everything, restart CORBA service/apache plugin

  9. Re-run Tcl test script (step 3) from Tcl CORBA/XML client

In the following figure, it is more clear which components are automatically generated and which are implemented ``by hand'':

Code Generation

Conculsion

Our CORBA-as-transport-only strategy which was initially motivated only by our desire to avoid as much as possible typically heavyweight CORBA development environments, became more and more attractive over time as it allowed for a relatively painless path to an XML interface, and other experimental interfaces to the same service.

Our scripting language strategy also paid off in that we were able to implement significant features in short amounts of time, with few resources. And we never had to worry about the robustness (e.g. bad memory management, pointers, etc) of other typical CORBA development languages.

Perhaps the most interesting results are gained with the hindsight of looking how things have evolved over time:


  Table of contents Author City Company Country State/Province Term Interchange