TMIP, A RESTful Topic Maps Interaction Protocol

Exchanging Whole Maps

To download a whole map addressed via /markup/xml/xpath/ from the server an HTTP request may look like this:

GET /markup/xml/xpath/ HTTP/1.1
Host: server1.farm.example.org
Accepts: application/xtm+xml

Should the map not exist on the given location, the server will return a message with the status code 404 (Not Found). Otherwise it will respond with

HTTP/1.1 200 OK
Server: TMIP server v0.3
Content-Type: application/xtm+xml

<?xml version="1.0"?>
<topicMap.....

Exchanging maps in XML form allows for platform-independent exchange but comes at a rather high performance cost. Using HTTP content negotiation clients can request also alternative formats. A Perl client, for instance, may request

GET /markup/xml/xpath/ HTTP/1.1
Host: server1.farm.example.org
Accepts: application/x-storable, application/xtm+xml

To figure out which formats the server supports for a given map, we can use the HTTP method OPTIONS:

OPTIONS /markup/xml/xpath/ HTTP/1.1
Host: server1.farm.example.org

HTTP/1.1 200 OK
Server: TMIP server v0.3
Allow: GET, PUT, OPTIONS
Accepts: application/x-storable, text/x-ltm, application/xtm+xml

To store a complete map under a particular location, it is natural to use the HTTP method PUT (here we use LTM as map format):

PUT /internet/web/ HTTP/1.1
Host: server1.farm.example.org
Content-Type: text/x-ltm

[ larsbot ]....

HTTP/1.1 201 Created
Server: TMIP server v0.3

Clearly, there could be many more reasons why such an import would fail. One of them is that the sent map information cannot be properly interpreted. In such a case the status code would be 415 (Unsupported Media Type). Such an error would also be returned if the client would send a map in a text format not understood by the server:

PUT /internet/web/
Content-Type: text/x-my-not-implemented-format

Map Fragments

More complex is the handling of map fragments. To retrieve a particular topic, the URL contains a simple path expression:

GET /internet/web/firefox HTTP/1.1
Host: server1.farm.example.org
Accepts: application/x-storable

If the platforms differ, though, the query result will have to be serialized into XML again. Here it is useful to understand that every TMQL path expression - when applied to a map - will return a sequence of tuples of items. To illustrate this, let us consider a query for all browser names:

GET /internet/web//browser/bn`s HTTP/1.1
Host: server1.farm.example.org
Accepts: text/xml

HTTP/1.1 200 OK
Server: TMIP server v0.3
Content-Type: application/x-tmql-sequence+xml

<seq xmlns="http://astma.it.bond.edu.au/ns/tmip/1.0/ts/">
  <t><s>Firefox</s></t>
  <t><s>Mozilla Firefox</s></t>
  <t><s>w3m</s></t>
  ....
</seq>

The response contains in the message body a tuple sequence (<seq>). Each of the tuples (<t>) in turn contains only only one component, a string (<s>, carrying the textual value of the topic name. Note, that - since a topic can have any number of names attached to it (possibly having a different scope) - one would expect to see an entry of each of these names in a separate tuple within the sequence, as is in fact the case with Firefox above.

The XML notation is itself defined as part of the protocol. With it, simple content like strings, identifiers and topic map fragments (serialized as XTM) can be organized into individual tuples, and these in turn into sequences. Since we did not ask for any ordering, the sequence itself is unordered.

As queries can be also issued within a particular scope context, that can be forwarded to the server via an additional HTTP header:

GET /internet/web//browser/bn`s HTTP/1.1
Host: server1.farm.example.org
X-TMIP-Accept-Scope: uc, en, de, *
Accepts: text/xml

At the end of this list, the wildcard * signals that any other scope can be used if no topic characteristic exists in the scopes before. Ignoring how that scoping context is used on the server-side, an important consequence for the result is that only one name, that in the appropriate scope, is now selected from the list of name items.

In the above examples we have always requested string representations of topic characteristics; here we used a stringification postfix `s. If a client is not interested in the string values, but needs the whole items (including scope and type information), then it would omit the prefix:

GET /internet/web//browser/bn HTTP/1.1
....

<seq ordered="no" xmlns="http://astma.it.bond.edu.au/ns/tmip/1.0/ts/" 
                  xmlns:xtm="http://www.topicmaps.org/xtm/1.1/"
                  xmlns:xlink="http://www.w3.org/1999/xlink">
  <t>
     <i>
       <xtm:association>
          <xtm:instanceOf><xtm:topicRef xlink:href="#has-basename"/>
          </xtm:instanceOf>
          <xtm:member>
              <xtm:roleSpec><topicRef xlink:href="#basename"/></xtm:roleSpec>
              <xtm:topicRef xlink:href="#x-string-436326"/>
          </xtm:member>
          <xtm:member>
              <xtm:roleSpec><topicRef xlink:href="#topic"/></xtm:roleSpec>
              <xtm:topicRef xlink:href="#firefox"/>
          </xtm:member>
       </xtm:association>
     </i>
  </t>
  ....
</seq>

The whole item is now embedded in the tuple as an association. Only structural information is sent, no additional unsolicited content is added, not even the string within the characteristics is sent. Otherwise the message has exactly the same structure as before.

This is also true if we ask for all involvements of a certain topic in particular associations:

GET /internet/web/firefox->software[*runs-on-platform] HTTP/1.1
...

Tabular Results

In general, path expressions can return tuples with more than one component. As an example, we consider the request:

GET /internet/web//browser < ./bn`s, ./oc[ *download ]`s > HTTP/1.1
Host: server1.farm.example.org
X-TMIP-Accept-Scope: uc, en, de, *
Accepts: text/xml

The response follows the general pattern of a tuple sequence:

<seq> 
   <t><s>Firefox</s><s>http://www.mozilla.org/products/firefox/download.html</s></t>
   <t><s>lynx</s><s>http://lynx.isc.org/release/</s></t>
   ....
</seq>

Scoped Names

Sor far, a client can retrieve mostly structural information. At some stage it may need topic names to present information eventually to a human user. The protocol allows to bulk-load topic names for a list of topic identifiers in a specific scope context:

GET /internet/web/ HTTP/1.1
Host: server1.farm.example.org
X-TMIP-Accept-Scope: uc, en, de, *
Accepts: text/xml
Content-Type: application/x-tmql-sequence+xml

<seq>
   <t><id>firefox</id></t>
   <t><id>w3m</id></t>
</seq>
    

<seq>
   <t><id>firefox</id><s>Firefox</s></t>
   <t><id>w3m</id><s>w3m</s></t>
</seq>
    

As names are usually quite static, applications may choose to manage them separately from the structural information. This also minimizes the traffic if users switch the scope frequently. Experiences have shown that significant responsiveness can be achieved if this information is cached on the client side.

Updating Fragments

Updating server-side TM content could be done by first pinpointing involved topics (or associations) and then by applying the update on them. Instead we consistently choose to leverage our use of TMQL path expressions to precisely define particular components of a target map.

Consider again the download occurrence of the topic firefox we had mentioned before. Should this be updated to a new value, that can be sent within a tuple sequence to the server:

PUT /internet/web/firefox/oc[*download] HTTP/1.1
Host: server1.farm.example.org
Content-Type: application/x-tmql-sequence+xml

<seq>
   <t><s>http://www.spreadfirefox.com/</s></t>
</seq>

On the server side the tuple sequence will be used to update the fragment addressed by the URL as follows: If the new value is a string (as in the example above), then the occurrence value will be set to this string. If the new value is a complete characteristics item, that will then replace any existing one. In a similar way, we can also make use of the method POST to add topic characteristic values to a map or DELETE some. If the sent data does not make any sense to the server, it will respond with an appropriate error code.

The same mechanism can be extended to extend a particular map with additional fragments. Using, for example a POST request

POST /internet/web/ HTTP/1.1
Host: server1.farm.example.org
Content-Type: application/xtm+xml

<xtm:topic id="safari" xmlns:xtm="...">
   <!-- topic information here -->
</xtm:topic>

Using 3rd Party XML

The TMQL queries so far were all path expressions, so that they can be easily embedded into a request URL. General TMQL queries can also generate XML content to simplify the integration into XML application servers.

The simplest way to do that is by using predefined queries. As example we show a query HALO which - when confronted with a specific topic - computes from the map the surrounding of the topic (all its occurrences, names, involvement in associations, etc.). It was especially designed for generic TM user interfaces.

Queries like these are then used as filters. First, a relevant part of a topic map is identified and then the filter is applied to this result:

GET /internet/web/firefox * HALO (occ=>[1,3]) HTTP/1.1
Host: server1.farm.example.org
Accepts: text/xml

The predefined queries themselves are not relevant to the protocol per-se. They just show how flexible the REST approach can be. What is relevant, though, is that all these functions will return XML content to the client. How its structure is, will depend on the function.

Client-Controlled XML Response

TMQL can be used to organize query results not only into tuple sequences, but also directly into XML. This is not possible using TMQL path expressions as we have done so far; for this we have to switch into TMQL FLWR (flower) mode:

GET /internet/web/ HTTP/1.1
Host: server1.farm.example.org
Accepts: text/xml
Content-Type: application/x-tmql

return
<browsers>{
   for $browser in %_ // browser
   return
      <browser id="{ $browser }">{
          for $occ in $browser / oc
          return
              <url href="{ $occ }">{ $occ * }</url>
      }</browser>
}</browsers>

Without going into any detail regarding the query language, the query expression above will iterate over all browsers (%_ identifies the queried map) while wrapping all subresults into a <browsers> root element. For each of the browsers we find all ocurrences and iterate over those. For each of the occurrences we issue a bit of XML, inserting the occurrence string (the URL) and the occurrences type (computed by $occ *).

While the client must be more skillful to add a full TMQL FLWR query, it can precisely control the XML structure which it will get back from the server.

Of course, servers may flag their unwillingness to perform certain queries if these violate a local policy.

Controlling the Server Side

One plausible extension to the above is to view the server itself as a (virtual) topic map[BaVirt04]. This allows to access operational data (such as statistics, current resource limit settings or access control) as topic map data and - given the necessary permissions - also to modify them.

For this purpose the protocol has reserved a special URI subspace, /.meta/ as it is used in the following where we access some statistical information:

GET /.meta/inbound-messages-per-sec HTTP/1.0

The ontology for the server is currently in flux, but the protocol specification will define a minimum every implementation has to provide.

Performance

To benchmark the protocol (and not the server implementation) we have to identify particular use cases and have to reflect on the costs involved. To store, for instance, a topic map instance into a TM server, the client has to prepare an HTTP request. Here it will embed the map into the message body whereby several format options exist. Then the message will have to cross the wire.

On the server side, the infrastructure will have to do a service dispatch after analyzing the incoming request. Depending on the format, the server will have to deserialize (parse) the map from the message body. Any acknowledgement message will be sent after that.

In Fig 1. we generated increasingly big maps in different formats (XTM, LTM, AsTMa and a binary format). The amount of content (assertions involved) is used as a common factor for comparability. The overhead processing time is then computed from the timing, factoring out message preparation (on the client and on the server side) and server method dispatch times as they are constant and effectively negligible. The overhead then only includes transfer time (on a local network) and the deserialization costs (on a typical contemporary hardware). While the absolute numbers may vary with the implementation it is obvious that the deserialization is the predominant cost factor.

Figure 1: Exchange Timing, PUT map

[Link to open this graphic in a separate page]

As expected, XTM performs worst; the binary format was the fastest, the other two formats occupy the middle range. To make the costs for the transfer more visible, Fig 2. shows the absolute size (in KB) of the maps.

Figure 2: Message Sizes, PUT map

[Link to open this graphic in a separate page]

While LTM and AsTMa= seem to be quite economic, the binary format for maps quickly reaches unacceptable sizes, at least in current WAN environments. What the diagrams seem to prove is that more compact map encodings or notations pay off as long as their deserialization effort is bounded.

A second series of experiments analyzes the cost structure of GET requests using query expressions. While the advantage of TMIP is a very fine-grained control over the requested content, this comes at a significant price.

For the preparation of the request, transferring it to the server and dispatching it there the costs are negligible. The time-consuming steps are the parsing, analyzing and pre-optimization of the query expression, the execution of the query on the named map and the serialization of the results. Finally, on the client-side the XML-encoded tuple sequence has to be built up in memory for further processing. Fig 3. shows these times over an increasing result size.

Figure 3: Processing Times, GET TMQL path

[Link to open this graphic in a separate page]

As expected, querying the backend is by far the dominant factor. One might justifiably argue that the used TMQL processor is not using any optimization and is only orthodoxically implementing the current TMQL draft. Still, even with the improvements to be expected, querying times will remain the bottleneck for the time being and will be pose the throughput-limiting factor of a TM server.

Security

The first aspect of security is authorization. Since we are using HTTP as transport we can use all the options associated with it, including adding TLS/SSL for certificate based authentication. TLS can also be used for encryption, if end-to-end privacy is an issue.

Access control can also be completely burdened onto the HTTP infrastructure. Request URIs can be used for a fine-grained control so that users can have access to complete maps or only to certain topics. It is difficult, though, to extend URI based authentication by including TMQL path expressions. This limits the granularity of access control.

Both aspects, authorization and access control we regard outside the TMIP protocol specification. This does not apply to the protocol's resilience to denial of service attacks. For this, servers are allowed to deploy a local resource limitation policy. Such a policy would include limits on the use of CPU resources, number of requests per second per client, etc.

Architecture

It is arguable, that the REST approach chosen for TMIP captures a considerable range of application scenarios. HTTP has proven to be sufficiently rich to cover most interactions; when necessary we added additional HTTP headers and reused existing MIME types. Still, there was no need to introduce new status codes.

This minimality partly justifies why we regard a RESTful solution superior compared to a SOAP-based protocol stack. While debatable, SOAP seems to be more equipped to closely-coupled, DCOM-style interaction between two parties. The fact that the interface has to be declared separately allows a very fine-grained information flow into remote methods and back. This is certainly not possible using REST. Interfaces using the latter architectural style also struggle somewhat with composition, especially with transactions.

What seems to be quite reassuring is that the chosen setup conforms largely with the one outlined in the WWWA document[WWWA]. Maps are treated as resources using a reproducable URI space, but also individual topics can be interpreted as individual resources. In a general way, even adding a path expression to a request URI can be seen as addressing fragments of map in a declarative way.

Different representations (in the WWWA sense) can be requested by the client as it seems fit. Servers can respond to these choices along the rules of HTTP negotiation[RFC2616]. A convenient side-effect of using the HTTP method GET together with the TMQL path expression language is that all conveyed responses to these requests can be cached consistently in downstream cache chains. There are no special characters, additional cookie information (as would be necessary for state management) which would undermine caching efforts. All the chosen URIs are also context free.