Factual topic maps can be written in AsTMa=, the first sublanguage in the AsTMa* family. A topic covering a particular object like the cooporate firewall could be written as

firewall is-a host
bn: firewall.example.org
in: protects the good from the evil, whoever is inside
	

firewall is the topic identifier which can be used to refer to this topic elsewhere in the map. As this topic represents a particular host the is-a inlined association connects the firewall topic with another topic, host. This expresses an is-instance-of relationship between the two.What follows then are basenames (bn), external occurrences (oc), internal occurrences (in) or - not used here - subject indicators. All this other information is optional.

Associations simply pull together various topics using their topic identifiers:

(is-A-record-for)
fqdn       : firewall
A-record   : ip-192-168-0-1
	

This association is of type is-A-record-for, the topic firewall plays the role fqdn (fully qualified domain name) and our previous topic plays the role of the IP address. The ip-192-168-0-1 topic can be defined explicitely as shown below; otherwise AsTMa= processors are also allowed to generate topics if only their topic id has been mentioned.

ip-192-168-0-1 is-a ip-address
bn: 192.168.0.1
	

It is up the the ontology engineer how much information about the base vocabulary is inside a topic map or whether this is stored in the affiliated ontology. The AsTMa! sublanguage contains AsTMa= as a subset and so allows to represent topics and associations like above.

Additionally, the ontology engineer can apply restrictions using simple logic expressions:

forall [ $ip is-a ip-address ]
  => exists [ $ip
              bn: /\d+\.\d+\.\d+\.\d+/ ]
	

In this case we specified that all topics which are an (indirect or direct) instance of ip-address also must have a basename which matches the given regular expression. Also the structure of associations can be constrained:

forall [ (is-A-record-for) ]
   => exists ] (is-A-record-for)
               fqdn     : $h
               A-record : $ip [
      and
      exists [ $h is-a host ]
      and
      exists [ $ip is-a ip-address ]
	

This makes sure that all associations of type is-A-record-for must exist exactly in the form given, namely that there are two members with the roles as specified. Additionally, we enforce that the topics playing the roles are hosts and IP addresses, respectively.

Aside from purely structural constraints which control how a map should be formed, the constraint language AsTMa! also allows to express more application specific rules. In our running example of DNS-related data we certainly would not want that the subdomain relationship can work bothways:

forall [ (is-subdomain-of)
         domain    : $dom1
         subdomain : $dom2 ]
  => not exists [ (is-subdomain-of)
                  domain    : $dom2
                  subdomain : $dom1 ]
	

Associations of type is-subdomain-of can actually be derived automatically given that the basenames hold the domain names as strings:

forall [ $a is-a fqdn
         bn: $dom1 ]
   => forall [ $b is-a fqdn
               bn: $dom2 =~ /[^.]+\.$dom1$/ ]
      => derived exists ] (is-subdomain-of)
                          domain    : $a
                          subdomain : $b [
	

If the ontology would be associated to a map, then two topics in a map, both being an instance of fqdn, which would happen to have basenames such as, example.org and www.example.org, then the pattern match would succeed. An AsTMa! processor would automatically defer an association of type is-subdomain-of without that being directly present in the map.

The third sub-language, AsTMa?, allows to query topic maps. In that it treats a topic map as a data store from which particular values should be extracted.

To retrieve all IP addresses of the host firewall from a map we would notate this as:

forall [ (is-A-record-for)
         fqdn     : firewall
         A-record : $ip ] in tm:/internet/dns/
return
     {$ip}
	

Hereby we characterize the patterns we are interested, namely those where the firewall topic is involved in an association with a A record. The map to query is in our case accessible via a URL tm:/internet/dns/; it is outside the scope of the language to detail the access.

To add more flexibility we parameterize the query and wrap it into a function:

function get_A_records (string $host) as list return {
  forall [ (is-A-record-for)
            fqdn     : $host
            A-record : $ip ] in tm:/internet/dns/
  return
         ({$host}, {$ip/bn})
}
	

The output for one match of the pattern will now be a tuple of values; the first value will be the string representing the topic id of the host in question, the second value shows the use of a path language which derives the basename value of the topic captured via the variable $ip in this match. Overall a list of these tuples will be returned into the calling application.

As a convenience for application developers AsTMa? queries can also directly return XML content. This is supposed to facilitate the integration of TM content into web portals and content syndicates.

More of importance here is AsTMa?'s ability to create also TM content. This positions the language as a TM transformation language, converting one topic map into another. As an example we generate a new topic map which contains only hosts on a given network. The network is passed as parameter into the query:

function lan-hosts (string $network) as map return {
   forall [ (is-A-record-for)
            fqdn     : $host
            A-record : $ip ]
      => forall [ $ip
                  bn: $bn =~ /^$network/ ] in tm:/internet/dns/
   return

     {$host} is-a lan-machine
     bn: {$host/bn}

     (is-on-network)
     network: $network
     host   : $host
	

The return clause now includes not a list or XML fragment, it uses an AsTMa= fragment as template to create a topic for the selected machine together with an association connecting the machine to the network in question. The end result is a topic map containing all topics and associations generated in the process of matching and evaluating the template. What is so returned to the application can adhere to the same or a different ontology as that used by the queried topic maps itself.

A sophisticated mapping is necessary when the involved data is potentially unbounded in size, like, for instance that stored in the Internet DNS system. The number of domain names (FQDNs) and IP addresses together with their change dynamics prohibits that the data is stored in any central database, let alone a topic map.

Again, the first step of integration is to build an interface ontology:

ip-address subclasses address
bn: IP Address

fqdn subclasses name
bn: Fully Qualified Domain Name

forall $a [ (has-lookup) ]
   => exists $a ] (has-lookup)
                  fqdn     : $n
                  A-record : $ip [
      and
      exists [ $n is-a fqdn
               bn: * ]
      and
      exists [ $ip is-a ip-address
               bn: /\d+\.\d+\.\d+\.\d+/ ]
	

Here we simply introduced first the terms ip-address and fqdn. Then we specified that whenever there is a has-lookup association it has to have exactly two players for the given roles; the players have to be of type A-record and a fqdn, respectively (we only look at DNS A records here). Also we constrained that every FQDN must have at least one basename. The same holds true for IP addresses although here we additionally restrict the values to follow the given pattern. Equipped with this ontology we can write software which exposes a TM interface to an application.

Figure 3. DNS Virtualizer

That application can do a DNS forward lookup via the following AsTMa? query:

forall [ (has-lookup)
         A-record : $ip
         fqdn     : some.example.org ]
return
         {$ip}
	

The virtualisation software will analyse this query and will identify the relevant pattern there:

      [ (has-lookup)
        A-record : $ip
        fqdn     : some.example.org ]
	

As the player of A-record is left variable and that for the fqdn is fixed, the virtualizer will decide to do a DNS forward lookup, finding all A records for some.example.org. If the pattern would have both players fixed as in

      [ (has-lookup)
        A-record : ip-123-234-123-234
        fqdn     : fqdn-some.example.org ]
	

the virtualizer will do the same forward lookup, but this time checking whether 123.234.123.234 appears in one of the A records. If both players were left variable, then it may signal an exception to the application that it is unwilling to enumerate all A records in the DNS. Another implementation might be more inclined to do so if there were a restriction to a particular domain:

forall [ (has-lookup)
         A-record : $ip
         fqdn     : $fqdn ]
       and
       [ $fqdn is-a fqdn
         bn: qr/\.example\.org$/ ]
return
       {$ip}
	

Here the player of the fqdn role in the has-lookup association can have only particular values for its basename as specified by the regular expression qr/\.example\.org$/. If the virtualizer can detect this dependency, it would change its approach to simply query for A records and would instead initiate a DNS zone transfer for the given domain. If that succeeds it would extract the list of names and would - one by one - retrieve the A records for these.

How much of such intelligence is implemented will depend on the context of the application. The example simply should demonstrate the benefit which is gained by lifting DNS querying into a TM framework: Provided the virtualizer is fit for some query combinations, much richer and connected queries can be executed in a standardized way.

Writing dedicated virtualization software for particular data sources like the DNS or Internet whois is feasible. These stores offer a fixed vocabulary and a rather restricted query functionality which simplifies the development of dedicated software. This is not true for relational or - in the same vein - object oriented databases, such as, for example, LDAP. These allow for an independent definition of a database schema.

Our original approach, namely to analyze AsTMa? queries and to translate them directly into SQL statements, proved to be impractical without the knowledge how the database schema and the interface ontology correspond. Consequently, we introduced a mapping between the two ontologies:

Figure 4. Relational Virtualizer

Our procedure is to translate the database schema into AsTMa! topic definitions and constraints so that we can use the same formalism as for the interface ontology. This allows us then to combine the two ontologies and to express relationships between concepts in the native database and those in the interface ontology. This information then can be used by the query virtualizer.

To illustrate this lengthy process we assume a highly trivial database with a single table labs:

A rather obvious representation of this data in a topic map on the interface level would involve to store one row in one topic:

lab1 is-a lab
bn: lab1
in (subnet): 192.168.0.64
in (equipment): windows, well

lab2 is-a lab
bn: lab2
in (subnet): 192.168.0.128
in (equipment): linux, ok
	  

This implies that we should set up a structure like this in the interface ontology:

forall [ $l is-a lab ]
  => exists [ $l is-a lab
              bn: *
              in (subnet): /\d+\.\d+\.\d+\.\d+/
              in (equipment): * ]
	  

Remaining simplistic, let us assume that the application issues the following query to return all labnames together with the equipment information:

forall [ $l is-a lab ]
return
     ({$l}, {$l/in(equipment)})
	  

If the query virtualizer would have to generate an SQL query then we would ideally expect something like this

SELECT name, equipment FROM labs;
	  

to be used with the underlying relational database. To facilitate automation we next rewrite the database schema into the AsTMa! ontology language. This is a rather stereotypical process as long as we only regard tables and attributes in tables:

labs is-a table

(has-attribute)
table: labs
attribute: subnet

(has-attribute)
table: labs
attribute: equipment

(has-attribute)
table: labs
attribute: name
	  

Having modelled our database substantially with an has-attribute association, we then trivially enrich this with a generic ontology about the concepts of a relational database. These are always the same: tables, which have any number of attributes and rows which contain data values for particular attributes. Here we will introduce a is-row-in association type between tables and rows and another is-attribute-in between rows and individual values in these rows.

If we now combine the database ontology with our original interface ontology, then we can set up constraints how their respective concepts should interact. First we observe that for every topic of type lab there must be a corresponding row in the relational database (Rule I):

forall [ $l is-a lab ]
  => exists [ (is-row-in)
              table: labs
              row  : $row ]
     and
     exists [ (is-attribute-in)
              row       : $row
              attribute : name
              value     : $l ]
	  

In our simplistic scenario also the converse is true: whenever we find a row in the table then there should be an appropriate topic on the interface level (Rule II):

forall [ (is-row-in)
         table: labs
         row  : $row ]
  => forall [ (is-attribute-in)
              row       : $row
              attribute : name
              value     : $l ]
     and
     forall [ (is-attribute-in)
              row       : $row
              attribute : subnet
              value     : $s ]
     and
     forall [ (is-attribute-in)
              row       : $row
              attribute : equipment
              value     : $e ]
     => exists [ $l is-a lab
                 bn: $l
                 in (subnet): $s
                 in (equipment): $e ]
	  

All these long-winding definitions provide now the basis for the query virtualizer whenever it is faced with a concrete query like above. It will first analyze that part of the query which captures results:

forall [ $l is-a lab ]
  ....
	  

According to rule I this implies that (a) there is a particular row in the table labs and that (b) the name attribute in this row has the same value as the topic id. The condition (a) can be easily translated into SQL:

   SELECT * FROM labs
	  

The condition (b) would translate into a WHERE clause:

   WHERE name = $l
	  

As $l itself is variable the WHERE clause is empty and be optimized away. Here the downward translation could stop and without any further consideration the query virtualizer would execute

   SELECT * FROM labs
	    

against the relational database. From then on the query virtualizer has to upward-translate the results. When retrieving the individual rows it will use rule II as it applies to all rows. It will further evaluate all forall subclauses and will in this process bind the appropriate column values to the variables $l, $s and $e. Once this is satisfiable - which in our case it is - the exists clause will tell the processor to generate an intermediate result, namely the given topic using the bound values for the id, basename, the subnet and equipment information. Only then it may look at the form of the result which is supposed to be returned into the application. It will evaluate all the path expressions in this tuple for each such topic:

   ....
   return
       ({$l}, {$l/in(equipment)})
	    

You may have noted that the whole process still generates intermediary topic map results and does not shortcut it to directly return the result of

   SELECT name, equipment FROM labs;
	    

This shortcoming of our method is subject to further investigation. In any case, the down- and upward translation can be automated to a large extent. If - which is not uncommon - the form of the queries is known in advance, then special code can be generated moving the comparatively expensive analysis phase from runtime to compile time.

One may speculated whether a similar approach as the one with relational databases would also work for integrating XML based stores. This is an area for future, challenging research as XML, its abstract structure and the operations based on XPath[XPath 1.0] or even XQuery[XQuery] provide a rather big base ontology. Consequently the translation process is much more complex.

It is worth noting, though, that AsTMa? potentially allows to shortcut results here as well: If an application issues an AsTMa? query which is supposed to return XML content, then the XML structure could be generated by the XML store and could be passed directly through to the application.

The simplest for of combination is certainly to merge maps according to the generic rules as provided by the Topic Map standard itself or to merge them in an application specific way.

One consideration here is that some of the maps which are supposed to be merged might be not materialized themeselves. Even if all maps were materialized, we still could decide NOT to merge them into a newly materialized whole. In this case merging is only a logical operation, the individual maps still remain as entities. Queries executed against the combined map have to be split and have to be executed - more or less independently - in the individual maps. The results are then combined, merged again in the TM sense.

If the federation software would know nothing about the individual maps, then every query against the whole map would have to be executed against all operand maps. In our framework, though, the federation software is aware of the ontologies of those maps. With some analysis effort the federator can route (sub)queries into the appropriate maps.

If, for example, we federate a virtual topic map covering DNS with another, covering local web services we run on particular machines, then the content of the two maps is rather disjunct. That part of a query which retrieves all services on a particular host with a given FQDN would not be routed into the virtual DNS map:

forall [ $s is-a web-service
         bn: $service-name ]
         and
       [ (is-hosted-on)
         host    : $host
         service : $s ]
return # find the IP address using a AsTMa Path expression
     {$host -> fqdn \ has-lookup / A-record}
	

The path expression in the return clause uses the current host topic as starting point and hops via the association of type has-lookup to the IP address(es). That last part can be routed into the DNS map.

A query can also be broken up into subqueries heading for different maps when conditions are OR'ed like in the following example:

forall [ $s is-a fqdn
         bn: qr/\.example\.org$/ ]
       or
       [ $s is-a web-service ]
return 
      {$s}
	

which simply should return all known web services and all hosts in the given domain. A "join" between two or more maps involves that two conditions have to be checked independently in different maps. A query like

forall [ $fqdn is-a fqdn            # this is directed into the DNS map
         bn: qr/\.example\.org$/ ]
       and
       [ (is-hosted-on)             # this is directed into the WS map
         host    : $fqdn
         service : $s ]
return
      {$s}
	

would involve that both maps return all FQDNs topics and all is-hosted-on associations. Only those pairings where the host role matches the topic id of a FQDN topic we can count into the overall result.

These examples make it clear that the exact rules when and how queries can be factorized is beyond the scope of this presentation and is subject to further research.

In some cases the application should not see a complete topic map but only a particular view, be it for reasons of confidentiality, limited scope or because of application-specific reasons. For this purpose it is useful to filter the topic map data by applying constraints. Conceptually, an ontology is exactly such a filter: It contains concepts, organized in a type hierarchy together with a set of application specific rules. Given a (virtual) map M, a constraint C can be applied to it, constraining it further.

Technically a constraint is nothing else than a query which asks for exactly that data in the topic map which satisfies the constraint. Obviously, the end result is a map again.

As a generalization of the above, it is also possible to transform any existing topic map data before the application sees it. A reason for this could be that the application is expecting an ontology at the interface with the TM infrastructure which is slightly (or considerably) different from the existing TM data.

Using a Perl interface, applications may access any existing topic map by creating an object representing the map:

use TM;
my $tm = new TM ('mymap.xtm');
	

Once the map is loaded, applications can navigate through the data structure using a dedicated interface to extract topics or associations. Alternatively, applications may also choose to query maps using AsTMa?

my $q = new TM ('myquery.atm');
my $r = $tm->apply ($q);
	

Depending on the return type of the query (AsTMa queries can return lists, XML content or complete topic maps), the variable $r contains a list (reference), a reference to an XML DOM object or a reference to the result map.

The notation

my $tm = new TM ('mymap.xtm');
	

is actually only a shorthand for

my $tm = new TM ('file:mymap.xtm');
	

Accordingly, this is also a means to load from other resource, such as from HTTP or FTP servers. It is also the mechanism to load a virtual map:

my $dns = new TM ('dns:mydns.example.org')
	

As URI for this resource we used dns:mydns.example.org which invokes a virtualizer for DNS (if such exists in the local installation). As part of the URL we also provide the name of the DNS server to be used.

As it can be expected that for particular resources there might be different implementations we use a mechanism whereby the application engineer can control which virtualizer should be used in particular situations:

use TM::Virtual::DNS;
$TM::schemes{'^dns:'} = 'TM::Virtual::DNS';
my $tm = new TM ('dns:mydns.example.org');
	

Above we load the named module and use that for all resources for which the URI matches the pattern ^dns:. The t-algebra parser will activate this particular implementation then.

To federate two maps using TM merging, the engineer would connect two map resources with the operator +:

my $tm = new TM (tau => 'file:mymap1.xtm + file:mymap2.atm');
	

This time we have merged a map coming from an XTM file with one from an AsTMa file. A similar expression also works in the case that one operand is not materialized as is the case with our DNS map:

my $tm = new TM (tau => 'file:mymap.xtm + dns:mydns.example.org');
	

A map can also be enriched by an ontology in the sense that it provides a type hierarchy relevant to the map. Also here we can use the merge operator +:

my $tm = new TM (tau => 'file:dns.onto + dns:mydns.example.org');
	

The operator + is overloaded in order to merge maps, maps and ontologies, but also ontologies themselves. In short, merging ontologies means that all the vocabulary of these ontologies are merged according to the standard merging rules. In terms of the constraints within the ontologies a merge corresponds to a logical OR between these constraints. Accordingly, given two ontologies O and P a map M validates against O+P if it validates either against O or P.

If a map has to be filtered by a constraint or even a list of constraints in an ontology, then we symbolize this operation by the binary operator *. In the following t-expression we form a new virtual map by filtering out all topics and associations which are not allowed according to the ontology dns.ont:

my $tm = new TM (tau => 'file:dns.ont * dns:mydns.example.org');
	

The operator also is used to express the application of a TM transformation:

my $tm = new TM (tau => 'file:simplify.trafo * dns:mydns.example.org');