Interaction Design System Use of XML

Topic Coverage

The User Centered Design principles will not be discussed directly in this paper. However, they will appear as an underlying theme in each section. In this paper the IDS architecture and reasoning will be presented briefly through a "Fill Prescription" example. The focus of the paper will be on the use of XML technologies and their role in making IDS a truly generic system. After an overview of the system in this Section, Task Modeling and its RDF representation will be presented in Section 2. Task Modeling will be shown in the context of the "Fill Prescription" example. In Section 3, Domain Modeling and its representation will be presented. In Section 4, the third model, Interaction Device Modeling, will be presented through a small browser example. IDS 's method for generating Presentations will then be shown in Section 5. In Section 6, the "Fill Presentation" example will be walked through the generation process. Finally, in Section 7, current and potential follow on work will be mentioned.

System Overview and Definitions

 Presentations  The system-generated interactions are called Presentations .

Generation of Presentations An Interaction Requirement is built of Domain Items, a Task and available Interaction Devices. The Interaction Requirement generates Qualifying Presentations. Presentations specify the Presentation Element and Interaction Device needed to generate the user interface.

A Presentation may specify the components needed to generate a visual display for a web browser, an audio message for a speaker system, a small Java program, a menu system for a touch tone phone, or any other method of interacting with a human through an Interaction Device.

Presentation Class (Note: In this paper, figures such as this one represent a Java Class associated with the term being defined. These figures do not include all the properties and methods available in that Class. The ones included will be addressed in this paper.)

Presentations bind a Presentation Element to a specific Interaction Device.

PresentationElement Class

Presentation Element The IDS Presentation Element corresponds to specific presentation methods suitable for satisfying the specified Interaction Requirement and defining the characteristics Interaction Devices must possess to be capable of supporting the Presentation Element.

The Presentation Element also specifies the type of Task Primitives and the type of information the Presentation Element is designed to support.

Interaction Device An IDS Interaction Device characterizes the user interaction capabilities of a real-world device.

InteractionDevice Class

These characteristics, as defined in a device signature, can be written in an RDF file and read into the IDS data store.

Device Signatures are one kind of Noun Signature.

Partial view of the NounSignature Class

Device Signature  Noun Signature   A Noun Signature is an abstraction of characteristics of a Domain Element. The Device Signature contains information about graphical capabilities, audio support, hardware buttons and the like. Device Signatures and other Noun Signatures can be marked up in an XML file adhering to the IDS Noun Signature DTD . More on this format will be mentioned later.

Domain Elements In an active Interaction Design System, the information being presented to the user is taken from the Domain Model. The Domain Model is based on a domain specific ontology and encoded in RDF for input into IDS . Domain Elements are representations of anything in the domain, including Tasks, Interaction Devices and Noun Signatures.

Domain Items Domain Items are Domain Elements relating directly back to the Domain Model. They often represent things like medications, pharmacies or people, but can also represent information such as drug interactions or cost of co-pays or they can represent a statement such as "Remember your Dr. appointment is at 3 o'clock" or "Did you take your medicine?"

DomainItem and DomainRelationship Classes

Domain Items serve as both the context of a Task and content of a Presentation. Its role of context is being defined at the time of this paper's writing. Its role of content will be addressed later in this paper.

 Task Primitives  Task Primitives are basic actions a user may take while interacting with the generated system. Task Primitives tend to fall into three types: Receive, Assert and Navigate. These generally cover information presentation, interactions with the underlying program or data source and movement through information and interactions, respectively. The structure of Tasks will be shown in Section 2.

Tasks Tasks define the user's goal and the subtasks needed to meet smaller goals along the way. They will be explained more fully in Section 2. All Tasks are comprised of one or more Task Primitives.

Match Requirement  Usability Criteria   Tasks contain a Match Requirement to specify the Usability Criteria for Presentations supporting the Task. Usability Criteria are based on a 10-point scale and specify both the way content is displayed and the way the user should interact with the content. Usability Criteria rank the importance of the Task, the obtrusiveness of the interaction, the scope of information and other control and display characteristics important to the Presentation selection. The methods for selection are based on User Centered Design principles.

InteractionRequirement Class

Interaction Requirement When a Presentation is requested, the Task needing a Presentation creates an Interaction Requirement. The Interaction Requirement specifies the Task, the Domain Item required for the interaction and the available Interaction Devices to support the Presentation.

The Interaction Requirement retrieves the Presentation Elements that work on an Available Device, support the Task Primitive and handle information of the type held in the Domain Element. IDS then provides the Presentations that best meet the Usability Criteria required by the Task.

Once a Presentation has been selected for generation, the Widget associated with the Presentation Element is executed.

Widget Class

Widget Each Presentation Element has an associated Widget. The Widget extracts information from the Interaction Requirement to create the XML string containing the content for eventual presentation. It then gives the content to the appropriate XSL Driver through the Interaction Device specified by the Presentation.

XSLDriver Class

XSL Driver The XSL Driver is the active part of the Interaction Device just as the Widget is the active part of the Presentation Element. The XSL Driver converts the Widget's produced XML to the appropriate format for the Interaction Device. This may be HTML or JSP for a web browser, WML for a WAP phone or a proprietary format supported by a domain specific device. The output is then written to the location held in the Interaction Device and is then available for actual presentation.

Generation of User Interface The actual generation of a User Interface is done by a Presentation Element and an Interaction Device. The Presentation Element executes its associated Widget and the Widget generates XML that specifies the content of the User Interface. This XML is handed off to the Interaction Device's associated XSL Driver. The XSL Driver converts the XML to the format expected by the Interaction Device. At this point the real-world Interaction Device can present the IDS generated User Interface.

Complex Task example

Lois Anderson wants to SELECT the medications she wants refilled, SELECT the pharmacy to fill the prescriptions and to ASSERT the person that she intends to pickup the medications. This could be modeled in several ways, however for the sake of example, assume that the pharmacies available are dependent on the medications to be filled and that the pickup person is not related to either the medication type or the pharmacy selected. The resulting task model could then be: "Fill Prescription" consists of an AND Junction followed by two Task paths, 1) "Select Medication" which is followed by "Select Pharmacy" and 2) "Assert Pickup Person." These paths are rejoined in another AND Junction.

To define the scope of the Junction's operator, all Junctions preceding Task paths are paired with another Junction following the Tasks. Graphically this would look like Figure 12, where the rectangles are User Tasks, the directed line segments are Relations and the half ellipses are AND Junctions. Each of these graphical objects has an associated Java object in IDS that is used in the Presentation selection process.

Graphical Task Model

The domain specific tasks can be directly created using Java or they can be read in via an RDF file or a JDBC data source. This paper will show how the RDF representation can be processed by IDS to conceptualize the graphical model and thus the underlying Java code.

Here is a snippet from an RDF file that describes the Fill Prescription User Task.

 <userTask rdf:about="idsTask_00025" name="Fill
             Prescription" primitive="NONPRIMITIVE"> <entryTask
             rdf:resource="idsTask_00026"/> <exitTaskrdf:resource="idsTask_00029"/>
             <matchRequirements rdf:resource="idsTask_00036"/> </userTask>
             

From the snippet we see that there exists a User Task (idsTask_0025), named "Fill Prescription" with primitive type NONPRIMITIVE. A primitive type of nonprimitive indicates that the User Task will contain subtasks, including a first (entry) Task and a last (exit) Task. The initial graphical representation may look like Figure 13. Note that identifiers have been added before the Task name and shorted to the last two digits to improve readability.

Creating a Nonprimitive User Task "Fill Prescription" has stubs for the entry and exit Tasks.

The entry Task references "idsTask_0026", so we look it up in the RDF file and create the appropriate object.

 <junction rdf:about="idsTask_00026" identifier="AND1"
             operator="AND" primitive="JUNCTION"> <parent
             rdf:resource="idsTask_00025"/> <mate rdf:resource= idsTask_00029"/>
             <followingRelations rdf:resource="idsTask_00030"/> <followingRelations
             rdf:resource="idsTask_00034"/> </junction> 

idsTask_0026 is identified as "AND1", has an operator AND and a Primitive of type JUNCTION. This allows us to add in an AND gate as the entry task, which graphically looks like Figure 14.

Adding a Junction "AND1" is the entry Task for "Fill Prescription"

"AND1" has two followingRelations. Our process follows the first Relation until we reach the mate of "AND1," which is resource idsTask_00029. The RDF look up of the followingRelation idsTask_00030 supplies the following.

 <relation rdf:about="idsTask_00030"
             identifier="Relation1" relationship="PRECEDENCE"> <source
             rdf:resource="idsTask_00026"/> <destination
             rdf:resource="idsTask_00027"/> </relation> 

The relationship type is one of PRECEDENCE, therefore the source Task is evaluated before the destination Task. Graphically this is shown as a directed line segment from the source task to the destination task. "AND1" is the source Task and the destination Task, idsTask_00027, is looked up in the RDF file.

 <userTask rdf:about="idsTask_00027"
             classElement="DomainItem" identifier="tskSelectMedication"
             itemType="medication" name="Select Medication" primitive="SELECT">
             <parent rdf:resource="idsTask_00025"/> <precedingRelations
             rdf:resource="idsTask_00030"/> <followingRelations
             rdf:resource="idsTask_00031"/> <matchRequirements
             rdf:resource="idsTask_00036"/> </userTask> 

The Task's name is "Select Medication" and it has a Task Primitive of type SELECT. Graphically this is a rectangle containing the task name placed after the relation arrow.

Adding a Relation and Subtask "Relation1" connects "AND1" to "Select Medication"

"Select Medication" has a followingRelation, idsTask_00031, which, if we continued tracing through the RDF file, we would find was "Relation2" and connected the Task "Select Medication" with the Task "Select Pharmacy." Further, we would find that "Relation3" connects "Select Pharmacy" with "AND2", a Junction.

Adding Relations and Tasks to Reach the Second Junction Completing the first Task path between "AND1" and "AND2"

"AND2" returns us to its mate "AND1" to look for more followingRelations.

From the RDF file we would find that "AND1" precedes "Assert Pickup Person" and that "Assert Pickup Person" precedes the "AND2" Junction. This is shown in Figure 17.

Adding Second Task Path Completing the second Task path between "AND1" and "AND2"

This time when returning to "AND1" from its mate "AND2," we find no more followingRelations to process. This causes us to return to "AND2" to look for its followingRelations. Upon finding none we check to see if "AND2" is its parent Task's exit Task. We find that "AND2" is the exit Task for its parent, "Fill Prescription" and that "Fill Prescription" has no followingRelations of its own. Therefore, we graphically connect "AND2" with the exit Task line segment and consider the Task, "Fill Prescription," completely specified.

 Ontology  For a non-programmer, a tool that can take the idsTask RDF schema and aid users in creating task models, this input method becomes critical. In the future we may write a graphical task-modeling tool, but for now we recommend using "Protege,"  an ontology builder created at Stanford University. An ontology provides a common vocabulary and common set of property names to describe relationships between vocabulary items. "Protege" outputs ontologies and domain models based on the ontologies in RDF Schema and RDF files respectively. The RDF and RDF Schema snippets shown in this paper are not exactly what "Protege" outputs. They have been simplified to make them easier for us to follow.

The Schema captures the Class names, properties and connectivity that become the framework and rules for specific domain generation. The RDF file captures Tasks modeled for a particular domain. Multiple Tasks may be included in one RDF file and multiple RDF files may be processed into the IDS Task Library.

Presentations

Generating Presentations   Presentations   getQualifyingPresentations   Presentations contain all the components needed to produce an interaction. They are generated by a method in the Interaction Requirement called getQualifyingPresentations. This method creates a collection of Presentations capable of supporting the Task, presenting on an available Device and meeting the Usability Requirements. Each Presentation connects to the Interaction Requirement, to a single Interaction Device and to a single Presentation Element. It also contains an overall score indicating the quality of the Presentation in handling the Task and its Usability Criteria Match Requirement.

A partial view of a DeviceSignature's characteristics

Presentation Elements

Presentation Elements Presentation Elements characterize how well their associated Widget can support an Interaction Requirement. It imposes constraints for selecting an Interaction Device capable of presenting the Widget. For example, a map Widget may require a 300 pixel by 400 pixel color display. If none of the available Devices have a color display of at least 300 by 400 pixels in size, the Presentation Element would not be considered as a candidate for use in a Qualifying Presentation.

Likewise, Presentation Elements declare which Task Primitives the associated Widget affords. Some examples are: listboxes afford selections, auditory alarms afford direction of attention, and edit boxes afford assertions, retractions and changes.

The Presentation Element's final constraint applies to the type of information it can present. Standard types are Numeric, Temporal, Text, Graphic and Enumeration. Each of these types is called a Signature and has properties that constrain the type further. For example a TextSignature specifies the minimum and maximum number of characters that can be presented and a graphic signature specifies the number of vertical and horizontal pixels as well as the number of colors that can be presented.

We have shown how a Device Signature can be entered into IDS with the XML for an Interaction Device. We can enter the other Signatures in a similar way using XML . The RDF Schema for input via an ontology generation program is in development now.

Currently Presentation Elements and associated Widgets are written in Java. A way to input Presentation Elements through a text file is being explored, as is the use of external Widget libraries.

Presentation Selection

Presentation SelectionThe getQualifyingPresentations method returns all the Widget-Device pairings that are capable of handling the Task. From this set of Presentations the associated Widgets will be queried to determine how well the Widget on the given Device rates on each of the Usability Criteria scales. For example, a message sent as a phone call will have a higher display obtrusiveness score than a message stored on a webpage. The level of support is then compared with the Task's Match Requirements. If the Task had a high requirement for display obtrusiveness, the webpage presentation may not be included in the set of Qualifying Presentations.

After comparing each Usability Criteria supported by the Widget-Device pairing to the Task's Match Requirement criteria, an overall score is assigned to the Presentation. Only the top scoring Presentations are returned to the Interaction Requirement as Qualifying Presentations.

Widgets

WidgetsEach Presentation Element has an associated Widget. The Widget is responsible for pulling information from the Interaction Requirement to create the XML string containing the content for Presentation. Once the specific Presentation to use has been selected, that Presentation's Widget is executed. Generally the Widget will use the Task's name as a label and collect most of the content from the Interaction Requirement's Domain Element.

Interaction Content The Widget's execute method formats the interaction content to be displayed in a well-formed XML statement that complies with the same Schema against which the XSL driver was written. If the task is to RECEIVE text or images, generally the Widget will read it from the Domain Element and be done with it. When the task is to RECEIVE something more complicated, the task's elementClass is checked for the type of information expected. For example, the listing of medications being taken by a client would have an elementClass of type "medication." If the Interaction Requirement's Domain Element is a list of medications, the Widget takes those and looks no further. However, if the Domain Element was of another type, for example "person," then the Domain Model is queried.

The Domain Item held in the Domain Element property holds a list of Domain Relationships. If any of them are to Domain Items of type "medication," those relations are followed and the Widget generates the medication list. If the Domain Relationships don't map to medications then the Domain Rules structure is queried to get all chains between a "person" type and a "medication" type. Then links of Domain Relationships following a rule chain are looked for and the medication list is constructed from the "medication" found on the chain.

For example, selection of a medication from a list of five medications could be done visually using radio buttons, a list box, a menu or a set of links or it could be done via the phone with a numbered menu system. For the sake of example, assume the best presentation is a listBox displayed on a small browser.

The listBox Widget may be coded to create a well-formed XML string that contains:

 <list name="X" displaySize="N"> <listItem
             value="Yi">Zi</listItem> </list> 

such that, X is the name of the Task that the Presentation's InteractionRequirement supports, N is the number of lines to show in the listbox and Yi and Zi are the names of the medications held either in the Presentation's InteractionRequirement's DomainElement property as a list of medications or formed by the method described above.

Assume the designers have decided that for usability and space constraint issues, the listbox displaySize cannot be less than 3 or greater than 9.

Here is an example XML string produced by the listbox Widget supporting the "Select Medication" Task.

 <list name="Select Medication" displaySize="5">
             <listItem value="Coumadin"> Coumadin </listItem> <listItem
             value="Dexasone"> Dexasone </listItem> <listItem
             value="Dilantin"> Dilantin </listItem> <listItem value="Lasix">
             Lasix </listItem> <listItem value="Prozac"> Prozac
             </listItem> </list> 

When the Widget has the content to be displayed, it encodes the information into a well-formed XML string, like the one above.

XSL Driver

Once the Presentation's content is provided in XML , the Interaction Device's XSL Driver completes the transformation into a format supported by the Interaction Device.

Consider an Interaction Device which is a simple web browser supporting basic HTML . The associated XSL Driver would then transform the XML string into an HTML page. The resulting HTML might be in the following format, given the XML string sent by the Widget in Section 5.5.

 <form name="formName" method="post"
             action="response.asp"> <p> <b>Select Medication</b>
             <br/> <select name="Select Medication" size="5"> <option
             value="Drug1"> Drug1 </option> </select> <br/> </p>
             <p align="center"> <input name="Submit" type="submit"
             value="Submit"/> </p> </form> 

The XSL Driver can then write the results of the transformation to a location specified by the associated Interaction Device, completing the interaction generation process.

Current and Future IDS Enhancements

Methods for easily adding Presentation Elements and using existing Widget Libraries are currently being explored.

In 2002 we intend to add User Modeling to the system and use the information to design interactions suited for the particular user or type of user. In the home healthcare environment user types might include the elderly, the informal caregivers, the medical community and emergency dispatchers.

We hope to include IDS in other systems supporting a variety of domains to emphasize its generic nature. For example, in a petrochemical plant, there are Plant Supervisors, Board Operators and Field Operators. A Plant Supervisor may carry a WAP phone to receive status reports or to navigate between reports from various units of the plant. A Board Operator monitors and controls a unit of the plant. This Operator generally has three or more large computer screens with which to interact. A Field Operator takes readings, inspects systems and manually controls equipment within the plant. This Operator may have a radio communication device and a handheld computer with Radio Frequency communication capabilities.

As you can see, the petrochemical plant has many Interaction Devices and several User Roles requiring different interaction access and presentation needs. This is one of many possible domains in which to apply IDS .

There are many ways the system can be enhanced. Adding visual tools for modeling Tasks and Domains would add to the ease of using IDS . Making the Domain Item selection reasoning 'smarter' would both simplify the developer's Task Modeling and would increase the system's ability to track down appropriate information for a given Task. These and many other enhancements are being considered for future work on IDS and we are always looking for new domains needing User Interface generation tools and techniques.