W3C's Multimodal Web

What Not To Do

Some attempts at making the Web mobile and multimodal have failed or are bound to. As appealing as they may seem to quickly solve the problem, they all run the risk of fragmenting the Web into several incompatible sub-webs. The sources of that risk are listed below:

• Rebuilding the Web from scratch: WAP had similar concerns regarding device capabilities (low arithmetic capability, small memory capacity, energy consumption control, etc.). As a result, The WAP Forum defined a complete OSI-like set of network layers, at the top of wich the WAP Markup Language, as well as WMLScript and the WBMP picture format, were going to be the equivalent (but not compatible) to HTML, JavaScript and GIF, but for WAP devices. The effort proved to be unsuccessful despite the hype and investments, and even if it is sometimes claimed that the reason was poor marketing and the limitations of 2G networks, it is doubtless that the complexity of the new design is a main cause for WAP's demise.
• Writing custom sets of HTML pages, each page adapted to a category of devices. It is well known that this approach works as long as the amount of categories is limited. Having different sets of URIs for different versions of the same site is an extra burden (shared bookmarks, etc.). Even if it often the case that the adapted pages are be generated from a single source, the burden of maintaining everything in sync remains very costly.
• Device specific portal: a web site (most often a provider's entry page) is written to cover "all" the customer's needs (weather, news, movies, etc.). That web site is usually designed for the specific devices that provider offers, so the types of devices supported are limited to the few the provider supports. Where this breaks is of course when the user tries to access any other web site, as it obviously will not have been designed specifically.
• Another cause for Web fragmentation is the tentative ".mobi" initiative, which, if it happens, will allow "mobi" as a top-level domain name. Company XYZ would thus be able to have a "mobile-enabled" version of its site at http://www.xzy.mobi. This approach is not only architecturally wrong (top-level domains describe the server, not the client) but would also create much confusion about what is "mobile" or not: should a laptop computer access acme.mobi or acme.com? Pushing the argument further would incite the creation of even more domains describing new classes of devices TBLMobi.

What to do

Despite the initial appeal of the solutions above, they all prove to be a poor choice. However, better alternatives and sound technical solutions exist. Some are simple and can be applied quickly, while others are further ahead but will hopefully enable a Web that does work across all environments.

• Writing accessible content: the initial step in order to have content that works across devices is to use web standards and accessiblity guidelines, which readers of this article will already be familiar with. A web site designed using a combination of techniques developped with device independence in mind (style sheets, WAI guidelines, etc.) will readily work on a large range of devices.
• Smart browsers: new browsers are written especially for small screens and add functionality and heuristics to display complex content. For instance, by ignoring fixed-width directives in the source HTML in order to avoid horizontal scrolling, or by linearising tables which are used for layout.
• Browser "sniffing": right now, the method of making the server detect browser information from the HTTP request and dynamically adapt its contents works. A few popular Web sites use that technique and will, for instance, serve Javascript-less pages if it detects a browser which known not to support Javascript. Doing it dynamically avoids the problem of synchronisation and storage mentioned above. But again, this works as long as the number of browsers remains small, as as long as the server can withstand the load caused by dynamically adapting content.

Advanced Delivery Methods

If used properly the solutions mentioned above could make the Web work on a fair portion of modern devices. It is to be expected that as more and more try to access the Web, site administrators will realise that they have tools to handle the new scenarios. Yet, these methods will work only to an extent. As mentioned in the introduction, the hardware market evolves at a fast pace and the web infrastructure has to adapt to it in more advanced ways.

A better way of handling server-side adaptation is to handle device characteristic on a feature by feature basis. Because a list of all browser is difficult to maintain, a mechanism to access the browser's characteristics from the server is much more scalable: instead of trying to adapt using what the server knows of the browser (if it knows the browser at all), the server can retrieve more explicit information such as screen size, number of colours, whether the device has a speech recogniser, etc.

The Device Independence Activity of the W3C DI is exploring this area. In 2004, the Content Capabilities/Preferences Profile specification was released CCPP, defining a framework for describing device capabilities and user preferences, and how they could be used to perform content adaptation tailored not only to the terminal but also to the person using it. The Open Mobile Alliance has adopted CC/PP as the standard for machine-readable mobile device description.

The DI Activity is also exploring ways the delivery context provided in CC/PP form is generated and travels back to the server, as well as other intermediaries which could perform partial adaptation to alleviate the load on the server and on the client.

Rationale

The principles we have described so far are addressed by the device independence activities, as well as by recent versions of well known specifications (HTML, CSS, XForms, etc). However, real multimodal interaction will require a bigger effort:

• The specificity of new input devices needs to be addressed: new input modes like voice recognition, handwriting, even keypads, are much more inaccurate than the usual keyboard and mouse combination. This problem has to be addressed at a higher lever than that of the browser: an application author must provide specific information in order to design multimodal web content (grammars, time-out values, etc.)
• The combination of different input modes must also be addressed at the application level. Contradiction can happen (e.g. the user says "yes" but clicks on "no"), or multi-input instructions (user says "I would like to go from home to there", while he points to a location on a map."). Again, the author needs to be able to tailor their application to specify a proper behaviour.
• The interaction mechanisms will appear as a result of certain combinations of input and output modes, making the currently ubiquitout "HTML page with links and forms" model of interaction no longer be sufficient. For instance, a speech only browser would need specific information from the Web page or service, about the reading order, or what to do if the user interrupts the aural rendering of the page, etc.
• As mentioned in the introduction, external parameters like ambient noise or battery level can influence the way the Web application is presented to the user.
• Input or output modes could change during the interaction with the application. For instance, the user may request to migrate a session with a roadmap Web application from his desktop computer to his car navigation system, thus migrating session data and changing the input mode from keyboard and mouse to speech input.

A study of requirements and use cases UseCases has been conducted by the multimodal interaction working group, leading to the definition of the MMI Framework Framework.

Multimodal browsers

The MMI framework describes a conceptual architecture which modularises the curent "monolithic" browser model, introducing components whose interaction fulfills the requirements listed above.

The goal of the work ensuing from the framework is, once essential components have been identified, to standardise the interfaces between these components. Interoperability in this area is crucial, because it is very likely that components (e.g. handwriting recogniser, speech synthesizer, GPS interface) will be built by different manufacturers. It is also worth noting that most of the resulting specifications will either reuse (or build on) existing Web standards

The MMI Framework

Figure 1 shows a simplified view of the Framework. Between the user and the Web are five components, each taking care of particular multimodal features:

• The input component comprises all the input modes: speech or gesture recognition engines, etc.
• The output component comprises all the output modes: screen, speech, audio, haptic devices, etc.
• The interaction manager connects all the components together: its role is to select the type of interaction (visual, linear, graphical or text-only) from the modes it finds connected to the user, or from the data it receives from the system-and-environment component
• The System and Environment component is in charge of providing values for environmental conditions: battery level, location coordinates, CPU speed, etc.
• The Sessions component is in charge of managing how a session with a Web page, or application, can migrate from one modality to another, or between multimodal browsers.

The MMI Framework (detailed view)

The next figure shows more details for the components listes above, as well as new ones which serve specific functions: the role of the "integration" component is to combine the data produced by each input component into one unified stream that the interaction manager receives. The component takes care of mixing and disambiguating input, handling the possible contradictions and uncertainties. The "Synthesis" component does the opposite: from one output stream the interaction manager produces, the component generates output data specific to the output modes available.

Note that the components are only conceptual and do not necessarily represent particular hardware or software components, nor does the framework mandate any implementation.

The MMI Framework (detailed input and output views)

The description of the framework goes deeper, and the following figures, copied from the W3C Note Framework, shows even more detail. We won't go into more detail here, but one thing to mention is that both figures show, in red, where existing W3C technology can be found: SSML, CSS, SVG, etc. for output; SRGS, and SI for input. The rest of the needed standards are being developed within the group.

Among the standards that are in development, or expected to be developed are:

• A specification of how input or output objects connect to the interaction manager: the Working Group has published the "Modality Component to Host Environment DOM Requirements and Capabilities Assessment" MID, which describes requirements for a DOM-based API between "modality components" (e.g voice, pen, visual display) and the "host environment", which coordinates the activities of the components, for instance: activate, deactivate, display or prompting.

• The specification of a wrapper format for transporting input data from the modality components to the interaction manager: the Extensible Multi Modal Annotation language EMMA is an XML vocabulary to represent recognition results. An instance typically contains one or several possible values of an interpretation, along with related metadata (time, device information, type of modality, confidence scores). The sample EMMA file below shows two alternative interpretations of a spoken utterance, each with different confidence values:

  <emma:emma version="1.0" xmlns:emma="http://www.w3.org/2003/04/emma"> 
    <emma:one-of id="r1" emma:start="1087995961542" emma:end="1087995963542">
  
      <emma:interpretation id="int1" emma:confidence="0.75"> 
        <origin>Boston</origin>
        <destination>Denver</destination>
        <date>03112003</date>  
      </emma:interpretation>
  
      <emma:interpretation id="int2" emma:confidence="0.68">
        <origin>Austin</origin>
        <destination>Denver</destination>
        <date>03112003</date>
      </emma:interpretation>
  
    </emma:one-of>
  </emma:emma>
  	  

  EMMA data is generated by input modality components and is later processed by integrations components, which perform operations like selecting from a list of alternatives, or merging the inputs from different modalities, eventually resulting in the interaction manager receiving a single unambiguous result.

• A compositing mechanism: the process of merging or selecting input must be described in some way by the application designer. It should be possible to specify a set of rules to indicate, for instance, that the interpretation with the highest confidence value should be selected, or that contradictory values between simultaneous voice and keyboard results should cause voice data to be ignored, etc. The rules would be expressed in a format that would allow an integration engine to apply those rules to EMMA documents. The Working Group has not yet published any document on this work item.

• System and Environment component: the "Dynamic Properties Framework" DPF is a specification which standardises IDL interfaces for the interaction manager to access system parameters or query environmental conditions and adapt the application accordingly. The example below shows DPF interface functions used in an XHTML document which reacts to the device's battery level:

  <html xmlns="http://www.w3.org/1999/xhtml"
   xmlns:dpf="http://www.w3.org/2004/11/dpf"
   xmlns:sel="http://www.w3.org/2004/06/diselect">
    <head>
      <title>Battery check</title>
    </head>
    <body>
      <sel:select>
        <!-- battery level is a number between 0 and 100 -->
        <sel:when sel:expr="dpf:component()/device/battery &lt; 20">
          <h1 class="alert">Low Battery</h1>
          <p>You are running low on power!</p>
        </sel:when>
        <sel:otherwise>
          <h1 class="alert">High Battery</h1>
          <p>You have plenty of power!</p>
        </sel:otherwise>
      </sel:select>
    </body>
  </html>

  The DPF specification doesn't provide a list of properties. Similarly to CC/PP, they are expected to be defined by other organisations, either for standard or for proprietary uses.

• InkML InkML: an XML language to represent "digital ink" traces, for handwriting recognition and other pen-based applications.

  <ink>
    <traceFormat id="xy">
      <regularChannels>
        <channel name="X" type="decimal" units="mm"/>
        <channel name="Y" type="decimal" units="mm"/>
        <channel name="P" type="decimal" units="N"/> <!-- pressure -->
     <trace>
       10 0 9 14 8 28 7 42 6 56 6 70 8 84 8 98 8 112 9 126 10 140
       13 154 14 168 17 182 18 188 23 174 30 160 38 147 49 135
       58 124 72 121 77 135 80 149 82 163 84 177 87 191 93 205
     </trace>
     <trace>
       130 155 144 159 158 160 170 154 179 143 179 129 166 125
       152 128 140 136 131 149 126 163 124 177 128 190 137 200
       150 208 163 210 178 208 192 201 205 192 214 180
     </trace>
  </ink>

  InkML is meant to represent either handwritten input, or pen gesture data, as sent from a graphics tablet to a handwriting or gesture recogniser. Alternatively, it can be used to specify handwriting or gesture grammars, with trace instances representing expected input from the user, expressed in a canonical form.

This list of specifications the MMI group is working on is growing, and should eventually cover the full framework. But as we mentioned before the Multimodal Web won't happen on the browser side only, and so the group is also investigating how multimodality is going to affect what's on the servers, in particular authoring languages for web application.

Multimodal Web languages

the Multimodal Web will not only be enabled by defining a new browser architecture, but also by creating a new generation of Web languages, suited for multimodal interaction. Existing Web pages do not yet provide modality dependent text (spoken or written), or modality dependent interaction (e.g. specify grammars or timing information).

New markup, preferably extensions to HTML, would therefore be needed. One can always add new functionality through scripts, but declarative markup is always preferable, and proto-multimodal extensions to HTML have already been published by other groups: XHTML+Voice X+V and Speech Application Language Tags SALT define extension markup to handle voice interaction: specify prompts, grammars, time-out values, etc. The listing below are examples of X+V and SALT, respectively, showing how grammars can be specified for user input:

    <vxml:form id="voice_city">
      <vxml:field name="field_city">
        <vxml:grammar src="city.srgf" type="application/x-srgf"/>
        <vxml:prompt id="city_prompt">
          Please choose a city.
        </vxml:prompt>
        <vxml:catch event="help nomatch noinput">
          For example, say Chicago.
        </vxml:catch>
      </vxml:field>
    </vxml:form>

<salt:listen id="lsnName" onreco="askTravel.start();">
   <salt:grammar id="gram2" name="gram2">
      <grammar version="1.0"  xml:lang="en-US" xmlns="http://www.w3.org/2001/06/grammar" root="aUser">
         <rule id="aUser" scope="public">
            <one-of>
               <item>Fred</item>
               <item>Sam</item>
            </one-of>
         </rule>
      </grammar>
   </salt:grammar>
   <salt:bind targetelement =" saltdebug" value="/" />
   <salt:bind targetelement =" userName" value="//"/>
</salt:listen>

Another way of extending Web content ensues from the observation that the concept of styling HTML pages could be generalised to that of "skinning Web applications": a standard web page can not only have its appearance defined by a stylesheet, but also the way the user can interact with it. Standard stylesheet selection mechanisms could allow controlling speech input or pen interaction with a page, for example for form filling or following links.

A language embracing this concept is CSS-MMI CSS-MMI, which adds new properties and selectors to CSS. For instance:

#wants-drink:focus {
   prompt: "do you want a drink?";
   grammar: yes | no;
   reprompt: 3s;
   next: "yes" #beverage "no" #food;
   }

Different modes of interaction can be specified by using CSS's @media rule:

@media speech {

  /* rules applicable to the speech media type */

}

@media handheld {

  /* rules applicable to handheld media type */
}

The three languages mentioned here part of a wider range of media-independent authoring markup languages, many of which are used internally and are usually transformed back to other media-specific languages like HTML or VoiceXML. See the papers presented at the W3C Workshop on Multimodal Interaction MMIWS for a few examples.

One of the goals of the multimodal working group is to develop an authoring language standardising the proposals above, to allow simple multimodal capabilities to be added to existing markup languages in a way that is backwards compatible with the current Web, and which builds upon widespread familarity with existing Web technologies.