Maintaining Ontology Implementations: The Value of Listening

Human Communication

The most basic model of human communication has a sender producing a set of symbols that are consumed by a receiver [Cummings, Long and Lewis, 1983].

Figure 1: Direct communication between two people

[Link to open this graphic in a separate page]

The reason direct human communication works (well, usually works) is because there is a feedback loop that allows a process of refining the symbols and their interpretation to increase alignment of the meaning of the symbols between the sender and the receiver. The nuances involved in creating shared meaning also rely on the sharing of context, experience, and secondary information that help to round out the concepts that are being shared.

Figure 2: Direct communication relies on feedback to achieve the intent

[Link to open this graphic in a separate page]

Shannon and Weaver [1949] extended the simple, direct communication model to represent the role of technology devices (in their case, the telephone).

Figure 3: Shannon and Weaver's Communication Model for Telephone Systems

[Link to open this graphic in a separate page]

Their focus was on the direct signal processing of audio via telephone equipment, and the need for technical quality (noise reduction) to limit the potential for miscommunication. They did not go into the limitations of audio-only communication on the quality of the communication experience more generally (lack of visual non-verbal cues, dependence on synchronous communication, etc.). If we applied that same perspective to computing, it would be like focusing on the screen’s pixel resolution and number of colors displayed – i.e. we are still left to explore what messages and symbols are being facilitated or filtered by the characteristics of the specific technology involved. Exploring these latter issues is increasingly important the more we mediate human communication through computers, the more we use terms like meaning, intelligence, and knowledge, and the more we load our interfaces with inconsistent, culturally-bound metaphors and representations.

Chandler [1994] identified gaps in the Shannon and Weaver model as it has been more generally applied to organizational communication. Of particular interest to us is our understanding of the goals and practical implementation challenges of ontologies in computer/information environments, and specifically the lack of feedback and context. Here, feedback refers to the active role of the “destination” in successful communication, and context refers to situational information (either explicitly or implicitly provided). Chandler goes on to say:

The Computer's Non-Neutral Role

There is a relentless drive toward using computers and the Web as the primary point of interaction between people and organizations. So many areas of our lives are now online experiences, whereas once we used to go to an office to conduct a transaction (buy goods, renew a drivers license, apply for benefits, manage money, find reference books in a library) or make a telephone call to receive/give information (product support, customer services, making travel arrangements, or complaining to our local Congressional representative). We used to interact with other people, not with e-forms and search engines and site maps as a proxy for people. Once communication was more synchronous and direct, now it is more often asynchronous, indirect, and impersonal.

What is the fundamental implication of this? We, the users, must fit our communication into the pre-determined constructs of the software we are interacting with. A huge burden is thus placed on designers (of software, interactions, and information) to predict and support widely divergent user needs. Computers are extremely weak in their ability to tune services or information based on recognizing and interpreting the context of our particular situation. What has been lost? Listening!

We have been exploring the impact of computer-mediated communication as part of our practical project work with both transactional systems (people carrying out tasks, often with more than one party involved) and content systems (people seeking and using formal/semi-formal written communication). Next, we’d like to explore the computer’s role and potential impact areas on communication. It has been by understanding where the impact lies that we begin to understand where to focus our design efforts and technologies.

Figure 4: The impact of computer involvement in human communication

[Link to open this graphic in a separate page]

As you see in Figure 4, the communication situation becomes much more complex. The Sender transmits the content/data into the computer, which has some form of standardized filing system. The structure of the filing system and the sophistication of the application that manages the activity dictate the level of the “conversation” and feedback that the Sender experiences. The conversation often consists of confirming whether the data provided conforms (or not) to a limited, standardized schema. Data conformance requirements are based on decisions (which are in many ways predictions) made by the designer of the data capture application months or years before the Sender decides to provide the information.

It is possible (in fact, likely) that the designer responsible for predicting how the content/data will be used is different from the storage designer who designs a retrieval application to support that intended use. Data may undergo transfer or translation, where again some aspects that are valuable for shared meaning may be altered in some way. This is increasingly the case when the idea of reusable content becomes a priority among technologists and user organizations.

Finally, the Receiver expresses a need in the limited language of the computer retrieval application. The feedback available to the Receiver is limited, in many cases today, to a list of available data items – in the case of no items being available there may be a message, which may describe error conditions. That is the extent of the feedback to the user, and often it is completely ignorant of the person’s context, or why they are asking to retrieve information.

Does an Ontology Framework Help?

We believe that there can be an improvement – and simplifying – of the model with the introduction of semantic technologies and an ontology framework for applications and content. We believe the model could begin to look more like this:

Figure 5: Ontology-based communication, using semantic technologies

[Link to open this graphic in a separate page]

The key improvement here is a framework for harmonizing the language and definition bases used by the computer during the course of interactions with both Sender and Receiver. The ontology in this case participates in two key activities:

• Providing a map to standardized language that represents the contexts and domains served by the computer application
• Providing a framework for managing language evolution – and potentially meaning – in order to maintain the mapping between the parties (human and computer) involved

It is important to understand that the simplification occurs regardless of whether the communication system is defined through a more informal localized system where agreements are reached through a handful of entities to perform a specific function, or through a global system pushed through industry standards and alignment, for example.

Ontology-driven applications can be a very good thing, and make the conversation with the computer better – if the ontology is relevant and up-to-date! If the ontology begins to ossify, it loses relevance to the human participants in the dialog, and would increasingly become a barrier to communication. This particular problem is a bigger risk to global standards where there will be more resistance to change, but that resistance could easily make the standard not useable and possibly replaced by improved versions. The key issue is that it is impossible to predict all possible language requirements and changes in advance, so the ontology must be able to learn and evolve. But how?

Feedback Paths in Computer Applications

What’s missing in the above models is feedback between the Sender and the Receiver. It is feedback that allows the content/data (and the descriptive mapping of that content to the Receiver’s situation – the ontology) to be refined and improved. What types of feedback mechanisms are available currently, between Sender and Receiver? Figure 6 illustrates the current, limited feedback in many applications.

Figure 6: Two types of feedback, both outside the communication channel

[Link to open this graphic in a separate page]

Feedback channels fall into two broad categories. One (increasingly of interest to the KM community, through a focus on collaboration and Communities of Practice, e.g. [Wenger, McDermott, and Snyder, 2002], [Preece, 2000]) is to return to the simpler model of direct communication – sending an e-mail to the author to start a conversation, calling a help desk, or asking your neighbor.

The other feedback mechanism is slower because it requires identifying meaning mismatch through the symptoms of poor task performance. Most MI (management information) systems today are not set up to accommodate the integration of performance data from transactional systems and information access systems to allow people to track and correlate findings. They also do not necessarily distinguish between types of interactions that could help distinguish ”conversational” problems from other forms of performance issues.

Neither of those two types of feedback lends itself to broad-based, systemic, ongoing improvement of the overall application, content or ontology. There is clearly a need for improved conversations between computers and users, and richer mechanisms for feedback to support refining the design (hence, dialog) of applications in a way that evolves naturally with continual improvement.

Ontology Feedback

So, if the ontology used by semantic technologies within/supporting applications, enhances the ability of the computer to listen in the course of a conversation with a user, then it has to become self-aware to be effective. The big question is: how well does it understand the language of the user?

Figure 7: Feedback from users refines the relevance of the ontology

[Link to open this graphic in a separate page]

The feedback paths for the ontology are both explicit (suggestions and direct entry) and subtle (pattern analysis, dynamic listening, clarification-seeking, meta-learning, alerting for alternatives and drifts). Both are important and must be considered in system design. Some of the ways of doing this are explored in the next section. The interesting thing is that the design approaches that help us maintain an ontology can also gather feedback that allows us to use the ontology to help with more general feedback between users, content creators, and application developers.

Relevance = Context = More Useful

KeeKeeping the ontology relevant to users means keeping it aligned with the user’s context. Everything else is unimportant if we want computers to work more effectively on our behalf and communicate with us in a more meaningful way. Understanding, interpreting and maintaining context (rather than just representing it or codifying it in markup) is the truly difficult problem to be solved.

Many communications problems with face-to-face conversation are due to semantic differences – when one person does not fully understand what the other is saying. Context is not only situational, it is also experiential. Two people in the same place at the same time still come away with different perceptions and knowledge. Our understanding of context is informed by our knowledge relating to the situation, our prior experiences, our expectations, and our emotions. How well will software agents operating on our behalf know all the rich aspects of context, including our goals, expectations, and mood? How well will computers be able to represent that situation to other agents and people, so they can carry out their own actions on our behalf?

We predict it will take a long time to refine not only how to properly represent context so that it can be used in a meaningful way, but also to develop the required learning techniques that keep the context relevant and current, and the social networks to refine them [Behrens and Kashyap, 2002]. This, too, will evolve through experience, but its importance cannot be lost or underestimated.

Expansion of scope for a particular subject

Subject definitions are not static, and their meaning changes based on legislative changes, colloquial uses or re-uses, cultural variations and general language adoption. For example, the introduction of ten new countries (and nine new languages) to the European Union will prompt more than just a huge demand for translation services. It also prompts a range of reinterpretations of existing laws and information, as the carefully crafted language that represents the intentions of legislators has to be aligned with phrases and cultural expectations of new member countries. This will include re-examinations when “we don’t have a word for that in our language” or “if we implement that directive, it will undermine these other tenets in our existing legislation.”

Changes in organizational structure

Mergers, acquisitions, reorganizations, selling off of business units happen every day for a variety of reasons. We often understand what the business rationale behind the decision was, yet we sometimes underestimate long-term language/meaning shifts when we bring entities together or split them apart. It is rare that separate entities have the same definitions and ontological views of even the same business. How do they come together? How do we align our old historic information? What constraints have been lifted where we can improve our definitions? Where are our relationships with new and old partners affected? How do we represent that? Where should we evolve and how?

The U.S. phone companies went through this entire process over the last 20 years since they first split up to be the “baby bells.” During the split, copies of the billing system were provided to each operating unit. Over time, the systems in these operating units evolved totally independently. Each established different descriptions to define the service being ordered and pricing models – contained in an ontology (that is, USOC/FID) – that triggered no less than 16 computing systems just to turn on one “plain old telephone” (that is, a POTs) line for a customer. Subsequently, the “baby bells” began to merge into new entities like Verizon, bringing together these systems that now contained totally different ontologies. In many cases the original staff who knew how the codes were interpreted had retired, so the actual meaning of the codes was lost. New pricing models and operational consolidation has been nearly impossible. They still struggle with the issue of finding a common meaning encapsulated within a single, comprehensive ontology that describes the full range of pricing and services in a single product catalog that characterizes and is constrained by the switch capabilities of the network (a hardware constraint).

Changes in law, court interpretations, precedent, and legislative intent

Some changes in law have major impact on the way we see the world and where we get the information we need. One example of this is the introduction of “accessibility” legislation (such as “Section 508” in the U.S.) – such legislation both broadens definitions of who computer users are and what rights of access they have. At the same time, many software managers are narrowing the concepts of access to software functionality – where they focus on disabled people rather than a wide range of definitions for what constitutes “access.” Another recent change was the creation of the U.S. Department of Homeland Security. This newly created agency took on the aggregation of multiple agencies and parts of other agencies bringing them under a single umbrella. It created new departments and reorganized others. Whole new definitions for homeland and security emerged, and these concepts continue to evolve.

Human events changing our perception of what words mean

Some definitions evolve over time to become mainstream. Take for example the word mouse. It did not originally mean the thing you use to drive a cursor around the screen of a computer – but it does now. At first, that meaning was uncommon and rarely used. Chances are when the word mouse is used today, it is more likely to be in the context of a computer instead of a rodent. On the other hand, some definitions change or become more commonly used quickly based on events. A good example of this was how we all personally redefined the term threat after 9/11. Words like terrorism became commonly used words overnight, with both more clarity of meaning (in reference to that event) and at the same time much broader boundaries to its use.

Discovery that prompts new world views

Every day, science and technology advance. New capabilities are discovered, old capabilities are combined with others to create something new, and definitions are clarified, added or replaced. Current areas of discovery where our basic information of what we think we know is challenged includes genetics, space, technology, and evolution (for example, not that long ago, dinosaurs were not thought to be related to birds). New information in any one area can have dramatic impact on entire bodies of knowledge as well as related bodies of knowledge. This is particularly noticeable in medicine – we’ve seen how quickly SARS and mad cow disease have entered the language.

Natural evolution of, and differences in, language and culture

Knowledge is constantly growing. At the same time, our access to both historic and new knowledge is expanding. As a result, we increasingly interpret information beyond the reference points of where we live, who we live with, and any cultural biases we carry. The collective meaning of words like retiree or family have changed quite a lot over the years, and so references to content on these subjects may need to have different alignments depending on the age of the information being referenced. The same knowledge can have slight or significant variations in meaning when the context is taken into account.

Additionally, we have to consider that in some languages we have words the express very different meaning depending on context and other languages have different words to convey the different conditions. For example, aloha can mean both hello and goodbye. So, we may have word equivalents, but we may need context to determine which meaning to use.

The need to monitor and incorporate change is critical to maintain the value and quality of knowledge. Entities that make their ontologies available to others will need to incorporate methods of presenting information about both meaning and evolution so that others can judge whether the data is acceptable for their use. To date, there has been little written into standards for creating the meta tags needed to present this rich – and subtle – information.

Going forward, value, quality and accuracy will be key elements in building trust associated with an ontology. More over, there is a need to continually validate the interpretations made and the use of any information source to maintain accuracy.

Capturing Commonly Used Phrases

Listening to the language of users isn’t as hard as it seems. We ask them to provide data all the time, and they liberally use both formal and informal language. What we fail to do is use that information to routinely refine our ontology.

We must continue to develop methods and frameworks to monitor, find patterns in, and reuse these common phrases. Today, we monitor Web usage logs to gain some insight in use patterns, but this doesn’t go far enough to monitor language patterns. Some applications will log search strings as they are used and even optimize queries based on the commonly used patterns, but we do not often use this information to refine the ontology that defines the organization’s view of the world.

The best information to tap into is the information that people create while simply doing their daily work assignments. It changes organically in response to the people who they interact with daily. We need to tap into the common language and identify the phrases, relationships, trends and shifts. Here are some ideas:

• Gather new information during the course of daily work. Prompt a user to provide information (via their topic selections and navigational choices) to understand how their current situation has prompted a need for information. If the data is found, then you have a clear match between commonly used terminology and available information, strengthening your awareness of the relevance of those terms. The most meaningful feedback from the user comes when there is no direct match. More information may then be solicited from the user to help locate information – and as a by-product to improve the ontology and/or the available content. Transporting the data and user suggestions using standard representation syntax (such as RDF or XTM) allows it to be more easily shared and further analyzed.
• Provide capabilities that allow people to link documents to each other. Today, the best we can do in most environment is ask designated subject experts or look at directory listings on computers of highly organized individuals to understand relationships between documents to improve how an ontology supports standard workflow. There is a growing need and opportunity to create frameworks and methods to improve how people share documents in common file systems. Unfortunately, most file sharing is accomplished through email attachments. With improved methods of sharing and linking documents (in ways that allow computers to read the resulting associations), an organization can gain powerful insights into what is really happening at a workflow level.
• Select commonly referenced topics and associations, and then re-use them to query content resources and facilitate navigation to content. There are relationships across topics that, when represented properly, can lead a user to more discrete topic areas or broader domains – whatever is most useful to them at the time. In this situation, each user can benefit by the actions of the collective. A secondary benefit from this is the ability to listen and detect a context different from the one most commonly used. In an ideal world, a user would be given the ability to have their own common patterns either as a separate source to draw upon or included in the retrieval methods. These do not have to be elaborate. For example, provide “myXYZ” and allow user’s to store their choices, or provide a search history to draw upon for similar situations.
• Monitor changes in selections over time. This begins to tell us where the language presented to the user may be in sync or out of sync with their situations. Shifts in selection patterns may refer to the whole (e.g., the affect of a legislative change or implementation of a new process) or to the individual (e.g., a job change). By aggregating from individual instances, we can begin to interpret what caused the shift. This is important to understand the nature of the change and the impact on the ontology.
• Understand when users switch to alternative search methods. This is a common behavior when user topics become misaligned with their situation. Reviewing usage patterns across various navigation methods helps identify when alternate search methods become preferred. Sustained changes in behavior are alarms that require further investigation.
• Improve “bookmarking” capabilities. In one of our applications, users are able to bookmark interesting sites and content items using a specialized bookmark application that maintains a repository of user-defined references to content (in the context of what they are doing at the time). At the time they mark the item, they can also describe it – thus providing us with further semantic information. Note that our approach to this has not used the standard bookmark capability of a browser, rather replaced it with a standards-based data process that provides the information we need to mine the information. There are also some sites like www.backflip.com that provide a shared service where favorites lists can be exported for daily use and future interpretation.

This area of information gathering, monitoring and interpretation is a major area of opportunity for individuals interested in improving the quality of ontologies that rely on common language in semi-changing environments. It is important to point out that sampling is an acceptable method of gathering insights.

Gathering Simple Feedback

For some reason, many application designers fear feedback from end users – either consciously or subconsciously. This is truly unfortunate, because the most relevant feedback available to support improvements is from end-user experiences. They are willing and able to tell you what you really need to know about the application – in context.

• Make sure the user has a chance to provide open, unstructured feedback at every point. Always provide a method of communication. Always respond that their message was received. Invite them to identify themselves so that you can follow up for additional information where relevant.
• Let users “speak their mind.” We have found that collectively a user community can clearly articulate priorities, shortcuts, and useful improvements that can define success. More generally, the success of discussion threads, blogs and open source development activities show this to be true, as well. Particularly in metadata-driven systems, users should be encouraged or even polled on areas of definition or description where designers are uncertain.
• Capture the context in which the feedback was called. Do not just the collect information about the page they are sitting on when they select the feedback button. There are tremendous insights in collecting context – the criteria that got them to that page, and preferably other session or user data that is able (and agreed with the user) to be captured.
• Separate feedback submissions that are questions about working processes from feedback about the application. In customer-facing roles, user questions often reflect the type of question the individual staff member is being asked by the customer. The language in the question is often reflective of the language of the “outside world” and so should be considered carefully.
• Allow users to nominate new terms. Synonyms become richer and the patterns inherent to different business units become apparent when users nominate new terms. Synonym language will reflect things that have meaning to groups within, and outside an organization.

Listening to Conversations

The best way to assess changes in language is really by listening to ordinary conversations, and then – most importantly – interpreting what you hear. Where are the human and organizational conversations taking place? Increasingly they are online, providing a unique opportunity to learn by monitoring (this is the same thing as listening).

• Monitor every type of “conversation” that you are allowed to monitor. Monitor for the purpose of understanding language and priorities. Increasingly, users have collaboration tools available to them beyond e-mail. The use of collaboration and discussion areas is a fabulous opportunity, because the language used is more dynamic, and closer to the conversations that staff hear when they interact with people outside the organization (customers and the public). With each new collaboration environment that evolves within the organization, there is a new source of information to understand how users see their world. We are exploring the use of new technologies to mine these conversations, and then map that back to the organizational ontology. Profiling and discrimination are key capabilities for automated tool sets.
• As new content is being created and reviewed, a collaborative conversation takes place around that new content. Capture the conversations surrounding changes in content, which helps to identify the potential impact on the metadata. Review and approval cycles present rich data about different interpretations and meanings, and also capture the reasoning behind why something is represented or phrased the way it is. Encourage the dialog where possible.

Applying Stealth Knowledge Management: Learning from What People Do

We actually know quite a bit about how people use applications and web sites. Stealth knowledge management is an approach where we learn about context and the decisions people make by observing the patterns of their transactional behavior when they use applications [Degler and Battle, 2000]. As best practices are identified through actual performance, they can be reflected back to less experienced application users in the form of guidance and support. When gathering this type of feedback, we don’t have to be concerned about each and every user, rather their overall behaviors. Discovering patterns, trends, shifts, and profiles really helps designers prioritize.

In particular, we need to monitor work tasks that may require multiple applications or information resources to understand user context at more than the individual software application level. Among other techniques, use periodic testing, job monitoring, and feedback loops. Optimize on the typical paths and monitor atypical patterns as this may be an early indication of knowledge shifts. When work patterns change, it is the time the new information is most needed.

We are beginning to see ontologies used to map between different management information systems and reports, but rarely see ontologies mapping across transactional systems and content management systems. Transactional systems and content management systems tend to be disassociated from each other semantically, yet both are key assets to most business. This has to change! It is important to monitor both task performance and information use by using the same ontology framework. This allows us to use the management information about each one to access insights and indicators from the other.