IBM Watson Assistant: Intents

IBM’s Watson Assistant is a software platform which enables users to develop conversation interfaces leveraging Artificial Intelligence. In my role as a UX Designer on the Watson Assistant team, I worked closely with data scientists to make the complex AI systems transparent and approachable to enable end users to build custom solutions.

Watson’s AI Assistants are composed of dialog skills which enables the assistant to understand a question or request and provide an appropriate response. Questions and requests are understood, by being broken down into intents and entities. For example, if a user tells the AI assistant “I want to order a pizza”, “order” (action to take) is the entity, and “pizza”, (object being ordered) is the entity.

Creating Intents

Intents can be added manually by the user by entering examples of what a user might say.
In the pizza ordering example, the user may train the system by manually entering a variety of phrases such as:
“I want to buy a pizza”, “Let me get a pizza”, or “I want a pizza delivered.”
The system would learn that each of these phrases indicate the intent to “order”.

Alternatively, Watson can use AI to automatically generate example phrases based on real phrases it learns from either training data, or once live, actual conversations with people interacting with the Watson Assistant.

Each of these methods have advantages and disadvantages.

 
 
 

Manually Intent Entry

Manually entering intents gives users full control, but is time consuming, does not take full advantage of the AI’s capabilities, and lacks accuracy because it often does not capture the wide range of ways someone interacting with the assistant may ask the same question.

 
 
 
 
 
 

Automated Intent Creation

Allowing the system to create intents on its own maximizes speed of learning, but does have the potential for errors in understanding end user queries. 

 
 
 
 

Intent Recomendations

With this in mind, our team created a third method called “Intent Recommendations” where Watson would surface recommended phrases to add to an  intent based on its machine learning. The user could then review and approve the recommendations. This method is not only faster than manual entry, but offers more diverse options based on real data. This method allows the user to take advantage of the machine learning while still offering more control than fully autonomous addition of intents.

 
 

The intent recommendations feature appears in two places:

  1. In the intent overview page which displays all of the intents build for that AI Assistant

  2. Within an individual intent, on the page that shows the example utterance that comprise that intent|

The recommendations function works a bit differently in each location.
In the overview page, at system will recommend new intents to be created. This feature requires the user to upload data in order to get recommendations.

 

To address this issue, the Assistant team developed an intent recommendations feature which leverages chat logs to automate the intent creations process. Assistant users can upload chat logs and similar utterances within these logs will be clustered to form intent recommendations. These recommendations can be turned into need intents, or added to existing intents.

To address the issue of accommodating both new and existing users, empathy and journey mapping were done for both user types in an effort to ensure that both user groups would have their unique needs met through an intuitive flow. Existing users still needed to be able to easily find intents they had already created, while being introduced to the new intent recommendations option and understanding how they can use this feature to make their training data more robust. Meanwhile, new users needed to be informed of both creation methods when they land on the intents page so that they can select the option they wish to start with, and be aware of how they can leverage both methods to generate training data for their system.

To address the challenge of users discovering the intent recommendations feature, a standard Watson Assistant banner was used. This is a common component used in the UI to highlight new features. In this instance, the empty state of the banner was used provide a brief description of the new recommendations feature and provide a link to learn more or upload files. Once chat logs have been ingested, the banner serves as a preview, displaying the top four intent recommendations.

To confirm user comprehension of the new feature, we set up a user test with IBM sponsor users. Some had worked with intents before, and some had not. In this test, participants were given a scenario in which they were to act as though they were an employee of a large telcom tasked with setting up an FAQ bot for the company. We provided them with a .csv file of chat logs from this imaginary telcom and asked them to build some entities. The main research questions from the test were ifs the feature discoverable, if users understood how to use the feature, and if they found value in this new method on intent creation.

Based on the user test, we were able to determine that participants were excited about the new feature and thought it could be a useful tool both during initial set-up, and to improve a more mature bot. Participants were able to understand the concept and upload the provided chat log easily. However, some findings uncovered room for improvement, which were addressed with immediate fixes and plans for longer term improvement. The documentation was revised to provide more information about how the chat log data should be structured. Top intent recommendations are now highlighted based on uniqueness and frequency of the utterances in the recommendation. This gives users more clear guidance on where they can start adding intents to have the greatest impact on their training data. Additionally, the ability to rename intent recommendations before adding them to training data was implemented and we worked on future plans to integrate more dynamic ways to explore recommendations such as searching and combining recommended intents.