Wizard of Oz Experiment

April 29, 2019
|By Daniel Rösch

–-> Go to BOTwiki

 

In the field of human-computer interaction, the "Wizard of Oz" method refers to a research experiment in which participants interact with a computer system that they perceive as autonomous, but which is in fact operated or partially controlled by an unseen person.

 

Chatbots and Wizard of Oz

Chatbots are enormously suitable for the Wizard of Oz experiment. This means that a use case can already be examined for its "chatbot suitability" before implementation. The findings can then be used to iteratively expand existing flows and define new communication strands. In addition, the collected data, such as utterances, can be used directly for the Chatbot Training be used. A mature human handover tool is even able to automatically convert the tested data into a chatbot.

 

Recommended procedure

  1. Define different chatbot flows
  2. Integration of a live chat or an empty chatbot that only triggers a human handover.
  3. Manual response to user queries
  4. Derive chatbot flows
  5. Answering the queries based on the chatbot flows
  6. Iterative revision and expansion of the chatbot flows
  7. Manual or automated transfer of the flows into a chatbot builder

> Back to BOTwiki

 

The Wizard of Oz experiment is a UX research method in which users interact with an apparently autonomous system whose responses are formulated by an unseen human. In the context of conversational AI, the method is used to realistically test the dialogues of a planned AI agent before the system goes live. This allows use cases to be validated and real user data to be collected without the NLU model having to be trained yet.

A WoZ test is particularly worthwhile when a new use case for an AI agent is being planned and the requirements for dialogue management or tone are still unclear. The method also helps identify risks early on in sensitive channels such as voice hotlines, where there is little room for error. It is also useful when there is internal disagreement about whether a topic can be automated at all.

The experiment yields both qualitative and quantitative data: typical user comments, response patterns, escalation rates, common misunderstandings, and knowledge gaps. This information is incorporated into intent models, dialogue flows, and Knowledge AI content. As a result, the experiment serves as a direct precursor to the productive training of an AI agent.

While a traditional prototype test evaluates a system that has already been implemented, the Wizard of Oz experiment examines a use case whose logic does not yet exist in technical form. The Wizard replaces the model and provides flexible responses, enabling dialogues that are much more realistic. This yields insights that can be addressed during the early conceptual phase, rather than only becoming apparent after technical implementation.