Cambridge Design Partnership (CDP) has developed an innovation capability called Potential Realised. This allows new products to be developed through a systematic, evidence based process that considers technology, stakeholders and commercial needs in a balanced way. Just as new technologies are developed through testing and iteration in the lab, so are product concepts developed through market testing.
Development teams must observe potential users and customers interacting with a product or service to gain direct feedback on its potential value and acceptance in the ‘real world’. This is a vital route to understanding what users like, aspects they may not value, things they struggle with, where improvements need to be made and, ultimately, if the proposition is likely to be successful.
Market testing as early as possible in the design process is critical. It means decisions influencing the final design can be made with confidence at the start of the design cycle, when the costs of change are low. However many companies believe this is not possible until fully working prototypes are complete, when it’s often too late to make changes. The result is they struggle to deliver really focused innovation.
Wizard of Oz testing
One of the techniques used by CDP to overcome this paradox is called ‘Wizard of Oz’ testing. If you are basing fundamental design decisions on market and user feedback at an early stage of a project, it is important that your user base gives unbiased and uninfluenced reactions.
However, the Hawthorn Effect says that subjects modify their behaviour when they know they are being tested. One solution is to make it appear as if it’s not a test or to make them unaware of what is being tested.
This is the theory behind Wizard of Oz testing. It’s a methodology for the evaluation of a new product or service, but where the product doesn’t need to work like or even look like the real thing. That said, the experience of interacting with the innovation feels real and less like a test, ultimately eliciting more natural responses.
The technique is named after the novel and classic film, where the Wizard is exposed as a normal man sitting behind a curtain.
Wizard of Oz testing sees the experimenter create a realistic illusion of a new product or device to discover how users interact with an idea, to reveal and record genuine and valid responses.
For example, users might believe they are communicating with a computer using voice recognition, when the words are being entered manually by the experimenter (the wizard). The goal is to observe how a user interacts with a voice recognition system, rather than measure the effectiveness of the technology.
The method relies on effective interaction prototypes. These ‘acts like’ prototypes don’t need to function as or look like the ‘real thing’; they simply need to work on a level where the concept can be given to target users to study how they behave and interact with the proposed product or service.
The method allows new ideas to be tested with the confidence that reactions are genuine. By testing the product at an early stage, experimenters can identify flaws before money and time is spent on implementation.
Validating your designs
In order to generate facts about what your customers and users want and need, not what developers think they might need, factual evidence from primary research with users must be prioritised.
Users are not designers but, by allowing them to interact, react and respond to an idea within a well designed Wizard of Oz experimental programme, they contribute to the vision and outcome of the product by being themselves.
Wizard of Oz testing validates ideas, propositions, features and their implementations so that organisations can learn enough to maximise their potential as early as possible in the development process.
In practice
Have you ever seen a fantastic new product and gone to purchase it, only to be added to a waiting list? In showing intent to buy, you are helping the innovators validate the potential commercial success of their product.
Wizard of Oz testing was applicable to a product announced by CDP in July 2015 to meet an identified need for a low-cost device that allows Army and disaster zone medics to measure many patients’ vital signs at once.
The small device clips onto a patient’s nose and monitors breathing and heart rates, giving an ‘at a glance’ indication of both. This data is added to a trends graph, which can then be transmitted using Bluetooth Low Energy to a smartphone or tablet for further analysis.
One Wizard of Oz approach is to use first response training scenarios familiar to medics, while introducing low fidelity, real time vital signs information via nothing more complicated than cardboard dog tags and sharpie markers around the necks of casualty actors. The actors change the numbers relating to their heart rate and breathing rate when the medics aren’t looking to provide real time visual physiological parameters and scenarios. Designers can see how the medics respond with this new real time data to hand.
This may seem simplistic, but data was collected at a critical phase in the product’s development that influenced the design directly by generating realistic insights into how medics would respond to this new data; helping to answer questions like ‘how much time and effort is saved?’, ‘how is triaging and treatment of the patients affected?’ and, ultimately, ‘what potential could this innovation have on the outcome of critically ill trauma patients?’.
No new initiative is without its challenges. For any new concept to progress beyond paper, it has to develop a business case that is built upon evidence, linking technical feasibility, commercial viability and user desirability.
The most difficult and important questions for innovators are ‘will people want your new product?’, ‘will they elicit value and enjoyment from it?’, ‘will they choose it over the competition?’ and ‘will they pay?’.
While CDP’s Potential Realised approach has an array of tools to answer these questions, Wizard of Oz testing is an important way to ensure the answer these questions is ‘yes’.
Author profile:
Lucy Sheldon is a consultant with Cambridge Design Partnership.
The benefits of Wizard of Oz testing * Focused innovation. New products become more focused on meeting customer’s needs, using technology to deliver features they value and will pay a premium for. * Speed to market. You can learn fairly early in the process whether design changes need to be made, so weeks or months are not added to the development process if altered further down the line * Cost savings. Through the development process, the cost to make changes increases exponentially and can become too expensive to change if uncovered too late in the process |
