Product user testing: The void between laboratory testing and field testing

Abstract

User testing will frequently make the difference between an excellent product and a poor one. Moreover, in certain fields such as medical device development or training, the defence field or automotive industry, such testing can literally be the difference between life and death. Unfortunately, design teams rarely have the luxury of either time or budget to user test every aspect of a design at every stage, and so knowing where and when to devote time to testing, and the fidelity required for accurate results are all critical to delivering a good result. This paper introduces research aimed at defining the optimum fidelity of mixed-reality user testing environments. It aims to develop knowledge enabling the optimisation of user testing environments by balancing effort vs. reward and thus developing critical and accurate data early in the design process. Testing in a laboratory setting brings advantages such as the ability to limit experimental variability, control confidentiality and measure performance in great detail. Its disadvantages over ‘in the wild’ approaches tend to be related to ecological validity and the small but vitally important changes in user behaviour in real life settings. Virtual reality and hybrid physical-virtual testing environments should theoretically give designers the best of both worlds, finding critical design flaws cheaply and early. However, many attempts have focussed on high fidelity, technology-rich approaches that make them simultaneously more expensive, less flexible and less accessible. The final result is that they are less viable and hence somewhat counter-productive. This paper presents the results of testing at a variety of fidelity levels within a mixed reality testing environment created by a team of artists and designers. It concludes with a series of recommendations regarding where and when fidelity is important.