In my previous article for Electronic Ink, I compared quantitative to qualitative research and described ways these methods could be combined effectively for better overall research. Building off the discussion of methodology, I’d like to share two examples that demonstrate the power of observation and importance of qualitative, in-person research.
People aren’t always good judges of their own behavior
Recently, I conducted a few days of in-person usability testing for a financial website. During the session, I observed participants struggle to complete tasks on the website and express frustration and confusion. Yet at the end of session, when I asked participants what they thought of the process, nearly all of them said it was easy and straightforward.
Why would they say a website was easy to use if it wasn’t? It’s possible they wanted to seem competent to an observer, or, having gone through the website once, they expect it to be easier the next time.
Paying attention not only to their words but also to their behaviors, however, it was clear that participants had difficulties at specific points of the workflow. Based on those observations, I made recommendations to clarify the process and make it easy for new users to use the website effectively. If I had simply relied on what participants told me, I would have missed these pain points and failed to meaningfully improve the user experience on the website.
Surveys are not always accurate or actionable
When conducting research, there is some information you can only gather through observation. For example, surveys can be a good way to solicit and receive feedback, but they also can have significant limitations. One of the biggest drawbacks with surveys is that people do not always accurately report their experiences. They may forget details of their experience or interpret a question or answer differently than the researcher intended, or they may not explain why they chose a particular answer.
I just received a usability survey from my hometown’s local library, asking me about my experience using the library. Here’s one of the questions they asked:
Twenty-seven drop-down menus: an excellent way to ensure participants resent filling out your survey.
Even if I did manage to slog through all 27 of these dropdowns, how would the results help the library? What does “Above Average” layout for a reference desk mean? If someone reports signage as “Needs Improvement” but they don’t fill out any additional comments, how will the library know what needs improvement and why? Furthermore, if the participant hasn’t visited the library recently, her response will be based on what she can remember, which may be incorrect or have changed since her last visit.
Library staff would learn more if they directly observed patrons in the library and asked them questions about their experiences. Asking people about their experiences in the moment provides much clearer, more useful feedback than asking them to rate different parts of the library on a meaningless scale after the fact.
Match the research method to the problem at hand
To reiterate, both qualitative and quantitative research can lead to important discoveries and need not be mutually exclusive. In general, it is best to select your research method based on the research goals for a specific problem. For example, you may want to distribute a survey or conduct data analysis if you want to understand the actions of and receive feedback from a large population size. I have found, however, that nothing beats the power of in-person observation when it comes to understanding people’s thoughts, behaviors and emotions and discovering not only what is transpiring, but also why. Sometimes you need both methods – it all depends on what you want to achieve through user research.