What do surveys, interviews and focus groups all have in common? They’re all ways of collecting self-reported data.
Anytime a visitor tells you about their experience, you are getting self-reported data.
This means they’re sharing information about themselves without any external verification. For example, if a visitor selects their age range in a survey you would not double check its accuracy by asking their family members or checking their birth certificate.
Self-reported data is relatively easy to gather but there can be issues with accuracy. Sometimes those accuracy issues are a big deal, and other times they won’t make or break the study. This is why it’s important to know how to balance what’s feasible to collect and useful to know.
Here’s a breakdown of what self-reported data does well and its limitations.
Self-reported data is good for:
1. Things that only the visitor knows.
Visitor thoughts, motivations and beliefs are all things you may want to know, but are not observable. If you want to know what motivated a visitor to attend a new program, you need to ask them directly. If you want to know why a visitor spent 30 minutes sitting on a bench in front of that painting, you need to ask them directly.
2. Things that are not feasible to verify externally.
There are some things that are theoretically possible to check, but actually getting that data is just not feasible. If you run a program designed to encourage families to reduce their use of single-use plastics it is technically possible to track their behaviour over the following year to see what they do – but that takes a ton of resources.
The same goes for collecting data on group sizes across a large historic site; you could track visitors as they arrive, count how many people are in their group at various points in their visit and compare that to ticketing data, but it is so much easier to just ask them.
However, self-reported data has issues with accuracy.
Some common issues are:
1. Social desirability bias (when people exaggerate, or even lie, to answer things in ways they think will make them look good).
2. Misremembering their behaviour.
3. Misunderstanding the meaning of a question.
(Bryman & Bell, 2019: 134)
Imagine you are studying visitor behaviour in your museum. You want to know if the visitors are using interactives, and how much time they’re spending in each exhibit. You could survey visitors about what they did and how long they spent. It would be quick and easy, but would you get accurate results?
Let’s break it down.
Social desirability bias:
Would visitors really tell you if they sped through the museum and didn’t use any interactives? Some may exaggerate the time they spent because they worry that their behaviour comes across as disrespectful of the curators. Or, maybe they’re embarrassed they weren’t a ‘good visitor.’
Misremembering:
Would visitors even know how much time they spent in different exhibits? Even if they checked the time upon arrival, would they be able to accurately remember how long they spent in different exhibits?
Misunderstanding:
How are you counting “using interactives”? Is there a minimum amount of time they have to spend? Does returning to the same interactive twice count as using an interactive once or twice?
Self-reported data is terrible at accurately tracking behaviour. On top of that, the visitors would likely feel judged, confused and stressed out because they’re being asked to report on things they don’t remember, don’t know, or are worried might make them look bad.
An Alternative to Self-Reported Data
If you need to capture data on visitor behaviour, the best thing you can do is observe and record what you see. This generally takes more time and resources, but the payoff is more accurate results (and less stressed out visitors).
The Case for Triangulation
You can check the accuracy of self-reported data through triangulation. This means you’re measuring the same phenomenon, using different methods. For example, combining a survey with observation and interviews. If you see results that support each other, across the different methods, you can be more confident that your data is accurate.
Self-reported data isn’t good or bad, but it has its limitations. Understanding those limitations will help you decide if it’s the right choice for your study.
Reference:
Bryman, A. & Bell, E. (2019). Social Research Methods (Fifth Canadian Edition). Oxford University Press Canada.
Learn more through professional development opportunities and the services I offer to support your evaluation projects.
TM Visitor Studies 