Beyond Surveys: Uncovering Hidden Barriers and Motivations

Surveys are a staple in market research, and for good reason: they’re adaptable and effective at gathering much of the data we need to generate reliable insights. However, surveys are also limiting and they represent a single arrow in the quiver. It’s time for some new arrows.

We’re getting ready to launch a first-of-its-kind research study to investigate the hidden drivers behind the decision to become a member, and we most certainly will not be surveying members, visitors, or donors. Why not? Let me explain.

Surveys are often a poor indicator of an individual’s true feelings, beliefs, needs, and intentions. This is due to a number of factors, chief among them is what social scientists refer to as “social desirability bias.” This kind of bias happens when people answer survey questions in a way that they think will be viewed more favorably by others. When people are asked about their behaviors, beliefs, or motives, they often feel a need to present themselves in a positive light. In some cases, respondents may not even be aware that they are responding in a biased way, or they may simply be lying to themselves out of guilt or an idealized self-identity. 

It can also be difficult for respondents to faithfully answer survey questions like “Why did you decide to become a member?” because we are often unable to articulate the underlying motivators for why we did or didn’t do something. Because our decisions are influenced by many things on a subconscious level, we are poor witnesses to our own behavior and motivations. 

Moreover, surveys tend to take place in an environment that is far removed from the actual decision-making experience. For example, asking someone why they decided to become a member when they’re at home six months after joining will not accurately reflect the frame of mind they were in at the time and place when they decided to join (e.g., on the museum’s website or standing in line at the box office). 

Another problem with surveys? Surveys tend to attract people who like to participate in surveys, leading to a biased sample that excludes a broad swath of non-respondents. Thus, there can be significant differences between those individuals who completed the survey and those who did not. 

And surveys that ask questions about future intentions—even when there is rigor built into the methodology—tend to be an unreliable indicator of future behavior. As we are all aspirational creatures by nature, surveys that ask about future intentions tend to draw out responses that are steeped in optimism and framed by an idealized version of our future selves.

All of this is not to say that surveys are useless. In fact, surveys can be incredibly valuable in certain circumstances. However, there are a growing number of research methods that can, in many cases, provide faster, cheaper, and better data. 

So how can we get at better data? 

Surveys are a great starting point, but a mature insights strategy goes beyond surveys to integrate more innovative and relevant methodologies into the mix such as predictive analytics, user studies, in-depth interviews, journaling, and card sorting exercises, to name a few. 

Another great way to get unbiased data? Observing people interacting with a product, experience, or marketing message in real-time. While most organizations are familiar with the concept of conducting A/B testing in direct mail, digital advertising, and email campaigns, few are leveraging this kind of experimentation to inform the way we frame the membership value proposition. 

This is the approach we will be applying in the Member Motivations Study — we are going to be conducting a large-scale A/B test across the cultural sector to test the resonance of different membership messages. For this initial phase of research, prospective members will be shown (at random) different versions of a membership landing page allowing our research team to analyze the influence of each value proposition. 

So, the next time you find yourself in need of data, and before you reach for that trusty survey, I hope you’ll consider adding a few new arrows to your quiver — each with its own unique shape and mighty aim to provide a deeper understanding and more actionable insights.

As always, reply to this email with your thoughts, leave a comment on the blog, or share your feedback.

Rosie

P.S. If you’d like to dive deeper into member motivations, please join me in the MAP Community’s Membership Innovation Group, where we spotlight exciting work happening in membership programs in cultural organizations around the country.