Can we even trust surveys? The last few elections could make people wonder if anybody can trust polls anymore. Time after time, election results were far different from polls, even for congressional seats and statehouses. When polling firms seem to get it wrong by five, seven, or even 19 percent, serious questions emerge about whether people can trust the results.
Because Campbell Rinker serves the nonprofit world with marketing research, surveys are part of our toolset. With their focus on the bottom line and vigilance for cost control, it’s reasonable for nonprofits to wonder if they can trust survey results. Thankfully, surveys are different from polls in some significant ways.
This article describes why nonprofits can still trust surveys when they’re conducted carefully.
What Makes a Trustworthy Survey?
Solid survey research is like a stool propped up by three legs – accuracy, reliability and validity. Accurate surveys ask enough people to get a statistically predictable result. Reliable surveys get response from a significant share of the target audience. Critically, recent election polls failed the validity test – quite simply, the people they polled didn’t represent the people who voted. In research jargon, the sample was biased.
Another concern with the recent political polling was whether people were willing to answer truthfully. Thankfully, the things a nonprofit might ask about are usually far less polarizing than political or religious views. While a few people may answer questions in ways they think are more socially desirable, there’s little to no stigma associated with their response.
As we see it, getting trustworthy survey results requires that we know our audience, know how to use our tools, and know where we’re going.
Knowing Your Audience
Many nonprofits seek to learn more about people who have interacted with the organization before, such as donors, members and alumni. They often routinely collect or append demographic or psychographic data to constituent records over time. With this data, nonprofits can work to study these groups proportionally. Alternately, its possible to adjust or ‘weight’ survey data to match these proportions to make the study more valid after fielding is done.
If you happen to be a nonprofit without a lot of detail about your donors or members, there are ways to develop that knowledge before your study. Nonprofits may easily append key demographics to a random selection of donor records (age, gender, marital status, kids in household, ethnicity, etc.), to help dial-in the sampling mix and guide overall research strategy. This is a key area for getting stakeholders at your nonprofit to trust surveys.
Knowing Your Research Tools
In any profession, it’s vital to use your tools effectively and use them well. Most research firms have a wide array of ‘tools’ on hand to ensure trustworthy survey results.
For some nonprofits with a diverse constituency, it is helpful to survey using more than one contact method. Phone surveys reach people who wouldn’t respond to mail or online, and vice versa. Older people more often use a landline as their primary phone. Younger people tend to use more sophisticated email platforms and thereby respond less using this channel. Sampling plans must consider these dynamics and carefully target potential respondents for the best reliability. Pursuing multi-mode surveying and stratified random sampling make your data more trustworthy by including the opinions of key sub-segments you might otherwise miss.
If one of your goals is to learn more about prospects, there are some other key factors to consider. Prospect questionnaires must include questions to confirm respondents are who they say they are. Our methods isolate and disqualify respondents who speed through the questions, rate all the questions with the same score, or who don’t care about your issues. When surveying unknown donor audiences, Campbell Rinker often rejects 40-50% of respondents because they fail these tests.
Also, to trust surveys of prospects, collect your data using the channel or channels that mirror the habits of your potential audience. Limit your questionnaire to 5-8 minutes so that more respondents stay engaged to the end. And finally, its essential to keep questions clear, short and neutral to make interpreting the results easier.
Accuracy Is Everyone’s Goal
Full-service research firms like Campbell Rinker consider accuracy our primary goal. So do the companies up and down the supply chain who point us to qualified respondents. None of us survive if we can’t deliver clear, actionable data. This is why our vendors applaud our efforts to alert them to bad survey-takers. (Here’s a good article on this topic).
Furthermore, we consider it a sacred trust when you allow us to contact your constituents. When we represent you well, we represent ourselves and our industry well, too. When respondents feel respected and known, they’re more comfortable being transparent with an interviewer, and this helps deliver trustworthy survey results.
For example, political pollsters typically blend data from multiple sources, including automated Interactive Voice Response (IVR) survey calls. They do this because they’re under tremendous pressure to get speedy results. We take our time. Campbell Rinker never uses IVR because it is highly impersonal, could reflect poorly on you as a client, and might be susceptible to bogus responses.
We’re in this business to deliver insights. Toward that goal, we strive to innovate and keep up with best practices in an evolving research world. We know there are some businesses that survive on headlines and clicks, but that’s just not us.
Three Keys to a Trustworthy Survey
In summary, despite the inaccurate headlines of the past several months, solid insights are still available from careful, rigorous research surveys. This is especially true for nonprofits who seek to learn from the constituents they already serve.
But beyond all the technical details involved in developing the right sampling strategy or settling on the best tactics for asking your audience, there are three overriding keys that we use to help develop valid research for our clients. These are…
- Establish and get executive buy-in on research objectives in advance. This keeps the goalpost from moving once the data are collected.
- Plan your research carefully – from survey design and sampling all the way through to analysis. This step helps ensure the results drive your conclusions and not the other way around.
- Finally, be willing to accept the results as they are, without prejudice to early hypotheses.
Following these guidelines can help your nonprofit trust surveys and gain valuable insights as well.