Association execs are being asked to do more with research these days, but there’s a temptation to play expert when it comes to analyzing data about your audience or your industry.

A new white paper makes the case to resist the temptation: Caveat Emptor: Becoming a Responsible Consumer of Research‌ [PDF], authored by Spark Consulting Chief Strategist Elizabeth Weaver Engel and consultant Polly Karpowicz. In an interview, Engel and Karpowicz made it clear that this challenge of responsible data consumption was widespread.

“Most of us don’t have any kind of a formal research training background,” Engel said. This is in part, she said, because association leadership is not generally a field people train for. “Nobody asks a 5-year-old, ‘What do you want to be when you grow up?’ and they say ‘I want to be an association executive,’ right?”

With that in mind, the white paper dives into a number of basic terms commonly used in data, such as samples, variables, reliability, margin of error, and bias. Part of the challenge, Karpowicz said, is that you shouldn’t cut corners when it comes to producing or using research.

“You can’t jump ahead. To translate research data into knowledge, you should take time to understand what results can actually tell you—whether it’s existing insight or newly collected data,” she said. “This is why we need research education for all association professionals.”

Understanding Flaws and Fallacies

Often, research done by associations can create a number of issues that can affect the outcome, both in terms of how the research is completed and conducted. Here are a few of them, as highlighted by the white paper:

• A perception that research is the same as a survey, when it is often one aspect of research.
• A lack of “no response” answers in quantitative research, which can discourage the completion of the survey or skew data.
• The Hawthorne effect, in reference to answers that are given intentionally to please the researcher—a major problem in qualitative research.
• Not accounting for different biases that can be introduced during the research process, such as a skew in the methodology or the sample used.
• A tendency to pull from low-quality secondary sources, with poor vetting, which can create quality and reliability issues for the findings you’re looking to highlight.

Taking a Responsible Approach

One of the major things that the white paper discusses is the responsibility associations have in both accepting and consuming research. A lot of this, Engel and Karpowicz noted, comes down to transparency and integrity throughout the process.

“Before you start any research,” Karpowicz said, “you really should have a plan for a number of things, including how you intend to communicate the results,” as well as strategies for data management after the fact.

An issue associations might run into is a natural tendency to want to soften the blow of negative findings—which sounds like a good idea to save face, but introduces bias, Engel said.

“If we ask people interview questions, and we get a lot of people who are upset about something, or who have a major complaint with the association, there’s a temptation to put a sunny face on things, and hide that information,” she said.

Among the strategies to help manage concerns around being too close to the research: Bring in third parties that can do the research for you, and implement processes that encourage a more objective approach, preventing the addition of biases where possible.

Understanding, Not Expertise

Engel and Karpowicz make it clear in the 51-page report that the goal is not to make you an expert, but to help you gain a broader understanding of how to more efficiently and effectively work with research. “Whether it’s something you’re using externally, within your job, or you are somehow involved in the research your association is doing,” Karpowicz said it’s important to have a basic grasp of how the research process works.

The white paper features case studies from associations such as the American Association of Colleges of Pharmacy and the Association of American Medical Colleges, highlighting how each group approached a specific research project—whether the goal was to better inform the public, improve the member experience, or optimize the association’s DEI strategy.

While improved research skills will not make you perfect, they can better equip you for a process that will ideally help you generate better decisions.

“The reason you want to develop skill in this is because, while good research doesn’t necessarily guarantee that you’re going to make good decisions, it certainly helps,” Engel said. “And bad research almost always guarantees bad decisions. So, you want to develop your skills.”

(Chainarong Prasertthai/iStock/Getty Images Plus)