Four Language Pitfalls Associations Should Avoid in Member Surveys

Associations surveys can produce a wellspring of data that can be used to better understand member needs and take decision-making beyond the gut. But poorly considered questions and careless phrasing can lead to member surveys that are exclusionary, biased, leading, or repetitive—undermining the usefulness of the results.

How can you avoid these traps when asking your members relevant questions?

Cynthia Simpson, CAE, manager of member services at the National Society for Histotechnology  (opens in a new tab), has focused on the role that survey questions play in member engagement over her roughly three decades in the association space. Read on for her insights on what to watch out for in the way you structure your questions.

Don’t Make Respondents Think Too Hard

Survey questions need to be easy to respond to. Concise, clear wording is key, but so is structure. For example, offering too many answer options for a multiple-choice question reduces respondents’ ability to focus on what you’re asking. A long list of choices can naturally bias respondents toward the ones that appear last on the list, Simpson says, especially if the survey is conducted over the phone.

She also warns about questions that lead the respondent down a certain line of thinking. She cites the example of a question stating that a website “isn’t easy to use unless I use the search function.”

“Having that word ‘isn’t’ in there implies that the website isn’t easy to use to begin with. Well, for some users, it may be easy to use,” Simpson says. “So you’re already misleading them and using that double negative to frame their response.”

To weed out potential biases, she recommends asking the same question in multiple ways. If one version of the question confuses or misleads respondents for a reason you haven’t considered, another version may capture the respondent’s true answer, preventing skewed results.

Be Wary of Gender Bias

Sometimes, phrasing may unintentionally reflect gender bias. Simpson, who wrote about this topic for Association Success in 2018  (opens in a new tab), says it’s important to consider which descriptive attributes are used in a question.For example, using ability-focused terms such as “brilliant,” “capable,” and “analytical” may subconsciously skew male for respondents; “grindstone” terms such as “hardworking” and “meticulous” may carry a female connotation. Using attributes traditionally associated with men or women can skew the response, she says.

“You need to be careful to not include those types of gendered questions because the picture that the person gets in [their] mind reflects back on the question,” she explains. “The best type of questions are free of that type of language.”

Avoid Unnecessary Implications

Sometimes wording can reflect other forms of bias and result in leading questions.For example, in a survey about COVID-19 attitudes, asking whether “concerned citizens” should wear a mask creates an implication about what the researcher believes.

“That implies that if you aren’t wearing a mask, you’re not a concerned citizen,” she says. “And so using that word, ‘concerned,’ already implies that only concerned citizens wear masks and that other citizens don’t wear masks, are not concerned, and that may not be true.”

This can go the other way as well: Survey results may be skewed by social desirability bias, in which the answer to a question—say, about a controversial political candidate—is affected by the respondent’s desire to be liked. For example, a participant might respond to the question “Who do you plan on voting for?” with the answer they believe the pollster wants to hear. “You want to be liked, whether [your answer is] true or not,” Simpson says.

When phrasing a question, remove words that imply value judgments, and ask yourself in what ways a respondent could potentially be misled by the question. If asking questions over the phone, take care to monitor your responses—for example, avoid offering encouragement when a respondent expresses an opinion you agree with.

Don’t Raise Expectations You Can’t Meet

Survey questions can sometimes set subtle (or overt) expectations in respondents. For example, if the phrasing of a question hints at a new member offering, it could put you on the spot for something you weren’ actually planning to do. Even general questions about improving the member experience can lead to unfulfilled expectations.

“Be very careful what you ask,” Simpson says. “If you’re unwilling or unable to make change [implied in the question], then it doesn’t do any good, and in fact it leaves a negative thought in the respondent’s mind.”

Ultimately, Simpson says, “if you aren’t able to implement the answer, then really think hard about asking the question.”

(Ekaterina-84/iStock/Getty Images Plus)