Large Language Models as a Member Engagement Tool

There’s plenty of hype around artificial intelligence these days, especially large language models. Even as multiplying lawsuits challenge big-name LLMs over copyright and training text usage, and regulators and advocates try to establish guidelines for rapidly advancing technologies, there’s a strong sense among CEOs (opens in a new tab) that their companies should be using AI to transform work.

For associations, there could be something valuable in what LLMs have to offer, especially at a time when we’re asking if there is too much content  (opens in a new tab). Several associations have moved forward with the development of their own LLMs to improve their technology and relieve some administrative burden on staff. Most importantly, they seek to elevate the value of membership by better connecting members to the information they’re looking for and creating customized content syntheses they can’t get anywhere else.

Creating New Value Through Existing Content

When considering what AI can do for your association, it’s good to start with a challenge you have. The Water Environment Federation  (opens in a new tab) needed to help members navigate a key benefit—an extensive content collection comprising technical papers, conference proceedings, books, magazine articles, web content, and more. Steve Spicer, WEF’s senior director of user experience and digital content strategy, compared the LLM they’re developing to a “supercharged all-knowing librarian.” He explains, “One of the reasons that we got into AI was that it can ‘know’ all of [that content], so it’s a huge help in unearthing all of the trapped knowledge.”

WEF took a “crawl, walk, run” approach to developing WEF AI  (opens in a new tab). They used what was easy to plug in for LLM training, including the RSS feed content from their Access Water website. Spicer says, “We made the very deliberate decision to avoid anything dealing with personally identifying information, because, for anyone starting, that is complicated. It can be legally fraught.”

The Missouri State Teachers Association  (opens in a new tab) started exploring LLM-based technology to replace a rule-based chatbot introduced during the pandemic. But Kara Potter, MSTA’s digital strategist, realized, “[helping people find information] was just scratching the surface of what [an LLM-based assistant] would do on our site. Yes, it would serve as a website concierge, but it can do so much more.” They trained their LLM, called Tillie  (opens in a new tab), on MSTA’s web content, anticipating Tillie would be tasked with queries about everything from starting salaries in certain towns to lesson plan guidance.

Leaders at the Association of Child Life Professionals  (opens in a new tab) turned to an AI-based solution to replace poorly performing search functionality on their website. Keri O’Keefe, ACLP’s director of communications and publications, says, “It felt like we were in this repeating pattern of just trying to catch up to what was current five years ago. We had a lot of conversations around AI advances in the last few years. We took a look at the ROI, and we thought [AI] was going to be a better long-term solution for us.”

When it came to identifying content to train their assistant, Scout, ACLP “went with the approach of adding everything that was relevant and what we thought would be worthwhile to our members” to train the LLM, O’Keefe explains. That included the blog, articles from their research journal, and webinar content.

The expectation across all three organizations is that such access will improve member engagement and perception of value. MSTA’s Tillie launched in the spring, and Potter has already been surprised by feedback from members interested in using Tillie in unexpected ways, like developing recommendation letters for students during scholarship cycles. Potter hopes Tillie will change the way some members engage with MSTA. “A lot of our members join for the liability insurance and that’s fine, but if we can provide them with more value on top of that, that’s really our mission,” she says.

ACLP is collecting beta-testing feedback from member volunteers and will be launching Scout at the end of July. O’Keefe will be monitoring two goals: improving member experience on the website and reducing the amount of staff time devoted to frequent inquiries. She’ll use call and email volume to monitor progress over time.

Although WEF AI will launch in July, Spicer has already learned a lot from testing. “It has shown me how differently people think, because when it’s just a blank chat line, the way that you form a question versus the way I form a question completely changes how [the LLM] is going to answer it.” Plus, Spicer notes, the AI provides useful content interest data. “If we get 1,000 people asking questions on a topic that we don’t have any content for, I know what our next is content is going to be.”