How Beta Testers Can Help Build Grade-A Exams
The tech association CompTIA improves its certification exams by enlisting IT experts to test the tests before they’re opened to all. How do they do it? Here are a few keys to making beta testing work.
Ready to get your certification or credentialing exam out to your members? Recruiting beta testers can help identify trouble spots, so long as you have enough testers—and offer them a few shiny incentives.
That’s a lesson that CompTIA, an association of information technology professionals, has learned through its exam development process. This week, for instance, it put out a call for beta testers for an updated version of the cloud-computing certification exam it released in 2013. According to Steven Ostrowski, director of corporate communications at CompTIA, the association has been using beta testers for more than a decade, in which time it’s learned a few lessons about how to attract them.
First, the exam needs to be nearly ready for prime time before it’s released to testers. CompTIA’s program is designed to reveal questions that need some tweaks, not broader structure-and-design concerns.
“What typically happens is we’ll find some questions where maybe the language isn’t quite clear, or the number of people missing a particular question is very high,” he says. “If a lot of people are missing a particular question, then maybe we’ll go back and look at the question and the responses to see if that needs to be adjusted.”
The key to confirming that a potentially problematic question is indeed problematic, Ostrowski says, is to have a critical mass of beta testers take part. For the current cloud-computing exam, for instance, CompTIA wants at least 400 completed exams—which means reaching out to more than 400 people, since inevitably some volunteers won’t complete the exam.
Strong vetting of participants is also important. People with work experience in the subject matter, if not an actual credential, are in the best position to judge the quality of the questions. For instance, for the cloud-computing exam, CompTIA is seeking testers with “two to three years of on-the-job experience in networking, storage, or IT data center administration as a systems administrator, systems engineer, or related position.”
Maintaining a narrow window of opportunity to complete the exam can also help improve results, Ostrowski says. “It can range from a month to two months,” he says. “After that it gets a little harder. People lose interest.”
But CompTIA offers a couple of incentives to complete the exam. For one thing, even though the test is in beta mode, it counts: People who pass the exam will receive the three-year certification, and they’ll do so earlier than those who will take the exam when it formally launches in early 2018.
“They are kind of an early adopter of the certification if they pass the exam. Then they’ve got that as a credential that they can tout,” he says.
Beta testers also get a discount: The fee for the cloud-computing exam is $50 for the first 800 people who sign up, much less than price of finished exams, which in this case is $285 but can range up to $400.
“It’s proven well for our exams, and if you do it right I think it would be a good way for other organizations to prepare their certifications and credentialing tests,” Ostrowski says.