A Tiny Reminder of the Value of Experiments

Don't let the recent headlines fool you: Online experiments, when done well, are effective and might teach you a lot about your audience. The secret is not to mess with user expectations.

Experiments by websites are apparently in vogue, so we thought it’d be great to get in on the fun.

Unlike our friends at Facebook and OKCupid, we’re not doing anything of questionable ethics here. Even so, I’m certainly an advocate of this line, recently used by OkCupid cofounder Christian Rudder in a blog post discussing his company’s own experiments: “But guess what, everybody: If you use the internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work.”

It’s true, but these experiments generally vary in scale from big to small. The ones Rudder discusses are big and risky for a dating site—removing photos from profiles, putting incompatible people together—and got some folks angry. (And, inevitably, led to a hilarious Clickhole parody.)

There’s no need for us to go big and risky—what we do already works for our audience, and it’d be out of character for us, anyway. So instead, we decided to do something much smaller in scale and involving something that’s becoming a bit of a trend these days: the use of emojis in emails. The content we delivered remained the same as every day and for all users, but the experiment would show whether we could draw a few extra readers by borrowing from a tactic common in the world of email marketing.

About the Experiment

A lot of marketing emails these days have started to take cues from the world of SMS, adding emoji symbols to subject lines as a way to “pop” a little more. A couple of recent examples from my inbox:




This made us curious, so we figured we’d test it out. On Thursday and Friday last week, we did a simple A/B email test using our email client, BlueHornet, with messages otherwise the same except for one modest difference: For half of the recipients, the first character in the subject line was an emoji.

Going in, here’s what we knew about the potential benefits and pitfalls of doing something like this:

The plus sides: There’s evidence that, based on the type of newsletter, an emoji can help boost open rates. It’s also a relatively untapped approach in certain spaces—as Vocus noted in 2012, B2B marketers had used them rarely, if at all. Support is on the rise for such symbols, thanks to mobile devices that support them natively. “Using symbols, or emoji, in subject lines isn’t a new idea,” Campaign Monitor notes on its website. “However, now that many of us have emoji-friendly mobile devices and regularly read our email on them, it’s become an increasingly popular tactic for attracting attention in crowded inboxes.”

The downsides: As the email marketing company Emma notes, not every email client supports them (Outlook 2003 being a notable example of one that doesn’t), so their usage is best left to places in the subject line where it won’t matter if they don’t show up. In other words, “Learn To ♥ Your Members” is probably off-limits, but “♥ Share the Love with Your Members” is probably OK. Also, you have to keep your audience in mind. For example, Associations Now leans heavily on the “news” end of “newsletter,” so we have to be careful not to get too cutesy with our messaging—especially when the news gets dead-serious. In other words, a symbol like 0805_cat (cute as it is) probably wouldn’t make sense for us.

It was still worth a try, though—because, who knows? We might find out something really fascinating about our audience. In the end, the results of our test were mixed.

First Test: Networking

The symbol: 0805_thumb

The subject line: Trying to Become Better at Networking? We Have Some Tips

The results: The thumbs-up won out solidly, adding to the open rate by almost 1.5 percent (23.30 percent versus 21.92 percent) and helping to increase the click rate by about 1 percent (25.85 percent versus 24.93 percent). The top five links were the same in both emails, with networking proving to be the most popular for both sets of users, but emoji-receiving readers were bigger on veterinarians than on libraries.

Why was the click rate up? When noting this to my editor, Julie Shoop, I pointed out that the thumbs-up felt like the right choice for this one because it “conveyed a sense of encouragement.” In other words, it accented something that our readers would care about (beefing up networking skills ahead of a major conference) while making sense within the context of the blog post.

Second Test: Friday Rewind

The symbol: 0805_rewind

The subject line: Friday Rewind: Pull New Members in With a Magnetic Draw

The results: The email without the emoji won—though it was effectively neck-and-neck. But with a 20.63 percent open rate and a 20.24 click rate, the emoji-laden email nonetheless fell behind the standard-issue one, which scored a 20.72 percent open rate and 20.84 percent click rate. Otherwise, the symbol didn’t have an effect—four of the five top links were the same in both emails.

Why was the click rate down? Part of this might have been the result of the symbol having less of an emotional appeal than a thumb, and as a result, it stood out less. It’s possible that there may have been another icon that could have worked better—say, 0805_mailbox. (And to answer your question, no, there isn’t an emoji for magnets. I checked.)

Would We Do This Again?

Never say never, but probably not. While becoming more common overall, the tactic really lends itself better to straight marketing emails than it does news-oriented ones like Associations Now Daily News. And the difference even on Thursday was only modest.

If the numbers had shown a dramatic difference—say, an unexpected 20 percent increase both days—we might have more to talk about. You also have to consider the emotional punch, though: If you do this every day, people may not care so much. But if there’s a time when it makes sense, it might be worth a try.

Obviously, on the grand scale of things, getting a few extra clicks because of a thumb is tiny, but it’s something that anyone in contact with members should always be doing in big ways and small. Whether it’s about changing up your tweet schedule, testing out question headlines over pithy ones, or even leaning on one type of content over another, we always do tests like these, sometimes subconsciously. And they’re generally worth it: They teach us important things about the people we want to reach. Don’t let OkCupid or Facebook scare you off from this point—just know that whatever you try needs to respect the user.

So my question to you: What’s the best experiment you’ve ever done involving your marketing efforts? I’d love to hear about it. Throw a note in the comments below.


Ernie Smith

By Ernie Smith

Ernie Smith is a former senior editor for Associations Now. MORE

Got an article tip for us? Contact us and let us know!