When you shouldn’t use open rates to evaluate subject line tests

Does a subject line test show which subject line got a higher open rate? Yes, but that does not necessarily mean that that is how you should determine the winner because opening an email might not be the final interaction you’re seeking. If you are sending an email with the goal of people clicking on and buying an item in your email, open rate doesn’t tell you which emails drove more purchases or at least which sent more contacts to the site.

Most of us have had this experience as a consumer. We receive an email, think it’s something interesting we want to see, open the email, and then see what we were expecting and buy the item we wanted to buy. Other times, we open the email and realize that it was not what we thought and are actually not interested in what the email is presenting.

Subject lines are not only about what gets people’s attention enough that they’ll open the email. They are also about whether the email experience matches up with the kinds of expectations we’re setting. If the email pushes back against expectations in a negative way, we could be doing ourselves more harm than good with the “better” subject line.

Using ecommerce emails as an example, in an A/B test, we could determine the winner by response rate, which is calculated by dividing the number of orders received by the number of emails delivered. This would enable us to evaluate which email drove a higher rate of conversion and whether those results are statistically significant.

On the other hand, if your email is about engagement, and your goal is to get people to read your emails, determining the winner based on subject lines makes sense.

Have you ever made this switch? How did it change your results or approach?