Email analytics: which numbers still mean something

Open rate is measured by a tracking pixel, and since 2021 Apple's Mail Privacy Protection pre-fetches that pixel for Apple Mail users whether or not the message was read. Because Apple Mail accounts for a large share of email clients, a substantial fraction of recorded opens are machine-generated.

The damage is not only to the headline number. Any segment or automation branch built on opens silently degrades: subscribers who never read anything still register as opening, so they survive the re-engagement rules meant to remove them — and then continue depressing the sender reputation those rules existed to protect.

Opens retain some value as a relative trend within one audience over time, provided the client mix is stable. They have no value as an absolute figure, and none at all for comparison against another sender's numbers.

The reliable metrics are the ones requiring deliberate human action. A click, a reply, a purchase, and an unsubscribe all mean someone decided something — none can be manufactured by a privacy proxy pre-fetching content.

Read them together rather than individually. A campaign with a high click rate and a high unsubscribe rate got attention at a cost; one with a low click rate and no unsubscribes was probably ignored rather than disliked. Neither number alone tells you which happened.

Attribution assigns a conversion to a message, and every method involves a choice that changes the result. Last-touch credits the final interaction, so an email that merely completed a decision already made absorbs the full value of the sale.

The attribution window compounds it. A thirty-day window credits an email for a purchase four weeks later; a one-day window credits almost nothing. Platform defaults differ, which is why two tools reporting on the same campaign routinely disagree — and why comparing attributed revenue across tools is meaningless.

When a number is driving a real decision, measure incrementality instead. Withhold a small random share of the segment, send to the rest, and compare. The difference is the effect the campaign actually caused, which is what attribution was standing in for all along.

A useful report shows the metrics broken down by segment, because a list-wide average is a weighted blend of groups behaving differently. A campaign can show a healthy overall click rate while performing terribly among new subscribers, and the average conceals exactly the problem worth acting on.

It also shows placement, not just delivery. Delivery says the receiving server accepted the message; placement says whether a person could see it. Reporting that omits placement is reporting on the wrong question.

And it shows the cost side. Unsubscribes and complaints belong next to clicks and conversions, because a campaign that produced revenue and burned two percent of the list is not obviously a success — that judgement needs both halves.

An A/B test is only informative when the difference exceeds what random variation would produce, and email tests routinely declare winners on differences well inside the noise. A few percentage points on a few thousand recipients is usually not a result.

Test one variable at a time and give the test enough volume to resolve the effect size you care about. Detecting a large difference needs relatively few recipients; detecting a small one needs far more than most senders have.

Be sceptical of subject-line tests judged on open rate specifically. That is the metric privacy pre-fetching distorts most, so the test measures the winner's effect on Apple's proxy servers as much as on people.