Reclaim 2 hours: how Superhuman classifies 500M+ emails a day

The shape of the problem

Most of what makes email slow isn't the writing, it's the triage: cold pitches, marketing blasts, and internal noise all land in the same inbox as the three messages that actually need a same-day reply, and weak spam filters mean a human still has to eyeball most of it. That review pass is the tax AI email tools are trying to remove — and Superhuman's version of it is built on a principle worth stealing for any classification system: don't trust the model everywhere, only where it's earned it.

What the system does

A message arrives, gets embedded and classified across three tasks at once, and only the predictions the system is confident about get acted on automatically. Everything else stays in the inbox for the user to see.

Custom models, not one generic one

Generic off-the-shelf embedding models underperform on the specific task of "is this email actually important to this specific person." Superhuman runs dozens of fine-tuned, domain-specific models instead of one general-purpose classifier, trained on features that are cheap to extract but genuinely predictive: subject-line content, sender reputation from prior interactions, body length (short bodies skew promotional), presence of links or attachments, and send/reply timing patterns.

The calibration pattern

A standard classifier lands around 90% accuracy — roughly 1 error in 10. Human review is closer to 98%, or 1 in 50. Deploying the 90%-accurate model to make every decision means living with an error rate ten times worse than a human's. Superhuman's answer, which they call Superhuman Calibration, is to not deploy the model everywhere: identify the subset of predictions the model is genuinely confident about — the obvious promotional newsletter, the clearly-internal reply — and automate only those. Everything below that confidence line goes back to the user.

# a rough sketch of the calibration pattern the source article describes,
# not Superhuman's actual implementation
def route_prediction(email, model):
    label, confidence = model.classify(email)
    if confidence >= HIGH_CONFIDENCE_THRESHOLD:
        return auto_route(label)   # archive, categorize, or draft a reply
    return leave_for_user(email)   # model wasn't sure enough to act alone

One shared encoder, three tasks

Importance (binary: important or not, trained on starred and replied-to emails), category (multi-class: pitch, marketing, news, calendar, personal, other — this is what powers split-inbox auto-organizing), and spam detection (binary, tuned hard for precision, since a false positive here means a real email silently disappears) all sit on top of one shared encoder. Training signal from spam detection improves pitch classification and vice versa, because the shared representation transfers across tasks instead of each classifier learning from scratch.

The infrastructure that got latency under 500ms

Classifying at 500M+ emails a day only works if inference is fast enough to feel instant. The rebuild that got them there used Baseten Embeddings Inference on NVIDIA's TensorRT LLM runtime — kernel fusion for embedding operations, custom CUDA kernels, multi-cloud routing with active-active failover, and request batching to cut per-call overhead.

That's an 80% latency cut, serving dozens of models through one unified interface rather than maintaining bespoke serving paths per model.

What they actually monitor

• Inference latency — p50/p95/p99 per model, with alerts if p95 crosses 750ms.
• Model accuracy — precision and recall per task, plus false-positive rate tracked separately, because false positives on spam are the failure mode users notice first.
• Rollout safety — new model versions ship to 5% of users before a full rollout.
• Business metrics — hours saved per user (target: 4+ a week), 30-day trial retention, and how often users actually accept an AI-drafted reply.

Why this isn't really a modeling story

The source article's own conclusion is the right one: Superhuman's edge isn't a more advanced model, it's architectural — the calibration pattern, the shared multi-task encoder, and an inference layer built for latency and offline resilience. At under $0.001 per email and 500M+ emails a day, the infrastructure decisions matter as much as the classifiers sitting on top of them.