Retrieval for support deflection: chunking, reranking, and the metric that matters

The shape of the problem

The support queue took about 5,800 tickets a week. Roughly 60% were answerable from the help centre; agents answered them by finding the right page and rewriting it in their own words. That is a retrieval problem wearing a writing problem's clothes, and it is the reason this pattern shows up first in almost every support deployment.

The first version deflected 21% of eligible tickets. It also produced a small number of confidently wrong answers about billing, which cost more trust than the deflection saved. Everything below is what we changed over the following two quarters.

What the system does

A ticket arrives. We retrieve candidate help-centre sections, rerank them, and ask the model to answer using only those sections. If it cannot quote a section, it abstains and the ticket goes to a human with the retrieved context attached. Abstention is a first-class outcome, not an error path.

Chunking: sections, not tokens

Fixed-size chunks split procedures in half. A 512-token window would routinely cut "click Save" away from the three steps that precede it, and the model would answer with a fragment that read complete. We switched to splitting on the document's own headings, then merged any section under 80 tokens into its parent.

# split on headings, then merge fragments upward
def to_sections(doc: Document, min_tokens: int = 80) -> list[Section]:
    parts = split_on_headings(doc.html, levels=(2, 3))
    out: list[Section] = []
    for p in parts:
        if out and count_tokens(p.text) < min_tokens:
            out[-1] = merge(out[-1], p)   # keep steps with their heading
        else:
            out.append(p.with_breadcrumb(doc.title))
    return out

Prefixing each chunk with its breadcrumb (Billing → Invoices → Editing a paid invoice) raised recall@6 by 4 points on its own, because it gives both the retriever and the reader the context the section assumes.

Reranking earns its latency

A cross-encoder over 40 candidates costs about 180 ms at our volume. It is the cheapest accuracy we bought all year. The table below is from the 340-ticket evaluation set described in the next section.

The metric that matters

Deflection rate is a business metric and a terrible gate. It rewards answering, and answering is exactly the behaviour you must not reward when the system is unsure. We gate deploys on citation-supported rate: the share of sent answers where every factual claim maps to a span in a retrieved section, judged by a second model and audited weekly against 40 human labels.

Once that gate was in place, deflection went from 21% to 30% over three releases, because we could raise coverage without wondering what we were shipping.

Failure modes in week three

• Stale pages. The help centre described a refund flow we had replaced. The system was correct with respect to its sources and wrong with respect to the product. We now surface last-reviewed dates in retrieval and drop sections older than 12 months from the eligible set.
• Multi-question tickets. Customers ask two things in one message. Answering the first and silently dropping the second reads as evasion. The router now escalates any ticket with more than one detected question.
• Near-duplicate sections. Three regional variants of the same policy page filled the top 6 and crowded out the answer. Deduplicating by content hash before rerank recovered 2 points of recall.

Cost per resolved ticket

Inference is not the interesting line. At 5,800 tickets a week the model spend is a rounding error next to the agent minutes; what moves the number is how many escalations arrive with usable context attached. Escalated tickets that carried the retrieved sections closed 40% faster than cold ones, which is most of the value the deployment produced.