Design Calibrated Trust and Human Handoffs

Calibration, not confidence

The goal of trust design is not to make users trust the system. It is to make their trust track its actual reliability — high where the system is dependable, low where it is not, and responsive when that changes.

This reframing matters because it makes under-trust a first-class failure. Teams instrument heavily for over-trust, since its consequences are visible and embarrassing. Under-trust is quieter and more common: the feature ships, adoption plateaus at fifteen percent, and nobody can explain why. What happened is that users checked the output carefully for two weeks, found enough errors to conclude that checking was mandatory, and worked out that checking costs as much as doing the work themselves.

Once that conclusion forms it is very hard to reverse, because the user has stopped generating the evidence that would update it. A system that is right ninety-four percent of the time and gives no signal about which six percent is wrong is, from the user's point of view, a system that must be fully verified every time. Its effective accuracy is zero.

That is the central insight: the value of an AI feature is not its accuracy, but the accuracy the user can act on. Closing that gap is what the rest of this lesson is about.

Set expectations in task language

Expectations are set by three things, in descending order of influence: what the system does on the first few uses, what the interface implies, and what the documentation says. Most teams invest in reverse order.

Name the capability in the vocabulary of the task, not the technology. "Drafts a reply from the account's order history" tells a user what to expect and what to check. "AI-powered support assistant" tells them nothing and implies everything.

Avoid labels that imply understanding, memory, authority, or confidentiality the product does not provide. Giving an assistant a human name and a personality is a design choice with a cost: it invites users to assume continuity across sessions and discretion about what they share.

Show the limits near first use, with examples, rather than in a policy page nobody opens. A short list of three things it does well and two it does not is worth more than a paragraph of hedging.

Be careful with confidence scores. A number displayed without an explanation of how it was produced or what action it should change is decoration, and worse than nothing: it lends an unearned appearance of rigor. If you cannot say "below sixty, read the source before sending," do not show the number. Prefer signals the user can verify directly — showing the source passage lets someone check in two seconds, which is faster and more honest than a percentage.

Match control to consequence

The amount of friction in an interaction should be set by the reversibility of what it does, and nothing else.

| Consequence | What the interface owes the user | | --- | --- | | Reversible, private | Fast path, easy undo. Do not interrupt. | | Reversible, visible to others | Preview before it leaves. Undo window. | | Costly to reverse | Evidence shown inline, explicit approval, named actor recorded. | | Irreversible | Preview, confirmation naming the specific effect, audit record, tested recovery path. |

Two failure modes bracket this table. Uniform friction — a confirmation dialog on everything — trains users to click through confirmations, which disarms the ones that mattered. Uniform speed treats an irreversible action like a draft.

The bottom row deserves particular care about what is confirmed. "Are you sure?" is not a confirmation; the user is confirming that they clicked a button. "Refund £240 to card ending 1234. This cannot be undone." is a confirmation, because it restates the effect in terms the user can check against their intent.

Handoffs are payloads

Design the escalation path for the conditions that will actually cause one: low confidence, missing data, policy conflict, an explicit user request, repeated failure, and unusual cost or scope.

Then make sure the handoff carries something. An escalation that arrives as "the assistant could not complete this request" is worse than never having automated the case, because the human now pays the cost of rebuilding the context from scratch, on top of the original work, while the user has already waited.

The fourth row is the one that changes the ergonomics most. Handing someone a blank page is expensive; handing them a specific question with two candidate answers is cheap. "This account has two active subscriptions and the policy is ambiguous about which one the refund applies to — A or B?" can be answered in ten seconds by someone who would have needed ten minutes to reconstruct the situation.

The fifth row is what makes the handoff worth building twice. If the human's resolution disappears into a reply, the system learns nothing. If it returns as a labeled example, every escalation improves the next one.

Make feedback diagnostic

A thumbs-down aggregates every possible failure into a single number that no team can act on. Was the answer wrong, ungrounded, stale, missing context you already provided, badly formatted, or unsafe? Those route to five different teams and one incident channel.

Six buttons cost an afternoon of design and convert complaints into routed work. Two implementation details make the difference between a taxonomy that works and one that is ignored.

Keep it to five or six options, phrased in the user's language rather than yours. "Not supported by the cited source" is a phrase a user can evaluate; "hallucination" is jargon that invites miscategorization.

Attach the trace. Feedback without the exact model version, prompt version, retrieval index, and retrieved context is a complaint. Feedback with them is a reproducible test case. Handle this carefully, since traces contain user content — but a feedback pipeline that discards provenance is generating sentiment, not signal.

Practice: the interaction-state review

Sketch every state your feature can be in: first use, normal result, low-confidence result, unsupported request, sensitive action, user correction, escalation, system unavailable, and recovery.

For each state, write down three things: what the user sees, what they can control, and what they can verify. The third column is the one that exposes gaps — most teams find several states where the user is shown a result they have no way of checking.

Then run it with five people who do the work. Ask each, unprompted: What can this do? What can it not do? Who is responsible if it is wrong? What would you do if it was?

If their answers to the first question exceed the real capability, you have an over-trust problem in the framing, and it will produce incidents. If their answer to the last is "redo it myself," you have an under-trust problem, and it will produce an adoption number you cannot explain.

Common mistake

The most common mistake is treating trust as a launch-time concern — a matter of onboarding copy, disclaimers, and a tooltip — rather than a property that has to be maintained.

Trust is recalibrated continuously by experience, and the calibration is asymmetric: a single visible failure moves it far more than a hundred quiet successes. This means the states that matter most are the rare ones. How the system behaves when it is uncertain, when it is wrong and told so, and when it is unavailable does more to set long-run trust than how it behaves on the happy path.

Budget design effort accordingly. The uncertainty state and the correction state deserve as much attention as the primary flow, and they usually receive a fraction of it.