Email triage is a decision about attention. A useful system helps someone see what needs a response, what is waiting on another person, and what can be read later. An LLM can assist with interpreting varied language, but the surrounding workflow determines whether its labels save time or hide important work.

Start with a decision the team already makes. In an illustrative community workshop inbox, two coordinators receive booking questions, equipment updates, newsletters, and follow-up replies. They want a suggested review order each morning. They do not need the model to write replies or manage bookings. Keeping the first task focused makes its quality easier to measure.

Design labels around the next human decision

For this workshop, begin with five labels. Each label needs a definition, an example, and a boundary that distinguishes it from its neighbors. Avoid creating a new label for every topic: woodworking, photography, and ceramics are subjects, while whether someone needs to reply is an operational decision. Mixing those ideas makes the system difficult to evaluate.

  • Reply needed: the latest message asks a question or requests an action that a coordinator must address. An unanswered question about wheelchair access belongs here.
  • Waiting for sender: the sender explicitly says they will provide the next required information. A participant promising to confirm their group size is an example.
  • Reference: the message provides useful information without an apparent pending response. A completed delivery confirmation can fit, unless it also reports a problem.
  • Promotion: the main purpose is advertising or a general promotional update, with no identified workshop obligation requiring attention.
  • Review required: the available text does not support one clear label, or the message falls outside the defined task.

Write a precedence rule for mixed messages. In this example, an explicit unanswered request takes priority over an informational update. A supplier email containing a newsletter and a question about tomorrow's delivery should receive Reply needed. Review required remains available when the question's status cannot be established from the supplied context.

Ask for a small, inspectable result

Use a narrow output contract instead of a free-form explanation. An illustrative result might contain a source reference, one allowed label, a short evidence excerpt, and an optional review reason. The evidence helps a coordinator check the decision quickly; it should point to relevant message text rather than present a lengthy account of the model's internal reasoning.

{
  "message_ref": "sample-014",
  "label": "reply_needed",
  "evidence": "Could you confirm step-free access?",
  "review_reason": null
}

This is a proposed application record, not a working endpoint or a claimed VirtualEmailBox.com API. Validate the allowed labels, field types, lengths, and source reference in code. Confirm that the evidence actually occurs in the supplied text. A valid-looking object with an invented reference should become an exception rather than a mailbox change.

For one provider's implementation approach, OpenAI's Structured Outputs documentation explains schema-constrained responses and handling refusals or incomplete responses. It also states that structured results can still contain mistakes. Schema compliance makes results easier to process; the workflow must separately evaluate whether the classification is useful and correct.

Prepare only the context the label needs

For a first pass, supply the latest message, its subject, and a clear indication of where older quoted material begins. Include earlier conversation only when it is necessary to determine whether a request remains unresolved. If you truncate a long thread, record that limitation so the result can go to review when the missing context matters.

Strip decorative signatures and duplicated quoted passages where your parser can do so reliably. Preserve meaningful dates and the words that change a request's status, such as confirmed, canceled, or still waiting. A preprocessing rule that removes too much can cause more errors than a model change can repair.

The LLM email planning guide discusses the relationship between input preparation and model selection. Use the same prepared representation during evaluation and routine operation so the test reflects the actual task.

Create a test set before tuning prompts

Start with a deliberately varied collection of synthetic examples or appropriately approved, minimized records. A practical workshop exercise could use sixty messages, with ordinary examples and difficult boundaries represented. This number is an illustrative starting point, not a claim that sixty cases prove production readiness. The useful size depends on how varied the inbox is and how costly mistakes become.

Have both coordinators label the examples independently. Where they disagree, improve the definition before asking the model to choose. If one person treats every supplier update as Reference and another treats delivery problems as Reply needed, the disagreement reveals a policy gap. A model cannot reliably satisfy two contradictory labeling rules.

Keep some examples aside for a final check. Repeatedly changing instructions until they fit every visible example can disguise poor behavior on new messages. Record the label guide, preparation rules, and configuration version alongside each evaluation so a later comparison measures an identifiable change.

Measure the mistakes people actually feel

Overall agreement is useful, but it can hide the error that matters most. Suppose most messages are routine updates. A classifier could appear successful while missing several requests that require replies. Track how often actual Reply needed messages are found, and how often messages assigned that label truly need attention. These answer different operational questions.

Inspect representative pairs. A message saying that a sender will confirm tomorrow should generally differ from one asking the coordinator to confirm tomorrow. A canceled class announcement differs from a customer asking whether a class is canceled. Include short messages whose meaning depends on the preceding question, and messages with several requests in one paragraph.

Measure Review required separately. A system that sends everything to review can avoid many confident mistakes while offering little time saving. A system that never abstains may produce an attractive dashboard while concealing uncertainty. Choose the balance through observed examples and the team's tolerance for missed work, rather than an untested confidence threshold.

Give the review queue a usable operating rhythm

Show each suggested label beside the relevant source text and make correction easy. A reviewer should be able to accept, change, or defer without navigating through several screens. Keep Review required visible, and assign someone to inspect it at a defined interval. An exception queue with no owner becomes a second neglected inbox.

During the first phase, review all suggestions. Later, sample accepted routine labels while continuing to review difficult cases. Retain the ability to compare a corrected label with the original suggestion. Group recurring disagreements by cause: missing context, unclear policy, preprocessing error, or classification error. Each cause calls for a different fix.

For the permissions around this workflow, see AI agent email safety. Triage quality and authority are separate design questions, and both need explicit decisions before an assistant takes additional actions.

Control processing cost without hiding the hard cases

Estimate workload from the number of messages, the amount of text per request, and the number of passes per message. Include retries and human review time in the estimate. The cheapest model call may be an expensive workflow if coordinators must repeatedly repair its labels. Compare total operating effort using your own representative batch.

Reduce repeated context and avoid reclassifying unchanged records unnecessarily. Cache results only under a key that includes the relevant message state and labeling configuration. A new reply can change the correct label even when the thread identifier stays the same. Keep a bounded retry policy and send persistent parsing failures to review.

Make labels reversible and recovery ordinary

In early use, store suggestions separately from existing mailbox organization. If you later apply labels, record the previous state and the responsible run. Avoid deleting or permanently archiving mail as an automatic consequence of a classification. The team should be able to undo one run without guessing which changes it made.

When a configuration performs poorly, pause it, restore affected labels where appropriate, and return unresolved messages to review. The automation workflow guide explains the broader recovery process. A good triage system earns trust through clear definitions, measurable decisions, and easy correction. Start there, then expand only when the next step improves the coordinator's actual work.