LLM Email Triage: Design Labels, Reviews, and Recovery
Create a practical LLM triage system with a small label set, structured results, realistic evaluation cases, a manageable review queue, and reversible changes.
Read the articleAutomation & AI / Field guide
LLM Model Virtual Email Box
An LLM model virtual email box workflow uses a language model for a defined text task, such as suggesting a label or extracting a request for review. The central design work is evaluation: deciding what the labels mean, selecting representative examples, and measuring mistakes that matter to the workflow. Begin with a small, permitted sample and a result that a human can inspect. Keep the classification output separate from any decision to send, delete, or otherwise change mail.
Define the labels before writing prompts
Choose categories that correspond to different useful next steps. For each label, write a short definition and examples of both inclusion and exclusion. If two labels lead to the same action and reviewers struggle to distinguish them, consider simplifying the set. Include an explicit uncertain or needs-review outcome for messages that lack enough context. Decide how to treat mixed-topic conversations and quoted history. These choices belong in the task definition, because inconsistent human expectations make it difficult to judge whether a model's classification is useful.
Build an evaluation set with difficult cases
Select messages you are authorized to use, with routine cases, near misses, incomplete requests, and unusual formatting. Remove unrelated sensitive material when it is not necessary to the test. Have a reviewer label the sample using the written definitions, recording genuine disagreements instead of forcing certainty. Keep some examples separate from prompt development so they can reveal whether a revision improves beyond the cases used to create it. Review individual errors as well as totals, especially when one kind of mistake has a more consequential downstream effect.
Validate outputs and watch for change
Request an output structure that the application can check, then validate required fields and allowed labels before using the result. Treat malformed or unsupported output as a review case rather than guessing its intended meaning. Record the model and prompt configuration used for each evaluation. After changing either, repeat the representative test and inspect the errors. Periodically review new message patterns too; a workflow may encounter topics and formats absent from its original sample. Add permitted examples deliberately and preserve the ability to compare revisions.
Go deeper / Virtual Email Box Lab
Put the idea to work.
Detailed examples and practical tradeoffs for the decisions behind this guide.
AI Agent Email: Keep Untrusted Messages Inside Clear Boundaries
Design an email assistant around read-only access, explicit authority, reviewable drafts, and recovery steps so incoming messages cannot quietly expand its role.
Read the article