This Bytechap technical analysis examines a deceptively small production failure: an application expects one of a few routing labels, but the language model answers “Billing.” with capitalization and punctuation. A person sees the intended category immediately. A switch statement comparing the response with the exact string “billing” may see an unknown value and send the request down the wrong path.
The defect is at the contract boundary
A prompt such as “reply with billing, technical, or account” expresses a preference, not a machine-enforced contract. Language models generate text and may add punctuation, explanations, Markdown, whitespace, or a synonym. Treating that unconstrained text as an enum makes the routing layer depend on formatting that the model was never required to guarantee.
Constrain the output before parsing it
When the model API supports structured output or tool calling, define the label as an enum in a schema and require the response to satisfy it. The application should deserialize a field such as category, validate it against the allowed set, and only then dispatch work. This is stronger than repeatedly emphasizing “one word only” in the prompt because the constraint becomes part of the interface rather than a suggestion in prose.
Normalize defensively, but do not guess freely
Older models and providers may not support reliable schemas. At that boundary, trim surrounding whitespace, apply a locale-independent case conversion, and accept a small explicit mapping for harmless variants such as a trailing period. Keep this mapping narrow. Fuzzy matching an arbitrary paragraph to a business action can turn an ambiguous response into a confident but incorrect decision.
Unknown output is a valid system state
The router needs an explicit invalid or uncertain branch. It can retry once with a constrained prompt, request human review, or use a safe default that does not perform an irreversible action. It should not silently coerce an unrecognized answer. Log the raw response, validated label, model and prompt versions, and chosen fallback so failures can be reproduced without exposing sensitive user content.
Keep reasoning separate from the decision
A classifier may benefit from internal reasoning or a short explanation for audit purposes, but downstream code should consume only the validated decision field. If explanations are needed, return them in a separate optional field. Parsing the last word of a paragraph or searching the response for category names is brittle: the explanation may mention several labels while rejecting all but one.
Test the seam, not just the model
Evaluation should include casing, punctuation, extra whitespace, Markdown fences, multiple labels, refusal text, empty responses, timeouts, and newly introduced categories. Contract tests can feed these responses directly into the parser without calling a model. Production metrics should track invalid-output rate, retry rate, fallback route, and label distribution; a sudden change can reveal prompt drift or a provider update before it becomes a support incident.
The broader lesson
LLM output should be handled like data arriving from any external service: constrained when possible, parsed once, validated before use, and rejected safely when it violates the contract. The period after “Billing” is not the real bug. The real bug is allowing free-form language to cross directly into deterministic control flow.