Grounded Support Triage

Problem

A support ticket either has an answer in the product documentation or it needs a human. A bot that guesses wrong in either direction does damage: it invents an answer, or it escalates something the docs already cover. HackerRank Orchestrate’s first edition (May 2026), a 24-hour build where the challenge itself is agentic AI, asked for a system that triages tickets across HackerRank, Claude and Visa support corpora. Entries were judged on output accuracy against hidden labels, code quality, the AI-collaboration chat transcript and a 30-minute AI-judge voice interview.

Approach

I built it as a LangGraph multi-agent pipeline with four jobs: classify, retrieve, verify, then reply or escalate. Every ticket leaves a traceable execution flow, so a wrong answer can be followed back to the step that produced it.

Classification starts cheap. Regex fast paths and structured routing handle the common shapes of ticket without a model, which keeps a typical ticket at 0–2 LLM calls. That matters for latency and cost, and it means fewer places where a model can be wrong.

Retrieval is hybrid over a 774-document corpus. BM25 catches exact product terms; dense embeddings (bge-small-en-v1.5) catch paraphrases; a cross-encoder reranks the candidates.

The part I care most about is the critic. Before a reply goes out, a deterministic layer checks its citations and hard facts against the retrieved documents and rejects answers they don’t support. The LLM steps use Claude and Gemini, but the final gate is code.

I also added an evaluation CLI, tracing and observability tooling, so I could see where tickets went during the build rather than after it.

Results

12th place of 1,349 participants with all four submissions (12,885 registered from 48 countries). Typical tickets ran at 0–2 LLM calls.

What I learned

Checks belong around the model, in code, not inside the prompt. The critic is a small amount of deterministic code, and it is what made a generated reply safe enough to send.

Not every ticket needs an LLM. Routing the easy cases with regex made the system cheaper and easier to debug, and left the model budget for tickets that needed it.

This was my first Orchestrate. The same idea, code around the model and a closed check before output, became the “signals, never verdicts” design of my August entry. What I hadn’t learned yet was to keep a hold-out set; that lesson came later and is written up in the router post-mortem.