WhatsApp Notification Router

Problem

Every WhatsApp message needs a routing decision for its user: notify now, hold for a digest, or mute. Messages arrive as text, image posters and voice notes, and some come from impersonator businesses or carry prompt injections. In HackerRank Orchestrate’s August 2026 edition, a 24-hour build where the challenge itself is agentic AI, the system had to return an action, a message type, a reason, a confidence and cited evidence for each message. The data was 13 relational CSVs, 412 historical interactions, 20 images and 13 voice notes, and output accuracy was graded against hidden labels.

Approach

The design rule was “signals, never verdicts”. Deterministic code does the retrieval (BM25 plus hard links between records) and extracts risk and trust signals. The LLM only picks a reason from a closed bank of 27 reason templates. Code then derives the action, confidence and evidence, so every output is schema-valid by construction.

I measured the noise before fighting it. temperature=0 was ruled out, and a fixed seed still changed 37 of 40 rows between runs. A 5-vote majority halved run-to-run variance, from about 3 rows in 40 to 1.3.

Safety does not depend on the LLM agreeing. Untrusted message content is fenced, and deterministic overrides handle prompt injection, impersonators and high-risk businesses. A resumable per-vote cache meant 5 mid-run process kills cost zero LLM calls.

Results

Ranked 20th of 1,983 submissions, with a final score of 78.3/100: chat transcript 9.6/10, code 25.8/30, AI-judge interview 24.6/30, output accuracy 18.3/30. All 6 impersonator-business messages and both injection attempts were muted.

After the competition I cut 30% of the code (the pipeline by 41%) and proved behavior byte-identical with a golden fingerprint of every prompt and context.

What I learned

Output accuracy was my weakest score, so I wrote a post-mortem. Locally I measured 0.925. On the hidden set it was 61%. The post-mortem traces the gap to a missing hold-out set: without one, 0.925 was never an estimate of performance on unseen data. Two things made it worse. My final changes before submission were never measured, and the distribution was skewed toward mute.

The architecture held up. Closed vocabularies, code-derived outputs and deterministic overrides made every decision auditable, and the safety cases all landed. But auditable is not the same as accurate on data I haven’t seen. Next time the hold-out set comes first, and every change gets measured against it, including the last one.