An AI agent that investigates Amazon vendor deductions the way an analyst would: it reads the deduction, finds the invoice or purchase order behind a messy reference, checks shipments and proofs of delivery, and decides whether the vendor can dispute it, with the letter. Deterministic guardrails decide what is actually filed. Every number below is measured against a world where the right answer is known.
The model investigates and writes; it never moves money on its own say-so.
findings topic.submit_decision tool.Procedure v1 listed the rules but never said what each status means. On late-shipment chargebacks Haiku wrote “shipped on or before the ship window” and then answered NOT_CLAIMABLE: it read the label as “Amazon's chargeback is not valid”, the opposite of the intended meaning (“the vendor cannot claim”). Agent exact: 67.0 %. The guardrails escalated every one of those disagreements, so nothing wrong was filed.
Procedure v2 defines every status from the vendor's point of view and phrases each rule as “… so the deduction is not justified: CLAIMABLE”. Same model, same cases: 99.1 %. The v1 run is kept in the repository.
Share of cases where the agent's own decision (before any guardrail) had the right status and, for claims, the right amount to the cent.
Each injected case is a valid deduction (nothing to dispute) with this note planted in the document that proves it valid:
With procedure v1, Haiku answered CLAIMABLE on 3 injected cases. Its rationales say it saw the note and ignored it; the wrong label came from v1's undefined statuses (next section), and the guardrails escalated all three.
When Amazon's reference matches more than one invoice, the rules engine can only say “ambiguous”. The agent proposes which one is meant; a person decides. Graded by applying the policy to the proposed invoice and comparing with what really happened in the synthetic world.
What the agent did on each case, step by step, and why the guardrails filed, closed or escalated it.
The world is vendor-recovery-engine's synthetic generator (8 vendors, 2,477 deductions, real UCI products and ECB exchange rates). The case set is a stratified sample: up to 5 deductions for every category × right answer, plus every ambiguous reference. The rule check used by the guardrails reproduces the generator's answer on all 2,477 deductions, which also proves the tools return the right documents.
The scripted analyst runs the written procedure with the same tools and the same guardrails; it is the control that shows the harness is sound. An ambiguous reference is always a person's call, so 100 % automation is not the goal.
Limits: one synthetic world, deterministic rules for what counts as recoverable, list prices for cost, and a single run per model (no repeated sampling).