CS lead's playbook: onboarding an AI support agent with refund permissions

An AI support agent with refund permissions sits at the intersection of three failure modes: the agent over-refunds and you discover it in the monthly P&L, the agent under-refunds and CSAT drops without anyone knowing why, or the agent issues a refund pattern that surfaces in audit and finance learns about the rollout from the external auditor instead of from you.

All three are preventable with a 30-day rollout plan that treats the agent as a new financial actor from day zero, not as a chat tool that happens to also move money.

Week 1: shadow the agent before it touches anything

Spin up the agent in read-only mode. It can read tickets, read customer history, propose a refund decision, and write the proposed decision to a queue. It cannot call Stripe. For five business days, your most experienced CS rep reviews the queue and tags each proposal as 'agree, would have refunded the same way' or 'disagree, would not have refunded this way' with a one-line reason.

Target 80% agreement before moving forward. Below that, the model needs more context (better prompt, better RAG over your refund policy doc, or both) before it gets near a live button. Above that, you have a baseline.

Do not skip this week. Teams that skip shadow review almost always discover an over-refund pattern in week 3 that would have been caught in week 1.

Week 2: write the agent's policy

The agent is a new identity in your refund policy. Treat it that way explicitly. Three rules to write before the agent goes live:

Identity-based threshold: the agent's auto-approve ceiling is lower than your most junior human rep's auto-approve ceiling, until the agent has proven out. A common starting point is $50 for the agent versus $200 for a rep. Easy to raise once the data is in.

Volume cap: cap the agent's refund count per hour and per day. Refund storms are the most common AI failure mode and the cheapest to prevent. A reasonable starting cap is 2x the average hourly refund volume of your highest-volume human rep.

Reason-code rules: explicit auto-approve list (duplicate, fraudulent) and explicit must-route-to-human list (goodwill, requested_by_customer above $20). The agent's training data probably blurs these; the policy must not.

Week 3: live enforcement with a tight feedback loop

Turn on live enforcement under the week-2 policy. The agent issues refunds within policy automatically and routes everything above the thresholds to a human for approval. Critical: the approval surface needs to land where the human approver actually works (Slack DM, in-app queue, or both), not in an email they will see Friday afternoon.

Two daily rituals for the week: a 10-minute morning review of yesterday's auto-approved agent refunds (a sample of 10 is enough; you are looking for patterns, not auditing each one) and a same-day investigation of any agent refund flagged by a customer in a follow-up ticket. Both rituals taper after the first week.

Decision ledger is doing the heavy lifting here. If you do not have one, you are flying blind on the pattern question; an LLM transcript is not the same artifact as a signed decision receipt with the policy version.

Week 4: tighten or relax the policy, not the model

By end of week 3 you have 5 to 10 business days of live agent decisions logged against policy. Pull the weekly exception report: which rules fired the most, which auto-approve buckets had any customer complaints, which routed-to-human approvals took longest.

Three common moves in week 4: raise the agent's auto-approve threshold by 50 to 100% if the rule never fired for a problem reason; lower the volume cap if the agent hit it on a normal-volume day (cap was too tight); add an explicit identity-based rule for a customer segment where the agent's behavior is consistently off (high-value enterprise customers, accounts in renewal, accounts with prior refunds in the last 30 days).

Note: the move is to update the policy, not to retrain the model. The model will keep doing what its training data says it should; the policy is the layer that translates your judgment into enforcement. Updating policy is hours; retraining is months.

Coordinate with finance before the agent goes live

One conversation, 30 minutes, before week 3. Show the head of finance or controller the agent's policy, the threshold settings, and the planned exception report. They do not need to approve every line; they need to know the surface exists and the controls are in place.

Two outcomes from that meeting that matter: finance signs off in writing (one line in Slack or email is enough) and you agree on the report cadence for the first 90 days (weekly is usually right; daily for the first two weeks is fine if finance is anxious about it).

This single conversation prevents the worst version of the rollout, which is finance discovering the agent from a refund anomaly two months later.

What 'going well' looks like at day 30

Agent is handling 60 to 80% of inbound refund decisions autonomously inside policy. Human approvers are touching only the 20 to 40% that exceeded thresholds. Refund volume per ticket has not materially changed (the agent is not over-refunding). CSAT on tickets the agent touched is at or above CSAT on tickets a junior rep would have handled.

Exception report fires on something interesting (a customer who has refunded three times in 30 days, a reason-code shift) once or twice a week and gets actioned. Decision ledger is queryable; if finance or audit asks 'what did the agent do on the 14th between 2pm and 4pm', the answer is one query, not a Slack archaeology session.

If a refund storm happens anyway

It will, eventually. Volume cap catches most; for the case where the cap was too high, the kill switch is the backstop. Pause the agent's refund permissions in one click, investigate, fix the policy or the prompt, restart. The decision ledger tells you exactly which refunds happened, to whom, and what reason codes were involved.

Storms are recoverable when the surface is governed and traceable. They are career events when neither is true. The 30-day plan above is the cheap version of governed and traceable.