Elson TanTwo customers, same plan, same price. One asked short questions and cost me almost nothing. The other uploaded 40-page contracts all day and cost me more than they paid.
I was charging per message. A message from the first customer and a message from the second were the same line on the invoice and nowhere near the same line on my bill.
Message count is a proxy for cost that stops tracking cost the moment your product does anything interesting. So I moved to tokens, which is what the providers charge me for.
| Billing unit | Tracks provider cost | Customer intuition | Fails when |
|---|---|---|---|
| Per message | Poorly | Good | Message size varies, which it always does |
| Per conversation | Worse | Good | One conversation runs all afternoon |
| Per seat | Not at all | Best | Usage per seat varies by orders of magnitude |
| Input plus output tokens | Directly | Weak, needs translating | Never, but you have to explain it |
Count model calls, not user messages
One question from a user is not one call to a model. It is a routing call to decide who handles it, three rounds of tool use, and a summarisation at the end. Five calls, five bills from my provider.
So that is what I record: one row per model call, tagged with the tenant, the agent and the role it was playing. Adding them up for an invoice is easy afterwards. Throwing the detail away first is not something you can undo.
type UsageEvent = {
idempotencyKey: string // stable per provider call
tenantId: string
agentId: string
role: 'orchestrator' | 'classifier' | 'summariser' | 'reasoner'
model: string
tokensIn: number
tokensOut: number
at: number
}The role tag paid for itself immediately. Aggregate token counts told me I was spending a lot. Per-role counts told me a third of it was going to a routing step whose entire output was a sub-agent name, which is the finding that led to routing that role to a cheaper model.
Why does the idempotency key matter so much?
Because writes fail after the work is done, and a retry without a key charges twice.
The sequence that bites: the model call succeeds, you write the usage event, the write times out, your retry logic runs it again. Now the tenant has been billed for one call twice. Nobody notices until a customer reconciles their invoice, and at that point every number you have produced is suspect.
The key has to be derived from the call, not generated at write time, or a retry produces a fresh key and the guard does nothing. I use the provider’s response id where one exists and a hash of the request otherwise. Stripe’s idempotent requests document the same contract from the other side.
Do this from the first line of billing code. Adding idempotency to a metering pipeline that has been running for months means auditing every event you have already written.
Check the allowance before you spend it
Metering tells you what happened. It does not stop a tenant blowing through their plan, and recording an overage you cannot collect is not a billing system.
Anything expensive gets an allowance check before it starts. A website import that will cost around 51,000 tokens checks that estimate against the remaining balance and refuses up front, rather than failing partway with half an import written and a partial charge.
| Check | When | On failure |
|---|---|---|
| Hard balance | Before any model call | Refuse, prompt to top up |
| Estimated cost | Before a bulk operation | Refuse with the estimate shown |
| Soft threshold | On crossing 80% | Notify, do not block |
| Plan feature limit | On configuration change | Refuse, explain the plan limit |
The soft threshold is the one that reduces support load. A tenant who gets a warning at 80% tops up. A tenant whose agent goes silent mid-conversation opens a ticket.
Where the limit bites is a product decision
A conversation is not atomic. When the balance runs out three tool calls into a five-call task, the technically correct answer is to stop immediately, and it produces a terrible experience: a half-finished task and no explanation.
| Where you cut off | Overage you absorb | What the user sees |
|---|---|---|
| Mid tool call | None | The agent stops mid-sentence, with no explanation |
| At the next turn boundary | Bounded by one turn | The task finishes, then a clear message about the balance |
| At the end of the conversation | Unbounded | Nothing, until the invoice |
I check at turn boundaries and let an in-flight turn complete, accepting a small overage. The overage is bounded by the cost of one turn, which is a number I can look at and decide is acceptable. It is not free and I would not pretend the alternative is obviously wrong. It is a choice between a small unbilled cost and a user watching an agent stop mid-sentence.
Storage for counters wants low-latency reads on every request. I use Workers KV for the wallet, which is eventually consistent, so two concurrent requests can both see enough balance. Under-counting by one turn is a trade I took deliberately over putting a strongly consistent read on the hot path of every message.
Connecting it to money
Usage-based billing has a shape that predates AI, and Stripe’s usage-based billing documentation covers the meter and subscription mechanics.
The AI-specific parts are that your cost basis moves under you, and that tokens are not a unit customers have intuition about. When a provider changes prices or you route a role to a cheaper model, your margin changes without anything in your billing code changing. Store the model and the provider on every usage event so you can recompute historical margin. Without that you can see what you charged and not what it cost you.
Sources
- Idempotent requests, Stripe API reference
- Usage-based billing, Stripe documentation
- Workers KV, Cloudflare
- AI Gateway, Cloudflare