Your LLM gateway and your provider bill are both right

Most teams put a gateway in front of their LLM providers for reasons that have nothing to do with cost: one place to rotate keys, one place to set rate limits, one place to fail over between models. The cost benefit is a side effect, and it’s a large one.

A provider invoice arrives keyed by API key, at best. A gateway knows which of your services made the call. Under which virtual key, on behalf of which team, against which model. That is the attribution question everyone eventually asks, answered for free by infrastructure you already run.

Then you connect both to the same cost tool, and quietly break your totals.

The same dollars, twice

If your gateway routes to Anthropic, and you also read Anthropic’s own billing data, the same spend arrives from two directions. Both records are accurate. Neither is a duplicate in any way software can detect. They are two ordinary cost lines with the same amount, and nothing about either one says “this is the other one, seen differently.”

Summed, your AI spend doubles. Not visibly. It doubles into a number that looks entirely plausible, in the direction people are already braced for, at exactly the moment AI spend is the line everyone is watching. It is the kind of error you discover in a board meeting.

The fix is not clever. You have to decide which record your totals are built from. Which one that should be is not obvious.

Neither source is simply better

The invoice is exact and blind. It is what you were charged, including your negotiated rates, credits, and any traffic that never went near your gateway: the notebook someone ran against a raw key, the vendored SDK in a service that predates the proxy. It is the number finance will reconcile against. It knows nothing about your teams.

The gateway is attributed and approximate. It knows the key, the team, the model and the service, per request. But its dollar figures are computed, not billed: a gateway prices each request by multiplying tokens against a published model price map. That map — public, generic, and updated by strangers — does not know your contract, your committed-use discount, or a price change made last Tuesday. Two identical workloads priced by a gateway and by an invoice will differ, usually slightly.

So: exact totals with no breakdown, or a full breakdown with approximate totals. Which side you want depends on what you are doing with the number. A chargeback model that bills teams internally wants the invoice as ground truth. A team trying to work out why inference costs tripled last month wants the breakdown, and will not care about a 2% variance.

The gap between them

The difference between the two figures is itself a measurement, once you have both in front of you.

If your gateway attributes $412 and the provider billed $400, one of a few things is true. The price map has drifted from your actual rates. Or a discount applies that the gateway doesn’t model. Neither is alarming.

If the gap runs the other way, and the invoice is materially higher than everything your gateway saw, that is a different finding. It means real spend is not going through your proxy. Someone is calling the provider directly with a key your gateway never issued, and no amount of gateway configuration will show it to you. That is the number worth chasing, and you can only see it by holding both records up against each other.

Decide per provider

Routing is rarely uniform, so one source rarely fits every provider. The instinct is to pick one and apply it everywhere.

A fairly typical setup: OpenAI and Anthropic traffic goes through the gateway, but Bedrock is called directly from a service that already has IAM credentials, and one team uses a provider nobody else touches. The right answer differs per upstream provider: gateway figures where the gateway sees everything, provider figures where it does not.

That also means the decision has to be revisited. Routing a new provider through the gateway changes which source should own it, and nothing will remind you.

Tokens are not dollars

The double-counting problem applies to money. It does not apply to token counts. Your gateway’s record of how many input and output tokens each team consumed has no counterpart on the invoice. Nothing to double, nothing to reconcile, so keeping it costs nothing.

You can take the invoice’s dollars as your source of truth and still keep the gateway’s per-team, per-key token attribution. “Which team is driving our token growth” and “what did we actually pay” are different questions, and they can be answered from different sources without either contaminating the other. Deciding that the invoice owns a provider’s spend does not mean throwing away everything the gateway knows about it.

How Plutus handles this

Plutus reads a self-hosted LiteLLM proxy through a collector that runs inside your own network, so nothing has to be exposed to us. It aggregates locally and sends one daily summary. Per-request data never leaves your infrastructure.

When it reports a provider for the first time, that provider’s spend is held out of your totals until you decide who owns it. Not counted, not silently added. The decision is per upstream provider, it is reversible, and reversing it removes the figures it was contributing rather than leaving them behind. Token counts are recorded either way, so a provider whose dollars you track through its own connector still shows per-team and per-key usage.

Holding spend out has an obvious cost: your totals are short until you decide. Every provider being held out is listed as “Not counted”, so you can see exactly which spend is missing.

Setting it up

Two things reliably go wrong first.

Your gateway needs an admin-scoped key to expose spend data. The key that makes inference calls will authenticate happily and then fail on every billing route. The same thing happens with the providers directly, which we wrote about in attributing AI API costs per key, model, and team.

And your gateway needs somewhere to persist spend records. A proxy running without a database serves traffic perfectly well and keeps no history, so there is nothing to read. If your first sync reports zero rows and no error, check that before anything else.