Self-Hosted LLM Cost per Token: GPU Utilization

What does a self-hosted LLM cost per million tokens on a GPU I already own?

In this example, an existing GPU costs $0.40 in electricity or $2.40 fully loaded per million output tokens. Use the first figure to estimate what running the machine adds to your power bill. Use the second when budgeting for inference as a service.

Hold the workload fixed. Each machine runs the same model and prompt mix for 100 hours a month. It averages 50 output tokens per second across that time, including time spent processing prompts. That gives 18 million output tokens a month (50 × 3,600 × 100). The owned machines are off between runs. The rental is released.

Power. Assume a wall meter reads 400 W for the whole machine. At $0.18/kWh, 0.4 kW × 100 hours costs $7.20. Divide by 18 to get $0.40 per million output tokens. If the machine stays on overnight, include those idle kWh. The renter’s electricity share requires a separate figure from the provider.

For a measured example, see how much electricity a local LLM used per million output tokens.

Hardware and operations. The existing machine has $360 of cost left to allocate over 20 months. The new machine costs $1,800 spread over 36 months, with zero resale value. Rent is an assumed $0.80 per hour. Each case includes $18 a month for operator time and assumes zero additional storage, networking, or cooling fees. Replace these assumptions with your own costs.

What number should I give my manager? Report both the power bill and the monthly service total. If your existing machine is fully depreciated and would otherwise sit unused, its hardware allocation falls to zero. In this example that brings fully loaded cost down to $1.40 per million output tokens. If inference displaces paid work on the GPU, include the value of that lost capacity in the decision.

What Revolut didn't say

David Abitbol, who leads operations and infrastructure at Revolut, said at a New York conference in October 2026 that about 2,000 engineers moved from Claude Code and Codex to self-hosted GLM models on 56 NVIDIA B200 GPUs, at about $0.30 per million tokens and roughly 90% less than before. Coverage from Resultsense and AI Weekly lists what the talk left out:

• How busy the GPUs are, which decides whether the cost per token holds as usage changes.
• The total cost of the 56-GPU deployment, and whether the GPUs are owned or rented.
• The size of the team that runs and tunes it.

None of this makes the result wrong. It makes $0.30 a number that holds for Revolut's traffic, on Revolut's GPUs, in the hours it was measured. We covered whether self-hosting pays off by team size in Self-hosting AI coding models: when it pays off. This post is about what happens after you decide it does.

Cost per token is a utilization number

A GPU costs the same per hour whether it serves one request or a hundred. The cost per token is what that hour costs divided by the tokens it served, so it moves with traffic, not with the GPU price alone.

A rough check on scale: 56 B200s rented at Lambda's on-demand $6.69 per GPU-hour cost about $273,487 a month. At $0.30 per million tokens, that bill would need about 912 billion tokens a month, around 22 million per engineer per working day. That is heavy agent use, not occasional autocomplete. Owned or reserved GPUs would need fewer tokens to reach the same rate.

Now hold the GPUs fixed and change how busy they are. Assume $0.30 was measured with the GPUs 80% busy:

Benchmarks agree. A June 2026 study running vLLM on H100s measured effective costs from $0.21 to $15.25 per million output tokens on the same hardware, depending only on how many requests arrived. At 1 to 10 requests per second, costs were 2.5 to 24 times higher than at saturation, and up to 36.3 times near idle. Its author argues that most cost calculators assume full utilization, which understates the real cost by a factor of one over utilization.

Working hours, 24-hour bills

Coding assistants follow the working day. Ten busy hours on five weekdays is 50 of the 168 hours a rented GPU bills each week, about 30%. A cluster sized to stay 80% busy at the daytime peak averages about 24% across the week, so its real cost per token is about 3.4 times the daytime figure.

There are three ways out, and all of them need hour-by-hour numbers to judge:

1. 01
   Fill the night

   Move batch work, evaluations, test generation or document processing onto the idle hours. It runs at almost no extra cost.

2. 02
   Scale with demand

   Release GPUs outside working hours if your provider lets you. You then pay for the GPUs you held, not the peak.

3. 03
   Move API traffic in

   Features still on paid APIs can use the spare capacity. Start with the ones that pay the most per token.

Tokens aren't GPU time

Once the bill is known, the next question is who it belongs to. Splitting by tokens is the obvious answer, but tokens don't all cost the same GPU time. A long prompt is read in one fast pass. Generated text comes out one token at a time. Queueing, batching and cache hits change the time again.

In one illustrative day on 8 B200s costing $1,284.48, a customer sending 1.64 billion prompt tokens and receiving 40 million was charged $911.57 by tokens. vLLM's traces showed it used 42% of the GPU time, which is $535.20. The customer generating long answers carried the rest. Splitting by tokens would have overcharged the first customer by $376 for that one day.

vLLM can report each request's time in the model through OpenTelemetry traces. Matched to the request that made the call, that gives a split by measured GPU time, with tokens as the fallback for requests without a trace.

"You no longer have to save your tokens"

That was Abitbol's point about the biggest gains: engineers handed debugging and ticket admin to the models once tokens stopped feeling scarce. A fixed GPU bill does that. It also hides who is using the capacity. Before, Revolut said some engineers were spending "$1,000 a month, $10,000 a month" on commercial tools. On your own GPUs, nobody sees that number unless you build it.

That matters when the cluster fills. The next GPUs come as a whole node, and the team asking for them should know which workloads used the last ones. If the GPUs serve your product rather than your engineers, the same split decides each customer's margin: a customer on a flat plan who fills the night with agent runs costs more than the token count shows.

What to measure once your GPUs are running

1. 01
   Utilization by the hour

   NVIDIA's DCGM exporter or vLLM's own metrics show how busy each GPU was each hour. Idle cost is the unused share of every hour you paid for.

2. 02
   GPU time per request

   vLLM's OpenTelemetry traces give each request's time in the model. That is the fair basis for splitting the bill.

3. 03
   Cost per team, customer and feature

   Tag every call with who made it and why. Split each day's bill by GPU time, and show it next to what those customers pay or what that team's API spend used to be.

4. 04
   Cost per million tokens, now and at target

   Track today's rate and the rate at your target utilization. The gap is what idle time costs you, and the real comparison with an API price.

5. 05
   What to move next

   Compare each API feature's spend with its share of your spare capacity, and check whether the next whole GPU still saves money once you round up.

Ganivra does each of these for DeepSeek, Qwen, GLM or any model on vLLM: see how self-hosted LLM cost per call works. For today's API prices against self-hosted cost, see Claude, OpenAI and Gemini API vs self-hosted LLM cost. If you haven't bought GPUs yet, start with the GPU vs API calculator.