A quiet hour costs as much as a busy one.
There is no list price per call, so per-token cost tools show self-hosted calls as zero or unpriced.
GPU metering for self-hosted LLMs
Your self-hosted LLM bill, split into a cost per call.
Running DeepSeek, Qwen or GLM on vLLM? GPUs bill by the hour, busy or idle, so self-hosted calls have no price. Ganivra divides the bill across the calls that used it, so you see cost per request, per customer and per feature next to your API spend.
- Split by measured GPU time from vLLM traces
- Idle cost from NVIDIA GPU metrics
- Works with your own or rented GPUs
- No proxy, no prompts stored
Start free 14-day trial
Not running GPUs yet? Estimate GPU vs API cost, free →No credit card · Python SDK, REST or OpenTelemetry · From $29/month after the trial
Why it is hard
An API bills per token. A GPU bills per hour.
Requests share the GPU in batches.
Token counts miss that generating an answer holds the GPU far longer than reading a prompt.
Unused capacity is a real cost.
Someone pays for the hours the GPUs sit idle. If no customer does, margins look better than they are.
Same GPUs, same day
Tokens and GPU time disagree about who costs more.
Two customers share $1,284.48 of B200 time. Split by tokens, the customer sending long documents pays most. Split by the GPU time vLLM measured, the one generating long answers does.
Why GPU utilization moves the cost per token
Claude, OpenAI and Gemini API vs self-hosted cost
What you get
Every self-hosted call priced. Every idle hour counted.
New vLLM traces
Cost per call from measured GPU time.
vLLM reports how long each request spent in the model. Ganivra matches each trace to its call and splits the bill by that time.
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Bearer gv_…" \
vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
--otlp-traces-endpoint https://api.ganivra.com/v1/otlp/v1/tracesUtilization
See what idle GPUs cost, hour by hour.
NVIDIA's DCGM exporter, or vLLM's own metrics, through an OpenTelemetry Collector. Each hour's unused share becomes idle cost.
55%of a day's GPU bill paid for idle capacity Example · pool running at 45%
Send GPU metrics Autoscaling
Billed for the GPUs that actually ran.
For clusters that scale up and down, each hour is billed for the GPUs the metrics saw, not a fixed count.
How it worksNext to your APIs
One margin view for API and GPU calls.
Self-hosted calls appear in customer margins, cost breakdowns and executions with OpenAI, Anthropic and Gemini calls.
See the dashboardCost per 1M tokens
What your GPUs cost per token, now and at 80%.
Your effective rate today, the rate if the GPUs were 80% busy, and the same tokens on a hosted API.
When self-hosting pays offOwn GPUs or API
Which API features would cost less on your GPUs?
Ganivra already knows each feature's API tokens and cost. It prices them on your GPUs at 80% utilization, fills spare capacity first, and checks whether whole new GPUs would pay off. It works before you buy GPUs too.
Run it on your trafficFeatureAPI, 30 daysOn your GPUsVerdict
support_chatbot$1,242.50Spare capacityMove: fits in spare capacity
report_writer$6,100.00$3,240.00Move, on new GPUs
tagging$31.00$278.00Keep on the API
Saving over 30 days$2,525.70
Illustrative figures · compares cost, not model qualitySetup
Priced the same day. Two steps, then two optional.
- 01
Add your GPUs
GPU type, count, price per GPU-hour and the date they started. Pick a list price from Lambda or fal, or enter your contract rate. Optionally list the models the pool serves and an API model to compare with.
- 02
Send self-hosted calls
With the Python SDK, name your model servers once: self_hosted_hosts=["vllm.internal"]. Calls to them carry the customer, feature and workflow like any API call. With REST, set provider to self_hosted.
- 03
Add vLLM traces for GPU time
vLLM reports how long each request spent in the model. Ganivra matches each trace to its call by the response id and splits the bill by that time instead of tokens. Optional.
- 04
Add GPU metrics for utilization
An OpenTelemetry Collector forwards NVIDIA's DCGM exporter, or vLLM's own metrics. Each hour's unused share becomes idle cost, and autoscaled clusters are billed for the GPUs actually running. Optional.
OpenTelemetry Collector settings for GPU metrics
receivers:
prometheus:
config:
scrape_configs:
- job_name: gpus
scrape_interval: 30s
static_configs:
- targets: ["dcgm-exporter:9400", "vllm:8000"]
processors:
resource:
attributes:
- { key: ganivra.gpu_pool, value: "B200 cluster", action: upsert }
exporters:
otlphttp:
endpoint: https://api.ganivra.com/v1/otlp
headers: { Authorization: "Bearer gv_…" }
service:
pipelines:
metrics: { receivers: [prometheus], processors: [resource], exporters: [otlphttp] }What these estimates can and cannot tell you
- Calls without a vLLM trace are split by tokens, which is an approximation. With a trace, a call's share is the time it spent in the model; requests batched together each count their full time, so this measures time on the GPU, not compute used alone.
- Idle cost needs GPU metrics to see quiet hours. Without them, only whole days with no calls count as idle. NVIDIA's utilization counts a GPU as busy whenever any work runs on it, and vLLM's signal only shows whether a request was running, so both can read higher than real use.
- A call's cost on a finished day is recorded in the audit history when it changes. During the running day the split changes every hour and isn't recorded; the call's breakdown shows its day, rate and whether the figure is still provisional.
- The move analysis compares cost only. An open model on your GPUs is not the model the API ran, its tokenizer counts the same text differently, and a pool's throughput is assumed to hold for the new traffic.
The terms set out the same limits.
Frequently asked questions
Self-hosted LLM cost, answered.
How the split works, which servers it supports, and when self-hosting pays off.
01How do you calculate the cost per request of a self-hosted LLM?
Ganivra takes each UTC day's GPU cost (GPUs × price per GPU-hour × hours running) and splits it across that day's calls. A call's share is its measured GPU time when vLLM reported it, and otherwise its weighted tokens: input tokens plus output tokens counted 4 times. The per-call costs add up to the GPU bill, so cost per customer and feature works as it does for API calls.
02Which model servers does it work with?
Any OpenAI-compatible server for the calls themselves: vLLM, SGLang, TGI or your own. GPU time per call currently comes from vLLM's OpenTelemetry request traces. Utilization comes from NVIDIA's DCGM exporter on any server, or from vLLM's running-request gauge.
03Is self-hosting cheaper than using an API?
It depends on how busy the GPUs are. A GPU costs the same per hour whether it serves one request or a thousand, so cost per million tokens falls as utilization rises. Ganivra shows your pool's utilization, its cost per million tokens now and at 80% utilization, and what the same tokens would cost on a hosted API.
04Which of our API features should we move to our own GPUs?
The move analysis prices each feature's API tokens from the last 30 days on your GPUs at 80% utilization. A pool's spare capacity is filled first, at no extra GPU cost. Features that would need new GPUs are checked with whole GPUs, so a feature that needs a tenth of a GPU is not counted as a saving. It compares cost only, not model quality.
05We own our GPUs. What hourly rate should we enter?
An amortized rate: purchase price divided by the hours over its useful life, plus power, hosting and support per hour. For example, a server bought for $300,000 and used for 3 years costs about $11.42 an hour before running costs, or $1.43 per GPU-hour across 8 GPUs.
06Do you see our prompts or responses?
No. Ganivra receives usage metadata only: model, tokens, timing, GPU time and the IDs you choose. vLLM traces and GPU metrics are read for timing and utilization, and nothing else from them is kept.
Start with one pool
See what your GPUs cost per customer.
Add a GPU pool, send your self-hosted calls, and see each one priced the same day. Add vLLM traces and GPU metrics when you want the split by GPU time and the idle cost. Full 14-day trial, no credit card, no sales call.
Python SDK: name your model servers once
import ganivra
ganivra.init(api_key="your_key", self_hosted_hosts=["vllm.internal"])Three steps
No credit cardFrom GPU bill to cost per customer
01Create a workspace and add your GPUs
02Send self-hosted calls with customer and feature
03Add vLLM traces and GPU metrics for GPU time and idle cost
Start free 14-day trial Not ready? Estimate GPU vs API cost · Integration guide