Open source · Apache 2.0 · Written in Go

One switchboard for every LLM you run

Keystone. The Model Switchboard. (KTMS) is a self-hosted LLM gateway: it keeps your vendor keys, routes each request by effort, falls back when a provider fails and caps spend. light-tokenmeter shows who burns the tokens. ktms-admin puts it all in your browser.

Get started in 1 minute Read the docs

No enterprise edition gRPC + OpenAI-compatible HTTP amd64 & arm64 images

~/your-service

How it fits together

One endpoint in. Every model out.

Your services keep a single address and a scoped ktms_… key. Everything behind it can change without a deploy.

Your servicesgRPC clients, typed Protobuf

OpenAI SDKs & agentsbase_url → :8080/v1

KTMS

Postgres for config & encrypted keys · Redis for limits

vendor keyseffort rulesfallbacksRPM & budgetsbatchesnode pool

Cloud vendorsOpenAI · Anthropic · Gemini · DeepSeek

Your GPUsOllama · vLLM · llama.cpp

↓ one usage event per call

light-tokenmeterwho spends, on what, and is it worth it

ktms-adminmodels, rules, keys, playground, spend

One key per service

Stop scattering vendor keys.
Hand out one per service.

Every service gets a single ktms_… key. Vendor credentials stay encrypted inside Keystone, and the admin decides which vendors, models and limits each key gets, without touching the service.

Without Keystonekeys everywhere

billing/.envOPENAI_API_KEY=sk-proj-…
ANTHROPIC_API_KEY=sk-ant-…

search/.envOPENAI_API_KEY=sk-proj-…
GEMINI_API_KEY=AIza…

support-bot/.envANTHROPIC_API_KEY=sk-ant-…
DEEPSEEK_API_KEY=sk-…

Which service holds which key? Who can call the expensive model? A rotation means a redeploy everywhere.

With Keystoneone key each

billing/.envKTMS_API_KEY=ktms_…

search/.envKTMS_API_KEY=ktms_…

support-bot/.envKTMS_API_KEY=ktms_…

keyvendorsmodels · limits

billingopenaigpt-mini · 120 rpm

searchgoogle, localgemini-flash · 200k tpm

support-botanthropicclaude · any

Set in ktms-admin or over the Admin API. Rotate a vendor key once, revoke a service key in one call.

~7 MiBidle memory

~0.6 msadded latency, p50

≥4,000RPS on 4 vCPU

~170 msfrom start to first answer

KTMS, measured in October 2026 on GCP c3-standard-4 against a mock upstream. Compiled proxies such as agentgateway, APISIX and Kong add even less latency; KTMS plays in their league and adds the features. Method and full numbers →

Products

Three small services. One production stack.

Use them together or one at a time. Each is a single Go binary with its own Docker image, Helm-friendly defaults and an Apache 2.0 licence.

Keystone. The Model Switchboard. (KTMS)

LLM gateway · gRPC & HTTP

Your services talk to one endpoint. Keystone holds the vendor keys, picks the model, retries elsewhere when a vendor is down and records every call.

Vendor keys encrypted in Postgres (AES-256-GCM): OpenAI, Anthropic, Gemini, DeepSeek
Routing rules by effort and attributes, changed live
RPM, daily token budgets and USD spend caps per key, user and team
Batches at 50% off, speech, images, typed Judge
Pool of your own GPU nodes with cloud fallback

Docs → GitHub Docker Hub

light-tokenmeter

Usage meter · Postgres / ClickHouse

Your apps, or KTMS, push one event per model call. Ask who is spending your tokens, and whether they are worth it.

Append-only, idempotent events: sending twice is harmless
Bearer tokens with write and read scopes, Prometheus metrics
Daily summaries with any filters and groupings
HTTP/JSON for dashboards, gRPC for services
Optional ClickHouse for large volumes, with no lost events

Docs → GitHub Docker Hub

ktms-admin

Web console · one static binary

Manage models, routing rules and keys, try a prompt in the playground and read the spend dashboard. No more grpcurl.

Password, OpenID Connect, GitHub or access-proxy sign-in, read-only users
Greedy / worthy verdict per project, user and model
Works with Keystone, light-tokenmeter or both
UI embedded in the Go binary, no front-end build
Light and dark themes, usable from a phone

Docs → GitHub Docker Hub

Why Keystone

Stop hardcoding model names.
Let the gateway decide.

Ask for effort, not for a model

Clients send effort: LOW | MEDIUM | HIGH and free-form attributes. An ordered rule list maps them to a chain of models with fallbacks.

Rules change at runtime over the admin API or in ktms-admin. A new model release is a config change, not a refactor across a dozen services.

Routing ruleslive

LOW

local · gemma→gemini-flash

MEDIUM

gpt-mini→deepseek-chat→gemini-flash

HIGH

claude→gpt→gemini-pro

HIGH

iftenant=eu→gemini-pro→local · llama

Your GPUs first, the cloud when they are busy

Run Ollama, vLLM or llama.cpp on your own hardware. ktms-node-agent registers each machine and reports VRAM, time to first token and tokens per second.

KTMS balances the pool with Smooth WRR, keeps hot or overloaded nodes out of rotation and, with nodes_first, spills over to cloud vendors only when local capacity runs out.

Node poolplacement: nodes_first

gpu-01vLLM · 48 GB

HEALTHY

gpu-02Ollama · 24 GB

HEALTHY

gpu-03llama.cpp · 24 GB

SUSPECT

cloudOpenAI · Gemini

overflow

Budgets that actually stop the spend

Per-model requests-per-minute and daily token budgets for every API key and every end user, plus hard USD spend caps per key, user and team, enforced in Redis before the vendor ever sees the request.

Non-interactive work goes through the vendors' Batch APIs at a 50% discount, with background polling and a synchronous fallback.

Daily token budgetstoday

key · support-bot1.2M / 3M

key · nightly-reports4.1M / 5M

user · u-4821200k / 200k

u-4821 → ResourceExhausted: daily token budget reached

Know who is greedy and who is worthy

KTMS sends one usage event per call to light-tokenmeter. Slice spend by project, user, action, vendor or model, with failure rates and latency next to the cost.

ktms-admin turns it into a verdict: the workloads that burn tokens for little value, and the ones that earn them.

Spend by actionlast 7 days

answer-ticketsupport-bot

8.4M tok

worthy

summarize-threadinbox

2.1M tok

worthy

retry-loopagent-sandbox

19.7M tok

greedy

rerank-allsearch

11.3M tok

greedy

Quick start

From `git clone` to a routed request

Start the stack

Docker Compose brings up KTMS with Postgres and Redis, seeds default models and rules and a dev key.

Point your SDK at it

Change base_url and use model: "auto". Routing, fallbacks and limits come for free.

Go to production

Docker images for amd64 and arm64, a Helm chart and a production checklist are in the KTMS docs.

git clone https://github.com/ai-process/ktms.git
cd ktms

# any of OPENAI_API_KEY, GEMINI_API_KEY, ANTHROPIC_API_KEY
OPENAI_API_KEY=sk-... docker compose up -d --build

curl -s http://localhost:8080/readyz   # {"status":"ready"}

curl -s http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer ktms_dev_default_key" \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"Say hi in three words."}]}'

# response headers tell you who answered and why:
# X-KTMS-Vendor, X-KTMS-Rule, X-KTMS-Placement

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="ktms_dev_default_key")
response = client.chat.completions.create(
    model="auto",  # or "auto:low", "auto:high"
    messages=[{"role": "user", "content": "Say hi in three words."}],
)
print(response.choices[0].message.content)

Honest open source

Everything is Apache 2.0

No enterprise edition, no paywalled security features, no licence keys and no telemetry. Fork it and run it forever.

Small on purpose

Behaves like a proxy

Static Go binaries and small images that start in milliseconds, fit scale-to-zero and leave the memory to your models.

Typed contracts

gRPC first

Protobuf APIs with streaming and reflection for services, plus an OpenAI-compatible HTTP API for everything else.

Documentation

Read it, run it, ship it

Guides for every service, from a single Docker command to a GPU pool in production.

Notes from running LLMs in production

Architecture patterns, benchmarks with the method attached, and what we learn while building KTMS.

Read the blog →

Put one switchboard
in front of all your models

Free, self-hosted and Apache-2.0-licensed. Run it next to your current setup and compare.

Star KTMS on GitHub Read the docs

One switchboard for every LLM you run

One endpoint in. Every model out.

Stop scattering vendor keys.
Hand out one per service.

Three small services. One production stack.

Keystone. The Model Switchboard. (KTMS)

light-tokenmeter

ktms-admin

Stop hardcoding model names.
Let the gateway decide.

Ask for effort, not for a model

Your GPUs first, the cloud when they are busy

Budgets that actually stop the spend

Know who is greedy and who is worthy

From `git clone` to a routed request

Start the stack

Point your SDK at it

Go to production

Everything is Apache 2.0

Behaves like a proxy

gRPC first

Read it, run it, ship it

Architecture & deployment

Keystone (KTMS) gateway

Local GPU node pool

Quick start: local node

light-tokenmeter

ktms-admin console

Command-line tools

MCP servers

Skills & plugin

Notes from running LLMs in production

Put one switchboard
in front of all your models

One switchboard for every LLM you run

One endpoint in. Every model out.

Stop scattering vendor keys.Hand out one per service.

Three small services. One production stack.

Keystone. The Model Switchboard. (KTMS)

light-tokenmeter

ktms-admin

Stop hardcoding model names.Let the gateway decide.

Ask for effort, not for a model

Your GPUs first, the cloud when they are busy

Budgets that actually stop the spend

Know who is greedy and who is worthy

From git clone to a routed request

Start the stack

Point your SDK at it

Go to production

Everything is Apache 2.0

Behaves like a proxy

gRPC first

Read it, run it, ship it

Architecture & deployment

Keystone (KTMS) gateway

Local GPU node pool

Quick start: local node

light-tokenmeter

ktms-admin console

Command-line tools

MCP servers

Skills & plugin

Notes from running LLMs in production

Put one switchboardin front of all your models

Stop scattering vendor keys.
Hand out one per service.

Stop hardcoding model names.
Let the gateway decide.

From `git clone` to a routed request

Put one switchboard
in front of all your models