KTMS Architecture

KTMS Architecture An architecture diagram generated by Archify. Client Services · gRPC / OpenAI SDKs · Architecture component Client Services gRPC / OpenAI SDKs gRPC & HTTP Ingress · :9090 RPC / :8080 /v1 · KTMS Runtime & Stores · Bearer ktms_* gRPC & HTTP Ingress :9090 RPC / :8080 /v1 Bearer ktms_* Admin Service · Models, Rules, Keys · KTMS Runtime & Stores · admin scope Admin Service Models, Rules, Keys admin scope Router & Switchboard · Rule match & fallback · KTMS Runtime & Stores Router & Switchboard Rule match & fallback Redis Throttle · RPM & daily budgets · KTMS Runtime & Stores · fail-open Redis Throttle RPM & daily budgets fail-open PostgreSQL + Keyring · Config, keys, batches · KTMS Runtime & Stores · AES-256-GCM PostgreSQL + Keyring Config, keys, batches AES-256-GCM Batch Poller · Async jobs & retention · KTMS Runtime & Stores Batch Poller Async jobs & retention LLM Vendors · OpenAI / Anthropic / Gemini / DeepSeek / Ollama / TypeSafe · Architecture component LLM Vendors OpenAI / Anthropic / Gemini / DeepSeek / Ollama / TypeSafe light-tokenmeter · gRPC Usage Sink · Architecture component light-tokenmeter gRPC Usage Sink gRPC / HTTP Generate / Batch Admin RPC Validated tx RPM & budget Snapshot & batch Fallback chain Lease & complete Poll batch RecordUsageEvent KTMS Runtime & Stores Legend Backend Database Cloud Security External

Routing & Fallbacks

  • • Rules match on effort (LOW/MEDIUM/HIGH) and free-form attributes
  • • Ordered fallback chains retry across vendors before locking streams
  • • Supports text, JSON-schema, tools, streaming SSE/gRPC, 50%-off batches, TTS, image, and Judge

Limits & Persistence

  • • Redis enforces per-key RPM and per-key / per-user daily token budgets
  • • Postgres stores models, rules, batch state, and AES-256-GCM vendor keys
  • • Replicas reload snapshots periodically or immediately on local admin writes