Skip to content

A department platform

This is the example behind the recording in the README. The answers are what an interview would collect; the tier and the sizing below were computed by tier.py and size.py when this page was generated.

Question Answer
org.users_total 180
org.departments 4
org.isolation per_department
org.audit_required true
org.downtime_tolerance planned_only
org.ops_capacity small_team
org.kubernetes_experience false
workload.usage {“chat”: 1.0}
workload.models [{“name”: “one very large chat model”, “role”: “chat”, “params_b”: 671, “precision”: “fp8”}]
hardware.gpus [{“model”: “unnamed 141 GB GPU”, “count”: 8, “vram_gb”: 141, “tdp_w”: 700}]
existing.identity ad_ldap
existing.api_gateway none
existing.kubernetes none
existing.registry none
transfer.channel removable_media
transfer.cadence unknown

Base tier: T2.

Layer Floor
llm-serving T2
llm-gateway T2
ui T2
identity T2
observability T2
orchestration T2
state T2
registry T2

Why:

  • org.audit_required sets the llm-gateway floor to at least T2 and requires a gateway
  • org.isolation = per_department sets the llm-gateway and identity floors to at least T2 and requires a gateway
Peak concurrency 9 to 18 requests in flight
GPUs 8
Servers 1
Power 10.1 kW
Fits the hardware yes

Sizing covers memory, not speed: the blueprint still asks for a benchmark before production.

One 671B 8-bit chat model needs all 8 GPUs for one replica (sizing: 738.1 GB required, 1015.2 GB usable, 277.1 GB left for KV cache), so the engine must split one model across GPUs; vLLM’s README lists tensor, pipeline, data, expert and context parallelism and continuous batching. Table row 4 applies (chat only, no code_assistant or rag share), so vllm is the default, not a measured winner over sglang. Concurrency is handled by continuous batching and the queue settings –max-num-seqs, –max-num-batched-tokens, –max-num-queued-reqs and –max-num-queued-tokens; sizing expects 9 to 18 requests in flight. The 0.90 GPU memory share is the sizing assumption, not an engine default (vLLM v0.34.0 documents 1.0 on CUDA and ROCm, 0.92 elsewhere). ISOLATION BETWEEN DEPARTMENTS: four departments share this one engine and one replica, and a dedicated vLLM instance per department is impossible (the model fills all 8 GPUs). vLLM’s security page says it does not isolate tenants that share a server process and that prefix caching shares cached prompt prefixes between callers. The design therefore switches prefix caching off (–no-enable-prefix-caching); its throughput cost is not documented and is measured in the benchmark. The other documented route, a gateway that scopes a secret cache_salt per department, is not used because the LiteLLM entry documents no per-team request parameters, and the page itself calls cache_salt a reduction of specific risks, not an isolation boundary. Chat is text only; image inputs are not offered, because media cache keys can alias between callers (GHSA-rh6f-3x46-j33q). The engine is reachable only from the gateway on an internal network: –api-key does not protect every endpoint. Not known: whether vLLM v0.34.0 supports this model’s architecture, and whether the GPUs are NVIDIA (decides the image).

Evidence: estimate.

Four departments need separate budgets and every request must be attributable and kept: table row 3 (audit true, isolation not none, 4 departments). LiteLLM documents teams with max_budget, budget_duration and rpm_limit, team keys that draw on the team budget, virtual keys, and per-request spend logs in PostgreSQL. Design: one LiteLLM team per department, one team key per Open WebUI instance. Open points that the catalog leaves open and the design depends on: (1) its documentation contradicts itself on whether teams, keys and budgets need a paid license, and audit logs are listed as a paid feature that is off without a license; confirm with the vendor which features run without a license and whether spend logs hold enough for the audit duty (the catalog does not say whether prompts or answers are stored). (2) If a license key is needed it must pass the local signature check before install, otherwise LiteLLM calls license.litellm.ai and the paid features switch off; the check is not stopped by offline mode. (3) Attribution to a person, not only to a department, needs the custom entry. One worker only, so valkey is not required by LiteLLM. SSO for the LiteLLM Admin UI is free up to 5 users; only the 3 operators use it. Offline: LITELLM_OFFLINE_MODE=true plus the switches in section 4 of the manifest.

Evidence: estimate.

Open WebUI is the only UI in the catalog. Four instances, one per department, each with its own data volume and its own LiteLLM team key: this keeps department chats and uploads in separate stores and gives each department its budget at the gateway using only documented LiteLLM features. Each instance runs in its single-instance default (embedded SQLite, one worker), so the multi-replica requirements (PostgreSQL, vector database, shared storage) do not apply. 180 users is above 50, so the Open WebUI name and logo stay as they are (license condition). OFFLINE_MODE=true and the telemetry switches in the manifest. Sign-in through Keycloak (OIDC).

Evidence: estimate.

Reused: Active Directory (existing), federated through Keycloak’s LDAP/AD provider.

Floor T2 (isolation per_department): Keycloak federated to the existing Active Directory; passwords are validated on the directory and never imported. One realm, AD groups per department mapped to Keycloak groups, one OIDC client per Open WebUI instance; each client admits only its department’s group (to be verified). ldaps needs a truststore on the Keycloak side; the production database is the shared PostgreSQL, because the default dev-file database is for development only. One Keycloak instance is enough for planned-only downtime.

Evidence: estimate.

T2 floor. Prometheus scrapes vLLM’s /metrics and the host; Grafana shows GPU, queue and latency panels. Prometheus storage is local and not replicated; metrics loss is acceptable and the logs that matter for audit live in LiteLLM’s PostgreSQL. Grafana open-source edition: its LDAP active sync and team sync need Enterprise, so the 3 operators use local accounts or generic OAuth through Keycloak. GPU hardware metrics need an exporter that is not in the catalog (vendor unknown); the vLLM metrics come first.

Evidence: estimate.

The team of three does not run Kubernetes (answer false), so at T2 the floor says docker-compose, not k3s. One server hosts everything, which is the case Compose’s production page calls the easiest. Compose reserves GPUs with deploy.resources.reservations.devices; its GPU page covers NVIDIA only, so an AMD or Intel GPU needs a different route and a blueprint change. Compose has no scheduler: a failed container is restarted by the engine’s restart policy, nothing moves to a second host.

Evidence: estimate.

T2 floor: postgresql and valkey. One PostgreSQL 18.7 instance with separate databases and roles for LiteLLM (keys, teams, budgets, spend logs) and Keycloak; this is the audit store, so it is backed up on a schedule and its retention period is an open question. Valkey holds Open WebUI session revocation and sign-in rate limits (the Open WebUI guide recommends a Redis-protocol store even for one instance) and would be needed by LiteLLM only if it ran several workers. No standby database: automatic failover needs tooling outside the catalog, and planned-only downtime does not demand it.

Evidence: estimate.

T2 floor: harbor or an existing registry; none exists. Images arrive on removable media, are loaded and pushed into Harbor, and the Compose files pull only from there. Harbor’s offline installer needs Docker Engine and Docker Compose on its host; Compose is in the orchestration decision. Harbor ships no certificates and a default admin password (Harbor12345): set both. Trivy scanning is optional; if used, its vulnerability database has to be carried in too, and its telemetry switched off. Harbor on the one production server is lost with it, so the carried-in archive on removable media stays the recovery copy. The lighter distribution-registry is T1 only; choosing it needs an override.

Evidence: estimate.

The four scripts run on the department example: base tier T2, eight GPUs in one server, the sections of the transfer manifest, and lint reporting no findings