LiteLLM
Python-based AI gateway (LiteLLM Proxy) that puts many LLM backends behind one OpenAI-compatible endpoint and adds virtual keys, teams, budgets and spend tracking.
| ID | litellm |
| Layer | llm-gateway |
| Version | v1.110.2 |
| Tiers | T1, T2, T3 |
| Requires | postgresql |
| License | MIT (the enterprise/ directory is under the BerriAI Enterprise license) |
| Evidence | docs |
| Last verified | 2026-11-22 |
Choose it when
Section titled “Choose it when”- Several backends (for example vLLM, SGLang or Ollama servers and external providers) must appear as one OpenAI-compatible endpoint with per-team keys and budgets: in config.yaml model_list, each entry has a model_name (the name clients send) and litellm_params (model, for example hosted_vllm/<model> for a vLLM OpenAI-compatible server, plus api_base and api_key). Several entries with the same model_name are load balanced by the built-in router; the default strategy is simple-shuffle, others are least-busy, usage-based-routing, latency-based-routing and cost-based-routing, and an order parameter prioritizes deployments (load balancing page and vLLM provider page).
- Teams and applications need their own keys, budgets and rate limits, and the license position described in the caveats is acceptable: the documented features are Virtual Keys (/key/generate), Teams (/team/new with max_budget, budget_duration and rpm_limit; team keys draw on the team budget), Team Members (max_budget_in_team, team_member_budget), Internal Users and per-key and per-team tpm_limit and rpm_limit (Budgets and Rate Limits, Virtual Keys and Setting Team Budgets pages). The Teams and budget pages read carry no license marker, but the Enterprise page lists teams and keys among license-enabled features (see caveats).
Avoid it when
Section titled “Avoid it when”- Single sign-on for more than 5 users, audit logs, team- or organization-scoped admin roles or the other features in the caveats are required and no paid license (or an offline-verifiable key) is available.
- No PostgreSQL can be run: without a database the proxy still routes requests, but virtual keys, teams, spend tracking and every budget are unavailable and budgets are not enforced (they fail open).
Caveats
Section titled “Caveats”- Paid license, from the Enterprise, Activate your license and feature pages: audit logs (with a license they are on by default; without one the proxy writes none), SSO for the Admin UI beyond 5 users (SSO is free for up to 5 users from v1.76.0), JWT-based authentication, key rotations, budget and rate limit tiers, model-specific budgets per key and per internal user, reserving TPM/RPM on keys by priority, tag-based budgets, project management, temporary budget increases, soft budget email alerts, team-based logging, team-managed models, IP address access control lists, secret managers, and the guardrail integrations llmguard_moderations, llamaguard_moderations, hide_secrets, openai_moderations, google_text_moderation, lakera_prompt_injection and aporia_prompt_injection (Presidio PII masking and custom guardrails are open source). Pages marked as free enterprise features, for example managed files and managed batches, need no license.
- License conditions: GitHub reports no SPDX license (NOASSERTION). The LICENSE file at the tag says the content under the enterprise/ directory is under the license in enterprise/LICENSE.md and everything else is MIT. enterprise/LICENSE.md says that the software may be used in production only if you have agreed to and comply with the BerriAI Subscription Terms of Service (or another agreement with BerriAI) and hold a valid BerriAI Enterprise license for the correct number of user seats. Subject to that, you may modify it and publish patches, but BerriAI and its licensors keep all rights in those modifications and patches, which may be used, copied, modified, displayed, distributed or otherwise exploited only with such a license. Notwithstanding this, you may copy and modify the software for development and testing without a subscription, and BerriAI keeps the rights in those modifications too. No other rights are granted, and, subject to the above, copying, merging, publishing, distributing, sublicensing and selling the software are forbidden. The enterprise/ directory ships in the repository; whether the container image includes it was not checked.
- Role-based access control: the Enterprise page lists it among the license-enabled features, while the access control page marks only the organization and team specific roles (org_admin, team_admin) as Premium and describes the global roles proxy_admin, proxy_admin_viewer and internal_user without a marker. The documentation does not draw the line more precisely; confirm with the vendor before relying on team admins without a license.
- Teams, keys and budgets: unresolved contradiction in the documentation, like the RBAC one. The Teams and team budget pages (users.md, team_budgets.md) and the Virtual Keys page (virtual_keys.md) carry no license marker for virtual keys, teams, team budgets and tpm and rpm limits, and the docs say every budget is enforced against spend read from the database. The Enterprise page (enterprise.md) says everything in its feature list is enabled by the license key and lists “Multi-tenant Architecture. Organizations, teams, projects, and keys” and “Tag-based Budgets” there, and the license marker appears on model-specific budgets, key rotation and rate limit tiers. The documentation does not reconcile the two; confirm with the vendor which team and key features run without a license before the design depends on them.
- License validation needs network access in one case: the gateway checks the LITELLM_LICENSE signature offline with a public key in the image, and a key that does not pass that check goes to the LiteLLM license server (the Activate page says “such as an older key”; from source at tag v1.110.2, litellm_license.py, any local verification failure goes there, an expired key included). From source at tag v1.110.2: the call is a GET to https://license.litellm.ai/verify_license/<key>, sent three times in a row (num_retries = 3 counts all attempts, and the loop does not stop after a successful answer, so a successful check also sends three requests); an HTTP error status on the first two attempts is passed over and the third response decides, while any other error, an unreachable server included, is not retried; any failure makes the check return false, so the paid features are switched off. Offline mode does not stop that check. The documentation does not say what happens when the server is unreachable; get a key that passes the offline check before an air-gapped deployment with paid features.
- State stores: PostgreSQL is required for virtual keys, teams, users, budgets and per-request spend logs (db_info page, listed in requires). Redis is documented as needed when more than one worker or gateway instance runs: without it rate limits, budgets, cooldowns and cache hits are counted per worker, so a limit of 100 requests per minute across 4 workers lets 400 through; budget checks read current spend from a cross-pod counter in Redis (the budgets page), which is the source of truth on the hot path, can come back stale after a Redis restart and let a key spend past its max_budget, and is cross-checked against the database only when it reads below the last-known recorded spend unless fail_closed_budget_enforcement is set; the Redis and Valkey cache page says Valkey, ElastiCache and Memorystore speak the Redis protocol so everything on that page applies to them, so valkey fits, but it is not in requires because a single worker works without it; add the valkey entry for any design with several workers or replicas.
- Offline mode (LITELLM_OFFLINE_MODE=true) stops catalog fetches but not the license check, the model info refresh, Hugging Face lookups or requests to destinations you configure; see airgap.notes. The pip route downloads Prisma and Node.js on first start.
- Usage telemetry is documented as off by default and, when switched on, as sending counts only to an endpoint you set; the outbound calls that are on by default are listed under telemetry. The documentation pages are the LiteLLM documentation repository at a commit just before this release, not a release tag; documentation of later versions (for example Customer Management from v1.111) is not part of this entry.
Alternatives
Section titled “Alternatives”Carrying it in
Section titled “Carrying it in”| Image | Hardware | Digest |
|---|---|---|
ghcr.io/berriai/litellm:v1.110.2 |
any | not pinned |
Downloads at run time
Section titled “Downloads at run time”What the software tries to fetch while it runs, and how to stage it instead.
- Model cost map (prices, context windows): fetched from raw.githubusercontent.com at startup, on /reload/model_cost_map and on the optional reload schedule; when the download fails the proxy uses the copy bundled with the installed version (frozen at release time: newer models are missing and price at 0). Use LITELLM_OFFLINE_MODE=true or LITELLM_LOCAL_MODEL_COST_MAP=true to skip the fetch, or mirror the file internally and set LITELLM_MODEL_COST_MAP_URL (HTTP(S), s3:// or gs://; read at startup, so it must be an environment variable). To update prices offline, carry the file model_prices_and_context_window.json from the LiteLLM repository.
- Other catalogs fetched by default and replaced by bundled copies in offline mode: Anthropic beta headers and auto router presets and policy templates (raw.githubusercontent.com), blog posts and latest-release information (docs.litellm.ai, api.github.com).
- Tokenizers: token counting for legacy models such as davinci, ada or gpt2 downloads a tokenizer from openaipublic.blob.core.windows.net unless the tiktoken files are placed in a folder named by CUSTOM_TIKTOKEN_CACHE_DIR (cl100k_base, o200k_base and p50k_base are bundled); token counting for Cohere, Llama 2 and Llama 3 models and for Bedrock Kimi K3 models uses Hugging Face tokenizers from huggingface.co unless litellm_settings.disable_hf_tokenizer_download is true.
- Hugging Face lookups with no documented switch: the context window of huggingface/ models (config.json), chat templates for providers that build a prompt from messages (SageMaker, vLLM, Replicate, Triton, watsonx, Bedrock invoke) and the task of huggingface/ embedding models. A custom prompt template for the model skips the chat template lookup; input_type in the request skips the embedding lookup.
- Model info refresh: for openai, openai_like, text-completion-openai and hosted_vllm deployments with an api_base, the proxy calls {api_base}/v1/models every 5 minutes (your own backend); general_settings.disable_model_info_refresh: true stops it.
- Image URLs in requests are downloaded by token counting and by some providers (Bedrock).
- Database tooling: on a pip install, the first start downloads the Prisma CLI and engines (registry.npmjs.org, binaries.prisma.sh) and Node.js (nodejs.org) when it is missing; the Docker images include them and set PRISMA_OFFLINE_MODE=true.
- Callback packages: the supabase, sentry and slack callbacks run pip install when their package is missing; install the package in the image instead.
- Admin UI and API docs: the Admin UI and the Swagger page serve their own scripts and fonts; the /redoc page loads from cdn.jsdelivr.net and fonts.googleapis.com in the browser (NO_REDOC=True turns it off), and the Admin UI loads MCP server icons from cdn.simpleicons.org and raw.githubusercontent.com (they do not show when blocked).
License server
Section titled “License server”Yes: some use of it contacts a license server. The notes below say when.
Telemetry and outbound calls
Section titled “Telemetry and outbound calls”| What | How to turn it off |
|---|---|
| Check for the latest LiteLLM release (docs.litellm.ai and api.github.com) when the Admin UI notification bell calls /get/latest_release_info; on by default. | LITELLM_DISABLE_LATEST_RELEASE_CHECK=true, or LITELLM_OFFLINE_MODE=true (returns null, no upgrade notice). |
| Breached-password check at api.pwnedpasswords.com when a user password is set or changed and on password login; sends only the first 5 characters of the password’s SHA-1 hash; on by default. | general_settings.password_policy_check_breached_passwords: false, or LITELLM_OFFLINE_MODE=true. |
| Fetches of the model cost map, Anthropic beta headers, auto router presets, policy templates and blog posts from raw.githubusercontent.com or docs.litellm.ai (catalog fetches, no usage data described); on by default. | LITELLM_LOCAL_MODEL_COST_MAP=true, LITELLM_LOCAL_ANTHROPIC_BETA_HEADERS=true, LITELLM_LOCAL_AUTOROUTER_PRESETS=true, LITELLM_LOCAL_POLICY_TEMPLATES=true and LITELLM_LOCAL_BLOG_POSTS=true one by one, or LITELLM_OFFLINE_MODE=true for all of them. |
| Prisma CLI update check at checkpoint.prisma.io, run by the Prisma CLI that applies database migrations; on by default. | CHECKPOINT_DISABLE=1 |
| License check: with LITELLM_LICENSE set, a key that fails the local signature check (the page says ‘such as an older key’; from source at tag v1.110.2, an expired key too) is checked against https://license.litellm.ai/verify_license/<key>. Offline mode does not stop it. No documented switch; without LITELLM_LICENSE the check is not described. | Not applicable: leave LITELLM_LICENSE unset (open-source features only) or use a key that passes the offline signature check. |
| Usage telemetry (counts of requests, providers and Admin UI pages; never prompts, keys, user or team ids): documented as off unless a group and a destination are chosen; the destination is local_table (own database) or http, and there is no default endpoint. | Nothing to disable by default; LITELLM_TELEMETRY_DISABLED=true forces it off whatever is saved in the Admin UI or API. |
Offline behavior is documented but not verified: the entry author did not run LiteLLM. The documentation has a page for it (Outbound requests and offline mode): set LITELLM_OFFLINE_MODE=true (only the value true, any letter case) as an environment variable on every proxy instance; it cannot be set in config.yaml. It stops the requests listed in runtime_downloads and telemetry that it covers, but not requests to destinations you configure (providers, logging callbacks, guardrail vendors, secret managers, SSO, email, alerting), not the model info refresh, not the license check, not the Hugging Face lookups and not Prisma and tokenizer downloads (see runtime_downloads). The one image row is the image the release notes of v1.110.2 name (ghcr.io/berriai/litellm:v1.110.2); the documentation names the same image as ghcr.io/berriai/litellm, mirrored at docker.litellm.ai/berriai/litellm, and says one Dockerfile builds the proxy, gateway, backend, UI, migrations, metrics and collector. Not listed: the release notes also name the images litellm-database and litellm-non_root (and docker.io/litellm/* copies) for signature verification, but no installation page read names them; docker.litellm.ai/berriai/litellm, the mirror the quickstart uses, with the same tag, is an alias of the same image; the quickstart compose file also names postgres:16 (the database; take it from the postgresql entry). Images are signed with cosign from v1.83.0 (Docker image security page); verify with the commands on that page against the key in cosign.pub. Alternative install routes, not carry items: the pip package litellm[proxy] (uv tool install ‘litellm[proxy]’ on the quickstart page; versions from 1.84.0 need the Python version stated there, and a bare pip install silently resolves an older release on an older Python); the Helm chart litellm at oci://ghcr.io/berriai/litellm/chart/litellm (it uses the image ghcr.io/berriai/litellm and expects PostgreSQL and Redis hosts) and the Terraform modules for AWS and GCP that the README at the tag names under terraform/litellm/aws and terraform/litellm/gcp; the documentation names no artifact for installing without internet access other than these and the images. The proxy listens on port 4000. Settings: config.yaml, environment variables and the Admin UI (stored in the database with STORE_MODEL_IN_DB). LITELLM_MASTER_KEY and LITELLM_SALT_KEY are required by the quickstart compose file; the salt key encrypts stored provider keys and cannot be rotated in place. DB-less operation (a config file, no database) is documented but gives no virtual keys, spend tracking or budgets. The license check: with LITELLM_LICENSE set, the gateway verifies the key signature offline with a public key shipped in the image, and a key that does not pass that check (the page says ‘such as an older key’; from source at tag v1.110.2, any local verification failure, an expired key included) is checked against license.litellm.ai; license_server is true for that case. Without LITELLM_LICENSE no license check is documented.
Deeper skills
Section titled “Deeper skills”For running the tool itself, these skills go further than the catalog:
litellmin ScaleDockHQ/llm-skillslitellm-apiin air-gapped/skillslitellm-valkeyin air-gapped/skills
Evidence
Section titled “Evidence”All values were read on 2026-11-21 from the pages in sources (enterprise/LICENSE.md and litellm/proxy/auth/litellm_license.py at tag v1.110.2 were re-read on 2026-11-22). Version v1.110.2 is what the GitHub API returns as the latest release (not a prerelease, published 2026-11-20); the project also publishes patch releases of older lines (v1.109.3, v1.108.14, v1.108.13, v1.106.12) and release candidates, and the container image tag v1.110.2 exists on ghcr.io (manifest checked, linux/amd64 and linux/arm64). The documentation of LiteLLM no longer lives in the main repository: it is in the repository BerriAI/litellm-docs, which has no tags, so the documentation pages are cited at the last commit before the v1.110.2 release (d6ae56ec, 2026-11-20T02:44Z) instead of at a release tag; the sentences that carry a ‘since v…’ marker in those pages were checked against that commit. The license: GitHub reports NOASSERTION; the LICENSE file at the tag says the enterprise/ directory is under the license in enterprise/LICENSE.md and everything else is MIT, and enterprise/LICENSE.md says that software may be used in production only with the BerriAI Subscription Terms of Service or an enterprise license, except for development and testing. The licensing answers rest on the documentation pages Enterprise, Activate your license, SSO, Audit logs, Access control and the feature pages listed in sources; where two pages disagree the caveats say so. The offline behavior (offline mode, bundled copies, license check) is documented but not verified: the entry author did not run LiteLLM.
Sources:
- https://github.com/BerriAI/litellm/releases/tag/v1.110.2
- https://github.com/BerriAI/litellm/blob/v1.110.2/LICENSE
- https://github.com/BerriAI/litellm/blob/v1.110.2/enterprise/LICENSE.md
- https://github.com/BerriAI/litellm/blob/v1.110.2/README.md
- https://github.com/BerriAI/litellm/blob/v1.110.2/litellm/proxy/auth/litellm_license.py
- https://github.com/BerriAI/litellm/blob/v1.110.2/docker/docker-compose.quickstart.yml
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/enterprise.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/enterprise/activate.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/data_security.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/outbound_requests.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/telemetry.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/custom_model_cost_map.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/docker_quick_start.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/docker_image_security.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/deploy/helm.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/quick_start.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/db_info.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/virtual_keys.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/users.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/team_budgets.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/access_control.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/audit_logs.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/admin_ui_sso.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/rate_limit_tiers.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/redis_requirements.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/caching_redis.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/load_balancing.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/providers/vllm.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/config_settings.md
- https://github.com/BerriAI/litellm-docs/blob/d6ae56ec9fa4b0c232b5cb88ccb25482bcdf3ee8/docs/proxy/prod.md
Source of this page: skills/airgap-architect/catalog/components/litellm.toml.