Skip to content
Install Core free
ProductsCompare Core, Hub, Connect and MDM pricing
Solutions
Docs
ResearchPricing
CompareCompare all AI data-leak tools
TrustBenchmarks
Start 30-day free trial

Shadow Hub for your business

Tell us about your team and we'll help you get started.

We use these details only to reply to your enquiry. No newsletter, no cookies, no trackers.

What is an LLM proxy? How it works and when you need one

An LLM proxy sits between your apps and AI providers to route, log, limit or protect requests. The types, how they differ, and how to pick one.

Shadow Research TeamUpdated 8 min read

An LLM proxy is a service that sits between an application and one or more large language model APIs (OpenAI, Anthropic, Google and others). Every request passes through it, so it can route traffic between providers, enforce keys and rate limits, log usage and cost, cache responses, or inspect and change the content, for example masking personal data before it reaches the model. Apps usually use it by changing one setting: the API base URL or the HTTPS proxy.

Why do teams put a proxy in front of LLMs?

  • One endpoint for many providers: switch models without changing application code.
  • Keys and access control: apps get a proxy key; the real provider keys stay in one place.
  • Cost and usage tracking: per team, per app, per model.
  • Reliability: retries, fallbacks and load balancing across providers.
  • Caching: reuse answers to identical requests.
  • Privacy and security: detect personal data, secrets and confidential terms, then block, redact or mask them before the request leaves.
  • Audit: a record of which tool sent what kind of data where.

What kinds of LLM proxy are there?

TypeWhere it runsHow apps connectBest for
LLM gateway (reverse proxy)A server or managed serviceChange the API base URLRouting, keys, cost, fallbacks across a whole platform
Forward proxyOn the device or network edgeHTTPS_PROXY settingCovering tools you cannot reconfigure, such as CLIs and desktop apps
Privacy proxyUsually on the deviceBase URL or HTTPS_PROXYKeeping sensitive data from ever reaching the provider

Many products combine these. A gateway can add guardrails; a privacy proxy can also offer a gateway route.

How does an LLM proxy work?

Using a privacy proxy as the example (Shadow’s request lifecycle):

  1. Intercept: the app sends its request to the proxy, either to a local gateway route such as /gw/openai/v1/chat/completions or through HTTPS_PROXY.
  2. Inspect: the proxy parses the JSON body and scans each text field with detectors for emails, phone numbers, card numbers, IBANs, secrets, names, addresses and custom terms.
  3. Transform: sensitive values are replaced, for example with format-preserving fakes, so the model still gets a coherent prompt.
  4. Forward: the masked request goes to the real provider over TLS.
  5. Restore: the response, including streamed tokens and tool calls, is translated back so the user and the app see the real values.

How is an LLM proxy different from an API gateway or AI firewall?

  • An API gateway handles generic HTTP concerns (auth, rate limits) and does not understand prompts or tokens.
  • An LLM gateway understands model APIs: providers, models, token cost, streaming.
  • An AI firewall or guardrail focuses on blocking: prompt injection, unsafe output, data leakage.
  • A masking proxy focuses on letting the request through safely: it changes the sensitive parts and restores them in the answer, instead of only blocking.

Should an LLM proxy be self-hosted or managed?

A managed gateway is quick to set up, but every prompt passes through one more third party. A self-hosted or on-device proxy keeps plaintext on your infrastructure. For privacy, the closer to the user the masking happens, the fewer systems ever see the real data.

How do you choose an LLM proxy?

  1. Which problem comes first: routing and cost, or data protection?
  2. Which tools must it cover? SDK-based apps can change a base URL; CLIs, IDEs and chat apps often need a forward proxy.
  3. Does it handle streaming and tool calls, or only simple completions?
  4. Where does plaintext live, and who can read the logs?
  5. Does it run locally, with no account and no telemetry, so your security team knows where data goes?
  6. Can it sit next to an existing gateway (for example LiteLLM) instead of replacing it?

Shadow: a local privacy proxy for LLMs

Shadow runs locally on 127.0.0.1:8877, written in Rust. It works as a gateway (change the base URL) or as a forward proxy (set HTTPS_PROXY), so it covers SDKs, coding agents such as Claude Code, Codex CLI and GitHub Copilot, and ChatGPT, Claude.ai and Gemini in the browser. Tools with their own binary protocol, such as Cursor, are recognised but not masked. It masks personal data and secrets with reversible fakes and restores them in responses, streams and tool calls included. For hosts that are not AI providers it only tunnels bytes and never sees the content (proxy mode).

On macOS or Linux (Windows and other options):

Terminal window
curl -fsSL https://hub.getshadow.dev/core/install.sh | sh

Running it across a team? Shadow Hub shows what was masked or blocked per user and tool. Hub is $3 per user per month, with a 30-day free trial and no card required. See Pricing.

Try Shadow freeAll research

Frequently asked questions

What is the difference between an LLM proxy and an LLM gateway?

"Gateway" usually means a central server that apps call by base URL, focused on routing, keys and cost. "Proxy" is broader and also covers forward proxies that intercept traffic on the device.

Does an LLM proxy add latency?

Some. A local proxy adds a network hop on loopback plus inspection time; a remote gateway adds a round trip to its server. Measure it with your own prompts.

Can an LLM proxy work with streaming responses?

Good ones do. Shadow restores values inside streamed responses and tool calls.

Is LiteLLM an LLM proxy?

Yes, LiteLLM offers a proxy server that gives many providers one OpenAI-style endpoint. A privacy proxy like Shadow can run alongside it.