One Front Door for All Your AI: The Case for a Gateway
One Front Door for All Your AI: The Case for a Gateway
Organisations have one huge issue when it comes to the adoption of AI in their Platform, it comes first with no standardisation of how LLMs are being deployed, secondly it's what access to models being deployed, thirdly it's how they secure and lock down these LLMs... you're starting to see the picture here.
Ask around your organisation for access to an AI model and you will, more often than not, be handed an API key. It arrives in a Teams message, lands in a pipeline variable, gets copied into a Key Vault, and quietly multiplies — one key per team, per environment, per proof of concept that never got decommissioned.
Everything works, nobody complains — right up until an access review or a cost query lands, and it turns out the estate cannot say which applications can reach which models, or what any given team spent on them last month. Not because anyone was careless. Because nothing in the solution was ever built to know.
I have been building this solution out for a public sector client and wanted to build the whole solution out myself prior to it and also to share the lessons learnt from a real Production POV.
The repo is here (okaneconnor/terraform-azurerm-ai-gateway) You can also find this officially published on the Terraform Registry here also (Terraform Module registry).
This first post is going to be part of a multiple series of posts which aims to cover the why, the what and the how of an AI Gateway and traditionally how organisations currently manage this - compared to how they should handle this.
What an AI Gateway Actually Is
An AI gateway is not a new product you go and buy. It is an ordinary API gateway — Azure API Management, in this case — taught to understand LLM traffic. Microsoft ships the teaching materials as a set of AI gateway capabilities in APIM: policies that count tokens rather than requests, screen prompts for harmful content, cache completions by meaning rather than by exact match, authenticate to backends with managed identity, and emit per-consumer token metrics.
The architectural idea fits in one sentence: every AI call in the organisation goes through one door, and the door decides. Clients never see a model endpoint or a key. The gateway authenticates the caller, works out what they are entitled to, enforces it, screens the traffic, forwards the request under its own identity, and writes down exactly what was consumed and by whom.
Once that door exists, a whole class of organisational problems stops being a policy document and becomes policy configuration — enforced on every request, whether the calling team read the wiki or not.
The Unit of Account Changed
Traditional API management thinks in requests, and for a long time that was fine — one request was roughly one unit of load and one unit of cost. AI traffic broke that quietly and completely. A ten-word question and a two-hundred-page document stuffed into a context window are identical at the HTTP layer: one POST each. One of them costs a fraction of a penny — the other, several orders of magnitude more. Any control plane that counts requests is measuring the wrong thing.
It gets worse, because the thing being consumed is genuinely scarce. Azure hands you model capacity as tokens-per-minute quota on a deployment — a fixed, regional pool that every consumer shares. Without something in the middle enforcing fairness, the first team to ship a chatty agent eats the pool and everyone else starts seeing 429s with no idea why. A request-counting gateway cannot even see this happening, let alone stop it.
So the unit is different, the resource is contended, and the bill arrives as one undifferentiated number. That is the whole case for the gateway, really — everything else is implementation.
The Billing Model Is an Architecture Decision
There is one more piece of economics to put on the table before the architecture makes sense: how Azure sells you model capacity in the first place, because the billing model you choose quietly becomes an architecture decision.
Provisioned Throughput Units (PTU) are reserved capacity. You pay a fixed hourly rate for guaranteed throughput on a model, whether you consume it or not. The upside is predictable bills and predictable latency. The downside is waste when utilisation is low — and a hard ceiling when you burst above what you reserved.
Pay-as-you-go (Standard and GlobalStandard SKUs) is the opposite trade. You pay per token, with no commitment and no reservation. Costs scale linearly with usage — which is wonderful right up until a runaway agent makes "linearly" the scariest word in the sentence. This is what the deployment in this post runs: a GlobalStandard deployment carrying 130k tokens-a-minute of capacity, a ceiling the platform itself enforces with its own 429s.
Mature estates usually end up running both — PTU for the predictable baseline, because reserved capacity is cheaper when it is actually used, and pay-as-you-go as the overflow layer for bursts. And the moment you have two backends with different economics, something has to decide which request goes where: a load-balanced backend pool with a circuit breaker, sitting exactly where the gateway already is.
The same control plane extends past Azure OpenAI too — APIM's llm-* policy family speaks to Mistral, Cohere, and other providers — though this setup fronts the Azure OpenAI surface only today. The spillover pattern is the one piece this deployment does not implement yet — the pool currently has a single member — and it gets its own post later in the series.
The Residency Trap: Where Inference Actually Runs
One more thing the SKU quietly decides: where your prompts are actually processed. Residency has two layers — where data rests, and where inference runs — and they are easy to conflate right up until a compliance team refuses to. GlobalStandard keeps data at rest in your region but runs inference on global capacity. The DataZone SKUs route within a US or EU data zone — and the EU one excludes the UK. If the requirement is full UK processing residency, the menu collapses to two options: regional Standard, or regional provisioned throughput.
On the engagement this work came from, that menu turned out to be shorter than it looks. The GPT-4 generation is deprecated for new deployments — the apply fails with ServiceModelDeprecating — and no current-generation chat model is offered on UK South regional Standard at all. UK-contained chat today means regional PTU (ProvisionedManaged), with per-model minimums of 25–50 PTUs. And then the sting in the tail: quota is not capacity.
The subscription held 250 PTUs of quota — plenty — while the region had roughly 23 PTUs of GPU capacity actually available, two short of the smallest model's minimum. No quota increase fixes that. The levers left are per-model capacity retries or a capacity commitment through your Microsoft account team. (The deployment in this post runs GlobalStandard knowingly — a demo has no residency requirement. A department handling citizen data does.)
The lesson: model, SKU, residency posture, and physical regional capacity are one decision, not four. Verify all of them before anyone promises "fully UK-resident AI" — and expect the answer to be a funding conversation (regional PTU) or a formally agreed relaxation to data-at-rest residency (GlobalStandard), because those are the only two doors out.
The Setup: One Door, No Keys
Here is the actual shape of the gateway I run — and the design decision that shapes all of it is that there are no API keys anywhere in the request path. Not on the backends, and not even APIM subscription keys on the front door.
-
Inbound, identity is Entra ID. A calling app requests a client-credentials token and presents a JWT. The gateway validates the audience and tenant, then reads the token's
rolesclaim — and that claim is the consumption tier. Sandbox, NonProd, Production, whatever environments you define: each one is an Entra app role on the gateway's app registration. Onboarding a team is an app-role assignment — offboarding is deleting it. Clients still hold their own Entra credentials, but there is no shared gateway key to rotate and nothing to pass around in a Teams message. -
Every limit keys on the caller's app ID. The token's
azpclaim — the verified client application — keys the rate limits, the token-per-minute limits, the longer-period token quotas, and the cache partition. Attribution is not inferred from which copy of a shared key was used — it is written into every log line as a validated identity. -
The policy chain does the governing. Every request walks the same gauntlet, cheapest checks first. Identity comes before anything else — the IP filter and JWT validation turn away anything that shouldn't be here before it costs a penny. Fairness comes next: the caller's tier sets its request rate, its tokens-per-minute, and its longer-period quota. Then safety — every prompt is screened by content safety before the semantic cache is consulted (the cache itself is opt-in), so a cached answer is never a way around screening. Only after all of that does the request reach the circuit-broken backend. One discovery worth sharing: APIM products — the traditional tiering mechanism — never fire for keyless consumers, because product-scope policies only execute when a subscription key is presented. The tier logic lives in API-scope policy instead, keyed on the token's
rolesclaim. -
Outbound, identity is the gateway's own. APIM swaps the caller's token for its managed identity and calls Azure AI Foundry — or Speech, Language, Content Safety, Document Intelligence, all behind the same door. The backends have key authentication disabled outright and no public network access. The only route to a model is through the gateway, over a private endpoint, as the gateway.
-
Governance is enforced, not documented. An Azure Policy allowlist pins which model-deployment SKUs are permitted — a data-residency guardrail that fails closed against whatever SKU Azure invents next, and catches deployments made around the gateway via the portal or CLI.
All of it — APIM VNet-injected, the private endpoints and DNS zones, the Entra apps and roles, the policies, the monitoring — is Terraform. No portal clicking, no manual steps.
Where This Series Goes Next
Each post that follows takes one capability apart — the problem, the policy mechanics, and what it looked like live:
- Keyless by design — Entra ID end to end: JWT validation, app roles as tiers, managed identity to the backend
- Carving up the token pool — rate limits, tokens-per-minute, and quotas across many teams
- Guardrails at the gate — content safety and Prompt Shield on prompts and completions
- Answering twice, paying once — semantic caching with Azure Managed Redis
- Staying up — backend pools, circuit breakers, and zone-redundant Premium
- Who spent what — token metrics, chargeback, and the observability stack
- Moving in — landing-zone adoption with bring-your-own everything
The Takeaway
AI did not just add another API to your estate. It changed the unit of account from requests to tokens, made capacity a contended resource, turned prompts into an attack surface, and made per-team spend invisible by default. Handing out model keys and hoping is not a strategy — it is the absence of one.
One door in front of everything turns the problem into configuration: identity instead of keys, fair shares instead of noisy neighbours, screening instead of trust, and a ledger instead of a mystery bill. The whole setup is open source and deploys with one apply — okaneconnor/terraform-azurerm-ai-gateway — and you can have your own front door standing by this time tomorrow. Thirty-odd of those minutes — budget up to forty-five — will be APIM provisioning. Bring a coffee.