Keyless by Design: Authentication and Authorization for an AI Gateway
Keyless by Design: Authentication and Authorisation for an AI Gateway
This is the second post in the series. The first one made the case for putting one door in front of every model you run, and closed on a design decision I glossed over quickly: there are no API keys anywhere in the request path — not on the backends, and not even APIM subscription keys on the front door. This post is where we can go and dig a little bit deeper into how the authentication side of things work.
So before any of the clever token-accounting and caching that the rest of this series is about, the gateway has to answer three questions on every single request, in order:
- Who is calling? — that is authentication.
- What are they allowed to do? — this is authorisation.
- How does the gateway prove itself to the model? — backend authentication.
The standard way to teach APIM security answers those three with three layers, and the first layer is almost always a subscription key. This is where our setup diverges from the textbook on the very first page — so that is where we will start.
Subscription Keys vs OAuth
A subscription key is APIM's standard credential. The caller puts it in an Ocp-Apim-Subscription-Key header, APIM checks it against a product subscription, and on that basis it will let the request through and attribute it to a consumer.
Here is the problem with just that: a subscription key answers "who is calling?" only in the weakest possible sense. It proves the caller holds a copy of a shared secret. It does not prove which caller, because keys get copied — that is the whole failure mode from the first post. It carries no tenant, no expiry, no signature, and no notion of what the holder is entitled to. It identifies but it does not authorise this is important to note.
For an AI gateway, where the thing behind the door is expensive and contended, "someone has a copy of the key" is not a strong enough answer to who is calling. So we removed the layer entirely. There is no product, no subscription, and no front-door key. Every question about identity is answered by Entra ID instead — which turns out to answer all three questions better than three separate mechanisms did.
Subscription Keys are still a valid solution without a doubt, each team can have their own Subscription Keys, Products and Backend etc and each team will be assigned their own. I however have decided to go against the grain, and for good reason which I want to show and tell you.
Microsoft's own Well-Architected Framework for API Management recommends exactly this. The Azure security baseline for API Management says the same thing.
Layer 1: Authentication — Entra ID at the Door
The first thing every request meets, after the network boundary, is a single policy: validate-azure-ad-token. This is the whole of authentication, and it is doing considerably more than a Subscription Key check ever could.
A calling application authenticates to Entra ID first — the client-credentials flow, app-to-app, no user in the loop — and receives a JWT. It presents that token to the gateway as a normal Bearer credential. The gateway then validates, on every request, that the token is from our tenant and issued for our audience.
I want you to notice what changed. There is no shared secret in the request path — the client holds its own Entra credentials and mints its own short-lived tokens, so there is nothing for you as the client to rotate. Just pure simple keyless/passwordless authentication.
The moment the token validates, the gateway reads the azp claim — the verified client application ID turns into a variable and treats that as the single source of truth for who is calling. Every downstream policy in the chain reads that variable rather than re-parsing the header, which matters for a reason we will come back to.
Layer 2: Authorisation — App Roles as Consumption Tiers
So far, we know authentication establishes the who.
Authorisation is the separate and more interesting question of what they are allowed to do — and this is where the keyless model shines, because the answer is carried inside the token that the caller already presented.
For the AI gateway we use an app registration, this defines one application app role per consumption tier.
When a caller's token arrives, the same validate-azure-ad-token policy inspects its roles claim and requires a match against any of the recognised tier roles. A token with no recognised app role gets a 401.
This is claims-based authorisation: the token says who you are and what you belong to in simple terms, the token is signed from an authority which the gateway trusts, and the gateway enforces the band. It also produced the single most surprising lesson of the build. APIM's traditional tiering mechanism — products — never fires for a keyless caller at all, because product-scope policies only execute when a subscription key is presented.
With the key gone, the entire tier apparatus has to live in API-scope policy, keyed on the token's roles claim. If you go keyless and keep reaching for products, you will wonder why none of your tier limits apply.
If we imagine we have loads of development teams on a Platform, everyone wants to get onboarded into the AI Gateway. So how do we do this?
The way we onboard a team in a real project is simple: the team brings their own managed identity or service principal, and we assign it the app role for its tier on the gateway's app registration. That one assignment is the onboarding — from here their identity appears in the roles claim of every token they present, and the gateway admits them at that tier. Off boarding is deleting the assignment. Nothing else moves.
As a side note, I will be also blogging how teams and Platforms can onboard themselves into the AI Gateway at a later day.
Layer 3: Backend Authentication — Managed Identity, No Secret
The caller is authenticated and authorised. Now the gateway has to call the model — and this is the third question.
The answer is a managed identity — which, conveniently, is also Microsoft's recommended way to authenticate an AI gateway to its models — and it is the cleanest layer of the three.
Once the caller is authorised and their limits are applied, the gateway asks Azure for a short-lived token for its own system-assigned identity, scoped to https://cognitiveservices.azure.com for example, and swaps that token into the Authorization header in place of the caller's.
From the model's point of view, the request arrives as the gateway, proven by a token just issued — there is no secret anywhere in this exchange. Nothing is stored and nothing rotates.
On the model side, that identity is granted exactly the Azure RBAC role it needs and no more. For Azure AI Foundry inference the gateway's identity holds Cognitive Services OpenAI User — the least-privilege role for calling deployments.
For the other services behind the same door — Speech, Language, Document Intelligence, Content Safety — it holds Cognitive Services User on each. And the backends themselves have key authentication disabled outright and no public network access. The only route to a model is through the gateway, over a private endpoint, as the gateway's own identity with the app role. That is the backend half of Zero Trust: the model trusts one caller, and refuses everyone else by construction. It's simple... right?
One quick clarification, because it trips people up: And by people I mean meant me for the lonmgest time!
The app role from Layer 2 and this role are not the same thing. The app role authorises the client calling the gateway — it rides inside the token's roles claim. This is an Azure RBAC role on the model resource, and it authorises the gateway's managed identity calling the model — it never appears in a token.
Two directions, two identities, two different role systems: the client never gets Azure RBAC on the model, and the gateway's identity never gets an app role.
The Perimeter: Keyless Is Not the Same as Open
It is worth being precise about one thing, because "no keys" can be misunderstood as no boundary. The three layers above are identity. Underneath them sits our network perimeter.
The gateway is VNet-injected — We have two different ways which this can work.
- External mode for a public front door gated by everything above with policy checks etc.
- Internal mode for a fully private ingress you can front with services such as Application Gateway and Azure Frontdoor.
In front of the JWT check, an IP allow-list turns away anything outside your expected egress ranges before the token is even examined — the cheapest possible rejection, first in the chain. The backends sit behind private endpoints with private DNS, with public network access switched off.
Order Matters: The Shape of the Policy Chain
Layers are a useful way to think about the design, but on the wire it is one ordered chain, and the order is load-bearing. Cheapest checks first, identity before spend, and a couple of orderings that exist for reasons you only discover by getting them wrong along the way. Luckily I have walked this path to show you.
The caller's JWT is validated once, up front, and the caller's app ID is stashed in a variable at that moment — not re-read from the header later. That is not an optimisation; it is a correctness requirement. By the time the request reaches the backend, the managed-identity step has overwritten the Authorization header with the gateway's own token, which carries no azp claim. Any policy that tried to re-derive the caller's identity from the header downstream would be reading the gateway's identity, not the client's.
There is a second ordering with the same flavour. The content-safety check and the model call both ride on the Authorization header, so the managed-identity swap has to happen in a specific relationship to them, and content safety has to run before the semantic cache is consulted — otherwise a cached answer would be a way to skip screening entirely. Identity, fairness, safety, backend: the sequence is the security posture. Shuffle it and you have the same policies enforcing a weaker guarantee.
Here is that chain in the order it actually executes, straight out of the policy file. Each stage carries the status code it returns when it refuses, so every red line is a request that never reached a model and never cost a token:
Stage four is the ordering that catches people out, including me: the managed identity is injected halfway through rather than at the end.
Putting the Three Layers Together
Follow one request all the way through, and every identity it passes through on the way (follow the numbers: the caller never holds anything but its own Entra credentials):
In words: A production application mints a client-credentials token from Entra ID and calls the gateway. The IP filter checks it came from an expected range. validate-azure-ad-token confirms the token is ours — right tenant, right audience, unexpired, properly signed — and reads its roles claim: this caller holds the Production tier role, so it is authorised, and its verified app ID is captured for everything that follows. The tier's rate and token limits are applied against that identity. The gateway exchanges the caller's token for its own managed-identity token, the prompt is screened by content safety, and it calls Azure AI Foundry over a private endpoint as itself, and the model — which trusts nothing else — answers. Not one key was involved at any step, and every log line names a cryptographically validated caller.
That is the architecture you want in place before you go to production, not the hardening pass you promise to do afterwards. Authentication that proves identity instead of possession, authorisation that lives where your access reviews already look, and a backend that trusts exactly one caller by construction.
What's Next in This Series
With identity settled, the next post takes apart the problem the gateway was really built to solve: carving one contended pool of model capacity fairly across many teams — the rate limits, the tokens-per-minute ceilings, and the longer-period quotas, all keyed on the validated identity this post established.
- Keyless by design (you are here) — Entra ID end to end: JWT validation, app roles as tiers, managed identity to the backend
- Carving up the token pool — rate limits, tokens-per-minute, and quotas across many teams
- Guardrails at the gate — content safety and Prompt Shield on prompts and completions
- Answering twice, paying once — semantic caching with Azure Managed Redis
- Staying up — backend pools, circuit breakers, and zone-redundant Premium
- Who spent what — token metrics, chargeback, and the observability stack
- Moving in — landing-zone adoption with bring-your-own everything
The Takeaway
The default way to secure a model endpoint is a key, and a key answers the wrong question — it proves possession, not identity, and possession is exactly the thing that leaks. Trading it for Entra ID collapses three separate problems into one primitive: authentication becomes a signed token from your tenant, authorization becomes an app role your access reviews already govern, and backend trust becomes a managed identity with a least-privilege role and no secret to lose.
Keyless is not less security. It is the same three layers with the weakest link — the shared secret — removed from all of them. The whole setup is open source and deploys with one apply: okaneconnor/terraform-azurerm-ai-gateway.