./blog/keycloak-oauth2-proxy-ingress-debug
Nginx Ingress + oauth2-proxy + Keycloak is three well-documented pieces that still manage to break in ways none of their docs individually cover.
Debugging Nginx Ingress, oauth2-proxy, and Keycloak SSO in Kubernetes
Notes from putting real SSO in front of an internal cluster
Fronting internal tools with SSO via Ingress-Nginx's auth_request support, oauth2-proxy,
and Keycloak as the OIDC provider is a well-trodden pattern — until it isn't. Every failure
mode below looked, from the browser, exactly the same: a login page, a spinner, and either a
redirect loop or a 500. The differences only show up once you're reading oauth2-proxy logs
and comparing hostnames byte for byte. This is the checklist I wish I'd had before the first
SSO rollout.
Almost every SSO failure in this stack traces back to one of four root causes: the issuer URL Keycloak advertises doesn't match the one oauth2-proxy was configured with; the session cookie is larger than nginx's 4KB header buffer; oauth2-proxy replicas don't share a cookie secret so the CSRF cookie set by one pod isn't readable by another; or the Ingress
auth-url/auth-signin
annotations point somewhere that doesn't match the hostname the browser is actually on. None
of these show up as a clean error — they all just look like "login doesn't work."
the-flow --what-should-happen
Ingress-Nginx doesn't do authentication itself — it delegates. Every request to a protected
path triggers an internal auth_request subrequest to oauth2-proxy first. If
that subrequest returns 200, the real request proceeds with oauth2-proxy's response headers
merged in. If it returns 401, Nginx redirects the browser to oauth2-proxy's sign-in
endpoint, which redirects again to Keycloak, which redirects back to oauth2-proxy's
callback with an authorization code, which gets exchanged for tokens and set as a session
cookie — and only then does the browser finally land back on the page it originally asked
for.
failures --four-root-causes
oauth2-proxy validates that the iss claim in Keycloak's tokens matches the OIDC issuer URL it discovered. Point oauth2-proxy at Keycloak's internal Service DNS name for reliability, and the issuer Keycloak advertises (set via KC_HOSTNAME, typically the public URL) won't match — token validation fails even though authentication itself succeeded. The fix is making them agree: either set KC_HOSTNAME to the internal name and never expose that Keycloak URL directly to browsers, or route oauth2-proxy through the same external hostname Keycloak is configured with, certificate CN included.
OIDC tokens — especially with a few scopes and group claims attached — routinely exceed 4KB. Nginx's default buffers choke on that, and the failure shows up as a generic 400 Bad Request: Request Header Or Cookie Too Large, or a successful login that silently loses part of the session on the next request because only the first Set-Cookie header survives the auth_request subrequest. Either raise proxy_buffer_size on the Ingress, or move oauth2-proxy to --session-store-type=redis so only a small session ID rides in the cookie instead of the full token set.
oauth2-proxy sets a short-lived CSRF cookie before redirecting to Keycloak, then checks for it on the callback. With more than one oauth2-proxy replica and no shared, stable --cookie-secret, the pod that handles the callback may not be the one that set the cookie — or can't decrypt what a different pod encrypted. Logs show a clean authentication followed immediately by unable to obtain CSRF cookie and a redirect to a 404. The fix is a shared cookie secret across all replicas (from a Secret, not per-pod generation) plus session affinity on the Service if you're not already centralizing sessions in Redis.
The nginx.ingress.kubernetes.io/auth-url and auth-signin annotations tell Nginx where to send the internal check and the external redirect respectively — and they need different values. Point auth-signin at an internal-only address, or let the rd= redirect parameter resolve to a hostname the browser can't reach, and you get the classic "redirected you too many times" loop instead of a login page.
debug-order --what-to-check-first
Work top to bottom — each check rules out one of the four causes above before you spend time on the next.
1. Diff the issuer
Curl Keycloak's /.well-known/openid-configuration from where oauth2-proxy actually runs, and compare issuer byte-for-byte against oauth2-proxy's configured OIDC issuer URL.
2. Measure the cookie
After login, check the actual Set-Cookie size in the browser's network tab. If it's near or over 4KB, that's your 400 before it even happens in prod traffic.
3. Confirm cookie-secret parity
Check that every oauth2-proxy replica was started with the identical --cookie-secret value from a shared Secret, not a value generated per-pod at startup.
4. Walk the annotations
Read auth-url, auth-signin, and auth-response-headers off the live Ingress object and confirm every hostname in them is one the browser can actually resolve and reach.