'404 Bad Gateway' Is a Legitimate Traefik Response (And Here's Why)

A 404 with an 11-byte body that reads Bad Gateway is not a 404. It's not a 502. It's a routing middleware silently replacing your response body with the string its fallback logic returns when its own upstream fails.

I didn't realize any of that until I ran curl -i on a path that should have returned a normal EmDash 404 page and got this:

HTTP/2 404
content-length: 11
date: Sun, 19 Apr 2026 16:09:00 GMT

Bad Gateway

Fifteen seconds of confusion. Then another ninety minutes of tracing before I understood what was actually happening.

The setup

Calegix runs EmDash (Astro SSR) behind Traefik for the public site. The apex route at calegix.com is handled by a Traefik router with priority 150 and this middleware chain:

emdash-apex-public:
  rule: 'Host(`calegix.com`)'
  entryPoints: [websecure]
  middlewares: [calegix-errors@file, public-perf-sec@file]
  service: emdash-host@file
  priority: 150

The calegix-errors@file middleware is a Traefik errors middleware. Its job is to intercept any 4xx/5xx response from the backend, fetch a replacement body from a dedicated error-pages service, and return that pretty error page to the user.

calegix-errors:
  errors:
    status: ['403', '404', '500-599']
    service: error-pages@file
    query: '/{status}'

That service definition:

error-pages:
  loadBalancer:
    passHostHeader: false
    servers:
      - url: 'http://127.0.0.1:8484'

So: EmDash returns a 404 for /about (because /about doesn't exist, it's /pages/about). Traefik intercepts the 404. Traefik fetches http://127.0.0.1:8484/404 to use as the body. If that fetch fails, Traefik emits its own fallback, which has content-length 11 and reads Bad Gateway.

The diagnosis

I spent a while assuming EmDash was returning the Bad Gateway text itself. It wasn't. Compare two routes:

curl -sS -i https://calegix.com/about | head
# HTTP/2 404
# content-length: 11
# (empty headers)
#
# Bad Gateway

curl -sS -i https://calegix.com/pages/about | head
# HTTP/2 200
# content-security-policy: default-src 'self'; ...
# content-type: text/html
# x-content-type-options: nosniff
# (many more headers)

The 404 response has no security headers. No CSP, no HSTS, no cross-origin policy. The 200 response has the full Traefik-added security envelope. That's the tell: the 404 bypassed the normal response pipeline. Something was replacing it wholesale, very early in the chain.

Next: check what error-pages@file actually resolves to in the live Traefik:

curl -sS http://127.0.0.1:8080/api/http/services/error-pages@file | jq
# {
#   "loadBalancer": {
#     "servers": [{"url": "http://127.0.0.1:8484"}]
#   },
#   "serverStatus": {
#     "http://127.0.0.1:8484": "UP"
#   }
# }

UP. Everything green. So why does the body look wrong?

ss -tlnp | grep 8484
# (nothing)

curl -sS http://127.0.0.1:8484/
# curl: (7) Failed to connect to 127.0.0.1 port 8484: Connection refused

Nothing is listening. Traefik's dashboard says UP because the passive health check only marks things down after N failed requests, and nothing has requested this in a while. The port died - probably a container restart or a service that used to be there but got decommissioned.

The fix - three layers

Layer 1 - stop the bleeding: either stand the error-pages service back up, or remove calegix-errors from emdash-apex-public. EmDash has its own 404.astro that returns proper HTML; the errors middleware is actively making things worse for EmDash-owned routes.

emdash-apex-public:
  # calegix-errors removed; EmDash returns its own 404 page.
  middlewares: [public-perf-sec@file]
  service: emdash-host@file

Layer 2 - fix the actual missing route. For /about specifically, the expected route is /pages/about because EmDash serves pages at /pages/<slug>. I added an Astro-level redirect:

// site/astro.config.mjs
export default defineConfig({
  redirects: {
    '/about': '/pages/about',
  },
  ...
});

Rebuild emdash, redeploy. curl /about now returns HTTP 301 to /pages/about. Clean.

Layer 3 - add passive health checks that actually care:

error-pages:
  loadBalancer:
    servers:
      - url: "http://127.0.0.1:8484"
    passiveHealthCheck:
      failureWindow: "15s"
      maxFailedAttempts: "1"   # go DOWN on first failure

Even better - add a healthCheck (active probe) so the dashboard reflects reality without waiting for traffic to expose the failure.

The generalizable rule

Any middleware that replaces response bodies is a hidden dependency. If the replacer breaks, everything downstream of its insertion point looks subtly broken - but not broken enough to wake up a pager. Your response bodies silently get smaller. Your error pages silently lose their nice styling. Your 404s turn into Bad Gateway.

Two rules fall out of this:

1. If a middleware can rewrite response bodies, it needs its own health check that's independent of whether anyone's hitting it.

2. If a middleware is optional (like errors middleware - most apps render their own error pages), default to not installing it. Only wire it up where the benefit is clearly worth the new dependency.

Ops notes

curl -i on a dead route is the fastest smoke test for this class of bug. Compare content-length of your 404 to a known-good route. If the 404 body is measurably smaller and missing headers, something is replacing it.

Traefik's serverStatus in /api/http/services/... can lie when traffic is absent. For middleware targets, poll them actively.

Astro's redirects: config is a clean way to handle legacy URL shortcuts without reintroducing a rewrite middleware.

Related posts

• The 499 in Your Iframe
• Reversibility Is the MVP
• Self-Sovereign Analytics with Matomo