HAProxy troubleshooting

HAProxy won't start or reload

Run haproxy -c -f /etc/haproxy/haproxy.cfg for the exact error line. Check journalctl -u haproxy -n 50. Common causes: bind address already in use, invalid keyword, or duplicate backend/server names.

Also haproxy -vv is useful when troubleshooting: SSL support, Lua support, compiled features or version-specific behavior.

Reload succeeded but traffic still behaves strangely

haproxy -c and systemctl reload haproxy both succeed, but routing, backends, or timeouts still look like the old config. HAProxy reloads gracefully — old worker processes may still serve in-flight connections — or you may be hitting a different process or config than the one you edited.

Useful checks:

Common causes:

• Old workers still draining connections — reload starts new workers with the new config; existing sessions stay on old workers until they finish. Wait, or use show info / stats to confirm worker generation; heavy long-lived connections delay the switch
• Edited wrong config file — systemd may point at /etc/haproxy/haproxy.cfg while you changed a snippet elsewhere; compare with systemctl cat haproxy and the path in ps aux
• Runtime socket changes not persisted — disable server, weight tweaks, and other admin-socket commands affect the running process only; a reload restores whatever is on disk
• Multiple HAProxy instances — container vs host, old process still bound to the port, or a second config started manually; check ss -tlnp | grep haproxy and every PID from ps aux

Dump effective runtime state with show info and show servers state on the admin socket (see debugging workflow below). If on-disk config is correct but behavior is wrong, traffic may be bypassing this LB entirely — CDN, DNS, or another tier in front.

503 Service Unavailable

No healthy backend servers available. Check the stats page or show servers state — all servers may be DOWN (Note it requires a stats enable block in the config to show this information). This command requires the runtime socket, in practice most operators use: echo "show stat" | socat stdio /run/haproxy/admin.sock

Verify backends are listening, health check URL returns 200 (or what HAProxy is configured to expect with http-check expect status), and network/firewall allows HAProxy to reach them.

503 can also occur because:

• backend name typo
• ACL routes to a non-existent backend
• backend has zero servers configured
• all servers are in maintenance mode

All backends marked DOWN

Health check failing. Confirm option httpchk path and expected response. Test from the HAProxy host: curl -v http://10.0.1.10:8080/health. TCP-only backends need check without httpchk, or a valid port check. For more advanced validation, use in HAProxy's configuration option tcp-check.

Note that by default, option httpchk sends OPTIONS / HTTP/1.1 without a Host header, which many backends reject with 400 or 404. The explicit form is almost always needed in practice:
option httpchk OPTIONS / HTTP/1.1\r\nHost:\ example.com. In HAProxy 2.2+ the syntax changed to use a dedicated http-check send directive. The opposite problem — checks pass but real traffic fails — is covered under Health checks succeed but application still fails below.

Health checks succeed but application still fails

Very common — stats show all servers UP, the health check returns 200, but users still see 500s or broken features. HAProxy only probes the endpoint you configured; it does not validate the whole application.

Health checks prove the specific check URL works — often a lightweight stub that skips database, auth, or downstream services. Real traffic hitting /api/* or POST endpoints can still fail while /health stays green.

Fix the application or dependencies, not the LB config. Optionally tighten checks — hit a path that exercises DB connectivity, or use http-check expect on response body — but a passing check never guarantees every route works. Use access logs and backend logs to diagnose failing paths separately from probe success.

Wrong backend receiving traffic

ACL or use_backend rule order matters — first match wins. Verify default_backend is not catching requests meant for another pool. Test with curl -H "Host: api.example.com" http://lb/. If ACLs never match at all, check Layer 4 vs Layer 7 mode confusion below — HTTP ACLs are ignored in mode tcp.

Layer 4 vs Layer 7 mode confusion

ACLs not matching requests — many ACLs require mode http (Layer 7). In mode tcp (Layer 4), HAProxy forwards bytes without parsing HTTP — path, Host header, and cookie ACLs are unavailable and rules that use them silently never match.

Fix: set mode http on the frontend (and matching backend) when you need path, Host, or header routing. Use mode tcp only for raw TCP passthrough (TLS passthrough, database proxies, non-HTTP protocols).

option httpchk and option forwardfor also require HTTP mode. TCP health checks use option tcp-check instead. Mixed configs — HTTP ACLs on a TCP frontend — pass haproxy -c but route everything to default_backend.

Connection timeouts

Tune timeout connect, timeout server, and timeout client in defaults or per-section. Slow backends may need higher server timeout; idle clients need higher client timeout.

Note that timeout client applies to the client-side connection; timeout server applies to the backend. But timeout tunnel (default unlimited in many versions) governs WebSocket and CONNECT tunnels — forgetting to set it leaves long-lived connections open indefinitely. Also, timeouts set in defaults are inherited but can be overridden per-frontend or per-backend. See Log-based debugging below — the termination state field in access logs pinpoints which side timed out or closed first.

Address already in use

Another process holds the bind port — nginx, another HAProxy instance, or a stale process. Find it: ss -tlnp | grep ':80 '. Remove duplicate bind lines in the config.

Backend sees wrong client IP

Without forwarding (in HTTP mode), backends see HAProxy's IP. Add to the backend or defaults (if you are using TCP mode, use send-proxy instead):

TLS / HTTPS issues

For TLS termination on the frontend:

Log-based debugging

HAProxy access logs are highly configurable and often the fastest way to see which backend and server handled a request, how long each phase took, whether retries occurred, and why a connection ended. By default logs go to syslog (journald on systemd hosts), not a file under /var/log/haproxy/ unless you configure one.

Minimum logging setup — declare syslog in global, inherit in defaults, and use HTTP log format for L7 frontends:

Default httplog lines include frontend, backend, and server names, status code, bytes, and timing fields:

• Tq — time waiting in queue (throttling / maxconn)
• Tw — time waiting for a free server slot
• Tc — time to connect to the backend
• Tr — backend response time (request sent → first byte)
• Tt — total session time

The termination state field (two characters before timings in the default format) is essential for timeout and connection-drop issues:

• -- — normal, clean termination
• cD — client timeout during data (slow upload/download on client side)
• sD — server timeout during data (backend too slow to respond or stream)
• SC — server closed during connection (refused, reset, or died before response)
• CD — client closed during connection (browser tab closed, aborted request)

Reproduce the failure, then grep access logs for the request path or client IP. Compare Tc vs Tr to separate connect problems from slow backends; use termination state to tell client timeout from server reset. Runtime errors also land in the same syslog stream — pair with show errors on the admin socket (see debugging workflow below).

Debugging workflow

1. Validate config

2. Check backend health and runtime state

show stat lists per-server UP/DOWN and counters.
show info is also extremely useful for runtime troubleshooting — version, uptime, process state, current/max connections, and whether the process hit resource limits.

3. Test backend directly from LB host

4. Access logs while reproducing