Envoy Troubleshooting Scenarios

Scenario: "Bangalore": Envoy Panics

Level: Medium

Type: Fix

Access: Email

Description: You have an Envoy proxy routing traffic to two unhealthy backend services.

When the number of healthy backends falls under a panic threshold (default 50%), Envoy enters a panic mode and it will either send traffic to all upstream hosts or to none at all.

We are simulating this condition by returning an HTTP 503 status code from the /health endpoint in both backends. In our case Envoy is sending traffic to all upstream services.

Your objective is to change the panic Envoy behaviour so that it does not route any traffic to unhealthy services and instead Envoy returns "no healthy upstream". (Do not change anything in the backend services). There can also be other Envoy configuration issues you need to fix.

Test: Running curl localhost:10000 should return no healthy upstream.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.