Case study

NGINX 502 errors from one load balancer: how a segfault caused persistent upstream failures

A client reported an unusual production problem: requests routed through one load balancer were intermittently returning 502 Bad Gateway, while the other nodes in the same cluster worked normally.

The backend services themselves appeared healthy.

Health checks passed. Direct requests from the affected host to the upstream servers worked. Network connectivity looked normal.

Yet NGINX on one node continued to return 502 responses.

The immediate fix turned out to be simple:

`1`	`systemctl restart nginx`

The important part was understanding why a restart fixed it, and making sure the same failure could not remain unnoticed again.

The symptoms

The infrastructure looked roughly like this:

  flowchart LR
    C[Clients] --> LB1[NGINX node 1]
    C --> LB2[NGINX node 2]
    C --> LB3[NGINX node 3]

    LB1 --> APP[Backend pool]
    LB2 --> APP
    LB3 --> APP

    LB1 -. "502 errors" .-> C

Only one load balancer was affected.

NGINX logs on that node contained upstream timeout errors, suggesting a backend or network problem.

That initially made the failure look straightforward:

NGINX
  ↓
upstream timeout
  ↓
backend/network problem?

But the evidence did not support that conclusion.

From the affected host, connections to the same backend servers succeeded:

`1`	`curl -v http://backend.example.internal/health`

Health checks were passing as well.

If the upstream application or network path had actually been unavailable, we would expect failures from more than a single NGINX node.

That discrepancy became the most useful diagnostic signal.

What actually happened

Kernel logs revealed a different event around the time the problem started:

`1`	`nginx[24317]: segfault at 8 ip 00000000004602b1 sp 00007ffe0a8183f0 error 4`

The crash happened while NGINX log rotation was being performed.

After the segfault, the affected NGINX instance remained in a degraded state. Connections accumulated in its listening/backlog path instead of being processed normally.

From outside, this produced a misleading combination of symptoms:

the host was reachable;
the backend servers were reachable;
backend health checks succeeded;
NGINX itself was still present;
but requests through this particular load balancer timed out and returned 502.

Restarting NGINX recreated its runtime state and immediately restored normal request processing.

  sequenceDiagram
    participant C as Client
    participant N as Affected NGINX
    participant B as Backend

    C->>N: HTTP request
    N--xB: Request processing stalls
    N-->>C: 502 / upstream timeout

    Note over N: NGINX had previously segfaulted

    B->>B: Backend remains healthy

    Note over N: systemctl restart nginx

    C->>N: HTTP request
    N->>B: Forward request
    B-->>N: 200 OK
    N-->>C: 200 OK

The restart solved the incident.

It did not solve the operational weakness that allowed the incident to continue unnoticed.

Why health checks were not enough

Traditional health monitoring was checking whether:

NGINX was running;
the server was reachable;
upstream services responded;
application health endpoints returned success.

All of those checks could pass during this incident.

What was missing was monitoring of the actual traffic outcome.

A load balancer returning an abnormal rate of 5xx responses is unhealthy from the user’s perspective even if its process exists and every backend health check is green.

This required a second layer of monitoring.

Alert on abnormal NGINX upstream 5xx rates

The environment already exposed NGINX VTS metrics to Prometheus.

A useful signal is:

sum(
  rate(
    nginx_vts_upstream_requests_total{
      code="5xx"
    }[5m]
  )
) by (instance, upstream)

Instead of looking only at the cluster-wide total, keeping instance in the aggregation is important.

In this incident, a global metric could have hidden the failure because only one load balancer was broken.

For example:

1
2
3

node-1    0.1 5xx/s
node-2   27.4 5xx/s
node-3    0.0 5xx/s

The cluster as a whole might still appear mostly functional, while every client hitting node-2 experiences errors.

A Prometheus alert can detect this directly:

additionalPrometheusRules:
  - name: nginx
    groups:
      - name: nginx.rules
        rules:
          - alert: NginxUpstream5xxRateHigh
            expr: |
              sum(
                rate(
                  nginx_vts_upstream_requests_total{
                    code="5xx"
                  }[5m]
                )
              ) by (instance, upstream) > 20
            for: 5m
            labels:
              severity: warning
            annotations:
              summary: "High NGINX upstream 5xx rate on {{ $labels.instance }}"
              description: >-
                NGINX on {{ $labels.instance }} is returning more than
                20 upstream 5xx responses per second for upstream
                {{ $labels.upstream }}.

The exact threshold depends on normal traffic volume, but the principle is more important than the number:

Alert per load-balancer instance, not only on the aggregated service.

This would have identified the failing node quickly even without knowing anything about the segfault.

Detect the root cause from Linux kernel logs

The second monitoring layer targets the failure itself.

Linux records application segmentation faults in kernel logs, commonly visible through dmesg, journalctl -k, /var/log/messages, or /var/log/syslog, depending on the distribution.

For example:

`1`	`grep -E 'nginx.*segfault' /var/log/messages`

If Monit is already installed, it can watch the kernel log and react to an NGINX segfault.

1
2

check file kernel_messages with path /var/log/messages
    if content = "nginx\[[0-9]+\]: segfault at [0-9a-f]+" then exec "/usr/bin/systemctl restart nginx"

Validate the configuration and reload Monit:

`1`	`monit -t && monit reload`

This provides two things that were missing during the original incident:

an explicit alert that NGINX crashed;
automatic recovery of the affected node.

The service is no longer allowed to remain indefinitely in the partially broken state we observed.

Two signals are better than one

The final monitoring design intentionally detects the incident from two different directions.

  flowchart TD
    S[NGINX failure]

    S --> U[User-visible symptom]
    S --> K[Kernel-level symptom]

    U --> P[Prometheus<br/>NGINX 5xx rate]
    K --> M[Monit<br/>segfault detection]

    P --> A[Alertmanager alert]
    M --> A
    M --> R[Restart NGINX]

    R --> OK[Traffic restored]

The Prometheus rule answers:

Is this load balancer actually serving traffic correctly?

The Monit rule answers:

Did the NGINX process crash in a way that requires immediate recovery?

These are complementary checks.

Monitoring only the segfault would miss other causes of elevated 502 responses.

Monitoring only 5xx rates would detect the problem but would not explain or automatically recover from this particular failure.

Why the original diagnostics were misleading

The most confusing part of the incident was that NGINX reported upstream timeouts while direct upstream connectivity worked.

It would have been easy to spend significantly more time investigating:

firewall rules;
connection tracking;
backend saturation;
DNS;
routing;
TCP timeouts;
application health.

Those were reasonable things to check, but the strongest clue was that the problem followed one load-balancer node rather than the backend service.

When only one node of an otherwise identical cluster behaves differently, inspect local state early:

journalctl -k
journalctl -u nginx
dmesg
ss -s
ss -lnt
ps auxf

A running process does not necessarily mean a correctly functioning service.

Result

Restarting NGINX immediately restored traffic on the affected node.

The permanent improvement was not the restart itself.

We added monitoring at both levels:

Prometheus and Alertmanager detect an abnormal upstream 5xx rate independently for every NGINX node;
Monit detects NGINX segmentation faults in kernel logs, sends an alert, and automatically restarts the service.

The same class of failure should therefore be detected within minutes and, for the known segfault scenario, recovered automatically.

The broader lesson applies well beyond NGINX:

Infrastructure health checks should measure both whether a component is alive and whether it is still producing the expected result.

A process can be running. Its dependencies can be healthy. The network can work perfectly.

And the service can still be broken.

Get a Quote Back to services