Gateway failure-rate and latency anomalies

Admins can now see when calls through the Enterprise LLM Gateway start failing or a model gets slower. Two new anomaly detectors, AD-017 (failure rate) and AD-018 (latency regression), compare each window with your own organization's recent history and raise a finding in the Signals page when the change is large.

What you get

AD-017, proxy failure rate. For each hour or day, it measures the share of gateway calls that failed. It scores the whole organization, calls with no operation label yet, and each operation title on its own. A stream that breaks after the first byte counts as a failure. Calls with an unknown status are never counted as failures; if too many are unknown, a separate capture-health finding says so.

AD-018, proxy latency regression. Once a day, it compares the 95th-percentile and median latency of successful, non-streaming calls for each model against that model's baseline. A finding needs both numbers to rise, so one slow request cannot trigger it.

Each finding links to up to 20 affected sessions: the failing ones for AD-017, the slowest for AD-018.

How it decides

• A window needs at least 20 calls to be scored, and AD-017 needs at least 5 failures.
• The baseline is the previous 14 windows, weighted toward recent ones. With fewer than 7 windows, AD-017 falls back to a fixed rule (20% failures); AD-018 does not fire until it has 7 days of history, because normal latency differs so much between models.
• Hourly failure findings appear only after two consecutive hours are abnormal. This delays real outages by up to an hour but removes most one-off spikes.
• Only an organization-wide failure rate of 50% or more, or five times the baseline, sends an immediate email. Everything else goes into the digest.

What changed in the gateway

Until now the gateway did not store calls that failed at the provider, so any failure rate would have been wrong. It now records them with the status it returned, how long they took, and a failure class (HTTP error, timeout, connection error, or interrupted stream). Streams that fail after the response has started are recorded as status 200 with an interrupted-stream class, because that is what the client saw.

Each request gets its ID when it arrives, so a record that is retried from the local fallback queue updates the same row instead of creating a second one.

Limitations

• This covers the Enterprise Gateway only. Calls through the local proxy are not scored.
• Thresholds are fixed in code. We have not added per-organization tuning.
• Streaming latency is not evaluated yet, because total stream time mostly reflects output length. We plan to measure time to first token first.
• Calls that fail before the gateway can identify your organization (for example, a bad gateway key) are not recorded.
• Providers' own rate-limit and out-of-credit responses count as failures. If the gateway later adds its own limits, those will be excluded.
• Models with very variable latency, such as small local models, may produce noisy AD-018 findings until they have a longer history.
• We have tested the scoring on synthetic data and read real traffic data to design it. Real failures in production will show where the thresholds are wrong.

Feedback

If a finding is noisy, or an outage did not produce one, send us the window and the organization's rough traffic volume. Those two facts are enough to tell whether the thresholds or the baseline are at fault.