Monitoring
Performance
Performance shows latency distributions per endpoint, so a slow tail is visible instead of hidden behind an average.
Why distributions, not averages
If 95% of requests take 40 ms and 5% take 3 seconds, the average is about 190 ms and describes no actual request. Meanwhile one user in twenty is watching a spinner.
Averages hide exactly the requests you need to find. Read the percentiles.
| Percentile | What it tells you |
|---|---|
| p50 | The typical request. Good for trend lines, bad for complaints |
| p95 | Where the pain starts. One request in twenty is at least this slow |
| p99 | Your worst regular experience. On a busy endpoint this is thousands of requests a day |
Reading it
- Sort by the tail, not the mean
The endpoint with the worst p99 is where your users are suffering, even if its p50 looks fine.
- Weigh it by traffic
A slow endpoint nobody calls is a curiosity. A slightly-slow endpoint on your hot path is the actual problem.
- Open a trace at that percentile
Aggregates tell you where to look. A trace tells you why.
Latency you cause vs latency you observe
The numbers here are what your service did. A client on a bad connection experiences more, and no server-side percentile will show you that.
If your SDK retries a failed request, the caller waits for every attempt plus backoff, while your server sees several fast failures. A maxAttempts of 5 with no maxBackoffMs can leave a user waiting far longer than any percentile here suggests.
Watching a deploy
Compare a percentile before and after a release to catch a regression that didn't error, just got slower. See Releases.