Monitoring
Traces
A trace follows one request across every service it touched, so "the API is slow" becomes "the API waited 1.9 seconds on one query".
What a trace shows
Each trace is a tree of spans: the request, and everything it caused. You get where the time went, in order, with the failures marked.
The interesting part of a trace is often the space between spans. A 300 ms gap with no span under it is time nobody instrumented, and it's usually where the problem lives.
Service map
The service map is the aggregate view of the same data: which services call which, and where failures propagate.
Useful for the question a trace can't answer on its own, which is whether what you're looking at is one bad request or a pattern.
From a trace to an issue
Traces and issues are two views of the same production:
| Start here | To answer |
|---|---|
| Issue | "What is broken and for whom?" |
| Trace | "What was this request doing?" |
| Service map | "Is this contained, or spreading?" |
An error in a trace links to its issue, and an issue links to the traces behind it.
Traces and logs
Logs are queryable and jump straight to their trace, which is what makes them worth reading. A log line in isolation says something happened; the same line in a trace says what it was part of.
High-volume services don't keep every trace. Errors are what you want kept, and a sampled-away trace is why an issue occasionally has no trace behind it.