A customer uploads a document and waits for the checked version. Support can see that the file was accepted. The API team reports a successful response, workers show no errors, and infrastructure has spare capacity. Yet the document is missing. The company needs a fairly simple answer: where did the work stop, and who else is affected?
If answering that question requires every developer to join a call and hear the story again, the system has an observability problem. Another dashboard will not necessarily solve it. The team needs to follow an operation, test a suspected cause and choose an action that helps the customer.
Define the work that must actually finish
Here, a successful HTTP response only confirms receipt of a file. Validation, processing and delivery of the result happen later. Each component can report local success while the handoff between two components is broken.
Measure document progress as well as API calls: accepted work, completed work, unfinished age and the stage where progress stopped. For asynchronous processing, the age of outstanding operations is especially useful. A quick “accepted” response says little about when the customer receives a result.
Define legitimate outcomes too. A document rejected for a clear validation reason is different from a processing failure, but the user needs to receive either outcome. Work that disappears must not look healthy simply because nobody recorded an error.
Different signals answer different questions
Metrics show scope: whether processing is slowing, how much work is unfinished and which versions or stages differ from the rest. A platform-wide average may hide a small affected group. Useful dimensions and latency distributions help expose cases that an average conceals.
Logs explain individual events: processing started, validation finished, a dependency rejected a request. Consistent field names and links to the operation make those entries useful. “Processing error”, without context, often means another conversation with whoever wrote the message.
Traces connect instrumented steps across components. They show where a particular execution waited, but do not replace an aggregate quality measure. One captured failure does not tell us how many customers experienced it.
These are not three mandatory purchases. A small application may need only good metrics and correlated logs. Nor is “monitoring detects; observability explains” a clean boundary: monitoring can support diagnosis too. What matters is which questions the evidence can answer, not the label on the product.
Connect evidence without confusing identities
A stable document-operation ID can connect acceptance, queueing, processing attempts and the outcome. A trace ID helps within a particular execution; it need not be the business ID. Another attempt may create a new trace while still belonging to the same operation.
Service versions, configuration changes and feature variants help compare behaviour around a release. Timing alone does not prove causation. The document mix or a supplier's behaviour may have changed at the same time. Diagnosis should help test those alternatives rather than automatically blame the last deployment.
Context must survive queues as well as direct calls. Otherwise the history stops exactly where the customer stopped waiting for a synchronous response. The OpenTelemetry context propagation guide describes how signals can be connected and why trust boundaries matter.
Not every graph deserves to wake someone
An alert is useful when it calls for an action. A sudden increase in document completion time may need immediate attention. Gradual storage growth may need a planned task. An isolated technical spike without customer impact may only need to remain available as evidence.
An urgent alert needs an owner, the affected workflow and a way to establish the extent of the problem. Dozens of notifications for one failure rarely make the situation clearer. Google SRE's monitoring guidance emphasises meaningful symptoms and actionable responses.
Missing data needs attention too. When collection fails, an error chart may look unusually quiet. Zero failures and unknown state must remain distinct. An independent check of the customer journey can also detect failures that occur before requests reach the application.
Detail has a cost and an access policy
Do not put every document identifier into the labels of every metric. In a time-series system, new label combinations create additional series with storage and processing costs. Individual investigations are usually better served by suitable logs or traces with controlled access. The Prometheus naming guidance explains this high-cardinality problem.
Logging the uploaded file “just in case” creates another copy of customer information. Diagnosis usually needs the stage, failure category and approved technical attributes, not the whole document. Retention, access restrictions and secret removal must cover telemetry too. The OpenTelemetry sensitive-data guidance is a useful reference.
Collection itself can slow down or fail. Exporting diagnostics should not create an unlimited queue inside the application. Buffers need limits, and dropped telemetry needs visibility. A required business audit record is a separate obligation; a sampled trace is not a quiet substitute for it.
Test the diagnostic path with one support case
Take a deliberately stuck test document and ask an on-call engineer who did not build the processor to investigate. Where did it stop? Which customers are affected? What is established, and what is still a hypothesis? How will the team know whether an intervention worked?
That exercise tends to produce a useful list of gaps: a missing queue identifier, unknown task age or no way to compare versions. Fixing those gaps is more valuable than adding another general health screen. Repeat the exercise after changes; the diagnostic path can break just as ordinary application behaviour does.