The microservices debugging problem that tracing answers is old enough to have folklore, the incident that involves five services and the engineer who debugs each one separately, reading each service's logs, correlating the timestamps by hand, and reconstructing the request's path from the pieces. The reconstruction is expensive and it is unreliable, because the path the engineer reconstructs is the path they expected rather than the path the request took, and the actual answer, that the checkout spent eight hundred of its nine hundred milliseconds waiting on a cache miss that cascaded, is invisible from any single service's logs. OpenTelemetry tracing makes the path the primary artifact, the spans that record every hop, every queue, every database call, with their parents and their timings, and the incident question becomes where did the time go rather than which service is lying. This article builds the tracing practice properly, the span model and the propagation contract, the instrumentation that produces the spans, the sampling economics, the backend choice, and the disciplines that make traces answer questions rather than decorate dashboards.
The span model and the propagation contract
The trace is a tree of spans, each span a named unit of work with its start, its end and its attributes, and the tree's assembly across services is the mechanism that makes distributed tracing work, the context propagation. The W3C traceparent header is the standard carrier, the trace identifier and the current span identifier riding the HTTP request or the message header, and every service that honors the contract contributes its spans to the same tree:
traceparent: 00-4bf92f3577b34da6a0ce9a85d1e6f5c3-00f067aa0ba902b7-01
version-traceid----------spanid----------flagsThe producer library creates the span for its operation, propagates the context through the request to the next service, and the consumer's library continues the tree, the parent's identifier arriving with the work, and the whole path assembles in the backend without any service knowing the others exist. The propagation contract's dark corners are the engineering discipline's business, the queue that must carry the header, the thread pool that must propagate the context across the handoff, the framework's async boundary that drops it, and the dropped context is the broken trace, the two half trees that the incident engineer has to join by hand, which is the folklore the tracing was purchased to retire.
The instrumentation, automatic first and manual where it counts
The instrumentation economy is the practice's pleasant surprise, because the automatic instrumentation in the mainstream languages produces the majority of the valuable spans with zero application code. The framework's inbound and outbound HTTP, the database drivers, the queue clients, the cache calls, instrumented by the runtime layers, and the service's first day with the SDK installed already shows the request's path through its dependencies with their latencies. The manual instrumentation adds the business meaning where the automatic layer cannot, the span that names the domain operation, place order, with the attributes that name its context, the order identifier, the customer tier, and the naming discipline that the observability vocabularies share, the operation as the verb, the attributes as the bounded dimensions:
span = tracer.start_span("orders.place")
span.set_attribute("order.id", order_id)
span.set_attribute("order.tier", customer_tier)The span attribute discipline is the same cardinality conversation the logging and metrics articles kept having, the bounded dimensions on the spans, the identifiers as the drill down values, and the span name is the closed vocabulary that the backend's queries depend on, orders.place rather than the dynamic string with the identifier embedded, because the dynamic span name is the label explosion that the tracing store indexes badly.
Sampling, the economics of the trace budget
The trace volume economics is the discipline's hard arithmetic, because the full fidelity tracing of a high traffic service is a storage bill that nobody pays without an argument, and the sampling decision is the practice's answer. The head sampling decides at the service, the percentage policy, the trace id based deterministic sampling that keeps whole traces while thinning the fleet, and the parent based policy that honors the upstream decision, and the head sample's honesty is its blindness, the slow request and the failed request sampled at the same rate as the boring majority, the eight hundred millisecond cache miss discarded at the same one percent as everything else.
The tail sampling fixes the blindness at the collector, where the completed trace is inspected and kept by its properties, the slow traces, the error traces, the traces that touched the expensive path, and the tail sampler's configuration is the practice's real policy work, the latency threshold that matches the objective, the error status that always keeps, the resource ceiling that the high volume service sets:
policies:
- type: status_code # errors always kept
status_code: {status_codes: [ERROR]}
- type: latency # slow traces kept
latency: {threshold_ms: 400}
- type: probabilistic # a sample of the healthy fast majority
probabilistic: {sampling_percentage: 1}The tail sampling's cost is the collector's memory, the trace buffered until its completion, and the very high volume services run hybrid policies, the head sample for the flood, the tail policy for the kept sample, and the economics land where the invoice and the diagnostic power balance, every error traced, every slow trace kept, the healthy majority sampled.
The backend and the query surface
The trace backend is the store and the query surface, and the menu is short. Jaeger, the original, the query interface that the debugging workflow shaped, and the storage backends from the bad old Cassandra to the modern click houses. Tempo, the Grafana stack's entry, the object storage backed store that cheapened the retention, the trade being the query surface's dependence on the identifier rather than the free text search. The cloud vendors' managed tracing, the least operational surface and the per gigabyte bill, and the choice prices against the same three variables the log store did, the ingestion, the retention and the query surface the team actually runs.
The query surface is where the backend earns or fails, and the two entry patterns cover the incident, the identifier path, the trace id from the log line, the structured logging article's correlation field, and the search path, the query by the service and the operation with the latency and status filters, the service map that the traces generate, the dependency graph with the live latencies. The backend that answers both paths serves the whole workflow, the report to the timeline, the dashboard to the suspect, and the metric to the trace, the exemplar link that connects the p99 spike to the traces that made it, which is the observability stack's connective tissue, the three disciplines meeting in one investigation.
One slow endpoint debugged from the trace instead of the folklore
The debugging story makes the practice concrete, and it is the story the tracing purchases. The checkout endpoint's p99 crossed its objective at 14:00, the alert from the service level discipline fired, and the engineer's first click is the exemplar, the metrics panel's link to the traces behind the spike. The trace opens as the waterfall, and the nine hundred milliseconds is drawn rather than guessed, the checkout service's own span a thin forty, the payment authorization a steady one hundred and twenty, and the cache span that the eye catches immediately, six hundred and eighty milliseconds named cache.get against the product catalog, the parent's time almost entirely inside the child. The span's attributes name the miss, the key pattern and the node, and the expanded view shows the cascade, the cache miss falling through to the database query on the cold path, the query that the second level cache should have absorbed.
The second click is the slow query's own span, the statement and the row count in the attributes, and the 14:06 diagnosis is the catalog service's cache expiry, the stampede after the daily invalidation window moved, and the fix is the cache layer's jitter, the same stampede pattern the caching discussions document. The whole investigation is the trace's waterfall, the two spans that named the six hundred milliseconds, and the total time from the alert to the diagnosis is eleven minutes, one engineer, no log correlation, no service by service archaeology, and the same investigation in the folklore era is the afternoon that the intro paragraph described.
The service map and the capacity questions the traces answer
The trace collection builds the dependency view as a by-product, the service map that the traces assemble, every edge annotated with the live request rate and the latency distribution, and the map is the architecture diagram that updates itself. The capacity conversations the map serves are the practice's quiet return, the dependency that nobody remembered is visible on the map with its traffic, the service that calls the database directly is visible in the spans, and the quarterly review that opens the map starts from the system's actual topology rather than the diagram the last architect drew. The failure planning uses the same edges, the blast radius question, which services feel it when the catalog degrades, answered by the map's edges with the request rates, and the tracing practice that started as the debugging tool ends as the architecture's living documentation, which is the observability stack's highest dividend.## The disciplines that make traces answer questions
The tracing practice's failure modes are recognizable and preventable, and the disciplines are short. The dropped context at the async boundary is the first, the framework's documented handoff configuration, the test that asserts the trace's continuity across the service's own queue, because the broken trace is the folklore returning. The span naming drift is the second, the closed vocabulary documented beside the logging events, the review that enforces the verb and the bounded attributes. The sampling policy that pages is the third, the policy that discards the error is the policy that fails the incident, and the sampler's config is reviewed with the same seriousness as the alerting rules. The connection disciplines complete the practice, the trace id in the logs, the exemplar in the metrics, the dashboard's link to the trace query, and the investigation that starts anywhere ends in the trace, which is the stack's whole purpose.
The working rules compress the practice, and the list is short:
- Propagate the context at every boundary, the header on the queue, the context over the thread handoff, and test the continuity;
- Instrument automatically, name manually, the closed span vocabulary with the bounded attributes;
- Tail sample for the diagnostic value, errors kept, slow traces kept, the healthy majority thinned;
- Put the trace id in the log events and the exemplars in the metrics, the investigation starts from anywhere;
- Price the backend on ingestion, retention and the query surface, and revisit with the volume;
- Review the sampler and the span names with the alerting rules' seriousness, they are the same class of contract.
The trace is the distributed system's confession, the whole path with its timings and its dependencies, recorded whether the path was fast or slow, healthy or failing, and the engineer who reads the trace reads the system's actual behavior rather than its expected behavior. The tracing that honors the propagation contract, names its spans deliberately and keeps the traces that diagnose is the observability practice that makes the five service incident a quarter hour's question, which is the promise the folklore was waiting for all along.