Prometheus and Grafana with SLI and SLO instead of metric hoarding

The monitoring stack usually starts collecting before it starts deciding, and the collecting instinct is the failure this article exists to correct. The team instruments everything it can think of, exports every counter the frameworks expose, builds dashboards because the tooling makes them cheap, and six months later the platform holds ten thousand series that nobody can name the meaning of, the alerts page on the metrics that are easy to threshold rather than the ones that matter, and the honest answer to is the service healthy is a feeling rather than a number. The service level discipline inverts the order, decide what the user experiences, define the indicators that measure it, set the objectives with error budgets, and let the collection serve the definitions. This article builds that inversion properly, the indicator selection, the Prometheus instrumentation that expresses it, the error budget arithmetic, the alerting that pages on burn rather than on noise, and the Grafana practice that makes the dashboards operational rather than decorative.

The indicators that actually represent the user

The service level indicator is a measurement of something the user experiences, and the selection discipline is the one that rejects the metric hoarding. The latency of the critical path, the p99 of the request that loads the product page, not the average across all endpoints. The availability of the critical path, the fraction of requests that succeeded, not the uptime of the instances. The correctness of the data, the orders that processed correctly, not the CPU of the worker. The saturation that predicts the failure, the queue depth or the connection pool exhaustion, not the disk's raw usage. The RED vocabulary, rate, errors, duration, per critical endpoint, and the USE vocabulary, utilization, saturation, errors, per resource, are the two catalogs that cover the surface, and the team's first exercise is naming its critical paths, the ten requests that are the product, and defining indicators for those and nothing else at the start:

# the critical path definition, the whole monitoring contract starts here
GET /api/orders        latency p99 < 400ms     availability 99.9%
POST /api/checkout     latency p99 < 800ms     availability 99.9%

The objective, the SLO, is the target the team commits to on that indicator, 99.9 percent of checkout requests under 800 milliseconds, and the objective's arithmetic is the error budget, the one in a thousand requests that may miss, or the 43 minutes a month the 99.9 availability may be down, and the budget is the release management tool the next sections build on.

The Prometheus instrumentation that expresses the definitions

Prometheus's model matches the discipline's needs, the time series scraped from the exporters and the application endpoints, the four metric types with their correct usage. The counter for the monotonic, requests total, and the rate function that turns it into the per second rate. The histogram for the distribution, the request duration in buckets, and the histogram_quantile that answers the p99 question:

http_request_duration_seconds_bucket{
  handler="/api/orders", le="0.4"} 2401
http_request_duration_seconds_bucket{
  handler="/api/orders", le="+Inf"} 2410

The application side instruments the critical paths with the labels that matter, the handler and the status, and the label discipline is the cardinality discipline, the labels that carry the dimensions the alerts and dashboards query, never the user identifier, never the request identifier, the unbounded labels that explode the series count that the Prometheus operational articles keep warning about. The recording rules precompute the definitions so the expensive expressions run once at scrape interval rather than on every dashboard load:

groups:
  - name: sli
    rules:
      - record: sli:orders:p99
        expr: histogram_quantile(0.99,
          sum by (le) (rate(http_request_duration_seconds_bucket{handler="/api/orders"}[5m])))
      - record: sli:orders:availability
        expr: sum(rate(http_requests_total{handler="/api/orders", status=~"5.."}[5m]))
            / sum(rate(http_requests_total{handler="/api/orders"}[5m]))

The recording rule is the indicator's definition written down in the system, the series name that the alerts and dashboards consume, and the definitions live in the repository with the application code, reviewed and versioned like the code that generates the data.

The error budget arithmetic and the release policy it drives

The error budget converts the objective into an operational resource, and the arithmetic is simple enough to keep in the head. The 99.9 percent availability objective allows a 0.1 percent error rate, and the budget consumption is the actual error rate against it, the burn rate, the speed at which the month's budget is being spent. The multi window burn rate alert is the standard expression, the fast window that catches the fast burn, the 14.4 times burn over five minutes that pages, and the slow window that catches the steady leak, the 1 times burn over six hours that creates a ticket, and the two together are the alerting profile that pages on the user's pain rather than the metric's noise:

- alert: CheckoutBudgetFastBurn
  expr: sli:checkout:error_ratio * 1000 > 14.4
    and ignoring(slow) (sli:checkout:error_ratio offset 1h * 1000 > 14.4)
  for: 5m

The budget's management power is the release policy it makes possible, the month that the budget is spent early is the month the feature train slows, the reliability work takes the budget's place, and the trade is explicit rather than political. The budget's reporting is the executive dashboard's job, the budget remaining, the burn trend, the objective's recent attainment, and the Grafana panel that the leadership team actually reads is the one that shows the budget, because it is the single number that summarizes the service's operational story.

The Grafana practice that serves on call rather than decoration

The dashboards follow the same discipline the definitions do, and the operating hierarchy is the on call's actual workflow. The first dashboard is the service's landing page, the SLI panels, the traffic, the error budget's remaining, the current p99 against the objective line, and the panels answer the page's first question, is this the service the alert named. The second layer is the diagnostic per service, the resource panels, the dependency latencies, the saturation signals, and the links from the first layer's panels carry the drill down. The third layer is the long horizon, the weekly and monthly attainment, the budget's history, and the review meetings' material.

dashboard: orders-service / status
  row:  SLIs          [p99 vs objective, availability vs objective]
  row:  error budget  [remaining, burn rate]
  row:  dependencies   [db latency, cache hit, downstream p99]

The dashboard anti-patterns are the ones the hoarding builds, the forty panel wall that hides the status in noise, the average latency panel that hides the p99 the users feel, the dashboard that shows the instances rather than the requests, and the panel that needs a manual legend to read. The dashboard's test is the on call drill, the alert fires, the engineer opens the dashboard, and the page answers the question in thirty seconds, or the dashboard gets rebuilt until it does.

The instrumentation toolkit that keeps the definitions honest

The instrumentation practice stands on a small toolkit, and the team that knows it stops inventing metric shapes. The client libraries in every mainstream language expose the four types, counter, gauge, histogram, summary, with the standard runtime and framework instrumentation included, the process metrics, the HTTP handlers, the garbage collection timings, and the application adds the business counters, the orders processed, the payments failed by reason. The exporters cover the infrastructure the application does not see, node exporter for the host, the database exporters for the query and replication metrics, the cloud provider's own exporters, and the blackbox exporter for the outside in view of the critical paths, the probe that measures what the user measures, the endpoint's availability and latency from outside.

The histogram configuration is the toolkit's one decision that matters, the buckets that the quantile calculation interpolates, and the buckets chosen for the objective's neighborhood are the ones that make the p99 meaningful, the checkout objective at 800 milliseconds wants the bucket edges around it, the default buckets that cluster below 100 milliseconds make every real p99 a flat line at the top bucket. The instrumentation rule that falls out is worth writing down, the bucket edges are part of the definition, chosen with the objective, and the bucket change is a review item rather than a library default nobody read.

The burn rate windows, the arithmetic behind the paging

The multi window alert deserves its arithmetic printed, because the numbers look arbitrary until the budget stands behind them. The month's budget at 99.9 percent is 43 minutes, and the burn rate is the speed the budget spends, the 14.4 times burn spends the month's budget in two hours, the 6 times burn spends it in half a day, the 1 times burn spends it across the month itself. The windows are the confidence mechanism, the fast window alone pages on the spike that recovers, the slow window alone misses the spike that spends the budget, and the pair confirms, the fast burn that is also burning an hour ago is the fire that pages:

window     burn     budget spent      response
5m/1h      14.4x    2% of the month   page immediately
30m/6h      6x      5% of the month   page, slower confirmation
6h/3d       1x      ~10% of the month  ticket, no page

The table is the alerting policy in one view, the fast burns page because the budget is being spent at the rate that exhausts it inside a working day, the slow burn tickets because the month needs rebalancing rather than the pager, and the raw threshold alert that predates the discipline is the alert that pages on the 0.1 percent that the objective already budgeted for.

One monthly review where the numbers drive the roadmap

The monthly reliability review is where the discipline cashes out, and the meeting's agenda is the numbers' agenda. The attainment line first, the checkout objective met at 99.93 against the 99.9 target, the month was good, the feature train ran the full sprint. The budget story second, the 43 minute budget spent at 78 percent, the two incidents that spent it, the slow query that spent twenty of the minutes and now has the index that the database article's discipline would have named earlier. The trend third, the three month attainment, 99.97, 99.92, 99.93, the steady state with the budget margin, and the one decision the meeting makes, the next month's release policy, full speed with the margin, or the reliability sprint because the trend is steepening.

The meeting that the discipline produces is twenty minutes with decisions, and the meeting that the hoarding produces is an hour with impressions, the same platform, the same teams, and the difference is the definitions, the indicators that mean something and the budget that the roadmap can spend.

The working rules for the level discipline

The stack serves the definitions and the definitions serve the user, and the practice's rules are short enough to keep:

1. Name the critical paths and define SLIs for those before instrumenting anything else;
2. Express the definitions as recording rules in the repository, reviewed like code;
3. Set objectives with the stakeholders, and let the error budget drive the release policy;
4. Page on multi window burn rates, the fast burn and the slow leak, never on raw thresholds;
5. Build dashboards as the on call's drill surface, status first, diagnostics linked, history below;
6. Prune the series the definitions do not consume, the hoarded metric is the cost the query time pays.

The monitoring that starts from the user's experience ends with the numbers that mean something, the p99 that a stakeholder can quote, the budget that a release manager can spend, and the alert that an engineer can trust, and the stack that started from the collecting instinct ends with the ten thousand series that mean nothing in particular. The discipline is the difference, and the indicators, written as rules and owned as contracts, are where the whole inversion starts.