12.1 Metric Engineering: Cardinality, Histograms, and PromQL

Observability cost and query quality are often determined when you name metrics and choose your labels. Metrics are not event repositories meant for future use; they are bounded data models designed for aggregation, alerting, and capacity planning.

A time series is determined by a complete set of labels

http_server_requests_total{
  service="checkout",
  route="/orders/{id}",
  method="GET",
  status_code="200"
}

The metric name combined with all label values forms a unique series. A rough estimate:

series ≈ services × routes × methods × status codes × instances

Ten services, 20 routes, four methods, five status codes, and 30 instances produce a Cartesian product of 120,000 series. A classic histogram with the same dimensions and ten finite boundaries adds +Inf, _sum, and _count: 13 times as many, or 1,560,000 series before other metadata. Not every combination necessarily exists, but one label can still dominate cost. Labels like user_id, email, original URL, trace ID, full error message, and random task ID should not be included in standard Prometheus labels; they are better suited for logging, tracing, or analytics systems.

Use templated routes such as /orders/{id} instead of /orders/817263. If status codes don't require per-code analysis, record only 2xx, 4xx, and 5xx; make a clear trade-off between diagnostic capability and cardinality control.

Counter and Gauge Have Different Query Semantics

A Counter only increments and may reset to zero upon process restart. When analyzing rates, first rate(), then aggregate across instances:

sum by (service) (
  rate(http_server_requests_total[5m])
)

rate() must be placed inside sum() so that each original series can individually detect resets. For alerts and smoothed trends, use rate(); irate() only considers the last two points and is better suited for rapidly changing interactive charts. It should not be used by default for stable alerting scenarios.

A Gauge represents a transient state that can increase or decrease, such as queue depth or concurrent requests. Using rate() for Gauges typically has no business meaning.

Latency Needs a Distribution

Average latency can hide a small fraction of very slow requests. A classic histogram accumulates observations into buckets:

histogram_quantile(
  0.99,
  sum by (service, le) (
    rate(http_server_request_duration_seconds_bucket[5m])
  )
)

le must be preserved in the histogram_quantile() input of the classic histogram. The result is an interpolated estimate within each bucket, with accuracy dependent on the bucket boundaries chosen during instrumentation. To directly measure the fraction completing within 300 ms, include an exact le="0.3" bucket and divide its observation increase by the _count increase. Nearby boundaries still require estimation. Quantile queries depend on the full set of boundaries; queries cannot recover discarded resolution.

Native histograms treat a histogram sample as a time series, so aggregation queries no longer require le:

histogram_quantile(
  0.99,
  sum by (service) (
    rate(http_server_request_duration_seconds[5m])
  )
)

Prometheus officially recommends native histograms when feasible, but adoption must be verified across the full stack: Prometheus, remote storage, query layer, client libraries, and telemetry pipelines. During migration, do not assume all downstream components handle classic and native data equivalently. One native-series sample may contain many buckets; fewer series do not mean fixed sample size or processing cost.

A summary with quantiles computes preset percentiles on the client. Averaging instance quantiles cannot produce a global quantile, and new percentiles cannot be chosen afterward. Its _sum and _count can still be aggregated to calculate a mean. Unless a specific need for client-side sliding window semantics exists, service-level latency should generally favor histograms.

Vector Matching Requires Clear Relationships

Compute the error rate for each service:

(
  sum by (service) (
    rate(http_server_requests_total{status_code=~"5.."}[5m])
  )
  or
  0 * sum by (service) (
    rate(http_server_requests_total[5m])
  )
)
/
sum by (service) (
  rate(http_server_requests_total[5m])
)

The example keeps numeric status codes, hence 5... The right side of or supplies a zero-error numerator only for services with request metrics, so a service that has never seen a 5xx does not lose its ratio. This assumes missing error counters mean no observed errors; the expression cannot distinguish that from partial telemetry loss, so the instrumentation contract and collection health still need checks. Zero request rate still produces 0/0, or NaN; completely missing telemetry is not filled in as healthy. Aggregating away region/cluster suits an overall view; retain those labels on both sides when diagnosing failure domains.

Both sides of vector arithmetic need matching labels. Before using on(), ignoring(), group_left, or group_right, clearly define whether the relationship is one-to-one, one-to-many, or many-to-one. Unexpected many-to-many matches often indicate issues with the model or aggregation hierarchy.

Missing series must also have explicit semantics. up == 0 indicates a target exists but failed to be retrieved, while absent(up{job="api"}) checks for the absence of all matching series. It cannot detect one missing target among several; that needs an expected-target inventory or another independent signal. Treating "no data" as healthy automatically can create misleading green dashboards during data collection chain failures.

Recording Rules Save Common Aggregations

Frequent dashboards and alerts should not scan all raw series on every execution:

groups:
  - name: atlas-api-recording
    interval: 30s
    rules:
      - record: service:http_server_requests:rate5m
        expr: sum by (service) (rate(http_server_requests_total[5m]))
      - record: service:http_server_errors:ratio_rate5m
        expr: |
          (
            sum by (service) (rate(http_server_requests_total{status_code=~"5.."}[5m]))
            or
            0 * sum by (service) (rate(http_server_requests_total[5m]))
          )
          /
          sum by (service) (rate(http_server_requests_total[5m]))

Rules must have consistent naming, unit tests, and a designated owner. Pre-aggregation can lose labels, making them irrecoverable later, so only stabilize query layers that are inherently stable. Never sacrifice critical dimensions needed for future diagnostics in favor of performance.

Cardinality Budgets Are Interface Constraints

Before any new metric enters production, it must answer these questions:

• Which user question or SLO is driving this metric?
• What is the finite set of label values, and what is the worst-case cardinality?
• How does the number of series scale with each additional instance, tenant, or region?
• How long should the data be retained, and is downsampling or remote storage required?
• Which dashboard, alert, or runbook consumes this metric?
• Who is responsible for deletion if the metric has no consumers?

We must also monitor the observability system itself: active series count, scrape sample volume, rule evaluation latency, query concurrency, WAL and disk usage, remote write backlog, and discarded samples. When an observability platform goes out of control, it is precisely the evidence of the incident that vanishes first.

References

• Prometheus, Instrumentation

• Prometheus, Metric and label naming

• Prometheus, Histograms and summaries

• Prometheus, Query functions

• Prometheus, Recording rules

• Prometheus, Vector matching and operators