Expose latency distributions, not averages alone

perf-010

Intent

Prevent teams from shipping monitoring or benchmark results that hide outliers and bimodal behavior behind a reassuring mean.

Applicability

Applies when the diff adds or changes latency metrics, latency dashboards, performance tests, benchmark reporting, or SLO evaluation for request paths, database calls, file I/O, or network I/O. Return unknown if the change does not measure latency or the workload is purely throughput-oriented.

What to inspect

Metric definitions, histogram and percentile exports, dashboards, benchmark output, perf-test assertions, alert logic, and documentation describing latency results.

Pass criteria

Direct evidence shows the change records or reports a latency distribution using histograms, heat maps, or percentiles, optionally alongside averages.

Fail criteria

The diff introduces or relies on average latency alone for a latency-sensitive path, with no repository evidence that percentiles or another distribution view is captured for the same measurement.

Do not flag

Do not flag counters, pure throughput benchmarks, or CPU-bound microbenchmarks that are not making latency claims. Do not flag places where percentiles or histograms are already emitted by shared instrumentation outside the diff and clearly used for the same path.

Confidence guidance

HIGH when the diff visibly reports only a mean or average for a latency-sensitive path. MEDIUM when distribution support may exist elsewhere in a shared telemetry layer. LOW when the workload’s latency sensitivity is only loosely implied.

Remediation

Add histogram or percentile reporting for the measured latency path instead of publishing only an average.

Pass example

REQUEST_LATENCY = Histogram(
    "request_latency_seconds",
    "End-to-end request latency",
    buckets=(0.01, 0.05, 0.1, 0.25, 0.5, 1, 2, 5),
)

Fail example

avg_latency_ms = total_latency_ms / request_count
print(f"average latency: {avg_latency_ms:.2f} ms")

Sources

  • Systems Performance: Enterprise and the Cloud — Brendan Gregg, 2020 book