Metrics Collection, Golden Signals & Alerting

Last Audited: 2026-08-19
Tier-1 Platform Core
In Plain Language

Real-time time-series instrumentation, statistical percentiles (p50/p95/p99), cardinality governance, and symptom-based alerting.

Metrics: Real-Time Numerical Pulses of System Health

Metrics are aggregated, numerical measurements recorded over regular time intervals. Unlike logs that record every discrete event, metrics aggregate events into counters, rates, and distributions. This makes metrics computationally lightweight to query, inexpensive to retain over multi-year horizons, and ideal for triggering real-time automated alerts before users notice service degradation.

The Four Standard Metric Types (Prometheus / OpenMetrics)

Every metric instrumented in modern software falls into one of four standard mathematical types:

1. Counter

Monotonically Increasing

A cumulative metric that only ever increases or resets to zero on restart. Use rate() to calculate per-second throughput.

http_requests_total{status="200"} 14920

2. Gauge

Fluctuates Up & Down

Represents a single numerical value that can go up and down at any moment (instantaneous snapshot).

memory_usage_bytes 2147483648

3. Histogram

Configurable Buckets

Samples observations (usually request durations or sizes) and counts them in configurable cumulative buckets for calculating percentiles.

http_req_duration_seconds_bucket{le="0.5"} 140

4. Summary

Client-Side Quantiles

Similar to a histogram, but calculates configurable quantiles directly on the client instance over a sliding time window.

rpc_duration_seconds{quantile="0.99"} 0.42
Core Measurement Principles

The Four Golden Signals (Google SRE Framework)

Measure What Matters

If you can only measure four metrics of a user-facing service, focus on these four signals. They provide complete high-level visibility into system health, user impact, and capacity limits.

Latency

Milliseconds (ms) / Seconds (s) tracked as histogram percentiles (p50, p90, p95, p99).

The time it takes to service a user request, differentiating between successful request latency and failed request latency.

Percentile Rule: Never rely on average (mean) latency, which conceals extreme tail outliers. A p99 of 2.5s means 1 in 100 users experiences a 2.5s delay.
Metric PromQL Pattern:histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
Alert Policy: Alert when p95 latency exceeds 500ms for >3 consecutive minutes, or when p99 latency degrades by >50% relative to baseline.
Pitfall: Lumping error response latency together with successful responses (a fast 500 Internal Error can artificially lower your average latency).

Traffic

Requests per second (RPS), Transactions per second (TPS), or Network I/O throughput (Mbps).

A measure of how much demand is being placed on your system, measured in high-level system-specific throughput units.

Percentile Rule: Tracked as rate over sliding windows (1m, 5m, 1h) compared against diurnal day-of-week seasonality baselines.
Metric PromQL Pattern:sum(rate(http_requests_total[1m])) by (service)
Alert Policy: Alert on sudden traffic drop-offs (>80% drop in 2 minutes indicates upstream DNS or CDN outage) or unexpected traffic spikes (>300% surge indicates DDoS or runaway retry loop).
Pitfall: Measuring traffic solely at the ingress gateway and ignoring downstream inter-service traffic amplification.

Errors

Error percentage (%) of total traffic, or absolute failed requests per second.

The rate of requests that fail, either explicitly (e.g., HTTP 500s, gRPC Internal), implicitly (e.g., HTTP 200 containing an error payload), or policy-based (e.g., response took >2s).

Percentile Rule: Calculated as: `sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100`.
Metric PromQL Pattern:sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
Alert Policy: Alert when 5xx error rate exceeds 1% of total traffic over 5 minutes (burns error budget at 14x rate).
Pitfall: Treating user-induced errors (HTTP 401/404) identically to system failures (HTTP 500/503), creating constant false-alarm alert noise.

Saturation

Percentage utilization (%) or queue backlog length (e.g., 85% DB pool utilized, 4,200 messages waiting in queue).

A measure of how full your service is, emphasizing the resources that are most constrained (CPU, memory, database connection pool, queue depth).

Percentile Rule: Measure leading indicators before 100% capacity is reached, because system degradation typically becomes non-linear above 80% saturation.
Metric PromQL Pattern:sum(container_memory_working_set_bytes) / sum(container_spec_memory_limit_bytes) * 100
Alert Policy: Alert when disk storage exceeds 85% full, or when thread pool queue depth exceeds 200 pending tasks for >5 minutes.
Pitfall: Monitoring only CPU and memory while forgetting disk I/O IOPS limits, database connection limits, and file descriptor exhaustion.

High-Cardinality Explosion Hazards & Governance

Cardinality is the total number of unique time-series created by multiplying all possible values of every label key on a metric. Adding high-entropy fields as metric labels will crash Prometheus or cause astronomical cloud monitoring bills:

❌ Prohibited High-Cardinality Labels
  • User IDs / Patient UUIDs (millions of unique values)
  • Raw Request URLs with query parameters
  • Authentication tokens, IP addresses, or Session IDs
✅ Safe Low-Cardinality Labels
  • HTTP Status Codes (e.g., 200, 400, 500)
  • Parameterized Route Templates (e.g., /api/v2/prescriptions/:id)
  • HTTP Methods (GET, POST, DELETE)
  • Environment & Region (production, us-east-1)

Alert Policy Design: Symptom-Based vs. Cause-Based Alerts

To avoid on-call alert fatigue, alert policies must prioritize symptoms that impact users over noisy internal causes:

⚠️ Cause-Based Alerting (Noisy Anti-Pattern)

Paging an engineer every time a single server reaches 85% CPU. If the cluster is autoscaling smoothly and user latency is unaffected, this alert generates useless wake-up pages.

Outcome: Alert fatigue, ignored pages, missed outages.

✅ Symptom-Based Alerting (Best Practice)

Paging only when user-facing SLOs are burning error budgets rapidly (e.g., 5xx error rate >1% or p95 latency >500ms for 5 consecutive minutes).

Outcome: High signal-to-noise, actionable incident response.
Try This with AI: Multi-Window Multi-Burn-Rate Prometheus Alert Rule Generator

Generate mathematically rigorous Prometheus alerting rules using multi-window error budget burn rates to eliminate false alerts.

You are a Site Reliability Engineer (SRE). Write production-grade Prometheus alerting rules (PromQL) for our microservice based on Google SRE's Multi-Window Multi-Burn-Rate alerting methodology. Target Service: prescription-api Service Level Objective (SLO): 99.9% of requests must succeed (HTTP non-5xx) and return in <500ms over 30 days. Requirements: 1. Short-window alert (14.4x burn rate over 1 hour & 5 minutes) for urgent page (SEV-1). 2. Long-window alert (6x burn rate over 6 hours & 30 minutes) for ticket/slack notification (SEV-2). 3. Include runbook URLs, alert annotations, and exact PromQL expressions with histogram quantiles.

Community Discussion & Feedback

Attributed peer feedback and official Netspective architecture notes.

Was this documentation helpful?(100% found this helpful • 0 ratings)

Leave Feedback or Question

○ Loading user info...
0/2000 chars

Discussion (0)

Loading discussion thread...