Monitoring with Prometheus and Grafana: Metrics and Alerting

KUBERNATES COURSE

Learning objectives

In this lesson you will learn to design and build a modern monitoring system based on Prometheus and Grafana. You will see how to collect reliable metrics, create effective queries, build useful dashboards, and implement an alerting system that reduces noise and speeds up incident resolution. We’ll wrap up with best practices for scalability, security, and governance.

Why monitoring is crucial

Monitoring isn’t just to “see if something is broken,” but to understand system behavior over time, anticipate issues, and measure user impact. With consistent metrics and well-designed alerts you reduce MTTR, protect SLOs, and enable informed decisions on capacity and cost optimization.

Architecture at a glance

Prometheus is a time series database that scrapes metrics exposed by service endpoints or exporters. Metrics are labeled with labels to enable filtering and aggregation. Grafana connects to Prometheus as a data source and visualizes data with panels and dashboards. Alerting rules live in Prometheus; alert notification and orchestration are handled by Alertmanager.

Metric types and semantics

Knowing the metric types is essential to write correct queries and useful alerts.

Counter

A value that increases monotonically (e.g., total HTTP requests). It resets after a restart. Analyze it with functions like rate() and increase().

Gauge

A value that can go up and down (e.g., memory usage, active connections). Observe it with instantaneous functions or windowed statistics.

Histogram

Distribution of values across buckets (e.g., latency). Enables approximate quantiles with histogram_quantile() by summing over le.

Summary

Client-side computed quantiles. Useful but less aggregable across instances than histograms.

Naming and label best practices

Use descriptive metric names in snake_case and include units in the name (e.g., http_request_duration_seconds). Keep label cardinality under control: avoid labels with unbounded values (e.g., user ID, full URL). Define a minimal set of standard labels such as job, instance, environment, region, service, version. Document meaning and expected ranges.

Metric exposition and exporters

Your services expose /metrics in a text format; for third-party components use official exporters: node_exporter (host), blackbox_exporter (HTTP/TCP/ICMP probes), cAdvisor (containers), kube-state-metrics (Kubernetes), postgres_exporter, mysql_exporter and others. Avoid using Pushgateway for long-running jobs; it is intended only for ephemeral batch jobs.

Scraping and service discovery

Prometheus supports dynamic discovery (Kubernetes, Consul, EC2) and static targets. Apply relabeling to clean up or enrich labels. Keep scrape intervals consistent (e.g., 15s for core services) and timeouts lower than the interval. Minimal configuration example:

global:
  scrape_interval: 15s
  evaluation_interval: 30s

scrape_configs:
- job_name: node
  static_configs:
  - targets: ['node1:9100','node2:9100']
    labels:
      environment: production

- job_name: app
  metrics_path: /metrics
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: "true"

PromQL: from basics to practical examples

Queries are the heart of everything: they power dashboards, recordings, and alerts.

# Request throughput per second
sum by (service) (rate(http_requests_total[5m]))

# Error rate (%)
100 * sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
  / sum by (service) (rate(http_requests_total[5m]))

# Average CPU usage per pod
avg by (pod) (rate(container_cpu_usage_seconds_total[5m]))

# P95 latency over 5 minutes
histogram_quantile(0.95,
  sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))
)

# Synthetic availability (blackbox)
avg_over_time(probe_success{job="blackbox"}[30m])

Use recording rules to precompute expensive queries, reduce latency, and standardize indicators (e.g., rates, quantiles). Example:

groups:
- name: recording.rules
  interval: 30s
  rules:
  - record: service:http_requests:rate5m
    expr: sum by (service) (rate(http_requests_total[5m]))
  - record: service:error_rate:percent5m
    expr: 100 * sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
         / sum by (service) (rate(http_requests_total[5m]))

Effective dashboards in Grafana

A useful dashboard answers concrete questions: is the user impacted? where is the bottleneck? Create panels with a hierarchical layout (overview at the top, detail below), use variables (service, cluster, region) for context, limit the default time range, and add automatic annotations for deployments.

Practical tips:

– “Golden Signals” panels: traffic, errors, latency (P50/P95/P99), resource saturation.

– Resource charts: CPU, memory, I/O, GC, connections.

– Top-N tables to identify outliers.

– Alert links: every critical panel should link to a runbook.

Alerting: from symptom to diagnosis

Alerts should reflect user impact or imminent risk. Use different levels (warning, critical), filters per environment, and histograms/ratios to avoid false positives. Example rules:

groups:
- name: alerting.rules
  rules:
  - alert: HighErrorRate
    expr: service:error_rate:percent5m > 5
    for: 10m
    labels:
      severity: warning
      team: web
    annotations:
      summary: Error rate > 5
      runbook: https://wiki/runbooks/error-rate

  - alert: HighLatencyP95
    expr: histogram_quantile(0.95, sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))) > 0.5
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: P95 latency > 500ms
      runbook: https://wiki/runbooks/latency

Alertmanager: routing, deduplication, and silences

Alertmanager receives alerts, groups them, deduplicates, and forwards them to channels such as Slack, email, PagerDuty. Configure routing per team, inhibiting rules (e.g., suppress “derived” alerts when there is a root cause), and use silences for maintenance windows.

route:
  group_by: ['alertname','service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 2h
  receiver: team-default

  routes:
  - matchers:
    - team="web"
    receiver: team-web

receivers:
- name: team-web
  slack_configs:
  - channel: '#web-alerts'
    username: Alertmanager
    send_resolved: true
    title: '[{{ .Status }}] {{ .CommonLabels.alertname }}'
    text: 'Details: {{ .CommonAnnotations.summary }}'

inhibit_rules:
- source_matchers: ['severity="critical"']
  target_matchers: ['severity="warning"']
  equal: ['service']

SLO, SLI and error budget

Define reliability indicators (SLIs) such as availability or latency, set objectives (SLOs), and monitor the accumulated error. Example: availability 99.9

# Burn rate over 1h and 6h
short = (1 - avg_over_time(probe_success[1h])) / 0.001
long  = (1 - avg_over_time(probe_success[6h])) / 0.001

Scalability and high availability

For growing environments, use HA with two Prometheus instances per shard, a replica label for logical deduplication, and horizontal storage like Thanos/Cortex/Mimir via remote_write. Apply appropriate retention (e.g., 15–30 days locally; historical data in object storage). Keep an eye on disk usage: TSDB compaction and the WAL can be intensive.

Security, compliance, and governance

Protect /metrics endpoints behind TLS or authenticated ingress, filter sensitive labels, and apply rate limiting to Internet-facing exporters. In Grafana enable RBAC, team folders, and as-code provisioning of dashboards. Version configurations and rules in Git with mandatory reviews and linting tests. Keep runbooks and postmortems close to the alerts.

Common anti-patterns and how to avoid them

– Explosive cardinality: avoid labels with per-request unique values.

– Single-point alerts: prefer aggregations per service/area.

– Rigid thresholds without context: use percentiles, rates, and time windows.

– “Wall of charts” dashboards: favor data story and narrative over quantity.

Guided exercise: from zero to your first alert

1) Start Prometheus and a node_exporter, then add Grafana.

docker network create monitor

docker run -d --name=node --net=monitor -p 9100:9100 prom/node-exporter

docker run -d --name=prom --net=monitor -p 9090:9090 
  -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml 
  prom/prometheus

docker run -d --name=grafana --net=monitor -p 3000:3000 grafana/grafana

2) Create an alerting rule for high CPU and configure Alertmanager.

# Rule
- alert: HighCPU
  expr: avg by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m])) > 0.8
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: CPU > 80

3) Connect Grafana to Prometheus, import a “Node Exporter Full” dashboard, create environment and instance variables, and add annotations for deployments.

Production rollout checklist

– Standardized exporters and aligned versions.

– Consistent scrape interval and timeout, scraping health monitored.

– Recording rules for critical queries.

– Alerts with runbooks, team routing, configured silences and inhibiting.

– Dashboards with variables and annotations, correct Grafana permissions.

– Defined backup and retention; capacity plans for the TSDB.

– Policies on naming, labels, and rule reviews.

Conclusions

With Prometheus and Grafana you can build a robust observability system, able to turn raw signals into operational decisions. Focus on meaningful metrics, well-designed queries, and alerts that tell a clear story, linked to runbooks and SLOs. Scale prudently, protect endpoints, and manage everything as-code to avoid drift.

Put it into practice now

Do you want to bring these principles to your team? Start by creating a “Golden Signals” dashboard and two alerts focused on key SLOs. If you want a free review of your Prometheus/Grafana setup or a ready-to-use rule template, contact me: I will help you set up effective metrics, dashboards, and alerting right away.

Se vuoi farmi qualche richiesta o contattarmi per un aiuto riempi il seguente form

    Comments