Monitoring with Prometheus and Grafana: Metrics and Alerting
Learning objectives
In this lesson you will learn to design and build a modern monitoring system based on Prometheus and Grafana. You will see how to collect reliable metrics, create effective queries, build useful dashboards, and implement an alerting system that reduces noise and speeds up incident resolution. We’ll wrap up with best practices for scalability, security, and governance.
Why monitoring is crucial
Monitoring isn’t just to “see if something is broken,” but to understand system behavior over time, anticipate issues, and measure user impact. With consistent metrics and well-designed alerts you reduce MTTR, protect SLOs, and enable informed decisions on capacity and cost optimization.
Architecture at a glance
Prometheus is a time series database that scrapes metrics exposed by service endpoints or exporters. Metrics are labeled with labels to enable filtering and aggregation. Grafana connects to Prometheus as a data source and visualizes data with panels and dashboards. Alerting rules live in Prometheus; alert notification and orchestration are handled by Alertmanager.
Metric types and semantics
Knowing the metric types is essential to write correct queries and useful alerts.
Counter
A value that increases monotonically (e.g., total HTTP requests). It resets after a restart. Analyze it with functions like rate() and increase().
Gauge
A value that can go up and down (e.g., memory usage, active connections). Observe it with instantaneous functions or windowed statistics.
Histogram
Distribution of values across buckets (e.g., latency). Enables approximate quantiles with histogram_quantile() by summing over le.
Summary
Client-side computed quantiles. Useful but less aggregable across instances than histograms.
Naming and label best practices
Use descriptive metric names in snake_case and include units in the name (e.g., http_request_duration_seconds). Keep label cardinality under control: avoid labels with unbounded values (e.g., user ID, full URL). Define a minimal set of standard labels such as job, instance, environment, region, service, version. Document meaning and expected ranges.
Metric exposition and exporters
Your services expose /metrics in a text format; for third-party components use official exporters: node_exporter (host), blackbox_exporter (HTTP/TCP/ICMP probes), cAdvisor (containers), kube-state-metrics (Kubernetes), postgres_exporter, mysql_exporter and others. Avoid using Pushgateway for long-running jobs; it is intended only for ephemeral batch jobs.
Scraping and service discovery
Prometheus supports dynamic discovery (Kubernetes, Consul, EC2) and static targets. Apply relabeling to clean up or enrich labels. Keep scrape intervals consistent (e.g., 15s for core services) and timeouts lower than the interval. Minimal configuration example:
global:
scrape_interval: 15s
evaluation_interval: 30s
scrape_configs:
- job_name: node
static_configs:
- targets: ['node1:9100','node2:9100']
labels:
environment: production
- job_name: app
metrics_path: /metrics
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: "true"
PromQL: from basics to practical examples
Queries are the heart of everything: they power dashboards, recordings, and alerts.
# Request throughput per second
sum by (service) (rate(http_requests_total[5m]))
# Error rate (%)
100 * sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/ sum by (service) (rate(http_requests_total[5m]))
# Average CPU usage per pod
avg by (pod) (rate(container_cpu_usage_seconds_total[5m]))
# P95 latency over 5 minutes
histogram_quantile(0.95,
sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))
)
# Synthetic availability (blackbox)
avg_over_time(probe_success{job="blackbox"}[30m])
Use recording rules to precompute expensive queries, reduce latency, and standardize indicators (e.g., rates, quantiles). Example:
groups:
- name: recording.rules
interval: 30s
rules:
- record: service:http_requests:rate5m
expr: sum by (service) (rate(http_requests_total[5m]))
- record: service:error_rate:percent5m
expr: 100 * sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/ sum by (service) (rate(http_requests_total[5m]))
Effective dashboards in Grafana
A useful dashboard answers concrete questions: is the user impacted? where is the bottleneck? Create panels with a hierarchical layout (overview at the top, detail below), use variables (service, cluster, region) for context, limit the default time range, and add automatic annotations for deployments.
Practical tips:
– “Golden Signals” panels: traffic, errors, latency (P50/P95/P99), resource saturation.
– Resource charts: CPU, memory, I/O, GC, connections.
– Top-N tables to identify outliers.
– Alert links: every critical panel should link to a runbook.
Alerting: from symptom to diagnosis
Alerts should reflect user impact or imminent risk. Use different levels (warning, critical), filters per environment, and histograms/ratios to avoid false positives. Example rules:
groups:
- name: alerting.rules
rules:
- alert: HighErrorRate
expr: service:error_rate:percent5m > 5
for: 10m
labels:
severity: warning
team: web
annotations:
summary: Error rate > 5
runbook: https://wiki/runbooks/error-rate
- alert: HighLatencyP95
expr: histogram_quantile(0.95, sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))) > 0.5
for: 5m
labels:
severity: critical
annotations:
summary: P95 latency > 500ms
runbook: https://wiki/runbooks/latency
Alertmanager: routing, deduplication, and silences
Alertmanager receives alerts, groups them, deduplicates, and forwards them to channels such as Slack, email, PagerDuty. Configure routing per team, inhibiting rules (e.g., suppress “derived” alerts when there is a root cause), and use silences for maintenance windows.
route:
group_by: ['alertname','service']
group_wait: 30s
group_interval: 5m
repeat_interval: 2h
receiver: team-default
routes:
- matchers:
- team="web"
receiver: team-web
receivers:
- name: team-web
slack_configs:
- channel: '#web-alerts'
username: Alertmanager
send_resolved: true
title: '[{{ .Status }}] {{ .CommonLabels.alertname }}'
text: 'Details: {{ .CommonAnnotations.summary }}'
inhibit_rules:
- source_matchers: ['severity="critical"']
target_matchers: ['severity="warning"']
equal: ['service']
SLO, SLI and error budget
Define reliability indicators (SLIs) such as availability or latency, set objectives (SLOs), and monitor the accumulated error. Example: availability 99.9
# Burn rate over 1h and 6h short = (1 - avg_over_time(probe_success[1h])) / 0.001 long = (1 - avg_over_time(probe_success[6h])) / 0.001
Scalability and high availability
For growing environments, use HA with two Prometheus instances per shard, a replica label for logical deduplication, and horizontal storage like Thanos/Cortex/Mimir via remote_write. Apply appropriate retention (e.g., 15–30 days locally; historical data in object storage). Keep an eye on disk usage: TSDB compaction and the WAL can be intensive.
Security, compliance, and governance
Protect /metrics endpoints behind TLS or authenticated ingress, filter sensitive labels, and apply rate limiting to Internet-facing exporters. In Grafana enable RBAC, team folders, and as-code provisioning of dashboards. Version configurations and rules in Git with mandatory reviews and linting tests. Keep runbooks and postmortems close to the alerts.
Common anti-patterns and how to avoid them
– Explosive cardinality: avoid labels with per-request unique values.
– Single-point alerts: prefer aggregations per service/area.
– Rigid thresholds without context: use percentiles, rates, and time windows.
– “Wall of charts” dashboards: favor data story and narrative over quantity.
Guided exercise: from zero to your first alert
1) Start Prometheus and a node_exporter, then add Grafana.
docker network create monitor docker run -d --name=node --net=monitor -p 9100:9100 prom/node-exporter docker run -d --name=prom --net=monitor -p 9090:9090 -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus docker run -d --name=grafana --net=monitor -p 3000:3000 grafana/grafana
2) Create an alerting rule for high CPU and configure Alertmanager.
# Rule
- alert: HighCPU
expr: avg by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m])) > 0.8
for: 10m
labels:
severity: warning
annotations:
summary: CPU > 80
3) Connect Grafana to Prometheus, import a “Node Exporter Full” dashboard, create environment and instance variables, and add annotations for deployments.
Production rollout checklist
– Standardized exporters and aligned versions.
– Consistent scrape interval and timeout, scraping health monitored.
– Recording rules for critical queries.
– Alerts with runbooks, team routing, configured silences and inhibiting.
– Dashboards with variables and annotations, correct Grafana permissions.
– Defined backup and retention; capacity plans for the TSDB.
– Policies on naming, labels, and rule reviews.
Conclusions
With Prometheus and Grafana you can build a robust observability system, able to turn raw signals into operational decisions. Focus on meaningful metrics, well-designed queries, and alerts that tell a clear story, linked to runbooks and SLOs. Scale prudently, protect endpoints, and manage everything as-code to avoid drift.
Put it into practice now
Do you want to bring these principles to your team? Start by creating a “Golden Signals” dashboard and two alerts focused on key SLOs. If you want a free review of your Prometheus/Grafana setup or a ready-to-use rule template, contact me: I will help you set up effective metrics, dashboards, and alerting right away.

Comments