Why can Grafana not reach Prometheus at localhost:9090?

Inside the Grafana container, localhost is the Grafana container itself. In Compose, use the service name, such as http://prometheus:9090.

What makes a good alert?

It should signal a real, actionable problem that someone needs to respond to, with enough context to act. Alerting on every minor fluctuation causes alert fatigue, so prefer symptoms users feel and use a for duration to avoid noise.

Grafana Dashboards and Prometheus Alert Rules

Introduction

Prometheus stores metrics, and Grafana turns them into dashboards people can read. Prometheus can also evaluate alert rules that tell you when something is wrong. This lesson adds Grafana to the Compose setup, provisions the data source from a file, builds a panel, and writes an alert rule.

Prerequisites

The Prometheus basics lesson with its Compose file running

Step-by-Step Guide

Step 1: Add Grafana to the Compose file

Add a Grafana service with a volume for its data and a folder mounted for data source provisioning. Inside Compose, services reach each other by service name, so Grafana will use the host name prometheus. Pin versions of both images in real projects.

services:
  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro

volumes:
  prometheus-data:
  grafana-data:

Step 2: Provision the data source as code

Instead of clicking through the interface, describe the data source in a YAML file in the provisioning folder, so the setup is repeatable and version controlled. A provisioned data source cannot be edited in the interface. The URL uses the Compose service name, because localhost inside the Grafana container would mean Grafana itself.

# grafana/provisioning/datasources/datasource.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Step 3: Sign in and build a first panel

Start the stack and open http://localhost:3000. Sign in with admin as both username and password, and change the password when prompted, which you should always do. Create a new dashboard, add a visualization, choose the Prometheus data source, and enter a query such as up or rate(prometheus_tsdb_head_chunks_created_total[1m]). Pick a visualization type, give the panel a title, and save the dashboard.

docker compose up -d
# open http://localhost:3000   (admin / admin, then change the password)

Step 4: Write an alert rule

Alert rules live in a rules file that Prometheus loads. This rule fires when a target has been down for five minutes. The for clause makes the alert pending first, which avoids reacting to a brief blip, and annotations can use template values from the alert's labels. Mount the file into the Prometheus container.

# alert.rules.yml
groups:
  - name: example
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "Instance {{ $labels.instance }} down"

Step 5: Load the rules and know where notifications come from

Reference the rules file in prometheus.yml and mount it in Compose. Prometheus then shows the alert as pending or firing on its Alerts page. Sending notifications by email or chat is the job of a separate component, Alertmanager, which receives alerts from Prometheus and handles grouping, silencing, and routing. Configure it by following the Alertmanager documentation.

# prometheus.yml (add)
rule_files:
  - "alert.rules.yml"

# compose.yaml, under prometheus volumes (add)
#   - ./alert.rules.yml:/etc/prometheus/alert.rules.yml:ro

FAQs

Why can Grafana not reach Prometheus at localhost:9090?: Inside the Grafana container, localhost is the Grafana container itself. In Compose, use the service name, such as http://prometheus:9090.
What makes a good alert?: It should signal a real, actionable problem that someone needs to respond to, with enough context to act. Alerting on every minor fluctuation causes alert fatigue, so prefer symptoms users feel and use a for duration to avoid noise.

Summary

Run Grafana beside Prometheus, provision the data source with a YAML file using the service name, build panels from PromQL queries, and define alert rules with a for duration. Alertmanager handles delivering the notifications.