Deploy a Complete Prometheus Monitoring Stack with Ansible

Why Deploy Monitoring as Code?

Monitoring is usually the last thing teams automate — and the first thing that drifts. Someone hand-edits prometheus.yml to add a host, someone else clicks together a Grafana dashboard that only exists on one server, and six months later nobody knows why alerts fire (or worse, why they don't).

Treating your monitoring stack as Ansible code fixes this at the root:

• Self-updating targets - Scrape configs are generated from your Ansible inventory. Add a host to a group, re-run the playbook, and Prometheus starts scraping it. No manual target lists.
• Reviewable alerts - Alert rules live in Git and go through pull requests, not midnight SSH sessions.
• Disaster recovery - Lose the monitoring server? One playbook run rebuilds everything: exporters, server, rules, and dashboards.
• Consistency - Every environment (dev, staging, prod) gets the same stack with only variables changing.

Architecture Overview

The stack has four components, each with a clear job:

• node_exporter (port 9100) - Runs on every host and exposes CPU, memory, disk, and network metrics over HTTP.
• Prometheus (port 9090) - The central server. Scrapes every exporter on an interval, stores time-series data, and evaluates alert rules.
• Alertmanager (port 9093) - Receives alerts from Prometheus, deduplicates and groups them, and routes notifications to Slack, email, or PagerDuty.
• Grafana (port 3000) - Dashboards. Queries Prometheus and visualizes everything.

In inventory terms, that maps to two groups: a monitoring group (one or two servers running Prometheus, Alertmanager, and Grafana) and everything else, which just gets node_exporter:

# inventory/hosts.ini
[monitoring]
monitor01.example.com

[webservers]
web01.example.com
web02.example.com

[databases]
db01.example.com

[all:vars]
ansible_user=ansible

Step 1: Deploy node_exporter Everywhere

Start with the exporters, since Prometheus is useless without something to scrape. We'll write a small role: download the binary, create a dedicated system user, install a systemd unit, and open the firewall.

Role Tasks

# roles/node_exporter/tasks/main.yml
---
- name: Create node_exporter system user
  ansible.builtin.user:
    name: node_exporter
    system: true
    shell: /usr/sbin/nologin
    create_home: false

- name: Download and unpack node_exporter
  ansible.builtin.unarchive:
    src: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/node_exporter-{{ node_exporter_version }}.linux-amd64.tar.gz"
    dest: /tmp
    remote_src: true
    creates: "/tmp/node_exporter-{{ node_exporter_version }}.linux-amd64"

- name: Install node_exporter binary
  ansible.builtin.copy:
    src: "/tmp/node_exporter-{{ node_exporter_version }}.linux-amd64/node_exporter"
    dest: /usr/local/bin/node_exporter
    remote_src: true
    owner: root
    group: root
    mode: "0755"
  notify: Restart node_exporter

- name: Deploy systemd unit
  ansible.builtin.template:
    src: node_exporter.service.j2
    dest: /etc/systemd/system/node_exporter.service
    mode: "0644"
  notify: Restart node_exporter

- name: Allow Prometheus server through the firewall
  ansible.posix.firewalld:
    port: 9100/tcp
    permanent: true
    immediate: true
    state: enabled
  when: ansible_facts['os_family'] == "RedHat"

- name: Enable and start node_exporter
  ansible.builtin.systemd:
    name: node_exporter
    state: started
    enabled: true
    daemon_reload: true

The systemd Unit Template

# roles/node_exporter/templates/node_exporter.service.j2
[Unit]
Description=Prometheus Node Exporter
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address={{ ansible_default_ipv4.address }}:9100
Restart=on-failure
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true

[Install]
WantedBy=multi-user.target

And the matching handler:

# roles/node_exporter/handlers/main.yml
---
- name: Restart node_exporter
  ansible.builtin.systemd:
    name: node_exporter
    state: restarted
    daemon_reload: true

Step 2: Deploy the Prometheus Server

This is where monitoring-as-code really shines. Instead of hardcoding a target list, we generate the scrape configuration from the Ansible inventory itself using Jinja2. Every host Ansible knows about automatically becomes a Prometheus target.

Templating prometheus.yml from Inventory Groups

# roles/prometheus/templates/prometheus.yml.j2
---
global:
  scrape_interval: {{ prometheus_scrape_interval | default('15s') }}
  evaluation_interval: 15s
  external_labels:
    environment: {{ deploy_env | default('production') }}

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - "localhost:9093"

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

{% for group in ['webservers', 'databases', 'monitoring'] %}
  - job_name: {{ group }}
    static_configs:
      - targets:
{% for host in groups[group] %}
          - "{{ hostvars[host]['ansible_default_ipv4']['address'] }}:9100"
{% endfor %}
        labels:
          group: {{ group }}
{% endfor %}

Read that middle section carefully — it loops over inventory groups, and inside each group it loops over every host, pulling the host's IP from gathered facts via hostvars. Add web03 to the [webservers] group tomorrow, re-run the playbook, and it's scraped and labeled with zero config edits. That single template is the payoff of the whole approach.

Prometheus Role Tasks (abridged)

# roles/prometheus/tasks/main.yml (key tasks)
---
- name: Create Prometheus directories
  ansible.builtin.file:
    path: "{{ item }}"
    state: directory
    owner: prometheus
    group: prometheus
    mode: "0755"
  loop:
    - /etc/prometheus
    - /etc/prometheus/rules
    - /var/lib/prometheus

- name: Deploy Prometheus configuration
  ansible.builtin.template:
    src: prometheus.yml.j2
    dest: /etc/prometheus/prometheus.yml
    owner: prometheus
    group: prometheus
    mode: "0644"
    validate: /usr/local/bin/promtool check config %s
  notify: Reload prometheus

Two details worth stealing: the validate parameter runs promtool against the rendered file before it replaces the live config, so a broken template can never take down monitoring. And the handler uses state: reloaded — Prometheus supports live config reload, so there's no need for a full restart:

# roles/prometheus/handlers/main.yml
---
- name: Reload prometheus
  ansible.builtin.systemd:
    name: prometheus
    state: reloaded

Step 3: Alertmanager and Alert Rules as Code

Alert rules are just YAML files, which makes them perfect Ansible payloads. Keep them in the role's files/ directory so they're diffable in Git:

# roles/prometheus/files/rules/node_alerts.yml
---
groups:
  - name: node_alerts
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} has been unreachable for 2 minutes."

      - alert: HighDiskUsage
        expr: |
          (node_filesystem_avail_bytes{mountpoint="/"}
           / node_filesystem_size_bytes{mountpoint="/"}) < 0.10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"

      - alert: HighMemoryUsage
        expr: |
          (1 - node_memory_MemAvailable_bytes
             / node_memory_MemTotal_bytes) > 0.90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Memory above 90% on {{ $labels.instance }}"

Ship the rules and Alertmanager routing config:

# roles/alertmanager/tasks/main.yml (key tasks)
---
- name: Deploy alert rules (copy, NOT template!)
  ansible.builtin.copy:
    src: rules/
    dest: /etc/prometheus/rules/
    owner: prometheus
    mode: "0644"
  notify: Reload prometheus

- name: Deploy Alertmanager configuration
  ansible.builtin.template:
    src: alertmanager.yml.j2
    dest: /etc/alertmanager/alertmanager.yml
    mode: "0640"
  notify: Restart alertmanager

# roles/alertmanager/templates/alertmanager.yml.j2
---
route:
  receiver: slack-default
  group_by: [alertname, group]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: pagerduty-oncall

receivers:
  - name: slack-default
    slack_configs:
      - api_url: "{{ vault_slack_webhook_url }}"
        channel: "#alerts"
        send_resolved: true

  - name: pagerduty-oncall
    pagerduty_configs:
      - service_key: "{{ vault_pagerduty_key }}"

Notice the webhook URL and PagerDuty key come from Ansible Vault variables — secrets never land in Git as plaintext.

Step 4: Grafana Provisioning — Dashboards from Files

Grafana has a killer feature for automation: provisioning. Drop YAML files into /etc/grafana/provisioning/ and Grafana configures datasources and loads dashboards on startup — no clicking, no API calls.

Provision the Prometheus Datasource

# roles/grafana/templates/datasource.yml.j2
---
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true
    editable: false

Provision Dashboards

# roles/grafana/templates/dashboards.yml.j2
---
apiVersion: 1
providers:
  - name: ansible-managed
    folder: Infrastructure
    type: file
    disableDeletion: true
    options:
      path: /var/lib/grafana/dashboards

Then copy dashboard JSON files (export any dashboard from a Grafana UI once, commit the JSON) into place:

# roles/grafana/tasks/main.yml (key tasks)
---
- name: Install Grafana
  ansible.builtin.package:
    name: grafana
    state: present

- name: Provision datasources and dashboard providers
  ansible.builtin.template:
    src: "{{ item.src }}"
    dest: "{{ item.dest }}"
    mode: "0640"
  loop:
    - { src: datasource.yml.j2, dest: /etc/grafana/provisioning/datasources/prometheus.yml }
    - { src: dashboards.yml.j2, dest: /etc/grafana/provisioning/dashboards/ansible.yml }
  notify: Restart grafana

- name: Deploy dashboard JSON files
  ansible.builtin.copy:
    src: dashboards/
    dest: /var/lib/grafana/dashboards/
    owner: grafana
    mode: "0640"
  notify: Restart grafana

A great starting point is the community "Node Exporter Full" dashboard (Grafana.com ID 1860) — download its JSON once and commit it to roles/grafana/files/dashboards/.

Shortcut: The prometheus.prometheus Galaxy Collection

Don't want to maintain your own roles? The official prometheus.prometheus collection (successor to the popular cloudalchemy roles) packages battle-tested roles for the entire stack:

# Install the collection
ansible-galaxy collection install prometheus.prometheus

---
- name: Exporters everywhere
  hosts: all
  become: true
  roles:
    - prometheus.prometheus.node_exporter

- name: Monitoring stack
  hosts: monitoring
  become: true
  roles:
    - role: prometheus.prometheus.prometheus
      vars:
        prometheus_scrape_configs:
          - job_name: node
            static_configs:
              - targets: "{{ groups['all'] | map('regex_replace', '$', ':9100') | list }}"
    - prometheus.prometheus.alertmanager

The same inventory-driven trick works here: the map filter turns your entire inventory into a target list in one line. The collection handles versions, checksums, users, and systemd units for you — writing your own roles is still worth doing once to understand what's happening underneath.

Tying It Together: One site.yml

# site.yml
---
- name: Deploy node_exporter to all hosts
  hosts: all
  become: true
  roles:
    - node_exporter

- name: Deploy the monitoring stack
  hosts: monitoring
  become: true
  roles:
    - prometheus
    - alertmanager
    - grafana

# One command, whole stack
ansible-playbook -i inventory/hosts.ini site.yml --ask-vault-pass

Verifying the Deployment

Don't trust green Ansible output alone — verify the stack actually sees your hosts. You can automate that check too:

# verify.yml
---
- name: Verify all Prometheus targets are up
  hosts: monitoring
  tasks:
    - name: Query the Prometheus targets API
      ansible.builtin.uri:
        url: http://localhost:9090/api/v1/targets
        return_content: true
      register: targets

    - name: Assert every target is healthy
      ansible.builtin.assert:
        that:
          - targets.json.data.activeTargets
            | selectattr('health', 'ne', 'up') | list | length == 0
        fail_msg: >-
          Unhealthy targets:
          {{ targets.json.data.activeTargets
             | selectattr('health', 'ne', 'up')
             | map(attribute='labels.instance') | list }}
        success_msg: "All {{ targets.json.data.activeTargets | length }} targets are up."

Manual spot checks if you prefer:

# Is a node exporting metrics?
curl -s http://web01.example.com:9100/metrics | head

# Prometheus UI: http://monitor01:9090/targets  (all should be UP)
# Run the query:  up
# Grafana:        http://monitor01:3000  (dashboards pre-loaded)

Bonus: Close the Loop with Event-Driven Ansible

Here's where it gets fun. Alertmanager can POST alerts to a webhook — and Event-Driven Ansible can listen on that webhook and run remediation playbooks automatically. Disk filling up? A rulebook triggers a cleanup playbook. Service down? EDA restarts it and only pages a human if that fails.

# rulebooks/auto-remediate.yml
---
- name: Remediate Prometheus alerts
  hosts: all
  sources:
    - ansible.eda.alertmanager:
        host: 0.0.0.0
        port: 5050
  rules:
    - name: Restart a service that went down
      condition: event.alert.labels.alertname == "InstanceDown"
      action:
        run_playbook:
          name: playbooks/restart_service.yml

Your monitoring stack stops being a passive dashboard and becomes a self-healing feedback loop — deployed, configured, and now acted upon entirely as code. We cover this pattern in depth in our Event-Driven Ansible guide.

Conclusion

You now have a complete monitoring stack where every moving part is code: exporters rolled out by a role, scrape configs generated from inventory groups, alert rules reviewed in pull requests, dashboards provisioned from JSON, and a single site.yml that rebuilds it all from scratch. The pattern generalizes — the inventory-driven templating trick works just as well for HAProxy backends, WireGuard peers, or anything else that needs to know about your fleet.