New to KubeDB? Please start here.

RabbitMQ Alerting with Prometheus

This tutorial shows you how to configure Prometheus-based alerting for a KubeDB-managed RabbitMQ instance using the rabbitmq-alerts Helm chart. This chart also bundles a Grafana dashboard that it imports automatically through a post-install Job — no separate dashboard chart is required.

Before You Begin

• Ensure you have a Kubernetes cluster and that kubectl is configured to communicate with it. If you do not already have a cluster, you can create one using kind.

• Install the KubeDB operator by following the steps here.

• Deploy the database in the alert-rabbitmq namespace:

  $ kubectl create ns alert-rabbitmq
  namespace/alert-rabbitmq created
  

• To learn more about how Prometheus monitoring works with KubeDB, see the overview here.

• You will also need a Grafana API key / token with Editor permission so the chart’s dashboard-import Job can push the dashboard. See Step 1 below.

Note: YAML files used in this tutorial are stored in docs/examples/rabbitmq folder in GitHub repository kubedb/docs.

Configuration

Step 1 (kube-prometheus-stack) is required to follow this tutorial. Step 2 (Panopticon) is required for the Provisioner Group alerts below (KubeDBRabbitMQPhase...) — skip it only if you just want the exporter-based Database Group alerts. If you have already completed the step(s) you need in another guide, skip ahead.

Step 1: Deploy kube-prometheus-stack

kube-prometheus-stack installs Prometheus, Prometheus Operator, Alertmanager, and Grafana together. This is the recommended way to get the full monitoring stack on Kubernetes.

Add the prometheus-community Helm repo and install:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update

$ helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set grafana.image.tag=7.5.5

Wait for all pods to be ready:

$ kubectl get pods -n monitoring
NAME                                                   READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2     Running   0          2m
prometheus-grafana-xxxx                                3/3     Running   0          2m
prometheus-kube-prometheus-operator-xxxx               1/1     Running   0          2m
prometheus-kube-prometheus-prometheus-0                2/2     Running   0          2m
prometheus-kube-state-metrics-xxxx                     1/1     Running   0          2m

Find the serviceMonitorSelector/ruleSelector labels that Prometheus uses to pick up ServiceMonitor/PrometheusRule objects — this is the release: prometheus label used throughout this tutorial.

$ kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.ruleSelector}'
{"matchLabels":{"release":"prometheus"}}

$ kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.serviceMonitorSelector}'
{"matchLabels":{"release":"prometheus"}}

Step 2: Install Panopticon (required for the Provisioner Group alerts)

Panopticon is the Appscode operator that exports the KubeDB operator’s own view of every resource — kubedb_com_rabbitmq_status_phase and related metrics. It’s what powers the Provisioner Group alerts below (KubeDBRabbitMQPhaseNotReady/KubeDBRabbitMQPhaseCritical). Skip this step if you only need the exporter-based Database Group alerts.

$ helm repo add appscode https://charts.appscode.com/stable/
$ helm repo update

$ helm upgrade --install panopticon appscode/panopticon \
  --version v2026.4.30 \
  --namespace kubeops --create-namespace \
  --set monitoring.enabled=true \
  --set monitoring.agent=prometheus.io/operator \
  --set monitoring.serviceMonitor.labels.release=prometheus \
  --set-file license=/path/to/kubedb-license.txt \
  --wait --timeout 5m0s

Verify Panopticon is running:

$ kubectl get pods -n kubeops
NAME                          READY   STATUS    RESTARTS   AGE
panopticon-xxxx               1/1     Running   0          1m

Overview

• KubeDB deploys RabbitMQ with the built-in rabbitmq_prometheus plugin enabled, which serves metrics directly from the rabbitmq container on port 15692 — unlike some other databases, RabbitMQ needs no separate exporter sidecar.
• ServiceMonitor (named {rabbitmq-name}-stats) is created automatically by KubeDB and tells Prometheus to scrape the metrics endpoint every 10 seconds.
• KubeDB operator (panopticon) also exposes the CR’s own status as a metric, kubedb_com_rabbitmq_status_phase. The RabbitMQDown and provisioner-group alerts key off this metric instead of the database’s own stats endpoint, so they fire purely based on what KubeDB itself observes about the resource — even if the metrics scrape target is otherwise healthy.
• PrometheusRule is created by the rabbitmq-alerts chart and contains RabbitMQ alert definitions grouped by concern: database health and provisioner.
• Dashboard-import Job — when grafana.enabled is true, the chart also creates a one-shot Job that POSTs a bundled dashboard JSON straight to your Grafana instance’s /api/dashboards/import endpoint.
• Prometheus Operator evaluates every rule expression every 30 seconds and fires matching alerts to AlertManager.
• AlertManager groups, inhibits, and silences alerts, then routes them to configured receivers (Slack, email, PagerDuty, webhook, etc.).

Deploy RabbitMQ with Monitoring Enabled

At first, let’s deploy a RabbitMQ database with monitoring enabled. Below is the RabbitMQ object we are going to create.

apiVersion: kubedb.com/v1alpha2
kind: RabbitMQ
metadata:
  name: rmq-alert-demo
  namespace: alert-rabbitmq
spec:
  version: "4.2.4"
  replicas: 3
  deletionPolicy: WipeOut
  storage:
    storageClassName: "local-path"
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi
  monitor:
    agent: prometheus.io/operator
    prometheus:
      serviceMonitor:
        labels:
          release: prometheus
        interval: 10s

Here,

• spec.monitor.agent: prometheus.io/operator tells KubeDB to create a ServiceMonitor resource managed by the Prometheus operator.
• spec.monitor.prometheus.serviceMonitor.labels.release: prometheus adds the release: prometheus label to the created ServiceMonitor, matching the Prometheus serviceMonitorSelector so the target is discovered automatically.

Let’s create the RabbitMQ resource.

$ kubectl apply -f https://github.com/kubedb/docs/raw/v2026.7.10/docs/examples/rabbitmq/monitoring/rmq-alert-demo.yaml
rabbitmq.kubedb.com/rmq-alert-demo created

Now, wait for the database to go into Ready state.

$ kubectl get rabbitmq -n alert-rabbitmq rmq-alert-demo
NAME             VERSION   STATUS   AGE
rmq-alert-demo   4.2.4     Ready    5m

KubeDB brings up 3 pods, one per RabbitMQ node:

$ kubectl get pods -n alert-rabbitmq
NAME               READY   STATUS    RESTARTS   AGE
rmq-alert-demo-0   1/1     Running   0          5m
rmq-alert-demo-1   1/1     Running   0          4m
rmq-alert-demo-2   1/1     Running   0          4m

KubeDB creates a dedicated stats service with the -stats suffix for monitoring.

$ kubectl get svc -n alert-rabbitmq --selector="app.kubernetes.io/instance=rmq-alert-demo"
NAME                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                           AGE
rmq-alert-demo               ClusterIP   10.43.174.187   <none>        5672/TCP,1883/TCP,61613/TCP,15675/TCP,15674/TCP   59s
rmq-alert-demo-dashboard     ClusterIP   10.43.236.58    <none>        15672/TCP                                         59s
rmq-alert-demo-pods          ClusterIP   None            <none>        4369/TCP,25672/TCP                                59s
rmq-alert-demo-stats         ClusterIP   10.43.128.229   <none>        15692/TCP                                         59s

KubeDB also creates a ServiceMonitor that tells Prometheus where to scrape.

$ kubectl get servicemonitor -n alert-rabbitmq
NAME                    AGE
rmq-alert-demo-stats    55s

Verify that the ServiceMonitor carries the release: prometheus label so Prometheus discovers it.

$ kubectl get servicemonitor -n alert-rabbitmq rmq-alert-demo-stats \
    -o jsonpath='{.metadata.labels.release}'
prometheus

Step 1 — Create a Grafana API Key

The chart’s dashboard-import Job authenticates to Grafana with a bearer token, so create one first.

• Grafana 9+: Administration → Service accounts → Add service account → role Editor → Add token. Copy the token.

• Grafana 8.x and earlier (no Service Accounts UI, e.g. the bundled kube-prometheus-stack Grafana 7.5.5): use the legacy API Keys endpoint instead:

  # Port-forward Grafana
  $ kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80&
  
  # Retrieve the admin password
  $ kubectl get secret -n monitoring prometheus-grafana \
      -o jsonpath='{.data.admin-password}' | base64 -d && echo
  
  # Create an API key with Editor role
  $ curl -s -X POST -H "Content-Type: application/json" \
      -u admin:<grafana_password> \
      http://localhost:3000/api/auth/keys \
      -d '{"name":"rabbitmq-alerts-demo","role":"Editor"}'
  # Note the returned "key"
  
  # Stop the port-forward
  $ kill %1
  

Either way, you end up with a bearer token to use as grafana.apikey below.

Step 2 — Install rabbitmq-alerts

The rabbitmq-alerts chart creates a PrometheusRule resource containing RabbitMQ alert definitions grouped by concern: database health and provisioner.

Why the Helm release name matters

The chart derives the PromQL job/instance scoping (and the PrometheusRule name) from the Helm release name, not from a values field — so the release name must match the RabbitMQ object’s name (rmq-alert-demo) for the rules to be correctly scoped to this instance.

The chart’s default label is release: kube-prometheus-stack, so we must also override it at install time to match the Prometheus ruleSelector.

Install

$ helm upgrade -i rmq-alert-demo appscode/rabbitmq-alerts \
    -n alert-rabbitmq \
    --create-namespace \
    --version=v2026.7.14 \
    --set form.alert.labels.release=prometheus \
    --set grafana.enabled=true \
    --set grafana.url="http://prometheus-grafana.monitoring.svc:80" \
    --set grafana.apikey="<token-from-above>" \
    --set grafana.jobName=rmq-alert-demo-stats \
    --set form.alert.appSuffix=rmq-grafana-demo

To install alerts only, without the dashboard, omit the grafana.* flags (or set --set grafana.enabled=false).

Verify the PrometheusRule is created

$ kubectl get prometheusrule -n alert-rabbitmq
NAME             AGE
rmq-alert-demo   30s

Confirm the release: prometheus label is present.

$ kubectl get prometheusrule -n alert-rabbitmq rmq-alert-demo \
    -o jsonpath='{.metadata.labels.release}'
prometheus

Verify the dashboard-import Job

$ kubectl get job -n alert-rabbitmq
NAME                     STATUS     COMPLETIONS   AGE
rmq-alert-demo-post-job  Complete   1/1           17s

$ kubectl logs -n alert-rabbitmq job/rmq-alert-demo-post-job
{"pluginId":"","title":"kubedb.com / RabbitMQ / alert-rabbitmq / rmq-alert-demo","imported":true, ...}

A "imported":true response confirms the dashboard kubedb.com / RabbitMQ / alert-rabbitmq / rmq-alert-demo now exists in Grafana.

Confirm Prometheus loaded the rules

Port-forward the Prometheus UI and open the Status → Rule health page.

$ kubectl port-forward -n monitoring \
    svc/prometheus-kube-prometheus-prometheus 9090:9090

Open http://localhost:9090/rules?search=rabbitmq.

The rabbitmq.database.alert-rabbitmq.rmq-alert-demo.rules group is visible with all rules showing OK, confirming that Prometheus has loaded and is evaluating the RabbitMQ alert definitions every 30 seconds.

Verify End-to-End

1. Check the metrics endpoint

Because RabbitMQ’s Prometheus plugin runs inside the rabbitmq container itself, there is no separate exporter container to check — query the plugin’s own endpoint directly.

$ kubectl exec -n alert-rabbitmq rmq-alert-demo-0 -c rabbitmq -- \

                                      wget -qO- http://127.0.0.1:15692/metrics | grep rabbitmq_identity_info
# TYPE rabbitmq_identity_info untyped
# HELP rabbitmq_identity_info RabbitMQ node & cluster identity info
rabbitmq_identity_info{rabbitmq_node="rabbit@rmq-alert-demo-0.rmq-alert-demo-pods.alert-rabbitmq",rabbitmq_cluster="rmq-alert-demo",rabbitmq_cluster_permanent_id="rabbitmq-cluster-id-lcFtUYEzj3-MLTpL-uEbcg",rabbitmq_endpoint="aggregated"} 1

2. Check the Prometheus target is UP

Open http://localhost:9090/targets.

The target serviceMonitor/alert-rabbitmq/rmq-alert-demo-stats/0 shows 3 / 3 up, confirming metrics are being scraped from all three pods — rmq-alert-demo-0, rmq-alert-demo-1, and rmq-alert-demo-2 — in the alert-rabbitmq namespace.

3. Confirm the RabbitMQ alerts are inactive

Open http://localhost:9090/alerts?search=rabbitmq to see the RabbitMQ alert groups.

All 11 rules in the rabbitmq.database group and both rules in the rabbitmq.provisioner group show INACTIVE, meaning the 3-node cluster is healthy and no thresholds are breached.

Note: this cluster uses the local-path storage class, whose PVCs are just directories on the node’s root filesystem rather than isolated volumes — so DiskUsageHigh/DiskAlmostFull (kubelet_volume_stats_used_bytes) can occasionally read the node’s actual disk usage instead of the small demo volume’s own usage, showing PENDING/FIRING even though the RabbitMQ data volume itself is nearly empty. Not observed in this run, but worth knowing if your own result differs from the screenshot above.

4. Check AlertManager

Port-forward AlertManager to view any currently firing alerts.

$ kubectl port-forward -n monitoring \
    svc/prometheus-kube-prometheus-alertmanager 9093:9093

Open http://localhost:9093. PENDING rules have not yet fired — only alerts that cross into FIRING are forwarded to AlertManager — so with a healthy RabbitMQ instance no alerts for rmq-alert-demo are listed here yet.

Simulating a Firing Alert

The previous section confirmed that the RabbitMQ alerts are healthy. This section walks through deliberately triggering the RabbitMQPhaseCritical alert so you can observe the full alert lifecycle — from firing in Prometheus through to the AlertManager dashboard — and then resolve it.

On a 3-node cluster, crashing a single pod degrades the cluster rather than taking it fully down — the KubeDB operator moves the resource’s status.phase to Critical (one or more nodes unhealthy, but the remaining nodes keep serving) rather than NotReady (which needs a majority/all of the nodes down). RabbitMQPhaseCritical is therefore the realistic alert to demonstrate here; RabbitMQDown/KubeDBRabbitMQPhaseNotReady would need all 3 nodes crashed simultaneously and held down.

1. Crash one RabbitMQ node repeatedly

Kill the RabbitMQ process inside rmq-alert-demo-0. A lone kill 1 restarts fast enough that the container becomes Ready again before KubeDB’s health check can even observe the outage, and RabbitMQPhaseCritical needs the phase held at Critical for a full for: 3m — so keep the pod crash-looping for several minutes, not just a handful of kills.

$ end=$(( $(date +%s) + 240 ))
  while [ $(date +%s) -lt $end ]; do
    kubectl exec -n alert-rabbitmq rmq-alert-demo-0 -c rabbitmq -- kill 1 >/dev/null 2>&1
    sleep 5
  done

Watch the CR phase move from Ready to Critical:

$ kubectl get rabbitmq -n alert-rabbitmq rmq-alert-demo -o jsonpath='{.status.phase}'
Critical

RabbitMQPhaseCritical keys off kubedb_com_rabbitmq_status_phase (a metric emitted by the KubeDB operator itself), so what matters is the CR’s status.phase staying at Critical continuously, not the exporter’s own scrape health — if the pod recovers even briefly between kills, the for: 3m timer resets and you’ll see the rule flip back to PENDING. Let the loop run for the full 4 minutes above before checking.

2. Watch the alert fire in Prometheus

Open http://localhost:9090/alerts?search=rabbitmq.

RabbitMQPhaseCritical moves from INACTIVE to FIRING once its for: 3m window elapses with the phase held at Critical, while the rest of the rabbitmq.database group stays INACTIVE. The provisioner-group KubeDBRabbitMQPhaseNotReady alert only reaches PENDING in this scenario — it needs the operator to view the resource as fully NotReady, which a single crashed node in a 3-node cluster doesn’t trigger.

3. Check the AlertManager dashboard

Open http://localhost:9093/#/alerts?filter=%7Bnamespace%3D%22alert-rabbitmq%22%7D.

AlertManager shows the RabbitMQPhaseCritical alert. The alert card displays labels including:

• alertname: RabbitMQPhaseCritical
• severity: warning
• app: rmq-alert-demo, app_namespace: alert-rabbitmq
• phase: Critical
• k8s_kind: RabbitMQ

Note that the instance/pod/job labels on this alert point at the KubeDB operator’s panopticon component (job="panopticon"), not at the RabbitMQ pod itself — because this alert is derived from the operator’s own status metric rather than from the database’s stats endpoint.

AlertManager routes this alert to every receiver configured in your alertmanagerConfig (Slack, email, PagerDuty, webhook, etc.) based on your routing tree. If no receiver is configured, the alert is visible here but silently dropped.

4. Restore RabbitMQ

Delete the pod so KubeDB recreates it cleanly.

$ kubectl delete pod -n alert-rabbitmq rmq-alert-demo-0
pod "rmq-alert-demo-0" deleted

Once status.phase returns to Ready, Prometheus marks both alerts INACTIVE again and AlertManager sends a resolved notification to all receivers.

Alert Reference

All alerts are scoped to the rmq-alert-demo instance in the alert-rabbitmq namespace via the PromQL label filters job="rmq-alert-demo-stats" / app="rmq-alert-demo" and namespace="alert-rabbitmq".

Database Group

Fired based on live metrics from RabbitMQ’s rabbitmq_prometheus plugin, plus the two persistent-volume rules and the two KubeDB status-phase rules.

Provisioner Group

Monitors the KubeDB operator’s view of the RabbitMQ resource phase.

Customising Alerts

To override thresholds or disable specific alert groups, create a custom values file and upgrade the chart.

# custom-alerts.yaml
form:
  alert:
    labels:
      release: prometheus
    groups:
      database:
        enabled: warning
        rules:
          rabbitmqHighConnectionChurn:
            enabled: true
            duration: "2m"
            severity: warning
          diskUsageHigh:
            enabled: true
            val: 90        # fire at 90% disk usage instead of the default 80%
            duration: "5m"
            severity: warning
      provisioner:
        enabled: "none"    # disable all provisioner alerts

$ helm upgrade rmq-alert-demo appscode/rabbitmq-alerts \
    -n alert-rabbitmq \
    --version=v2026.7.14 \
    -f custom-alerts.yaml

Cleaning up

To remove all resources created in this tutorial, run the following commands.

# Remove the rabbitmq-alerts release (PrometheusRule + dashboard-import Job)
$ helm uninstall rmq-alert-demo -n alert-rabbitmq

# Remove the imported Grafana dashboard (it is not removed by helm uninstall)
$ curl -s -X DELETE -H "Authorization: Bearer <grafana-token>" \
    http://localhost:3000/api/dashboards/uid/<uid>

# Remove the RabbitMQ instance
$ kubectl delete rabbitmq -n alert-rabbitmq rmq-alert-demo

# Delete namespace
$ kubectl delete ns alert-rabbitmq

# Uninstall monitoring stack (optional — skip if other tutorials on this cluster still need them)
$ helm uninstall panopticon -n kubeops
$ helm uninstall prometheus -n monitoring

Next Steps

• Monitor your RabbitMQ database with KubeDB using builtin Prometheus.
• Monitor your RabbitMQ database with KubeDB using Prometheus operator.
• Detail concepts of RabbitMQ object.
• Want to hack on KubeDB? Check our contribution guidelines.