New to KubeDB? Please start here.

PerconaXtraDB Alerting with Prometheus

This tutorial shows you how to configure Prometheus-based alerting for a KubeDB-managed PerconaXtraDB instance using the perconaxtradb-alerts Helm chart.

Before You Begin

• Ensure you have a Kubernetes cluster and that kubectl is configured to communicate with it. If you do not already have a cluster, you can create one using kind.

• Install the KubeDB operator by following the steps here.

• Deploy the database in the alert-perconaxtradb namespace:

  $ kubectl create ns alert-perconaxtradb
  namespace/alert-perconaxtradb created
  

• To learn more about how Prometheus monitoring works with KubeDB, see the overview here.

Note: YAML files used in this tutorial are stored in docs/examples/percona-xtradb folder in GitHub repository kubedb/docs.

Configuration

Step 1 (kube-prometheus-stack) is required to follow this tutorial. Step 2 (Panopticon) is required for the Provisioner Group alerts below (KubeDBPerconaXtraDBPhase...) — skip it only if you just want the exporter-based Database Group alerts. If you have already completed the step(s) you need in another guide, skip ahead.

Step 1: Deploy kube-prometheus-stack

kube-prometheus-stack installs Prometheus, Prometheus Operator, Alertmanager, and Grafana together. This is the recommended way to get the full monitoring stack on Kubernetes.

Add the prometheus-community Helm repo and install:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update

$ helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set grafana.image.tag=7.5.5

Wait for all pods to be ready:

$ kubectl get pods -n monitoring
NAME                                                   READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2     Running   0          2m
prometheus-grafana-xxxx                                3/3     Running   0          2m
prometheus-kube-prometheus-operator-xxxx               1/1     Running   0          2m
prometheus-kube-prometheus-prometheus-0                2/2     Running   0          2m
prometheus-kube-state-metrics-xxxx                     1/1     Running   0          2m

Find the serviceMonitorSelector/ruleSelector labels that Prometheus uses to pick up ServiceMonitor/PrometheusRule objects — this is the release: prometheus label used throughout this tutorial.

$ kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.ruleSelector}'
{"matchLabels":{"release":"prometheus"}}

$ kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.serviceMonitorSelector}'
{"matchLabels":{"release":"prometheus"}}

Step 2: Install Panopticon (required for the Provisioner Group alerts)

Panopticon is the Appscode operator that exports the KubeDB operator’s own view of every resource — kubedb_com_perconaxtradb_status_phase and related metrics. It’s what powers the Provisioner Group alerts below (KubeDBPerconaXtraDBPhaseNotReady/KubeDBPerconaXtraDBPhaseCritical). Skip this step if you only need the exporter-based Database Group alerts.

$ helm repo add appscode https://charts.appscode.com/stable/
$ helm repo update

$ helm upgrade --install panopticon appscode/panopticon \
  --version v2026.4.30 \
  --namespace kubeops --create-namespace \
  --set monitoring.enabled=true \
  --set monitoring.agent=prometheus.io/operator \
  --set monitoring.serviceMonitor.labels.release=prometheus \
  --set-file license=/path/to/kubedb-license.txt \
  --wait --timeout 5m0s

Verify Panopticon is running:

$ kubectl get pods -n kubeops
NAME                          READY   STATUS    RESTARTS   AGE
panopticon-xxxx               1/1     Running   0          1m

Overview

• KubeDB deploys PerconaXtraDB with a mysqld_exporter-compatible sidecar (container exporter) that exposes metrics (mysql_*), the same exporter family used by MySQL/MariaDB.
• ServiceMonitor (named {perconaxtradb-name}-stats) is created automatically by KubeDB and tells Prometheus to scrape the exporter every 10 seconds.
• PrometheusRule is created by the perconaxtradb-alerts chart and contains alert definitions grouped by concern: database health, Galera cluster, provisioner, ops-manager, Stash backup/restore, and schema manager.
• Grafana dashboards for PerconaXtraDB can be provisioned via the kubedb-grafana-dashboards chart (--set featureGates.PerconaXtraDB=true) rather than duplicated here.
• Prometheus Operator evaluates every rule expression every 30 seconds and fires matching alerts to AlertManager.
• AlertManager groups, inhibits, and silences alerts, then routes them to configured receivers (Slack, email, PagerDuty, webhook, etc.).

Deploy PerconaXtraDB with Monitoring Enabled

Below is the PerconaXtraDB object we are going to create — a single standalone instance with monitoring enabled. (The chart’s cluster group only produces data if you deploy a Galera cluster via spec.topology; a standalone instance simply leaves that alert INACTIVE.)

apiVersion: kubedb.com/v1
kind: PerconaXtraDB
metadata:
  name: perconaxtradb-alert-demo
  namespace: alert-perconaxtradb
spec:
  version: "8.0.40"
  storageType: Durable
  storage:
    storageClassName: "local-path"
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi
  deletionPolicy: WipeOut
  monitor:
    agent: prometheus.io/operator
    prometheus:
      serviceMonitor:
        labels:
          release: prometheus
        interval: 10s

Here,

• spec.monitor.agent: prometheus.io/operator tells KubeDB to create a ServiceMonitor resource managed by the Prometheus operator.
• spec.monitor.prometheus.serviceMonitor.labels.release: prometheus adds the release: prometheus label to the created ServiceMonitor, matching the Prometheus serviceMonitorSelector so the target is discovered automatically.

Let’s create the PerconaXtraDB resource.

$ kubectl apply -f https://github.com/kubedb/docs/raw/v2026.7.10/docs/examples/percona-xtradb/monitoring/perconaxtradb-alert-demo.yaml
perconaxtradb.kubedb.com/perconaxtradb-alert-demo created

Now, wait for the database to go into Ready state.

$ kubectl get perconaxtradb -n alert-perconaxtradb perconaxtradb-alert-demo
NAME                       VERSION   STATUS   AGE
perconaxtradb-alert-demo   8.0.40    Ready    3m

KubeDB creates a dedicated stats service with the -stats suffix for monitoring.

$ kubectl get svc -n alert-perconaxtradb --selector="app.kubernetes.io/instance=perconaxtradb-alert-demo"
NAME                                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
perconaxtradb-alert-demo               ClusterIP   10.43.10.20    <none>        3306/TCP    3m
perconaxtradb-alert-demo-pods          ClusterIP   None           <none>        3306/TCP    3m
perconaxtradb-alert-demo-stats         ClusterIP   10.43.10.21    <none>        56790/TCP   3m

KubeDB also creates a ServiceMonitor that tells Prometheus where to scrape.

$ kubectl get servicemonitor -n alert-perconaxtradb
NAME                             AGE
perconaxtradb-alert-demo-stats   3m

Verify that the ServiceMonitor carries the release: prometheus label so Prometheus discovers it.

$ kubectl get servicemonitor -n alert-perconaxtradb perconaxtradb-alert-demo-stats \
    -o jsonpath='{.metadata.labels.release}'
prometheus

Step 1 — Install perconaxtradb-alerts

Why the Helm release name matters

The chart derives the PrometheusRule name and scopes every PromQL expression (via job="{release-name}-stats" / app="{release-name}") from the Helm release name — so the release name must match the PerconaXtraDB object’s name (perconaxtradb-alert-demo). Note the chart/repo name has no hyphen (perconaxtradb-alerts), even though the KubeDB guides directory uses percona-xtradb.

Install

$ helm upgrade -i perconaxtradb-alert-demo oci://ghcr.io/appscode-charts/perconaxtradb-alerts \
    -n alert-perconaxtradb \
    --create-namespace \
    --version=v2026.7.14 \
    --set form.alert.labels.release=prometheus \
    --set form.alert.appSuffix=pxdb-grafana-demo

Verify the PrometheusRule is created

$ kubectl get prometheusrule -n alert-perconaxtradb
NAME                       AGE
perconaxtradb-alert-demo   30s

$ kubectl get prometheusrule -n alert-perconaxtradb perconaxtradb-alert-demo \
    -o jsonpath='{.metadata.labels.release}'
prometheus

Confirm Prometheus loaded the rules

$ kubectl port-forward -n monitoring \
    svc/prometheus-kube-prometheus-prometheus 9090:9090

Open http://localhost:9090/rules and locate the perconaxtradb.database, perconaxtradb.cluster, perconaxtradb.provisioner, perconaxtradb.opsManager, perconaxtradb.stash, and perconaxtradb.schemaManager groups.

All groups should show OK. Unlike MariaDB’s chart, perconaxtradb-alerts v2026.7.14 has a stash group but no kubeStash group — every group it does declare in values.yaml renders correctly.

Verify End-to-End

1. Check the Prometheus target is UP

Open http://localhost:9090/query?g0.expr=up%7Bnamespace%3D%22alert-perconaxtradb%22%7D&g0.tab=1.

2. Confirm all PerconaXtraDB alerts are inactive

Open http://localhost:9090/alerts.

All rules should show INACTIVE. GaleraReplicationLatencyTooLong has no data on a standalone instance.

3. Check AlertManager

$ kubectl port-forward -n monitoring \
    svc/prometheus-kube-prometheus-alertmanager 9093:9093

Open http://localhost:9093.

4. Explore the Grafana dashboard

Provision the PerconaXtraDB dashboards via the kubedb-grafana-dashboards chart (--set featureGates.PerconaXtraDB=true), then explore them in Grafana under Dashboards.

Simulating a Firing Alert

This section deliberately triggers PerconaXtraDBInstanceDown (instant, for: 0m) by crashing the main database process.

1. Crash the PerconaXtraDB process

$ kubectl exec -n alert-perconaxtradb perconaxtradb-alert-demo-0 -c perconaxtradb -- sh -c '
    end=$(( $(date +%s) + 30 ));
    while [ $(date +%s) -lt $end ]; do
      pid=$(pgrep -x mysqld | head -1);
      [ -n "$pid" ] && kill -9 "$pid" 2>/dev/null;
      sleep 1;
    done'

2. Watch the alert fire in Prometheus

Open http://localhost:9090/alerts.

PerconaXtraDBInstanceDown (mysql_up == 0) should transition straight to FIRING.

3. Check the AlertManager dashboard

Open http://localhost:9093.

4. Restore PerconaXtraDB

Stop the loop from step 1.

$ kubectl get perconaxtradb -n alert-perconaxtradb perconaxtradb-alert-demo -w
NAME                       VERSION   STATUS   AGE
perconaxtradb-alert-demo   8.0.40    Ready    24m

If PerconaXtraDB does not recover on its own within a minute or two, force a clean restart: kubectl delete pod -n alert-perconaxtradb perconaxtradb-alert-demo-0.

Alert Reference

All alerts are scoped to the perconaxtradb-alert-demo instance in the alert-perconaxtradb namespace via the PromQL label filters job="perconaxtradb-alert-demo-stats" / namespace="alert-perconaxtradb" (database/cluster groups), or app="perconaxtradb-alert-demo" / namespace="alert-perconaxtradb" (provisioner/opsManager/stash/schemaManager groups).

Database Group

Cluster Group

Only produces data when spec.topology (Galera) is configured.

Provisioner Group

OpsManager Group

Stash Group

Only meaningful once Stash backup/restore is configured.

SchemaManager Group

Customising Alerts

# custom-alerts.yaml
form:
  alert:
    labels:
      release: prometheus
    groups:
      database:
        enabled: warning
        rules:
          perconaxtradbTooManyConnections:
            enabled: true
            duration: "5m"
            severity: warning
      cluster:
        enabled: "none"    # disable if you don't run Galera

$ helm upgrade perconaxtradb-alert-demo oci://ghcr.io/appscode-charts/perconaxtradb-alerts \
    -n alert-perconaxtradb \
    --version=v2026.7.14 \
    -f custom-alerts.yaml

Cleaning up

$ helm uninstall perconaxtradb-alert-demo -n alert-perconaxtradb
$ kubectl delete perconaxtradb -n alert-perconaxtradb perconaxtradb-alert-demo
$ kubectl delete ns alert-perconaxtradb

# Uninstall monitoring stack (optional — skip if other tutorials on this cluster still need them)
$ helm uninstall panopticon -n kubeops
$ helm uninstall prometheus -n monitoring

Next Steps

• Monitor your PerconaXtraDB instance with KubeDB using built-in Prometheus.
• Monitor your PerconaXtraDB instance with KubeDB using Prometheus operator.
• Want to hack on KubeDB? Check our contribution guidelines.