New to KubeDB? Please start here.

MySQL Alerting with Prometheus

This tutorial shows you how to configure Prometheus-based alerting for a KubeDB-managed MySQL instance using the mysql-alerts Helm chart.

Before You Begin

• Ensure you have a Kubernetes cluster and that kubectl is configured to communicate with it. If you do not already have a cluster, you can create one using kind.

• Install the KubeDB operator by following the steps here.

• Deploy the database in the alert-mysql namespace:

  $ kubectl create ns alert-mysql
  namespace/alert-mysql created
  

• This tutorial assumes you already have a kube-prometheus-stack running in your cluster, with Prometheus configured so that both serviceMonitorSelector and ruleSelector match the label release: prometheus. See the Grafana Dashboard guide for how to deploy kube-prometheus-stack if you don’t have it yet.

  To verify the selectors:

  $ kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.ruleSelector}'
  {"matchLabels":{"release":"prometheus"}}
  
  $ kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.serviceMonitorSelector}'
  {"matchLabels":{"release":"prometheus"}}
  

• To learn more about how Prometheus monitoring works with KubeDB, see the overview here.

• For dashboards and visualisation, see Grafana Dashboard for MySQL.

Note: YAML files used in this tutorial are stored in docs/examples/mysql folder in GitHub repository kubedb/docs.

Overview

• KubeDB deploys MySQL with a built-in mysqld_exporter sidecar (container name exporter) that exposes metrics on port 56790.
• ServiceMonitor (named {mysql-name}-stats) is created automatically by KubeDB and tells Prometheus to scrape the exporter every 10 seconds.
• PrometheusRule is created by the mysql-alerts chart and contains all MySQL alert definitions grouped by concern: database health, group replication, provisioner, ops-manager, and backups (Stash and KubeStash).
• Prometheus Operator evaluates every rule expression every 30 seconds and fires matching alerts to AlertManager.
• AlertManager groups, inhibits, and silences alerts, then routes them to configured receivers (Slack, email, PagerDuty, webhook, etc.).

Deploy MySQL with Monitoring Enabled

At first, let’s deploy a MySQL database with monitoring enabled. Below is the MySQL object we are going to create. Note that spec.storage.storageClassName is set to longhorn so the database’s data volume is backed by Longhorn-replicated block storage rather than node-local storage.

apiVersion: kubedb.com/v1
kind: MySQL
metadata:
  name: mysql-alert-alert-mysql
  namespace: alert-mysql
spec:
  version: "8.4.8"
  deletionPolicy: WipeOut
  storage:
    storageClassName: "longhorn"
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi
  monitor:
    agent: prometheus.io/operator
    prometheus:
      serviceMonitor:
        labels:
          release: prometheus
        interval: 10s

Here,

• spec.storage.storageClassName: "longhorn" provisions the data volume from the longhorn StorageClass instead of the default local-path, giving the volume replicated, node-independent storage.
• spec.monitor.agent: prometheus.io/operator tells KubeDB to create a ServiceMonitor resource managed by the Prometheus operator.
• spec.monitor.prometheus.serviceMonitor.labels.release: prometheus adds the release: prometheus label to the created ServiceMonitor, matching the Prometheus serviceMonitorSelector so the target is discovered automatically.

Let’s create the namespace and the MySQL resource.

$ kubectl create ns alert-mysql
namespace/alert-mysql created

$ kubectl apply -f https://github.com/kubedb/docs/raw/v2026.6.19/docs/examples/mysql/monitoring/mysql-alert-alert-mysql.yaml
mysql.kubedb.com/mysql-alert-alert-mysql created

Now, wait for the database to go into Ready state.

$ kubectl get mysql -n alert-mysql mysql-alert-alert-mysql
NAME                      VERSION   STATUS   AGE
mysql-alert-alert-mysql   8.4.8     Ready    17m

Confirm the data volume is actually Bound on the longhorn StorageClass.

$ kubectl get pvc -n alert-mysql
NAME                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-mysql-alert-alert-mysql-0   Bound    pvc-dee12333-ab1f-40df-958d-40edc156b30c   1Gi        RWO            longhorn       17m

KubeDB creates a dedicated stats service with the -stats suffix for monitoring.

$ kubectl get svc -n alert-mysql --selector="app.kubernetes.io/instance=mysql-alert-alert-mysql"
NAME                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
mysql-alert-alert-mysql         ClusterIP   10.43.149.251   <none>        3306/TCP    17m
mysql-alert-alert-mysql-pods    ClusterIP   None            <none>        3306/TCP    17m
mysql-alert-alert-mysql-stats   ClusterIP   10.43.91.232    <none>        56790/TCP   17m

KubeDB also creates a ServiceMonitor that tells Prometheus where to scrape.

$ kubectl get servicemonitor -n alert-mysql
NAME                            AGE
mysql-alert-alert-mysql-stats   17m

Verify that the ServiceMonitor carries the release: prometheus label so Prometheus discovers it.

$ kubectl get servicemonitor -n alert-mysql mysql-alert-alert-mysql-stats \
    -o jsonpath='{.metadata.labels.release}'
prometheus

Step 1 — Install mysql-alerts

The mysql-alerts chart creates a PrometheusRule resource containing all MySQL alert definitions grouped by concern: database health, group replication, provisioner, ops-manager, and backups (Stash / KubeStash).

Why the Helm release name matters

The chart derives the PromQL job/instance scoping (and the PrometheusRule name) from the Helm release name, not from a values field — so the release name must match the MySQL object’s name (mysql-alert-alert-mysql) for the rules to be correctly scoped to this instance.

The chart’s default label is release: kube-prometheus-stack, so we must also override it at install time to match the Prometheus ruleSelector.

Install

$ helm upgrade -i mysql-alert-alert-mysql oci://ghcr.io/appscode-charts/mysql-alerts \
    -n alert-mysql \
    --create-namespace \
    --version=v2026.7.14 \
    --set form.alert.labels.release=prometheus

Verify the PrometheusRule is created

$ kubectl get prometheusrule -n alert-mysql
NAME                      AGE
mysql-alert-alert-mysql   30s

Confirm the release: prometheus label is present.

$ kubectl get prometheusrule -n alert-mysql mysql-alert-alert-mysql \
    -o jsonpath='{.metadata.labels.release}'
prometheus

Confirm Prometheus loaded the rules

Port-forward the Prometheus UI.

$ kubectl port-forward -n monitoring \
    svc/prometheus-kube-prometheus-prometheus 9090:9090

Open http://localhost:9090/rules and search for mysql-alert-alert-mysql.

The mysql.database.alert-mysql.mysql-alert-alert-mysql.rules group (and the accompanying mysql.group, mysql.provisioner, mysql.opsManager, mysql.stash, mysql.kubeStash, and mysql.schemaManager groups) are visible with all rules showing OK, confirming that Prometheus has loaded and is evaluating the MySQL alert definitions every 30 seconds.

Chart note: unlike some other *-alerts charts, every alert group declared in mysql-alerts’ values.yaml (database, group, provisioner, opsManager, stash, kubeStash, schemaManager) is actually rendered into the PrometheusRule at v2026.7.14 — there is no group silently missing from the template.

Verify End-to-End

1. Check the exporter is running

The exporter sidecar inside the MySQL pod serves metrics at :56790/metrics. A value of mysql_up 1 confirms the exporter can reach MySQL.

$ kubectl exec -n alert-mysql mysql-alert-alert-mysql-0 -c exporter -- \
    wget -qO- localhost:56790/metrics | grep mysql_up
mysql_up 1

2. Check the Prometheus target is UP

Prometheus discovers more than 20 scrape pools on a shared cluster, so instead of the Target health page, query up directly for a reliable view.

Open http://localhost:9090/query?g0.expr=up%7Bnamespace%3D%22alert-mysql%22%7D&g0.tab=1.

The target reports up == 1 for mysql-alert-alert-mysql-0 in the alert-mysql namespace, confirming Prometheus is scraping the exporter on the longhorn-backed pod.

3. Confirm all MySQL alerts are inactive

Open http://localhost:9090/alerts and locate the mysql-alert-alert-mysql groups.

All 13 rules in the mysql.database group show INACTIVE (13), meaning the database is healthy and no thresholds are breached. This also confirms DiskUsageHigh/DiskAlmostFull are inactive — this chart’s disk-usage PromQL correctly divides against kubelet_volume_stats_capacity_bytes, so (unlike some other *-alerts charts) it does not falsely fire on a healthy volume regardless of storage backend.

4. Check AlertManager

Port-forward AlertManager to view any currently firing alerts.

$ kubectl port-forward -n monitoring \
    svc/prometheus-kube-prometheus-alertmanager 9093:9093

Open http://localhost:9093. With a healthy MySQL instance, no alerts for mysql-alert-alert-mysql will be listed here.

5. Visualise metrics with Grafana

MySQL metrics can be visualised on the pre-built KubeDB Grafana dashboards. See Grafana Dashboard for MySQL for how to install and explore them — they are not duplicated here.

Simulating a Firing Alert

The previous section confirmed that all alerts are INACTIVE while the database is healthy. This section walks through deliberately triggering the MySQLInstanceDown critical alert so you can observe the full alert lifecycle — from firing in Prometheus through to the AlertManager dashboard — and then resolve it.

The exporter sidecar runs as a separate container from mysql, so it keeps running even after the main mysql container crashes and gets restarted by Kubernetes. MySQLInstanceDown/MySQLServiceDown fire as soon as a single scrape observes mysql_up == 0 (for: 0m), but Kubernetes tends to restart a crashed container within a few seconds, so a single kill can recover before the next 10-second scrape catches it. Repeatedly killing the mysql process for a short window makes the outage reliably visible to Prometheus. The provisioner alert KubeDBMySQLPhaseNotReady has for: 1m, so keep the loop running for at least a minute to also observe that alert fire.

1. Crash the MySQL process repeatedly

$ while true; do
    kubectl exec -n alert-mysql mysql-alert-alert-mysql-0 -c mysql -- sh -c "kill 1" >/dev/null 2>&1
    sleep 3
  done

Let this loop run for a couple of minutes (leave it running while you check the next steps), then stop it once you’ve captured the firing state.

2. Watch the alert fire in Prometheus

Open http://localhost:9090/alerts.

MySQLInstanceDown and MySQLServiceDown transition from INACTIVE to FIRING once the exporter reports mysql_up == 0 on a scrape — since both have for: 0m, they fire on the very next evaluation cycle after the metric goes stale/zero.

3. Check the AlertManager dashboard

Open http://localhost:9093.

AlertManager shows both MySQLInstanceDown (sourced from the exporter’s mysql_up metric) and KubeDBMySQLPhaseNotReady (sourced from the KubeDB operator’s own view of the resource, exported via Panopticon) once the crash loop has persisted past KubeDBMySQLPhaseNotReady’s for: 1m window. The alert cards display:

• Severity: critical
• pod / mysql: mysql-alert-alert-mysql-0 / mysql-alert-alert-mysql in the alert-mysql namespace
• job: mysql-alert-alert-mysql-stats
• Started: timestamp when the alert first fired

AlertManager routes this alert to every receiver configured in your alertmanagerConfig (Slack, email, PagerDuty, webhook, etc.) based on your routing tree. If no receiver is configured, the alert is visible here but silently dropped.

4. Restore MySQL

Stop the loop from step 1. The container recovers on its own — Kubernetes just needs a few uninterrupted seconds without a fresh kill to let mysqld finish starting up.

$ kubectl get pods -n alert-mysql -w
NAME                        READY   STATUS    RESTARTS   AGE
mysql-alert-alert-mysql-0   2/2     Running   19         42m

Once mysql_up returns to 1 continuously and the resource phase returns to Ready, Prometheus marks the alerts INACTIVE again and AlertManager sends a resolved notification to all receivers.

Alert Reference

All alerts are scoped to the mysql-alert-alert-mysql instance in the alert-mysql namespace via the PromQL label filters job="mysql-alert-alert-mysql-stats" and namespace="alert-mysql" (database/group groups), or app="mysql-alert-alert-mysql" and namespace="alert-mysql" (provisioner/opsManager/stash/kubeStash/schemaManager groups).

Database Group

Fired based on live metrics from mysqld_exporter.

Group Replication Group

Fired based on MySQL Group Replication performance-schema metrics; only meaningful for clustered/group-replication deployments.

Chart bug (v2026.7.14): MySQLHighReplicationDelay and MySQLHighReplicationApplyTime render with the identical PromQL expression (mysql_perf_schema_replication_group_worker_apply_time_seconds) — they will always fire and clear together. MySQLHighReplicationDelay was almost certainly intended to alert on a different metric (e.g. an actual replication-lag/delay gauge). Neither alert can be exercised on this tutorial’s single-node MySQL instance regardless, since Group Replication metrics only populate on a clustered deployment.

Provisioner Group

Monitors the KubeDB operator’s view of the MySQL resource phase (sourced from Panopticon, not the MySQL metrics endpoint).

OpsManager Group

Tracks MySQLOpsRequest lifecycle during upgrades, scaling, reconfiguration, and certificate rotations.

Stash Group

Tracks Stash-driven backup/restore health for this instance. (You do not need to configure backups to see these rules; they are included in the PrometheusRule regardless.)

KubeStash Group

Tracks KubeStash-driven backup/restore health for this instance. Same semantics as the Stash group above, sourced from KubeStash metrics instead.

SchemaManager Group

Monitors MySQLDatabase schema lifecycle objects managed by KubeDB Schema Manager.

Customising Alerts

To override thresholds or disable specific alert groups, create a custom values file and upgrade the chart.

# custom-alerts.yaml
form:
  alert:
    labels:
      release: prometheus
    groups:
      database:
        enabled: warning
        rules:
          mysqlTooManyConnections:
            enabled: true
            duration: "5m"
            val: 90        # fire at 90% instead of the default 80%
            severity: warning
      opsManager:
        enabled: "none"    # disable all ops-manager alerts

$ helm upgrade mysql-alert-alert-mysql oci://ghcr.io/appscode-charts/mysql-alerts \
    -n alert-mysql \
    --version=v2026.7.14 \
    -f custom-alerts.yaml

Cleaning up

To remove all resources created in this tutorial, run the following commands.

# Remove the mysql-alerts release
$ helm uninstall mysql-alert-alert-mysql -n alert-mysql

# Remove the MySQL instance
$ kubectl delete mysql -n alert-mysql mysql-alert-alert-mysql

# Delete namespace
$ kubectl delete ns alert-mysql

Next Steps

• Monitor your MySQL database with KubeDB using builtin Prometheus.
• Monitor your MySQL database with KubeDB using Prometheus operator.
• Visualise MySQL metrics with Grafana Dashboard.
• Detail concepts of MySQL object.
• Want to hack on KubeDB? Check our contribution guidelines.