New to KubeDB? Please start here.

Elasticsearch Alerting with Prometheus

This tutorial shows you how to configure Prometheus-based alerting for a KubeDB-managed Elasticsearch instance using the elasticsearch-alerts Helm chart, and how to visualise live metrics using the kubedb-grafana-dashboards chart.

Before You Begin

  • Ensure you have a Kubernetes cluster and that kubectl is configured to communicate with it. If you do not already have a cluster, you can create one using kind.

  • Install the KubeDB operator by following the steps here.

  • Deploy the database in a dedicated namespace, so the alerting resources created in this tutorial stay isolated from other workloads:

    $ kubectl create ns alert-elasticsearch
    namespace/alert-elasticsearch created
    
  • To learn more about how Prometheus monitoring works with KubeDB, see the overview here.

  • You will also need a Grafana API key / token with Admin permission so the kubedb-grafana-dashboards chart’s grafana-operator integration can push dashboards into Grafana. See Step 2 below.

Note: YAML files used in this tutorial are stored in docs/examples/elasticsearch folder in GitHub repository kubedb/docs.

Configuration

Step 1 (kube-prometheus-stack) is required to follow this tutorial. Step 2 (Panopticon) is required for the Provisioner Group alerts below (KubeDBElasticsearchPhase...) — skip it only if you just want the exporter-based Database Group alerts. If you have already completed the step(s) you need in another guide, skip ahead.

Step 1: Deploy kube-prometheus-stack

kube-prometheus-stack installs Prometheus, Prometheus Operator, Alertmanager, and Grafana together. This is the recommended way to get the full monitoring stack on Kubernetes.

Add the prometheus-community Helm repo and install:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update

$ helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set grafana.image.tag=7.5.5

Wait for all pods to be ready:

$ kubectl get pods -n monitoring
NAME                                                   READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2     Running   0          2m
prometheus-grafana-xxxx                                3/3     Running   0          2m
prometheus-kube-prometheus-operator-xxxx               1/1     Running   0          2m
prometheus-kube-prometheus-prometheus-0                2/2     Running   0          2m
prometheus-kube-state-metrics-xxxx                     1/1     Running   0          2m

Find the serviceMonitorSelector/ruleSelector labels that Prometheus uses to pick up ServiceMonitor/PrometheusRule objects — this is the release: prometheus label used throughout this tutorial.

$ kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.ruleSelector}'
{"matchLabels":{"release":"prometheus"}}

$ kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.serviceMonitorSelector}'
{"matchLabels":{"release":"prometheus"}}

Step 2: Install Panopticon (required for the Provisioner Group alerts)

Panopticon is the Appscode operator that exports the KubeDB operator’s own view of every resource — kubedb_com_elasticsearch_status_phase and related metrics. It’s what powers the Provisioner Group alerts below (KubeDBElasticsearchPhaseNotReady/KubeDBElasticsearchPhaseCritical). Skip this step if you only need the exporter-based Database Group alerts.

$ helm repo add appscode https://charts.appscode.com/stable/
$ helm repo update

$ helm upgrade --install panopticon appscode/panopticon \
  --version v2026.4.30 \
  --namespace kubeops --create-namespace \
  --set monitoring.enabled=true \
  --set monitoring.agent=prometheus.io/operator \
  --set monitoring.serviceMonitor.labels.release=prometheus \
  --set-file license=/path/to/kubedb-license.txt \
  --wait --timeout 5m0s

Verify Panopticon is running:

$ kubectl get pods -n kubeops
NAME                          READY   STATUS    RESTARTS   AGE
panopticon-xxxx               1/1     Running   0          1m

Overview

The diagram below shows the full alerting architecture — from Elasticsearch metric export through to alert delivery and Grafana visualisation.

Elasticsearch Alerting Architecture

  • KubeDB deploys Elasticsearch with a built-in elasticsearch_exporter sidecar that exposes metrics on port 56790.
  • ServiceMonitor (named {elasticsearch-name}-stats) is created automatically by KubeDB and tells Prometheus to scrape the exporter every 10 seconds.
  • PrometheusRule is created by the elasticsearch-alerts chart and contains all Elasticsearch alert definitions grouped by concern: database health, provisioner, ops-manager, Stash backup/restore, and KubeStash backup/restore.
  • Grafana visualises metrics through pre-built dashboards provisioned by the kubedb-grafana-dashboards chart.
  • Prometheus Operator evaluates every rule expression every 30 seconds and fires matching alerts to AlertManager.
  • AlertManager groups, inhibits, and silences alerts, then routes them to configured receivers (Slack, email, PagerDuty, webhook, etc.).

Unlike some KubeDB databases, Elasticsearch’s exporter does not publish a single boolean “is the database up” gauge. Instead, the chart watches the health signals a real Elasticsearch cluster actually exposes — JVM heap usage, filesystem usage on the data path, cluster health color (green/yellow/red), node/data-node counts, and shard state — and fires alerts when any of those cross a threshold.


Deploy Elasticsearch with Monitoring Enabled

At first, let’s deploy an Elasticsearch database with monitoring enabled. This tutorial uses a topology cluster (dedicated master, data, and ingest nodes) rather than a single-node instance, since that’s representative of a real deployment and is what the rest of this guide’s screenshots are taken from. Below is the Elasticsearch object we are going to create.

apiVersion: kubedb.com/v1
kind: Elasticsearch
metadata:
  name: es-alert
  namespace: alert-elasticsearch
spec:
  version: xpack-9.2.3
  deletionPolicy: WipeOut
  topology:
    master:
      replicas: 2
      storage:
        storageClassName: "local-path"
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
    data:
      replicas: 2
      storage:
        storageClassName: "local-path"
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
    ingest:
      replicas: 2
      storage:
        storageClassName: "local-path"
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
  monitor:
    agent: prometheus.io/operator
    prometheus:
      serviceMonitor:
        labels:
          release: prometheus
        interval: 10s

Here,

  • spec.monitor.agent: prometheus.io/operator tells KubeDB to create a ServiceMonitor resource managed by the Prometheus operator.
  • spec.monitor.prometheus.serviceMonitor.labels.release: prometheus adds the release: prometheus label to the created ServiceMonitor, matching the Prometheus serviceMonitorSelector so the target is discovered automatically.
  • spec.topology.*.storage.storageClassName: "local-path" — use whichever storage class is available/default in your cluster (kubectl get storageclass). Note that local-path is a hostPath-backed class with no capacity quota — the PVC’s 1Gi request is only used for scheduling, and the volume is really backed by however much space is free on the node’s own disk. That’s fine throughout this tutorial, including the firing-alert simulation later, since that simulation scales the data-node count rather than filling disk.

Let’s create the Elasticsearch resource.

$ kubectl apply -f https://github.com/kubedb/docs/raw/v2026.7.10/docs/examples/elasticsearch/monitoring/es-alert.yaml
elasticsearch.kubedb.com/es-alert created

Now, wait for the database to go into Ready state.

$ kubectl get elasticsearch -n alert-elasticsearch es-alert
NAME       VERSION       STATUS   AGE
es-alert   xpack-9.2.3   Ready    37m

KubeDB brings up 2 master, 2 data, and 2 ingest pods for this topology — 6 nodes total:

$ kubectl get pods -n alert-elasticsearch
NAME                READY   STATUS    RESTARTS   AGE
es-alert-data-0     2/2     Running   0          37m
es-alert-data-1     2/2     Running   0          37m
es-alert-ingest-0   2/2     Running   0          37m
es-alert-ingest-1   2/2     Running   0          37m
es-alert-master-0   2/2     Running   0          37m
es-alert-master-1   2/2     Running   0          37m

KubeDB creates a dedicated stats service with the -stats suffix for monitoring.

$ kubectl get svc -n alert-elasticsearch --selector="app.kubernetes.io/instance=es-alert"
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
es-alert          ClusterIP   10.43.126.120   <none>        9200/TCP    37m
es-alert-master   ClusterIP   None            <none>        9300/TCP    37m
es-alert-pods     ClusterIP   None            <none>        9200/TCP    37m
es-alert-stats    ClusterIP   10.43.49.18     <none>        56790/TCP   37m

KubeDB also creates a ServiceMonitor that tells Prometheus where to scrape.

$ kubectl get servicemonitor -n alert-elasticsearch
NAME             AGE
es-alert-stats   115s

Verify that the ServiceMonitor carries the release: prometheus label so Prometheus discovers it.

$ kubectl get servicemonitor -n alert-elasticsearch es-alert-stats \
    -o jsonpath='{.metadata.labels.release}'
prometheus

Step 1 — Install elasticsearch-alerts

The elasticsearch-alerts chart creates a PrometheusRule resource containing all Elasticsearch alert definitions grouped by concern: database health, provisioner, ops-manager, Stash, and KubeStash.

Why the Helm release name matters

The chart derives the PromQL job/instance scoping (and the PrometheusRule name) from the Helm release name, not from a values field — so the release name must match the Elasticsearch object’s name (es-alert) for the rules to be correctly scoped to this instance.

The chart’s default label is release: kube-prometheus-stack, so we must also override it at install time to match the Prometheus ruleSelector.

A note on chart defaults

The chart’s default database group rules assume a specific minimum topology: elasticsearchHealthyNodes and elasticsearchHealthyDataNodes both default to val: 3 — i.e. “fire if fewer than 3 nodes / fewer than 3 data nodes are healthy.” Always override both vals at install time to match your actual node counts, as this tutorial does for its own es-alert topology (2 master + 2 data + 2 ingest = 6 nodes total, 2 data nodes).

One rule pair needs overriding regardless of topology:

  • diskUsageHigh / diskAlmostFull are disabled in this tutorial. Disk-space monitoring is instead handled by elasticsearchDiskOutOfSpace / elasticsearchDiskSpaceLow, which are computed from the exporter’s own elasticsearch_filesystem_data_available_bytes / elasticsearch_filesystem_data_size_bytes metrics.

Install

$ helm repo add appscode oci://ghcr.io/appscode-charts
$ helm repo update
$ helm search repo appscode/elasticsearch-alerts --version=v2026.7.14
NAME                         	CHART VERSION	APP VERSION	DESCRIPTION                                     
appscode/elasticsearch-alerts	v2026.7.14   	v0.7.0     	A Helm chart for Elasticsearch Alert by AppsCode

$ helm upgrade -i es-alert appscode/elasticsearch-alerts -n alert-elasticsearch --create-namespace --version=v2026.7.14 \
  --set form.alert.labels.release=prometheus \
  --set form.alert.groups.database.rules.diskUsageHigh.enabled=false \
  --set form.alert.groups.database.rules.diskAlmostFull.enabled=false \
  --set form.alert.groups.database.rules.elasticsearchHealthyNodes.val=6 \
  --set form.alert.groups.database.rules.elasticsearchHealthyDataNodes.val=2 \
  --set form.alert.appSuffix=es-grafana-demo
FlagValuePurpose
es-alert (release name)Scopes every PromQL expression to this instance (job="es-alert-stats"). This must exactly match the Elasticsearch object’s name — see above. A mismatched release name is the most common cause of alerts silently never firing (and Grafana/Prometheus showing nothing for a healthy instance): the chart’s rules end up scoped to a job label that no target ever carries.
-n alert-elasticsearchalert-elasticsearchInstalls the PrometheusRule in the same namespace as the database
form.alert.labels.releaseprometheusMatches the Prometheus ruleSelector so the rules are loaded
...diskUsageHigh.enabled / ...diskAlmostFull.enabledfalseDisk-usage alerts are disabled for this tutorial — see above
...elasticsearchHealthyNodes.val6Matches this tutorial’s real total node count (2 master + 2 data + 2 ingest)
...elasticsearchHealthyDataNodes.val2Matches this tutorial’s real data-node count

Whatever topology you actually deploy, set both vals to your real node counts — total nodes for elasticsearchHealthyNodes, data nodes for elasticsearchHealthyDataNodes. For a single-node instance that means val: 1 for both, and you should also disable elasticsearchUnassignedShards (a single-node cluster can never assign a replica shard, so this rule fires permanently).

Verify the PrometheusRule is created

$ kubectl get prometheusrule -n alert-elasticsearch
NAME       AGE
es-alert   22s

Confirm the release: prometheus label is present.

$ kubectl get prometheusrule -n alert-elasticsearch es-alert \
    -o jsonpath='{.metadata.labels.release}'
prometheus

Confirm Prometheus loaded the rules

Port-forward the Prometheus UI and open the Status → Rule health page.

$ kubectl port-forward -n monitoring \
    svc/prometheus-kube-prometheus-prometheus 9090:9090

Open http://localhost:9090/rules?search=elasticsearch.

Prometheus Rule Health

The elasticsearch.database.alert-elasticsearch.es-alert.rules group is visible with all rules showing OK, confirming that Prometheus has loaded and is evaluating the Elasticsearch alert definitions every 30 seconds.

Step 2 — Install kubedb-grafana-dashboards

The kubedb-grafana-dashboards chart creates GrafanaDashboard CRDs containing pre-built Elasticsearch dashboard JSON. A separate controller, grafana-operator, watches these CRDs and pushes the dashboards into Grafana over its HTTP API — both pieces are required.

Install grafana-operator

If your cluster doesn’t already have it (check with kubectl get crd grafanadashboards.openviz.dev), install the operator that reconciles GrafanaDashboard/GrafanaDatasource objects into a real Grafana instance:

$ helm upgrade -i grafana-operator appscode/grafana-operator \
    -n kubeops --create-namespace \
    --version=v2026.6.12 \
    --wait

Mark your Grafana instance as the cluster default

The chart looks up Grafana connection details from an AppBinding annotated as the cluster’s default Grafana. If you deployed Grafana via kube-prometheus-stack (as in this tutorial), that AppBinding doesn’t exist yet and must be created once per cluster:

# Create a Grafana API key (adjust the endpoint/payload shape for your Grafana version)
$ kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80&
$ GRAFANA_PW=$(kubectl get secret -n monitoring prometheus-grafana -o jsonpath='{.data.admin-password}' | base64 -d)
$ curl -s -X POST -H "Content-Type: application/json" -u admin:$GRAFANA_PW \
    http://localhost:3000/api/auth/keys \
    -d '{"name":"kubedb-dashboards","role":"Admin"}'
# Note the returned "key"
$ kill %1
# grafana-appbinding.yaml
apiVersion: v1
kind: Secret
metadata:
  name: grafana-admin-token
  namespace: kubeops
type: Opaque
stringData:
  token: "<api-key-from-above>"
---
apiVersion: appcatalog.appscode.com/v1alpha1
kind: AppBinding
metadata:
  name: grafana
  namespace: kubeops
  annotations:
    monitoring.appscode.com/is-default-grafana: "true"   # must be an ANNOTATION, not a label
spec:
  type: monitoring.appscode.com/grafana
  clientConfig:
    url: "http://prometheus-grafana.monitoring.svc:80"
  secret:
    name: grafana-admin-token
$ kubectl apply -f grafana-appbinding.yaml

Why an AppBinding at all? GrafanaDashboard objects don’t carry connection details themselves — grafana-operator looks up the one AppBinding across the cluster marked with the monitoring.appscode.com/is-default-grafana: "true" annotation and uses its clientConfig.url + referenced secret (must contain a token key) to talk to Grafana. Skip this step only if your cluster already provisions Grafana through an Appscode-managed chart that creates this AppBinding automatically.

Install the dashboards

$ helm repo add appscode https://charts.appscode.com/stable/
$ helm repo update appscode

$ helm template kubedb-grafana-dashboards appscode/kubedb-grafana-dashboards \
    -n kubeops \
    --version=v2026.7.10 \
    --set featureGates.Elasticsearch=true \
    --set grafana.url="http://prometheus-grafana.monitoring.svc:80" \
    --set grafana.apikey="<api-key-from-above>" \
  | 

Note: The kubedb-grafana-dashboards chart bundles many large Grafana dashboard JSON files. Even with a single featureGate enabled, the rendered manifests can exceed Kubernetes’ hard 1 MB Secret limit that Helm uses to store release state. To work around this, render the chart locally with helm template and apply the output directly with kubectl apply, which bypasses Helm’s Secret storage entirely. Also note that featureGates.<DB> defaults to true for almost every database in this chart (only Aerospike defaults false), so one helm template | kubectl apply installs dashboards for many databases at once, not just Elasticsearch — this is expected.

Verify dashboards are created

$ kubectl get grafanadashboards -n kubeops | grep elasticsearch
NAME                            TITLE                            STATUS    AGE
kubedb-elasticsearch-database   KubeDB / Elasticsearch / Database   Current   2m
kubedb-elasticsearch-pod        KubeDB / Elasticsearch / Pod         Current   2m
kubedb-elasticsearch-summary    KubeDB / Elasticsearch / Summary     Current   2m

Current means grafana-operator successfully pushed the dashboard into Grafana. If a dashboard stays Failed with a message like no default Grafana appbinding found, revisit the AppBinding step above.


Verify End-to-End

1. Check the exporter is running

The exporter sidecar inside the Elasticsearch pod serves metrics at :56790/metrics. The elasticsearch_cluster_health_status series confirms the exporter can reach Elasticsearch and report cluster health.

$ kubectl exec -n alert-elasticsearch es-alert-data-0 -c exporter -- \
    wget -qO- localhost:56790/metrics | grep elasticsearch_cluster_health_status
elasticsearch_cluster_health_status{cluster="es-alert",color="green"} 1
elasticsearch_cluster_health_status{cluster="es-alert",color="red"} 0
elasticsearch_cluster_health_status{cluster="es-alert",color="yellow"} 0

With master, data, and ingest nodes all up, the cluster can fully assign both primary and replica shards, so it reports green. (A single-node cluster would instead report yellow — it can never assign replica shards without a second node to place them on — which is expected and not an outage.)

2. Check the Prometheus target is UP

Open http://localhost:9090/query?g0.expr=up%7Bnamespace%3D%22alert-elasticsearch%22%7D&g0.tab=1.

Prometheus Target UP

All 6 series report up == 1 — one entry per master/data/ingest pod, confirming metrics are being scraped from every node in the alert-elasticsearch namespace.

3. Confirm all Elasticsearch alerts are inactive

Open http://localhost:9090/alerts?search=elasticsearch to see the Elasticsearch alert groups.

Prometheus Alerts — All Inactive

All 6 rules in the elasticsearch.database group show INACTIVE (6), meaning the database is healthy and no thresholds are breached.

4. Check AlertManager

Port-forward AlertManager to view any currently firing alerts.

$ kubectl port-forward -n monitoring \
    svc/prometheus-kube-prometheus-alertmanager 9093:9093

Open http://localhost:9093. With a healthy Elasticsearch instance, no alerts for es-alert will be listed here.

AlertManager — No Active Alerts


Simulating a Firing Alert

The previous section confirmed that all alerts are INACTIVE while the database is healthy. This section walks through deliberately triggering ElasticsearchHealthyDataNodes — along with two alerts it drags along with it, see the note below — so you can observe the full alert lifecycle and then resolve it.

Elasticsearch doesn’t have a single “process down” style alert the way some other databases do — its exporter reports live cluster metrics rather than a boolean liveness gauge. Killing the elasticsearch process inside a pod doesn’t work either: the container restarts in under 2 seconds (faster than the cluster’s fault-detection window), so the other nodes never actually perceive the node as gone. Instead, we shrink the data role from 2 nodes to 1 — a real, sustained, cleanly-reversible change that reliably crosses this tutorial’s elasticsearchHealthyDataNodes.val: 2 threshold set in Step 1.

1. Scale down the data nodes

$ kubectl patch elasticsearch -n alert-elasticsearch es-alert \
    --type=merge -p '{"spec":{"topology":{"data":{"replicas":1}}}}'
elasticsearch.kubedb.com/es-alert patched

KubeDB terminates one data pod to bring the topology down to the new desired count:

$ kubectl get pods -n alert-elasticsearch -l app.kubernetes.io/instance=es-alert
NAME                READY   STATUS    RESTARTS   AGE
es-alert-data-0     2/2     Running   0          37m
es-alert-ingest-0   2/2     Running   0          37m
es-alert-ingest-1   2/2     Running   0          37m
es-alert-master-0   2/2     Running   0          37m
es-alert-master-1   2/2     Running   0          37m

Wait 30–60 seconds for the next Prometheus scrape cycle (configured at 10 s) and rule-evaluation cycle (30 s) to register the smaller data-node count.

2. Watch the alert fire in Prometheus

Open http://localhost:9090/alerts?search=elasticsearch.

Prometheus Alerts — ElasticsearchHealthyDataNodes Firing

Dropping to 5 total nodes (1 data + 2 master + 2 ingest) crosses three thresholds at once, confirmed live — all for: instant, so all three move directly from INACTIVE to FIRING within one evaluation cycle, while the rest of the elasticsearch.database group stays INACTIVE:

  • ElasticsearchHealthyDataNodes — data-node count (1) is below val: 2.
  • ElasticsearchHealthyNodes — total node count (5) is below val: 6.
  • ElasticsearchUnassignedShards — with only one data node left, replica shards have nowhere to be placed.

Each fires once per surviving node’s exporter (5 series each here — one per remaining pod), since every node independently reports its own view of cluster-wide state; that’s 15 alert instances in total, not 15 separate incidents.

3. Check the AlertManager dashboard

Open http://localhost:9093/#/alerts?filter={namespace="alert-elasticsearch"}.

AlertManager — ElasticsearchHealthyDataNodes Firing

AlertManager shows all three alerts grouped by namespace (15 alerts total). Each alert card displays:

  • Severity: critical
  • app / job: es-alert / es-alert-stats
  • pod: the surviving node reporting the condition (e.g. es-alert-data-0, es-alert-master-1, …)
  • Started: timestamp when the alert first fired

AlertManager routes these alerts to every receiver configured in your alertmanagerConfig (Slack, email, PagerDuty, webhook, etc.) based on your routing tree. If no receiver is configured, the alerts are visible here but silently dropped.

4. Restore the data nodes

Scale the data role back to 2 to resolve the alert.

$ kubectl patch elasticsearch -n alert-elasticsearch es-alert \
    --type=merge -p '{"spec":{"topology":{"data":{"replicas":2}}}}'
elasticsearch.kubedb.com/es-alert patched

Wait for the pod to rejoin and for the next scrape cycle to register the recovered count.

$ kubectl get elasticsearch -n alert-elasticsearch es-alert
NAME       VERSION       STATUS   AGE
es-alert   xpack-9.2.3   Ready    41m

Once the Elasticsearch resource returns to Ready and elasticsearch_cluster_health_number_of_data_nodes reports 2 again, Prometheus marks all three alerts INACTIVE and AlertManager sends resolved notifications to all receivers.


Alert Reference

All alerts are scoped to the es-alert instance in the alert-elasticsearch namespace via the PromQL label filters job="es-alert-stats" and namespace="alert-elasticsearch".

Database Group

Fired based on live metrics from the Elasticsearch exporter.

AlertSeverityForWhat It Means
ElasticsearchHeapUsageTooHighcritical2mThe JVM heap usage is over 90%.
ElasticsearchHeapUsageWarningwarning2mThe JVM heap usage is over 80%.
ElasticsearchDiskOutOfSpacecriticalinstantThe disk usage is over 90%.
ElasticsearchDiskSpaceLowwarning2mThe disk usage is over 80%.
ElasticsearchClusterRedcriticalinstantElastic Cluster Red status — one or more primary shards are not allocated.
ElasticsearchClusterYellowwarninginstantElastic Cluster Yellow status — one or more replica shards are not allocated.
ElasticsearchHealthyNodescriticalinstantFewer than the configured minimum number of nodes are healthy in the cluster (default val: 3; this tutorial overrides it to 6 — see Step 1).
ElasticsearchHealthyDataNodescriticalinstantFewer than the configured minimum number of data nodes are healthy in the cluster (default val: 3; this tutorial overrides it to 2).
ElasticsearchRelocatingShardsinfoinstantElasticsearch is relocating shards.
ElasticsearchInitializingShardsinfoinstantElasticsearch is initializing shards.
ElasticsearchUnassignedShardscriticalinstantElasticsearch has unassigned shards.
ElasticsearchPendingTaskswarning15mElasticsearch has pending tasks — the cluster is working slowly.
ElasticsearchNoNewDocuments10minfoinstantNo new documents were indexed in the last 10 minutes (disabled by default).
DiskUsageHighwarning1mDisabled in this tutorial — see above.
DiskAlmostFullcritical1mDisabled in this tutorial — same as DiskUsageHigh.

Provisioner Group

Monitors the KubeDB operator’s view of the Elasticsearch resource phase.

AlertSeverityForWhat It Means
KubeDBElasticsearchPhaseNotReadycritical1mKubeDB marked the Elasticsearch resource NotReady — operator cannot reach the database.
KubeDBElasticsearchPhaseCriticalwarning15mThe instance is in a degraded/critical phase.

OpsManager Group

Tracks ElasticsearchOpsRequest lifecycle during upgrades, scaling, and reconfiguration.

AlertSeverityForWhat It Means
KubeDBElasticsearchOpsRequestOnProgressinfoinstantAn ops request is currently in progress.
KubeDBElasticsearchOpsRequestStatusProgressingToLongcritical30mAn ops request has been running for 30+ minutes — likely stuck.
KubeDBElasticsearchOpsRequestFailedcriticalinstantAn ops request failed — check the ElasticsearchOpsRequest object for the error.

Stash Group

Tracks backup/restore health for Elasticsearch instances backed up with Stash.

AlertSeverityForWhat It Means
ElasticsearchStashBackupSessionFailedcriticalinstantThe most recent Stash backup session failed.
ElasticsearchStashRestoreSessionFailedcriticalinstantThe most recent Stash restore session failed.
ElasticsearchStashNoBackupSessionForTooLongwarninginstantNo successful backup session in the last 18000s (5 hours).
ElasticsearchStashRepositoryCorruptedcritical5mThe Stash backup repository failed its integrity check.
ElasticsearchStashRepositoryStorageRunningLowwarning5mThe Stash repository has grown beyond 10 GB.
ElasticsearchStashBackupSessionPeriodTooLongwarninginstantA backup session took longer than 1800s (30 minutes) to complete.
ElasticsearchStashRestoreSessionPeriodTooLongwarninginstantA restore session took longer than 1800s (30 minutes) to complete.

KubeStash Group

Tracks backup/restore health for Elasticsearch instances backed up with KubeStash.

AlertSeverityForWhat It Means
ElasticsearchKubeStashBackupSessionFailedcriticalinstantThe most recent KubeStash backup session failed.
ElasticsearchKubeStashRestoreSessionFailedcriticalinstantThe most recent KubeStash restore session failed.
ElasticsearchKubeStashNoBackupSessionForTooLongwarninginstantNo successful backup session in the last 18000s (5 hours).
ElasticsearchKubeStashRepositoryCorruptedcritical5mThe KubeStash repository failed its integrity check.
ElasticsearchKubeStashRepositoryStorageRunningLowwarning5mThe KubeStash repository has grown beyond 10 GB.
ElasticsearchKubeStashBackupSessionPeriodTooLongwarninginstantA backup session took longer than 1800s (30 minutes) to complete.
ElasticsearchKubeStashRestoreSessionPeriodTooLongwarninginstantA restore session took longer than 1800s (30 minutes) to complete.

Stash and KubeStash alerts are only relevant if you’ve configured backups for this Elasticsearch instance. This tutorial doesn’t set up backups — the tables above are included so you know what’s available in the chart if you do.


Customising Alerts

To override thresholds or disable specific alert groups, create a custom values file and upgrade the chart.

# custom-alerts.yaml
form:
  alert:
    labels:
      release: prometheus
    groups:
      database:
        enabled: warning
        rules:
          elasticsearchHeapUsageWarning:
            enabled: true
            duration: "5m"
            val: 70        # fire at 70% heap usage instead of the default 80%
            severity: warning
      opsManager:
        enabled: "none"    # disable all ops-manager alerts
$ helm upgrade es-alert oci://ghcr.io/appscode-charts/elasticsearch-alerts \
    -n alert-elasticsearch \
    --version=v2026.7.14 \
    -f custom-alerts.yaml

Cleaning up

To remove all resources created in this tutorial, run the following commands.

# Remove the Grafana dashboards (installed via helm template | kubectl apply, not helm install)
$ helm template kubedb-grafana-dashboards appscode/kubedb-grafana-dashboards \
    -n kubeops \
    --version=v2026.7.10 \
    --set featureGates.Elasticsearch=true \
    --set grafana.url="http://prometheus-grafana.monitoring.svc:80" \
    --set grafana.apikey="<api-key>" \
  | kubectl delete -n kubeops -f - --ignore-not-found

# Remove the elasticsearch-alerts release
$ helm uninstall es-alert -n alert-elasticsearch

# Remove the Elasticsearch instance
$ kubectl delete elasticsearch -n alert-elasticsearch es-alert

# Delete namespace
$ kubectl delete ns alert-elasticsearch

# Optional: only if nothing else in the cluster depends on them
$ kubectl delete appbinding -n kubeops grafana
$ kubectl delete secret -n kubeops grafana-admin-token
$ helm uninstall grafana-operator -n kubeops

# Uninstall monitoring stack (optional — skip if other tutorials on this cluster still need them)
$ helm uninstall panopticon -n kubeops
$ helm uninstall prometheus -n monitoring

Next Steps