Version: 9.5

Set Up Grafana Alerts

Use the Grafana web interface to set up alert rules and notification channels to detect environment outages and performance degradation in a Qrvey Platform deployment. Alerts are routed to AWS SNS topics for flexible notification delivery.

Grafana version: 12.3.1 (Unified Alerting). Prometheus datasource: pre-configured as Prometheus. Audience: Platform administrators with Grafana Admin or Editor access.

Before You Begin

Obtain Grafana URL and Credentials

Grafana is only deployed when Monitoring is enabled in your config.json before running terraform apply. For more information, see Configure Monitoring and Logging.

After deployment, the Grafana URL and initial admin credentials are printed in the Deployment Output at the end of the terraform apply run. If you are upgrading an existing instance, the same outputs are printed after the upgrade completes.

Managing contact points and notification policies requires the Admin role. Creating alert rules within an accessible folder requires at least the Editor role.

SNS notifications require AWS SNS topics to be created and Grafana to be configured with AWS credentials. This one-time configuration is performed by your platform administrator.

Create SNS topics in AWS for each alert severity level:
- qrvey-critical-alerts — for critical priority alerts
- qrvey-operational-digest — for info/digest alerts
Subscribe to these topics through email, SMS, or other endpoints:

a. Go to each topic: Subscriptions > Create subscription.

b. Select a protocol (Email, SMS, HTTP, and so on) and enter an endpoint.

c. Confirm the subscription (check email for confirmation link if using email).
Record the SNS topic ARNs (format: arn:aws:sns:region:account-id:topic-name) for Grafana configuration.

Slack Webhook URL (required for Slack notifications)

To receive Slack notifications, create an Incoming Webhook in your Slack workspace:

Slack > Your workspace > Apps > Incoming Webhooks > Add to Slack > Choose channel > Copy Webhook URL

Record the webhook URL for use in your Slack Contact Point.

Verify Prometheus Datasource

Before creating alert rules, confirm that the Prometheus datasource is available:

In the left sidebar, select Connections > Data sources.
Verify that a datasource named Prometheus is listed and its status shows Data source connected and labels found.

Step 1: Configure Notification Channels

In the left sidebar, select Alerting (bell icon) > Contact points.
Select + Add contact point.
Complete the form:
- Name: SNS - Critical Alerts
- Integration: Select AWS SNS from the dropdown.
- Topic ARN: Paste the ARN for your critical alerts topic (such as arn:aws:sns:us-west-2:123456789012:qrvey-critical-alerts)
- AWS Region: Select the region where your SNS topic is deployed (such as us-west-2)
To customize the message format, expand Optional settings. The defaults are suitable for most deployments.
Select Test to send a test message and confirm delivery to your SNS topic subscribers.
Select Save contact point.

Slack Contact Point

In the left sidebar, select Alerting > Contact points.
Select + Add contact point.
Complete the form:
- Name: Slack - Platform Alerts
- Integration: Select Slack from the dropdown.
- Webhook URL: Paste the Slack Incoming Webhook URL from your workspace.
- Channel: Enter your Slack channel name, such as #platform-alerts.
Select Test to send a test message to the Slack channel.
Select Save contact point.

The digest contact point uses the same SNS integration as critical alerts but targets a different SNS topic — typically one subscribed to by operations or engineering teams who review a daily summary rather than responding immediately.

In the left sidebar, select Alerting > Contact points.
Select + Add contact point.
Complete the form:
- Name: SNS - Operational Digest
- Integration: Select AWS SNS from the dropdown.
- Topic ARN: Paste the ARN for your operational digest topic (such as arn:aws:sns:us-west-2:123456789012:qrvey-operational-digest).
- AWS Region: Select the region where your SNS topic is deployed.
Select Test to confirm delivery.
Select Save contact point.

Notification Routing Policy

Notification policies control which contact points receive which alerts based on label matching. The following policy routes critical alerts to both SNS and Slack, warning alerts to Slack only, and info alerts to the SNS digest.

In the left sidebar, select Alerting > Notification policies.
If the policy tree is not visible, or if the page displays a No policies matching filters message, select Clear filters at the top of the page. The search by matchers and search by contact point fields both act as filters. Any active filter hides the default policy tree.
Make sure the Choose Alertmanager dropdown (top right) is set to Grafana. If it displays a different Alertmanager, switch to Grafana to view the policies managed by Grafana's built-in alerting.
The default (root) policy displays at the top of the tree with a grey background. Select Edit to update the policy:

a. Set Default contact point to Slack - Platform Alerts.

b. Select Update default policy.
Select + new child policy to add a policy for critical alerts.

a. Add the following policy settings:
- Matching labels: Set Label = severity, Operator = =, Value = critical.
- Contact point SNS - Critical Alerts.
- Continue matching subsequent sibling nodes: Enable this toggle so the alert also reaches Slack.
- Override grouping: Leave disabled.
- Override general timings: Leave disabled (default timings work for critical alerts).
b. Select Save policy to save your changes. The resulting hierarchy ensures critical alerts always reach SNS.
Select + new child policy to create the Slack policy for critical alerts:

a. Add the following policy settings:
- Matching labels: severity = critical.
- Contact point: Slack - Platform Alerts.
- Leave all toggles disabled.
b. Select Save policy to save your changes. The resulting hierarchy ensures critical alerts always reach Slack, while warnings go to Slack only through the root policy.
Select + new child policy for the operational digest:

a. Add the following policy settings:
- Matching labels: severity = info.
- Contact point: SNS - Operational Digest.
- Enable the Override general timings toggle and complete the following fields:
  - Group wait: 1h
  - Group interval: 6h
  - Repeat interval: 24h
- Enable the Override grouping toggle to display a Group by field. Set it to severity. This batches all info alerts fired within the same hour into one SNS message.
b. Select Save policy to save your changes.

When the first info-severity alert fires, Grafana waits 1 hour before publishing to the SNS topic. Any other info-severity alerts that fire during that hour are bundled into the same message. A follow-up message is sent every 6 hours while alerts remain active, and a daily re-notification is sent if alerts are still firing after 24 hours.

Step 2: Create an Alert Folder

Keeping alert rules in a dedicated folder makes them easier to manage. Use one of the following options:

Default Alerts & Incidents folder created during Grafana deployment.
An existing folder (such as Alerts & Incidents). When setting the evaluation behavior of an alert rule, select the folder at the prompt.
A new folder:
1. In the left sidebar, select Dashboards.
2. Select New (top-right) > New folder.
3. Enter a folder name (such as Qrvey Platform Alerts) and select Create.

All alert rules are saved into the selected folder.

Step 3: Set Up Environment Down Alerts

These alerts detect conditions when the Qrvey Platform is completely or critically unavailable to end users.

Create an Alert Rule

Use the following general steps to create alerts.

Go to Alerting > Alert rules > + New alert rule.
At the top, enter the Rule name.
Define query and alert condition:

a. Select the Prometheus datasource selector and choose Prometheus.

b. Switch the query editor to Code mode (toggle in the top-right of the query panel).

c. Paste the PromQL expression into the query field (labelled A).

d. In the Alert condition panel, Set the condition (IS ABOVE / IS BELOW) and threshold value as specified in each alert.
Set evaluation behavior:
- Folder: Select your folder (such as Alerts & Incidents).
- Evaluation group: Create a new group named platform-alerts-1m with Evaluation interval: 1m (or choose an existing group).
- Pending period: Specified for each alert.
- No data: Alerting (so the alert fires if the metric disappears entirely).
- Error: Alerting.
An additional Keep Last State option is available for both No data and Error. This preserves the alert's previous state rather than changing the state when data is missing or an evaluation error occurs. For production monitoring, use Alerting. Use Keep Last State only if your metrics have frequent short gaps and you want to avoid noisy state changes.
Add labels and annotations:
- Add a label severity with the value specified for each alert.
- Add an annotation summary with the short summary text.
- Add an annotation description with the longer description text.
Select Save rule and exit.

The following alert sections list the PromQL expression and then give step-by-step Grafana UI instructions. The UI instructions follow the same pattern — only the name, query, threshold, and labels change.

KongDown — API Gateway Unavailable

The Kong public-facing API gateway has no ready pods. All inbound traffic to the Qrvey Platform fails.

PromQL Expression

Paste the following expression into query A:

sum(kube_pod_status_ready{namespace="kong", condition="true"})

Alert Condition

The following condition corresponds to Classic condition B:

IS BELOW 1

Evaluation Settings

Setting	Value
Pending period	`2m`

Labels

Key	Value
`severity`	`critical`
`component`	`kong`

Annotations

Key	Value
`summary`	`Kong API gateway has no ready pods`
`description`	`The Kong public ingress controller in namespace kong has 0 ready pods. All external traffic to the Qrvey Platform is blocked.`

Note: If you also run kong-private for internal traffic, create a second identical rule, replacing namespace="kong" with namespace="kong-private" and adjusting the name to KongPrivateDown.

QrveyAppPodDown — Application Unavailable

The total number of available deployment replicas across the qrveyapps namespace has dropped to zero, meaning no application workloads are serving requests.

PromQL Expression

Paste the following expression into query A:

sum(kube_deployment_status_replicas_available{namespace="qrveyapps"})

Alert Condition

The following condition corresponds to Classic condition B:

IS BELOW 1

Evaluation Settings

Setting	Value
Pending period	`3m`

Labels

Key	Value
`severity`	`critical`
`component`	`qrvey-application`

Annotations

Key	Value
`summary`	`Qrvey application has no available pods`
`description`	`All deployments in the qrveyapps namespace report zero available replicas. The Qrvey Platform is not serving any requests.`

Tip: If your deployment runs multiple replicas and you want an earlier warning, change the threshold from IS BELOW 1 to IS BELOW 3 to alert when the replica count drops below 3.

IngressEndpointUnreachable — No Backend Traffic Endpoints

Kong's Prometheus scrape target has gone down, meaning Prometheus can no longer reach Kong's metrics endpoint. This is a reliable proxy for "Kong is not accepting traffic" — if Kong is unhealthy or unreachable, its scrape target will either report up = 0 or disappear entirely.

PromQL Expression

Paste the following expression into query A:

up{job="kong-prometheus"} == 0

Before saving this alert, verify the job name by opening Explore > Prometheus and running up{job="kong-prometheus"} (without == 0). It should return a series with value 1. If it returns no data, run up and look for a series with a job label that matches your Kong scrape job. Update the filter above with that value.

Alert Condition

The following condition corresponds to Classic condition B:

IS ABOVE 0

Important — set No data to Alerting: In Section 3 of the alert rule form, set No data to Alerting. This catches the case where Kong has crashed completely and its scrape target disappears from Prometheus entirely (the metric is absent rather than 0).

Evaluation Settings

Setting	Value
Pending period	`2m`
No data	`Alerting`

Labels

Key	Value
`severity`	`critical`
`component`	`ingress`

Annotations

Key	Value
`summary`	`Kong ingress is unreachable — Prometheus cannot scrape its metrics endpoint`
`description`	`The kong-prometheus scrape target is down or absent. Kong is likely not accepting traffic. Check Kong pod health and confirm the Prometheus plugin is enabled.`

Note: This check detects Kong being unreachable from inside the cluster. For a full external HTTP probe (testing reachability from outside the network), install the Grafana Synthetic Monitoring plugin or the Blackbox Exporter.

RabbitMQPodDown — Message Queue Unavailable

All RabbitMQ pods in the rabbitmq namespace are not ready. This stops all data ingestion pipelines (file pump, database pump, transformations, and related services).

PromQL Expression

Paste the following expression into query A:

sum(kube_pod_status_ready{namespace="rabbitmq", condition="true"})

Alert Condition

The following condition corresponds to Classic condition B:

IS BELOW 1

Evaluation Settings

Setting	Value
Pending period	`2m`

Labels

Key	Value
`severity`	`critical`
`component`	`rabbitmq`

Annotations

Key	Value
`summary`	`RabbitMQ has no ready pods`
`description`	`All pods in the rabbitmq namespace are not ready. Data ingestion pipelines (pumps, transformations) will stop processing.`

Step 4: Set Up Early Warning Alerts

The following alerts detect degraded performance or approaching resource limits before an outage occurs. While less urgent than Priority 1 alerts, they are important for proactive monitoring.

KongHighLatency

The 99th percentile request latency through the Kong API gateway has exceeded 5 seconds over a 5-minute window — indicating the platform is responding very slowly from the user's perspective.

PromQL Expression

Paste the following expression into query A:

histogram_quantile(
  0.99,
  sum(rate(kong_request_latency_ms_bucket[5m])) by (le)
)

Alert Condition

The following condition corresponds to Classic condition B:

IS ABOVE 5000

(5000 milliseconds = 5 seconds)

Evaluation Settings

Setting	Value
Pending period	`5m`

Labels

Key	Value
`severity`	`warning`
`component`	`kong`

Annotations

Key	Value
`summary`	`Kong P99 request latency exceeds 5 seconds`
`description`	`The 99th percentile end-to-end request latency through Kong is above 5000ms. Users may be experiencing significant slowness.`

Tip: Adjust the threshold to match your SLA. If your target P99 is 2 seconds, change the threshold to 2000.

DataloadPumpBacklog

One or more data ingestion services (pumps) in the qrveyapps or qrveyapps-jobs namespace have been running at their maximum autoscaler replica limit — a sign that the system cannot scale further and a processing backlog is building.

PromQL Expression

Paste the following expression into query A:

count(
  kube_horizontalpodautoscaler_status_current_replicas{namespace=~"qrveyapps|qrveyapps-jobs"}
  >=
  kube_horizontalpodautoscaler_spec_max_replicas{namespace=~"qrveyapps|qrveyapps-jobs"}
)

Alert Condition

The following condition corresponds to Classic condition B:

IS ABOVE 0

Evaluation Settings

Setting	Value
Pending period	`10m`

Labels

Key	Value
`severity`	`warning`
`component`	`dataload`

Annotations

Key	Value
`summary`	`Data ingestion pump is at maximum replicas`
`description`	`At least one autoscaled service in qrveyapps or qrveyapps-jobs has been at its maximum replica count for 10+ minutes. A processing backlog may be forming.`

PVCSpaceWarning

A Persistent Volume Claim (PVC) — used by Prometheus, Loki, Elasticsearch, or other stateful services — has exceeded 70% disk usage. This provides time to act before the existing critical alert fires at 80%.

PromQL Expression

Paste the following expression into query A:

kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes

Alert Condition

The following condition corresponds to Classic condition B:

IS ABOVE 0.70

Evaluation Settings

Setting	Value
Pending period	`10m`

Labels

Key	Value
`severity`	`warning`
`component`	`storage`

Annotations

Key	Value
`summary`	`PVC {{ $labels.persistentvolumeclaim }} usage above 70%`
`description`	`PVC {{ $labels.persistentvolumeclaim }} in namespace {{ $labels.namespace }} is at {{ $value \| humanizePercentage }} capacity. The critical threshold is 80%.`

Note: This alert fires for each PVC. You see one alert instance for each PVC that exceeds 70%.

PrometheusTargetDown

A Prometheus scrape target cannot be reached. When a target is down, you lose visibility into that component's metrics, which means other alerts might also stop firing. This alert relates to the monitoring infrastructure itself.

PromQL Expression

Paste the following expression into query A:

up == 0

Alert Condition

The following condition corresponds to Classic condition B:

IS ABOVE 0

Evaluation Settings

Setting	Value
Pending period	`5m`

Labels

Key	Value
`severity`	`warning`
`component`	`monitoring`

Annotations

Key	Value
`summary`	`Prometheus target {{ $labels.job }} is down`
`description`	`Prometheus cannot scrape the target {{ $labels.job }} (instance: {{ $labels.instance }}). Metrics from this component are missing.`

LokiIngestionErrors

The Loki log aggregation service is returning HTTP 5xx errors on its push endpoint, meaning logs from application pods may be dropped and log-based visibility is degraded.

PromQL Expression

Paste the following expression into query A:

sum(
  rate(
    loki_request_duration_seconds_count{
      status_code=~"5..",
      route="/loki/api/v1/push"
    }[5m]
  )
)

Alert Condition

The following condition corresponds to Classic condition B:

IS ABOVE 0

Evaluation Settings

Setting	Value
Pending period	`5m`

Labels

Key	Value
`severity`	`warning`
`component`	`loki`

Annotations

Key	Value
`summary`	`Loki is rejecting log ingestion requests`
`description`	`Loki is returning 5xx errors on the log push endpoint. Application logs may be lost. Check Loki pod health and disk space in the monitoring namespace.`

NodeCPUHighUsage

A cluster node's CPU utilization has exceeded 90% for 5 minutes. Sustained high CPU leads to request throttling, pod OOMKills, and eventual scheduling failures.

PromQL Expression

Paste the following expression into query A:

1 - avg by(node) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

Alert Condition

The following condition corresponds to Classic condition B:

IS ABOVE 0.90

Evaluation Settings

Setting	Value
Pending period	`5m`

Labels

Key	Value
`severity`	`warning`
`component`	`node`

Annotations

Key	Value
`summary`	`Node {{ $labels.node }} CPU usage above 90%`
`description`	`Node {{ $labels.node }} has been at {{ $value \| humanizePercentage }} CPU utilization for 5 minutes. Pod throttling and scheduling issues are likely.`

KarpenterNodeProvisionFailed

In AWS, pods are waiting to be scheduled but Karpenter is not provisioning new nodes to accommodate them. This can occur due to EC2 capacity limits, misconfigured node pools, or IAM permission issues.

PromQL Expression

Paste the following expression into query A:

count(kube_pod_status_scheduled{condition="false"} == 1)

This counts pods that Kubernetes has determined cannot be scheduled (no suitable node exists or Karpenter has failed to provision one).

Tip: Before using this expression above, open Explore > Prometheus and search for metrics beginning with karpenter_nodeclaims. If you find karpenter_nodeclaims_disrupted_total, you can use a more targeted expression:
increase(karpenter_nodeclaims_disrupted_total{reason="failed"}[10m]) > 0
Use the Karpenter-native metric if it is present; otherwise, use the kube_pod_status_scheduled expression.

Alert Condition

The following condition corresponds to Classic condition B:

IS ABOVE 0

Evaluation Settings

Setting	Value
Pending period	`10m`

Labels

Key	Value
`severity`	`warning`
`component`	`karpenter`

Annotations

Key	Value
`summary`	`Pods are unschedulable — Karpenter may have failed to provision nodes`
`description`	`One or more pods cannot be scheduled for 10+ minutes. Karpenter may be failing to provision EC2 nodes. Check Karpenter logs and EC2 service quotas in the AWS console.`

PostgreSQLConnectionsHigh

In Azure, the PostgreSQL Flexible Server is approaching its maximum connection limit. When connections are exhausted, application queries fail with FATAL: sorry, too many clients.

This alert requires the prometheus-community/postgres-exporter to be deployed and scraping your PostgreSQL instance. If it is not yet installed, contact your platform administrator. The required metric is pg_stat_activity_count.

PromQL Expression

Paste the following expression into query (requires postgres-exporter):

sum(pg_stat_activity_count) by (datname)

Alert Condition

The following condition corresponds to Classic condition B:

IS ABOVE 80

Adjust this number based on the max_connections setting of your PostgreSQL Flexible Server instance. The default for B_Standard_B4ms is 179. Firing at 80 provides a safe margin.

Evaluation Settings

Setting	Value
Pending period	`5m`

Labels

Key	Value
`severity`	`warning`
`component`	`postgresql`

Annotations

Key	Value
`summary`	`PostgreSQL database {{ $labels.datname }} has high active connections`
`description`	`Database {{ $labels.datname }} has {{ $value }} active connections. This may be approaching the server's max_connections limit. Review connection pooling configuration.`

If the postgres-exporter is not yet available, you can configure an Azure Monitor alert on the active_connections metric directly from the Azure portal as an interim measure, configuring the notification channels as needed. This does not require changes to Grafana.

These alerts cover operational and infrastructure health conditions that are important to track but do not require immediate on-call response. They are routed to the SNS - Operational Digest contact point using the severity = info label, which combines them into a single hourly SNS message rather than firing individual notifications.

Use the same general procedure to create each rule. If one does not exist, create a new evaluation group with a platform-alerts-5m label that uses Evaluation interval: 5m.

ElasticsearchDiskWatermark

Elasticsearch data node disk usage has exceeded 75%. At this level, Elasticsearch activates its low watermark and begins relocating shards away from the affected node. Left unaddressed, this leads to index throttling and eventually a red cluster state.

PromQL Expression

Paste the following expression into query A:

1 - (
  elasticsearch_filesystem_data_available_bytes{job="elasticsearch-exporter"}
  / elasticsearch_filesystem_data_size_bytes{job="elasticsearch-exporter"}
)

Alert Condition

The following condition corresponds to Classic condition B:

IS ABOVE 0.75

Evaluation Settings

Setting	Value
Pending period	`10m`

Labels

Key	Value
`severity`	`info`
`component`	`elasticsearch`

Annotations

Key	Value
`summary`	`Elasticsearch node {{ $labels.name }} disk usage above 75%`
`description`	`Elasticsearch data node {{ $labels.name }} is using {{ $value \| humanizePercentage }} of its disk. The low watermark has been reached and shard relocation will begin. Expand storage or delete old indices.`

Note: This alert fires for each data node (multi-dimensional). In a three-node cluster, you can see up to three alert instances in the digest. The name label identifies each node.

Verify the job label by opening Explore > Prometheus and running elasticsearch_filesystem_data_available_bytes. Confirm the job label value as elasticsearch-exporter. If it differs, update the filter in the PromQL.

RabbitMQQueueDepthHigh

One or more RabbitMQ queues have more than 1000 ready (undelivered) messages for 10 consecutive minutes, indicating that data ingestion consumers are falling behind.

PromQL Expression

Paste the following expression into query A:

rabbitmq_queue_messages_ready

Alert Condition

The following condition corresponds to Classic condition B:

IS ABOVE 1000

Evaluation Settings

Setting	Value
Pending period	`10m`

Labels

Key	Value
`severity`	`info`
`component`	`rabbitmq`

Annotations

Key	Value
`summary`	`RabbitMQ queue {{ $labels.queue }} has a high message backlog`
`description`	`Queue {{ $labels.queue }} (vhost: {{ $labels.vhost }}) has {{ $value }} ready messages. Consumers may be slow or down. Check the dataload pump pods in the qrveyapps-jobs namespace.`

Adjust the threshold as needed. The default threshold of 1000 is a starting point. Review your normal queue depths in the RabbitMQ Cluster Overview dashboard to set a threshold appropriate for your data volumes.

This metric is served by the RabbitMQ exporter. If the query returns no data, open Explore > Prometheus and search for rabbitmq_queue. If no metrics appear, confirm the RabbitMQ exporter pod is running in the monitoring namespace.

KongPrivateDown — Internal API Gateway Unavailable

The Kong private (internal) API gateway has no ready pods. Internal service-to-service communication that routes through kong-private will fail. This does not affect external user traffic but impacts internal workflows and integrations.

The KongPrivateDown alert is operationally important but does not immediately block end users, so it is routed to the daily digest rather than triggering an on-call notification. If your deployment relies heavily on kong-private for critical internal integrations, consider promoting this to severity = critical.

PromQL Expression

Paste the following expression into query A:

sum(kube_pod_status_ready{namespace="kong-private", condition="true"})

Alert Condition

The following condition corresponds to Classic condition B:

IS BELOW 1

Evaluation Settings

Setting	Value
Pending period	`5m`

Labels

Key	Value
`severity`	`info`
`component`	`kong-private`

Annotations

Key	Value
`summary`	`Kong private ingress has no ready pods`
`description`	`All pods in the kong-private namespace are not ready. Internal API routing will be disrupted. External user-facing traffic is unaffected.`

OtelCollectorErrors

The OpenTelemetry Collector is failing to export log records — either dropping them at the receiver or failing to deliver them to the Loki exporter. When this alert fires, application log data can be silently lost.

PromQL Expression Option A

The following option applies to exporter failures:

sum(rate(otelcol_exporter_send_failed_log_records_total[5m]))

PromQL Expression Option B

The following option applies to receiver refusals:

sum(rate(otelcol_receiver_refused_log_records_total[5m]))

Alert Condition

The following condition corresponds to Classic condition B:

IS ABOVE 0

To determine which expression to use, open Explore > Prometheus and run both queries. Use the query that returns data.

If both return data, create separate rules for each expression, both using severity = info.
If neither returns data, the OTel collector metrics scraping might not be configured. Verify that the collector's Prometheus metrics endpoint can be reached from Prometheus.

Evaluation Settings

Setting	Value
Pending period	`5m`

Labels

Key	Value
`severity`	`info`
`component`	`otel-collector`

Annotations

Key	Value
`summary`	`OpenTelemetry Collector is dropping log records`
`description`	`The OTel Collector is failing to deliver log records to Loki. Application logs may be incomplete. Check the otel-collector pod in the monitoring namespace and verify Loki is healthy.`

Step 6: Verify Your Alerts

After creating all rules, confirm that the full alert pipeline is working.

Verify Rule Evaluations

Go to Alerting > Alert rules.
Expand the Qrvey Platform Alerts folder.
Each rule should show a green Normal state badge. Available states include the following:
- Pending — the condition is met but the pending period has not elapsed yet. This is expected immediately after creating a rule.
- Recovering — the alert was firing and has now returned within threshold, but is waiting for a full evaluation cycle to confirm recovery before returning to Normal. This is a new state in Grafana 12.
- Error — the rule failed to evaluate (such as wrong data source or PromQL syntax error). Select the rule and review the query.
If any rule shows Error, select the rule and review the query — the most common cause is an incorrect data source selection or an error in the PromQL.

Tip: If you accidentally delete a rule, go to Alerting > Recently deleted to restore it within the retention window.

Verify Prometheus Data

For any alert that shows No data:

Open Explore > select Prometheus data source.
Paste the PromQL expression from that alert.
Select Run query.
If no results are returned, the underlying metric is not being scraped. Verify that the relevant exporter or service monitor is running.

Send a Test Notification

Go to Alerting > Contact points.
Select the Test notification button (paper airplane icon) next to each new contact.
Confirm the test SNS message arrives at your topic subscribers (check email, SMS, or HTTP endpoint depending on how you subscribed) and the test Slack message displays in your channel.

Trigger a Test Alert

You can temporarily lower a threshold to trigger a real alert firing, confirm routing works to SNS and Slack, then restore the threshold.

For example, you can change the PVCSpaceWarning threshold from IS ABOVE 0.70 to IS ABOVE 0.01 to fire immediately. Observe the SNS notification at your subscribed endpoint and the Slack message, then revert.

Reference Table

Alert Name	Namespace / Component	Severity	Pending Period	Fires When
KongDown	`kong`	critical	2m	Kong pods = 0 ready
QrveyAppPodDown	`qrveyapps`	critical	3m	App deployment replicas = 0
IngressEndpointUnreachable	`kong`	critical	2m	Kong proxy endpoints = 0
RabbitMQPodDown	`rabbitmq`	critical	2m	RabbitMQ pods = 0 ready
KongHighLatency	`kong`	warning	5m	P99 latency > 5000ms
DataloadPumpBacklog	`qrveyapps*`	warning	10m	Any HPA at max replicas for 10m
PVCSpaceWarning	all namespaces	warning	10m	PVC disk usage > 70%
PrometheusTargetDown	all targets	warning	5m	Any scrape target unreachable
LokiIngestionErrors	`monitoring`	warning	5m	Loki push endpoint returning 5xx
NodeCPUHighUsage	all nodes	warning	5m	Node CPU > 90%
KarpenterNodeProvisionFailed	`karpenter` (AWS)	warning	10m	Pods unschedulable for 10m
PostgreSQLConnectionsHigh	PostgreSQL (Azure)	warning	5m	Active connections > 80
ElasticsearchDiskWatermark	`elastic-system`	info	10m	ES node disk usage > 75%
RabbitMQQueueDepthHigh	`rabbitmq`	info	10m	Any queue > 1,000 ready messages
KongPrivateDown	`kong-private`	info	5m	Kong private pods = 0 ready
OtelCollectorErrors	`monitoring`	info	5m	OTel collector dropping log records

Notification Routing Summary

Severity	SNS	Slack	Delivery mode
`critical`	Yes — `SNS - Critical Alerts`	Yes — `Slack - Platform Alerts`	Immediate (group wait 30s)
`warning`	No	Yes — `Slack - Platform Alerts`	Immediate (root policy)
`info`	Yes — `SNS - Operational Digest`	No	Batched hourly message, resent every 6h

Before You Begin​

Obtain Grafana URL and Credentials​

Set Up SNS Topics​

Slack Webhook URL (required for Slack notifications)​

Verify Prometheus Datasource​

Step 1: Configure Notification Channels​

SNS Critical Alerts Contact Point​

Slack Contact Point​

SNS Operational Digest Contact Point​

Notification Routing Policy​

Step 2: Create an Alert Folder​

Step 3: Set Up Environment Down Alerts​

Create an Alert Rule​

KongDown — API Gateway Unavailable​

PromQL Expression​

Alert Condition​

Evaluation Settings​

Labels​

Annotations​

QrveyAppPodDown — Application Unavailable​

PromQL Expression​

Alert Condition​

Evaluation Settings​

Labels​

Annotations​

IngressEndpointUnreachable — No Backend Traffic Endpoints​

PromQL Expression​

Alert Condition​

Evaluation Settings​

Labels​

Annotations​

RabbitMQPodDown — Message Queue Unavailable​

PromQL Expression​

Alert Condition​

Evaluation Settings​

Labels​

Annotations​

Step 4: Set Up Early Warning Alerts​

KongHighLatency​

PromQL Expression​

Alert Condition​

Evaluation Settings​

Labels​

Annotations​

DataloadPumpBacklog​

PromQL Expression​

Alert Condition​

Evaluation Settings​

Labels​

Annotations​

PVCSpaceWarning​

PromQL Expression​

Alert Condition​

Evaluation Settings​

Labels​

Annotations​

PrometheusTargetDown​

PromQL Expression​

Alert Condition​

Evaluation Settings​

Labels​

Annotations​

LokiIngestionErrors​

PromQL Expression​

Alert Condition​

Evaluation Settings​

Labels​

Annotations​

NodeCPUHighUsage​

PromQL Expression​

Alert Condition​

Evaluation Settings​

Labels​

Annotations​

KarpenterNodeProvisionFailed​

PromQL Expression​

Alert Condition​

Evaluation Settings​

Labels​

Annotations​

Before You Begin

Obtain Grafana URL and Credentials

Set Up SNS Topics

Slack Webhook URL (required for Slack notifications)

Verify Prometheus Datasource

Step 1: Configure Notification Channels

SNS Critical Alerts Contact Point

Slack Contact Point

SNS Operational Digest Contact Point

Notification Routing Policy

Step 2: Create an Alert Folder

Step 3: Set Up Environment Down Alerts

Create an Alert Rule

KongDown — API Gateway Unavailable

PromQL Expression

Alert Condition

Evaluation Settings

Labels

Annotations

QrveyAppPodDown — Application Unavailable

PromQL Expression

Alert Condition

Evaluation Settings

Labels

Annotations

IngressEndpointUnreachable — No Backend Traffic Endpoints

PromQL Expression

Alert Condition

Evaluation Settings

Labels

Annotations

RabbitMQPodDown — Message Queue Unavailable

PromQL Expression

Alert Condition

Evaluation Settings

Labels

Annotations

Step 4: Set Up Early Warning Alerts

KongHighLatency

PromQL Expression

Alert Condition

Evaluation Settings

Labels

Annotations

DataloadPumpBacklog

PromQL Expression

Alert Condition

Evaluation Settings

Labels

Annotations

PVCSpaceWarning

PromQL Expression

Alert Condition

Evaluation Settings

Labels

Annotations

PrometheusTargetDown

PromQL Expression

Alert Condition

Evaluation Settings

Labels

Annotations

LokiIngestionErrors

PromQL Expression

Alert Condition

Evaluation Settings

Labels

Annotations

NodeCPUHighUsage

PromQL Expression

Alert Condition

Evaluation Settings

Labels

Annotations

KarpenterNodeProvisionFailed

PromQL Expression

Alert Condition

Evaluation Settings

Labels

Annotations