Use Grafana for Outage Triage

You can use Grafana to investigate a Qrvey environment outage without extensive Kubernetes or infrastructure knowledge. After gathering the right information, you can then contact Qrvey Support.

Your Grafana instance is available at the URL provided by your Qrvey onboarding team.

Step 1: Confirm the Time Window

Before opening Grafana, record the exact time the issue started and (if applicable) when it recovered. Use the timezone your team works in, then convert to UTC — all Grafana metrics and logs are stored in UTC.

Example: Issue reported at 9:23 AM US Eastern (13:23 UTC).

Use this time range during the investigation.

Step 2: Open Grafana and Set the Time Range

1. Open your Grafana URL in a browser and log in.
2. In the top-right corner, select the time picker.
3. Set a custom range that covers 10 minutes before the outage started through 30 minutes after it recovered (or the current time if it is still ongoing).
4. Select Apply time range.

Tip: Keep this window open during your investigation. Use this range each time you run a query.

Step 3: Check Elasticsearch Health

The Elasticsearch database powers Qrvey analytics. Poor Elasticsearch health can produce slow queries or failures.

1. Open the Explore view.

   a. In the left sidebar, select the compass icon (Explore). b. From the datasource dropdown, select Prometheus.

2. Check Elasticsearch CPU usage.

   a. In the field provided, paste the following query:

   elasticsearch_os_cpu_percent

   b. Select Run query.

   What to look for:

   • Normal: values between 0–60%
   • Warning: any node consistently above 80%
   • Critical: any node at 95–100% → this will cause analytics to be slow or time out

   If you see a node at 100% CPU, note the name label shown in the results — this is the ES node identifier. Share this with Qrvey Support.

3. Check Elasticsearch cluster status.

   elasticsearch_cluster_health_status

   Values: 0 = green (healthy), 1 = yellow (some replicas missing), 2 = red (data loss risk).

   Any value above 0 is worth noting.

4. Check Elasticsearch disk usage.

   elasticsearch_indices_store_size_bytes_total

   Compare across nodes. If one node has significantly more data than others, it may be unbalanced.

5. Check the Elasticsearch search queue (backlog).

   elasticsearch_thread_pool_queue_count{type="search"}

Normal: 0. Any value above 0 means searches are waiting to be processed — ES is overloaded.

Step 4: Check Kong Gateway Health

The Kong API gateway routes all traffic to Qrvey services. If Kong is restarting or misconfigured, the frontend and APIs return errors.

1. In Explore, change the datasource dropdown to Loki.

2. Check for 5xx responses from Kong.

   Paste this query:

   {namespace="kong"} | pattern `<ip> - - [<_>] "<method> <uri> HTTP<_>" <status> <_>` | status >= 500

   What to look for:

   • Any results here mean Kong was returning server errors to users.
   • Note the URI and timestamp of the errors.

3. Check for Kong gateway restarts or config errors.

   {namespace="kong", pod=~"kong-controller.*"} |~ "error|Retrying|connection refused|failed"

   What to look for:

   • connection refused to an admin port (:8444) means new Kong pods were starting up during a rolling update — this is the most common cause of brief (~1 minute) frontend outages.
   • Retrying ... 0/60 means the controller just started trying to connect to new pods.

   If you see these messages clustered around your outage time window, a Kong gateway rolling update was in progress. Record the timestamps and share with Qrvey Support.

4. Check for a log gap in Kong pods (gateway was completely down).

   {namespace="kong", pod=~"kong-gateway.*"}

   If there are no log lines for a period during your outage window, the gateway pods were not running at all during that time. This is more severe than a rolling update.

Step 5: Check Application Health

(Dashboard / Frontend Service)

1. Check dashboard-x health errors.

   {namespace="qrveyapps", pod=~"qrvey-dashboard-x.*"} |~ "error|timeout|fetch failed|HealthChecker"

   What to look for:

   • fetch failed or HealthChecker timeout messages at the time of the outage — these mean the dashboard service could not reach internal services and started returning errors to users.
   • Note which service URL appears in the error (e.g. system_status_gateway, kong-private-gateway-proxy).

2. Check admin-security service (login issues).

   {namespace="qrveyapps", pod=~"qrvey-admin-security.*"} |~ "500|error"

   If users reported being unable to log in, check whether this service was returning 500 errors.

Step 6: Check RabbitMQ Health

Qrvey's data pipeline uses the RabbitMQ message broker. If data loads are failing or stuck, RabbitMQ health check errors are usually present.

1. Check system-status-service for RabbitMQ alarms

   {namespace="qrveyapps", container="qrvey-system-status-service"} |= "error"

   What to look for:**

   • Missing subscriptions for queues: ... — a data pipeline service pod was replaced and hasn't yet re-subscribed to its queue. This is usually temporary and self-resolves within a few minutes.
   • High CPU usage: 100% — Elasticsearch is overloaded (see Step 3).
   • Health check failed on multiple queues at the same time — a broader RabbitMQ disruption.

2. Check RabbitMQ server logs directly:

   {namespace="rabbitmq"} |~ "error|warning|partition|node.*down|alarm"

   What to look for:

   • partition — a network split between RabbitMQ nodes (serious, contact Support immediately).
   • alarm — memory or disk alarm triggered on a RabbitMQ node.

Step 7: Check Pod Restarts

Frequent pod restarts indicate a service is crashing and restarting — this causes brief gaps in availability.

1. Switch to Prometheus datasource.

   Run this query to see which pods have restarted recently:

   sort_desc(kube_pod_container_status_restarts_total{namespace=~"qrveyapps|kong|kong-private|rabbitmq"})

   What to look for:

   • Any pod with a restart count above 0 is worth noting.
   • Pods with 3+ restarts in your time window are likely crashing repeatedly — this is a service-level issue that requires Qrvey Support.

2. Check for pods in a bad state.

   count by (namespace, phase) (kube_pod_status_phase{namespace=~"qrveyapps|kong|kong-private", phase!="Running"})

   Pods should be in Running state. If you see Pending, Failed, or CrashLoopBackOff, note the namespace and pod name.

Step 8: Check Node Health

If many services are affected at once, the issue may be at the infrastructure (Kubernetes node) level.

1. Check for nodes not ready.

   kube_node_status_condition{condition="Ready", status="true"}

   All values should be 1. Any node showing 0 is offline. Contact Qrvey Support immediately if any node is not ready.

2. Check for node memory pressure.

   kube_node_status_condition{condition="MemoryPressure", status="true"}

   Any value of 1 means a node is running low on memory and Kubernetes is evicting pods. This cascades into pod restarts and service disruptions. Note the node name and contact Qrvey Support.

3. Check for node disk pressure.

   kube_node_status_condition{condition="DiskPressure", status="true"}

   Any value of 1 means a node is running low on memory and Kubernetes is evicting pods. This cascades into pod restarts and service disruptions. Note the node name and contact Qrvey Support.

Quick Diagnosis Reference

Use the following table to match symptoms to the most likely cause:

Symptom	First place to check	Likely cause
Frontend not loading / 500 errors	Kong controller logs	Kong gateway rolling update
Analytics slow / timing out	Elasticsearch CPU	Elasticsearch node overloaded
Data loads not completing	RabbitMQ health	Pod replacement / RabbitMQ disruption
Login not working	admin-security	Admin-security pod crash
Everything down at once	node ready status	Node failure / eviction
Brief outage (1–5 min) then self-recovered	Kong controller	Rolling update during pod churn
Outage ongoing for hours	ES CPU or memory pressure	ES overload or node memory pressure

What to Send Qrvey Support

When contacting support after running through this guide, provide the following information:

• Exact time window of the outage (start, end/ongoing, timezone).
• Screenshots of any query results from the steps above that showed anomalies.
• Log snippets - in Explore, after running a Loki query, select the log line to expand it and copy the full text.
• Environment information (NV, AU, your environment name).
• Which Step first showed an issue, and the specific error message you saw.

This allows Qrvey Support to begin investigation immediately without repeating the basic triage you have already done.

Useful Loki Log Queries by Service

Service	Query
All errors in qrveyapps	{namespace="qrveyapps", detected_level="error"}
Kong access log (all requests)	{namespace="kong"} | pattern `<ip> - - [<_>] "<method> <uri> HTTP<_>" <status> <_>`
kong-private controller events	{namespace="kong-private", pod=~"kong-private-controller.*"}
RabbitMQ all logs	{namespace="rabbitmq"}
Specific pod logs	{namespace="qrveyapps", pod="<pod-name>"}
Search all namespaces for a keyword	{namespace=~"qrveyapps|kong|rabbitmq"} |= "your search term"

To find pod names:

1. From Explore with Loki selected:Select Label browser.
2. Select namespace > qrveyapps.
3. Select pod to browse all running pod names.