Infrastructure Configuration Requirements

This document describes the minimum infrastructure and platform configuration required to ensure stable execution of all workloads and avoid processing errors in the environment. Any deviation from these values can cause performance degradation, synchronization issues, delayed processing, or workload execution failures.

After applying the configuration changes, you should validate autoscaling behavior and resource utilization.

OpenSearch Configuration

The OpenSearch cluster must be configured with the following specifications:

Setting	Value
Data Nodes	2
Instance Type	r8g.xlarge.search
Snapshots	Disabled

OpenSearch Cluster Settings

Apply the following persistent cluster setting:

Setting	Value
cluster.max_shards_per_node	20000

RDS – Aurora PostgreSQL Configuration

The Aurora PostgreSQL cluster must use the following instance type:

Setting	Value
Instance Type	db.r6g.large

EKS Cluster Configuration

KEDA Configuration

ScaledObject: qrvey-dp-dr-transformation-scaledobject

Setting	Value
Min Replicas	2
Max Replicas	10

ScaledObject: dashboard-x-scaledobject

Setting	Value
Min Replicas	2
Max Replicas	2

ScaledObject: cd-manager-service-scaledobject

Setting	Value
Min Replicas	2
Max Replicas	2

ScaledObject: dp-dr-db-datasource-pumps-service-scaledobject

Setting	Value
Min Replicas	5
Max Replicas	5

Kueue Configuration - qrvey-jobs-cluster-queue-syncs

Kueue Controller

Setting	Value
Memory	1024 MiB to execute ~1000 data syncs per minute.

Resource Quota

Setting	Value
nominalQuota	22

Qrvey Pod Deployments

Deployment: qrvey-job-manager

Set the following environment variable:

Name	Value
JOB_STALL_TIMEOUT_MINUTES	55

Deployment: qrvey-dp-dr-management-service

Set the following environment variable:

Name	Value
ENABLE_SYNCS_IN_KUEUE	true

If enabled, update the cronjob labels so existing jobs can run on Kueue.

Cronjob: qrvey-metadata-vacuum-fast

Set the following environment variables:

Name	Value
METADATA_VACUUM_ES_DELETE_LIMIT	100
METADATA_VACUUM_ES_DELETE_LIMIT_DEGRADED	25
METADATA_VACUUM_S3_DELETE_LIMIT	50
METADATA_DELETED_RETENTION_DAYS	10
METADATA_CLEANUP_BATCH_SIZE	100
MULTIPLATFORM_PG_ALLOW_EXIT_ON_IDLE	true

Deployment: qrvey-dp-dr-transformation

Set the following environment variables:

Name	Value
TRANSFORM_CHUNK_QUEUE_HP_PREFETCH	20
TRANSFORM_CHUNK_QUEUE_LP_PREFETCH	20

Deployment/Kong

Private Controller

Name	Value
sync-period	0 (seconds)
update-status	false
minReadySeconds	200
initialDelaySeconds	60 for readinessProbe

HorizontalPodAutoscaler

Use the following replica settings for kong-private-gateway and kong-gateway.

Name	Value
spec.minReplicas	2
spec.maxReplicas	2

Deployment

CoreDNS

For a deployment, five replicas are recommended.

Deployment Cluster Settings

Use the following settings for Elasticsearch clusters:

Name	Value
CD_DEPLOY_MAX_INDICES_PER_CLUSTER	20000
RUNTIME_HEALTH_EVENT_LOOP_DELAY_MS	1500

Monitoring

Set the following Grafana log entries:

• ENABLE_OTLP_LOG: true
• OTLP_TRACE_URL
• OTLP_LOG_URL