Migrate Public OpenSearch to VPC OpenSearch

This page describes how to migrate an existing AWS OpenSearch Service cluster from a public endpoint configuration to a VPC-private deployment. This migration applies to customers upgrading from Qrvey v8 or earlier, where the AWS OpenSearch domain was deployed with a publicly accessible endpoint and all traffic between the EKS cluster and the OpenSearch domain traveled through a NAT Gateway.

Qrvey supports deploying the AWS OpenSearch Service domain in private VPC subnets using the opensearch_config variable. For new deployment setup, see Deployment on AWS.

Overview

In Qrvey v8 and earlier, the AWS OpenSearch cluster was deployed with a public endpoint by default. This meant all traffic between the EKS cluster and the OpenSearch domain traveled through a NAT Gateway, incurring data-transfer costs and exposing the endpoint outside the VPC.

Starting in v9.3, Qrvey supports migrating the AWS OpenSearch Service domain to private subnets of your VPC. This eliminates NAT Gateway traversal and makes the domain accessible only from within the VPC.

Key Benefits

• Cost reduction — Eliminates NAT Gateway data-transfer charges for OpenSearch traffic.
• Security — The domain is only accessible from within the VPC (no public endpoint).
• Encryption — The system enforces TLS v1.2. Encryption at rest and node-to-node encryption are enabled by default.
• Low-downtime migration — Both clusters can coexist during the migration window, allowing the platform to continue operating.

IAM Policy Dual-Cluster Compatibility

During the migration window, the old public domain and the new VPC domain coexist. The IAM policies (elasticsearch_policy and opensearch_policy) include Resource ARNs for both domains, ensuring pods can access either cluster without permission errors.

When migration is complete and the old domain is removed, the old ARN entries become inert (no matching resource). There is no need to immediately remove them.

Before You Begin

• The existing environment must have a working public AWS OpenSearch domain.
• opensearch_config.enabled must be set to true in config.json.
• elasticsearch.host must be set to the current public domain endpoint (non-empty).
• Ensure sufficient EBS storage on the new domain to hold all existing indices.

Migrate a Public OpenSearch Domain to VPC OpenSearch

Step 1: Update config.json

Add the opensearch_config block while keeping the existing elasticsearch block intact:

{
  "variables": {
    "elasticsearch": {
      "host": "https://search-qrvey-xxxxx-xxxxxxxxxx.us-west-2.es.amazonaws.com",
      "cluster_name": "qrvey-xxxxx",
      "auth_user": "",
      "auth_password": ""
    },
    "opensearch_config": {
      "enabled": true,
      "instance_type": "r6g.large.search",
      "instance_count": 2,
      "volume_size": 100
    }
  }
}

Important: Do not remove the elasticsearch block during this step. Both clusters must coexist during migration.

Step 2: Deploy the VPC Domain

Run the apply command to create the VPC OpenSearch domain without triggering migration yet:

docker run --platform=linux/amd64 -v $(pwd)/config.json:/app/qrvey/config.json -it --rm qrvey.azurecr.io/qrvey-terraform-aws:${qrvey_version} apply

This command performs the following tasks:

1. Creates the VPC OpenSearch domain and all supporting infrastructure.
2. Updates the platform ConfigMap to point to the new VPC domain.
3. Restarts services to pick up the new endpoint.

After this step, the platform starts writing new data to the VPC domain. However, historical data from the old public domain is not yet available.

Step 3: Run the Migration

Trigger the data migration from the old public domain to the new VPC domain using the --migrate-opensearch flag:

docker run --platform=linux/amd64 -v $(pwd)/config.json:/app/qrvey/config.json -it --rm qrvey.azurecr.io/qrvey-terraform-aws:${qrvey_version} apply --migrate-opensearch

This command performs the following tasks:

1. Creates the S3 snapshot bucket and IAM snapshot role (if not already present).
2. Deploys a Kubernetes Job that runs the migration script.
3. Takes a snapshot of all indices from the old public cluster (stored in S3) and restores them to the new VPC cluster.

The command waits for the job to complete (timeout: 2 hours) and streams logs to the terminal.

To monitor progress separately:

kubectl logs -f job/<migration-job-name> -n qrveyapps

Note: Previously loaded data is not available in the platform until this step completes. Loading data for new datasets is also blocked during migration.

Step 4: Verify the Migration

After the job completes, verify that the indices are present on the new cluster:

# Get the new domain endpoint from Terraform outputs
docker run --platform=linux/amd64 -v $(pwd)/config.json:/app/qrvey/config.json -it --rm qrvey.azurecr.io/qrvey-terraform-aws:${qrvey_version} output

# Check indices (run from within the VPC or cluster)
curl -s "https://vpc-qrvey-xxxxx-vpc-xxxxxxxxxx.us-west-2.es.amazonaws.com/_cat/indices?v&s=index"

Step 5: Remove the Old Domain Reference

1. After verifying the indices, clear the elasticsearch values in config.json:

   {
     "variables": {
       "elasticsearch": {
         "host": "",
         "cluster_name": "",
         "auth_user": "",
         "auth_password": ""
       },
       "opensearch_config": {
         "enabled": true,
         "instance_type": "r6g.large.search",
         "instance_count": 2,
         "volume_size": 100
       }
     }
   }

2. Run the apply command:

   docker run --platform=linux/amd64 -v $(pwd)/config.json:/app/qrvey/config.json -it --rm qrvey.azurecr.io/qrvey-terraform-aws:${qrvey_version} apply

3. After confirming the migration was successful, remove the old public OpenSearch domain:

   a. Delete the old domain from the AWS Console or its Terraform/CloudFormation stack.

   b. Confirm that elasticsearch.host is cleared in config.json.

   The IAM policies still contain ARN entries for the old domain. This is harmless because the old domain no longer exists.

Entrypoint Flags Reference

--migrate-opensearch

Command: docker run ... apply --migrate-opensearch

Triggers the OpenSearch S3 snapshot migration job. Only runs when both conditions are met:

• opensearch_config.enabled = true
• elasticsearch.host is set (non-empty)

The entrypoint performs the following tasks:

1. Removes previous migration jobs from the Terraform state.
2. Deletes pre-existing migration jobs from the cluster.
3. Runs terraform apply to create the migration job (timeout: two hours).
4. Streams job logs to the terminal in real-time.

--skip-all-hooks

Command: docker run ... apply --skip-all-hooks

Skips all pre/post hooks, validations, restarts, and cleanup. Equivalent to passing all of the following flags at once:

• --skip-restart
• --skip-data-sync-jobs
• --skip-cleanup-orphans
• --skip-workflows-hooks

Additionally skips:

• Network CIDR validation
• EC2 instance quota validation
• Pre-apply hooks (suspend cronjobs, disable data sync)
• Post-apply hooks (translation sync, permissions, re-enable data sync)
• Rollout restart
• Helm deployment readiness wait

Use this flag when running targeted applies (--target) where you only need Terraform to apply specific resources without the full deployment lifecycle:

# Example: Apply only the general configmap, skip everything else
docker run ... apply --target=kubectl_manifest.general_configmap --skip-all-hooks

Terraform Outputs

The following outputs are available after deploying with opensearch_config.enabled = true:

Output	Description
opensearch_domain_endpoint	The HTTPS endpoint of the VPC OpenSearch domain
opensearch_domain_arn	The ARN of the VPC OpenSearch domain
opensearch_domain_name	The domain name (qrvey-<deployment_id>-vpc)
opensearch_snapshot_bucket	The S3 bucket used for migration snapshots
opensearch_snapshot_role_arn	The IAM role ARN used by OpenSearch for S3 access

All outputs return empty strings when opensearch_config.enabled = false.