Managed Datasets

Managed Datasets in Qrvey provide scalable, secure storage and fast analytics using a high-performance, serverless OpenSearch cluster. They support combining multiple data sources and offer advanced features for data transformation, synchronization, and visualization.

Managed Datasets, Dataset Views, and Base Datasets are only available in Qrvey Ultra.

Why Use Managed Datasets?

Managed Datasets are ideal for:

• Fast, scalable analytics and reporting
• Combining multiple data sources (joins, unions)
• Advanced data transformations and formatting
• Secure, centralized data management

Key Features:

• High-Performance Storage: Data is stored in Qrvey’s serverless OpenSearch cluster for fast analytics.
• Multi-Source Support: Combine multiple data sources using joins and unions.
• Advanced Transformations: Apply formatting, calculated fields, and data transformations during dataset design.
• Automated Synchronization: Supports scheduled syncs, incremental updates, and automatic column discovery.
• Secure and Centralized: Centralized data management with granular access controls.
• Integration: Use as sources for dashboards, reports, Dataset Views, and Base Datasets.
• Scalability: Designed to handle large data volumes (for example, 1M+ records) and complex data models.
• Sharing: Managed Datasets can be shared across applications for collaborative analytics.

Use Case

Best for large-scale analytics, dashboards, and reports that do not require real-time data.

Create a Managed Dataset

1. Go to Data > Datasets.
2. Select Create New Dataset > New Managed Dataset.
3. Select a data source (existing connection, dataset, or shared structure). To add a new connection, select Create New Connection.
4. Select your data source, then select Next.
   • For databases, pick a table/view or enter a custom SQL query.
   • To preview data, hover over a column and select View Source Data.
5. (Optional) For long queries (10+ min), select This is a long-running query.
6. Select Save to open the Design page.
7. Configure the dataset as desired.
   For more information, see Dataset Design.
8. The dataset saves in draft mode with a default name, which you can edit.
9. Select Load Dataset to open a modal.
   • Select estimated dataset size and select Load. Progress displays at the top-right.
   • You can continue working while loading.
   • You can adjust the estimate in Advanced Settings.
   • After loading, the Summary view appears.
   • Check the Activity Log for events or errors.

Save as Base Dataset

You can save any Managed Dataset as a Base Dataset, including those with geolocation or internationalization enabled, or those that are shared. If you save a shared Managed Dataset as a Base Dataset, the new Base Dataset are not shared by default.

Saving as Base Dataset optimizes storage and performance for downstream Managed Datasets. See Base Datasets for details and limitations (including DatasetView restrictions).

After saving, set up Data Synchronization to keep your data current.

View Source Data

Displays the first 100 rows from the data source.

Modify the Connection

1. Select the three-dot menu on the data source pill.

   

2. Choose:

   • Edit Connection Settings — Change connection properties.
   • Edit Query — Change table/view or SQL query (for databases/MongoDB).
   • Join to Another Data Source — Add another source. See Data Joins.
   • Union with Another Data Source — Append data. See Data Unions.
   • Switch for Another Data Source — Use a different connection.
   • Rename — Change data source name.
   • Delete — Remove a joined data source.

Note: Removing/renaming columns or changing data types can break existing visualizations.

Automatically Add New Columns

Enable Qrvey to add new columns found during data reloads (disabled by default):

1. Scroll to the bottom of the page.
2. Select Automatically add new columns discovered during data loads.

Select Dataset Columns

Use the checkboxes to include and exclude columns:

• Dataset: Include in dataset. Required columns for sync/join should not be removed.
• Visualizations: Show/hide in visualizations (for example, hide ID columns).

Note: Columns with Geolocation cannot be removed.

Change Column Name

Select the column name box to edit. Hover over the field for details (friendly name, original name, data type).

Tip: Use the Search box to find columns quickly.

Change Data Type

Qrvey auto-detects data types. You can change them by selecting the Column Type field and selecting a new type.

If the data has already been loaded, perform a reload.

Change Input Format

If the type of data in a column is unclear to Qrvey, you can adjust as needed. Requires data reload.

Change Visualization Format

Change how dates/numbers display in visualizations (no reload needed). Useful for formatting currencies, dates, and so on.

Custom date format syntax:

Syntax	Type
DD	day
HH	hour
mm	minute
ss	second
SSS	milliseconds
YYYY	year
MM	month

Select Unique Identifier Columns

For Data Synchronization in Append and Update mode, set a unique identifier:

1. Select the column's three-dot menu.
2. Go to Unique ID > On.
   • Requires data reload.

Automatic Column Links

Configure columns to display as links in visualizations (for example, map links). See Column Links.

Configure Geolocation Groups

See Geolocation Groups for displaying addresses on maps.

Transform Column Data

Apply Transformations (for example, uppercase, parse JSON) to modify data after loading.

Comparing Custom and Managed Datasets

Custom Datasets are created from exported Dashboard data and are managed separately from Managed Datasets. They are automatically kept in sync with their source dataset (every 12 hours at most) and cannot be edited in the Design tab. Management actions for Custom Datasets (delete, turn sync on/off) are available from the Data section or Dashboards. Their sync schedule is tied to the source dataset.

Limitations and Considerations

• Available in Qrvey Ultra tier only.
• Performance depends on data volume and transformation complexity.
• High-priority/low-priority queue mechanism is executed during both the data ingestion and transformation stages.
• Data must be synced with its data source.

See Also

• Base Datasets
• Live Connect Datasets
• Dataset Views
• Introduction to Datasets