Extraction and Discovery Intervals

Customize how often Veza updates your Authorization Graph

Veza periodically connects to integrated systems to maintain an up-to-date graph of authorization and relationships between entities in your environment. This process involves two main activities: discovering new data sources and extracting authorization metadata.

By customizing extraction intervals globally or per-integration, you can achieve both:

• Cost Reduction: Longer extraction intervals can reduce compute costs for systems like Snowflake, where each extraction may incur API charges.

• Performance Improvement: Increasing intervals for integrations with large data sources can prevent bottlenecks and reduce delays in updating other integrations and services.

All integrations have a default auto setting which can vary depending on the integration type. For most integrations it is 1h. Some integrations are adjusted to reduce cost and potential bottlenecks, such as for Sharepoint Sites (24h), Snowflake (6h), and Box (24h).

The auto setting allows Veza to manage extraction timing based on system load and integration type. When you override with a manual setting, Veza will schedule extraction precisely at that interval regardless of system conditions.

An administrator can change the default value for each integration, or set global extraction and discovery intervals.

Discovery Interval

• Determines how often Veza checks for newly added data sources in your integrated systems.

• The discovery interval can be set between 15 minutes and 30 days.

Extraction Interval

• Determines how often Veza collects authorization metadata to update entities in the Authorization Graph.

• Extraction intervals can be set between 1 hour and 30 days.

• Gathers supported entities and their attributes, which can take some time for large data sources.

• More frequent extractions ensure entity relationships and attribute updates are current but may impact system performance.

Manual/On-Demand Extraction

In addition to scheduled extractions, you can manually trigger synchronization for individual data sources. This is particularly useful for immediate updates, testing, and time-sensitive changes, such as updating Access Graph metadata for new Access Reviews or Lifecycle Management workflows.

How to Trigger Manual Extraction

You can manually start extraction in two ways:

From the All Data Sources page:

1. Navigate to Integrations > All Data Sources

2. Filter to locate the data source you want to extract

3. Click the Start Extraction button in the Actions column

4. The extraction will be queued and the status will update to show progress

Triggering a manual extraction from All Data Sources view.

From Integration Details:

1. Navigate to Integrations and select a specific integration

2. Click the Data Sources tab

3. Click the Start Extraction button next to the desired data source

4. Monitor the extraction status in the Status column

Manual extractions run with the same permissions and configuration as scheduled extractions. The next scheduled extraction will still occur at its regular interval.

Prioritization: User-initiated extractions are automatically prioritized in the extraction queue with higher priority than scheduled extractions.

Availability: Manual extraction is available for all native integrations (e.g., Okta, AWS, Azure, Snowflake). After initiating extraction the status will show "Extraction In Progress" with a timestamp. You can monitor progress via the status indicators and click on error messages for full details.

How to Customize Intervals

1. Open the Administration page on the main Veza navigation bar.

2. Go to the System Settings tab.

3. Scroll down to the Integrations section.

4. Use the dropdown menus to set global values for discovery and extraction intervals.

   • Discovery Interval: How frequently to discover datasource instances.

   • Extraction Interval: How frequently to extract datasources.

5. To customize intervals for specific providers:

   • Use the Search field to filter for a specific integration.

   • Set custom values for individual providers as needed (from auto to 1-30 days).

After updating an override, it will be listed beneath the global values, along with any other active overrides. Changes take effect immediately, with the next scheduled extraction/discovery following the new interval settings.

Considerations

Tuning extraction and discovery intervals can optimize performance while maintaining an up-to-date view of your authorization landscape:

• Balance frequency of updates with system performance and cost.

• Monitor the impact of your changes to ensure they meet your organization's needs.

• Remember that less frequent updates may result in slightly outdated data between extractions.