> For the complete documentation index, see [llms.txt](https://docs.veza.com/4yItIzMvkpAvMVFAamTf/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.veza.com/4yItIzMvkpAvMVFAamTf/features/lifecycle-management/how-to/duplicate-resolution.md). # Duplicate Resolution Rules Consolidate multiple records for the same person in a single source of identity into a single authoritative record before policy workflows act on them. A Duplicate Resolution Rule consolidates multiple records for the same person within a single source of identity into one authoritative record. The rule applies during workflow processing, so policy workflows act on a single resolved identity rather than once per duplicate. The most common case is a contractor-to-employee conversion in an HRIS, where the contractor record and the new employee record coexist until the contractor record is closed. Configure a rule in the Veza UI, where it appears as **Duplicate identity resolution** in a policy's identity source settings, or through the policy update API. Both write the same fields. ## When to use a rule Use a Duplicate Resolution Rule when one source of identity can produce more than one record for the same person, and a policy should act on only one of them: * **Contractor-to-employee conversion**: an HRIS keeps the open contractor record while creating a new employee record for the same person. * **System-generated duplicates**: a source of identity emits a placeholder record alongside the real one during onboarding. * **Merged data sources**: a CSV upload or custom OAA connector merges feeds that share individuals. A rule resolves duplicates within **one source of identity**. Each source carries its own rule, and a rule never matches a person across two sources. Cross-source correlation is handled by the policy's primary and secondary source configuration. See [Identities](/4yItIzMvkpAvMVFAamTf/features/lifecycle-management/identities-overview/identities.md). ## How resolution works For each extraction of the source of identity: 1. Veza groups identities that share the same values for all of the configured **deduplication key properties**. 2. Veza applies the **ranking rules** to choose the authoritative record in each group. When no rule produces a single winner, the record with the lowest entity ID is kept. Entity IDs are compared as text, not as numbers, so `10` sorts before `9`. 3. Veza hard-deletes the other records in the group before workflows evaluate the resulting identity. No soft-deleted record remains, so a dropped duplicate cannot be restored through the Lifecycle Management rehire flow. 4. Policy workflows process the authoritative record. Veza skips any group larger than the configured `max_duplicate_group_size`. Every record in a skipped group passes through unresolved, and workflows run for each one. A large group usually means that a deduplication key property is not unique to one person. {% hint style="warning" %} Enabling a rule on an existing policy, or changing ranking rules so that a different record wins, hard-deletes the non-authoritative identities and removes them from their access profiles. Those records cannot be recovered through the rehire flow, and the UI does not ask for confirmation before saving a rule. Test a rule on a separate, non-production policy first. {% endhint %} Clearing a rule does not restore the records it dropped. If the source still emits them, they return on the next extraction as **new** identities, and no downstream state from the original records is carried over. ### Confirm that a rule took effect Duplicate resolution produces few visible signals: * **No deletion event is emitted for a dropped record.** An **Identity Deleted From Sources** notification does not fire for records dropped by a rule. * **The kept record is marked.** Veza sets the system attribute `sys_attr__has_duplicates` to `true` on the surviving identity. This attribute is excluded from change detection, so setting it does not trigger a workflow. * **Counts are recorded internally.** Veza records backend metrics, including `lcm.duplicate_identities_resolved_total`, and trace logs for each resolved and skipped group. Contact Veza Support to obtain these figures for an investigation. * **The Identities page changes only after the next extraction.** Until the source of identity is extracted again, the page still lists the records that the rule will drop. ## Configure a rule in the Veza UI **Available in Veza v2026.5.25 and later.** ### Before you start * A permission set that allows updating Lifecycle Management, such as an administrator role. * A policy with a source of identity selected. The controls stay disabled until a source is chosen, because the property list comes from that source's entity type. * At least one completed extraction of that source, so that its properties are available. ### Open the duplicate resolution settings 1. Open **Lifecycle Management** and select a policy. 2. Click **Policy Settings**. 3. Open the step for the source of identity: * For the primary source, open the **Primary Identity Source** step. * For an additional source, open the **Additional Identity Sources** step. The section is at the bottom of that source's card, below the enrichment options. 4. Locate the **Duplicate identity resolution** section. If no rule exists, it reads "Not configured. Workflows treat every record as a distinct identity." 5. Click **Add Resolution**. The policy creation wizard includes both steps, so a rule can also be configured when a policy is created. ### Configure the rule The **Duplicate identity resolution** dialog contains three settings: 1. In **Match on these properties**, select the properties whose combined values identify one person. Records that share the same values for **all** selected properties are duplicates. The list contains the properties Veza discovered for the source's entity type; internal `sys_attr_` properties are not selectable. 2. In **Maximum group size**, set the largest group that Veza resolves. The field appears after at least one match property is selected, with a default of `3` and a range of 2 to 20. Veza resolves no records in a group that exceeds the limit. 3. Under **Ranking Rules**, click **Add ranking rule**, then set a **Filter**, a **Sort** property, or both: * **Filter**: a SCIM filter expression, such as `is_active eq true`. Records that match rank above records that do not. The filter evaluates only the candidate record, so it cannot reference related entities. * **Sort**: a property to sort by, such as `hire_date`. Select **Descending** to prefer the highest value. **Save** stays disabled until every rule sets a filter or a sort property. Ranking rules are evaluated in order. The first rule that produces a single winner decides the outcome. A rule that leaves a tie, or matches no records, passes the group to the next rule. Use the arrows beside a rule to reorder it, and the delete icon to remove it. For example, to keep the most recently hired active record, and the most recently updated record when none is active: * **Rule #1**: filter `is_active eq true`, sort by `hire_date`, **Descending** selected. * **Rule #2**: sort by `last_updated`, **Descending** selected. ### Save the rule 1. Click **Save** in the dialog. This stores the rule in the settings form but does not apply it. 2. Click **Save** on the **Policy Settings** page. {% hint style="info" %} Changes made in Policy Settings affect all versions of the policy, including published, draft, and retired versions. {% endhint %} The rule takes effect on the next extraction of that source of identity. See [Confirm that a rule took effect](#confirm-that-a-rule-took-effect). ### Review, edit, or remove a rule The **Duplicate identity resolution** section summarizes a configured rule: its match properties, maximum group size, and a numbered table of ranking rules. With no ranking rules, the summary reads "None — record with the smallest entity ID is kept." Click **Edit** to change the rule, or **Remove** to clear it. **Remove** does not ask for confirmation. ## Rule configuration through the API The rule is stored on the policy, not on a policy version: * `primary_duplicate_resolution_rule` on the `Policy`, for the primary source of identity. * `duplicate_resolution_rule` on each entry in `secondary_source_of_identities`, for secondary sources. Set both through the policy update endpoint, with the policy object as the request body: ```sh PUT /api/private/lifecycle_management/policies/{policy_id} PATCH /api/private/lifecycle_management/policies/{policy_id} ``` Validation rejects a rule that sets `ranking_rules` or `max_duplicate_group_size` without `dedup_key_properties`. ### Fields | Field | Type | Description | | -------------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dedup_key_properties` | array of strings | Property names whose combined values identify a person. All properties must match for two records to be duplicates; list order does not matter. Names are matched exactly against the record's own properties, so nested or dotted paths are not supported. Reserved `sys_attr_` names, empty or whitespace-only names, and duplicate entries are rejected. | | `ranking_rules` | array of `RankingRule` | Ordered rules for choosing the authoritative record. Each rule applies to ties left by the previous one. If empty, the lowest entity ID wins. | | `max_duplicate_group_size` | integer | Largest group that Veza resolves. Range: 2-20. Send `0` or omit the field to use the default of 3. Out-of-range values are clamped when the rule runs. Larger groups pass through unresolved. | #### RankingRule | Field | Type | Description | | ------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `scim_filter` | string | SCIM filter expression. Matching candidates rank above those that do not. The filter sees only the candidate identity, not related entities. | | `order_by` | string | Property to sort by within the `scim_filter` match group, or the primary sort when `scim_filter` is empty. Supports strings, numbers, booleans, and lists of strings. Matched by exact name; nested or dotted paths are not supported. | | `descending` | bool | If `true`, `order_by` sorts in descending order. | {% hint style="info" %} A `scim_filter` can reference `id` and `type`, but they are not valid for `dedup_key_properties` or `order_by` unless the record also carries them as properties. {% endhint %} ### Matching behavior Matching on `dedup_key_properties` is exact and type-aware: * Strings are compared byte for byte, so case and whitespace are significant. * Values of different types do not match: the string `"1"` and the number `1` are different. * Element order in lists and structs is significant: `[1, 2]` does not match `[2, 1]`. * A record that lacks a key property, or holds a null value for it, is not grouped and passes through unchanged. Sorting by `order_by` is **case-insensitive**. Two sort values that differ only in case are a tie, which passes to the next ranking rule. ### SCIM filter handling for API callers Validation trims whitespace around a `scim_filter` before parsing it, but the rule does not trim it at run time. A padded filter can therefore pass validation and then fail to parse during an extraction. An unparseable filter matches every candidate, which makes that ranking rule ineffective without failing the run. Send filters without leading or trailing whitespace. The Veza UI trims filters on save. A candidate against which the filter cannot be evaluated is scored as not matching, so it can lose its group and be deleted. ## API reference * [Policies](/4yItIzMvkpAvMVFAamTf/developers/api/lifecycle-management/policies.md): policy endpoints, including the update endpoint above. [Update Policy Configuration](/4yItIzMvkpAvMVFAamTf/developers/api/lifecycle-management/policies/updatepolicyconfiguration.md) updates a policy *version* and does not carry these fields. * [Policies](/4yItIzMvkpAvMVFAamTf/features/lifecycle-management/policies-workflows/policies.md): policy structure and lifecycle. * [Identities](/4yItIzMvkpAvMVFAamTf/features/lifecycle-management/identities-overview/identities.md): identity behavior, including reconciliation of primary and secondary sources of identity. ## Examples ### Match on a single field, prefer active records Group identities that share an `email`. Prefer records where `is_active` is `true`, then the most recently hired. ```json { "primary_duplicate_resolution_rule": { "dedup_key_properties": ["email"], "ranking_rules": [ { "scim_filter": "is_active eq true" }, { "order_by": "hire_date", "descending": true } ], "max_duplicate_group_size": 3 } } ``` ### Stack multiple ranking criteria Group identities that share an `employee_id`. Prefer active records, then employees over contractors, then the most recently hired. ```json { "primary_duplicate_resolution_rule": { "dedup_key_properties": ["employee_id"], "ranking_rules": [ { "scim_filter": "is_active eq true" }, { "scim_filter": "worker_type eq \"Employee\"" }, { "order_by": "hire_date", "descending": true } ], "max_duplicate_group_size": 3 } } ``` ### Match on a composite of two fields Group identities that share both `employee_id` and `country`, with the default ranking. ```json { "primary_duplicate_resolution_rule": { "dedup_key_properties": ["employee_id", "country"], "ranking_rules": [], "max_duplicate_group_size": 3 } } ``` ### Clear an existing rule Send an empty object: ```json { "primary_duplicate_resolution_rule": {} } ``` An empty object is the only way to clear a rule. When a request omits the rule or sends `null`, the API keeps the rule already stored on the policy, because the Policy Settings page does not include the rule in every request it sends. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.veza.com/4yItIzMvkpAvMVFAamTf/features/lifecycle-management/how-to/duplicate-resolution.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.