Whether you have already worked with PII data or heard of requirements to handle PII data, you are probably aware that there are several ways or techniques in which you can make your data compliant with certain frameworks or just ensuring that selected personas within your organization can access certain types of data (such as HR data or the clients’ personal information). In this article, I will be talking about two ways in which we can approach this challenge in databricks
What is PII Data?
Personally Identifiable Information (PII) encompasses any data element—or combination of elements—that can be used to distinguish, contact, or locate an individual. In modern enterprise lakehouses, PII ranges from direct identifiers (such as Social Security numbers, email addresses, and phone numbers) to indirect identifiers (such as IP addresses, geospatial coordinates, or combined demographic traits).
Restricting PII is not merely a technical hygiene exercise; it is a critical requirement driven by strict data privacy compliance frameworks like GDPR, CCPA, and HIPAA. Unrestricted access to PII creates severe regulatory liabilities and security exposure. Within a unified analytics platform like Databricks, securing PII requires balancing two competing goals: ensuring sensitive data remains completely masked or filtered for unauthorized users, while ensuring compliance rules do not create massive query latency or operational overhead for data engineering teams.
What are Ruled-Based Materialized Views in Databricks?
Ruled-Based Materialized Views (often implemented using dynamic views, manual row filters, column masks, or pre-filtered Delta materialized views) rely on explicit declarative SQL logic or pre-computed structures bound to specific dataset definitions. In this architecture, security logic—such as filtering rows via current_user() or is_member() conditions or masking columns with dynamic SQL functions—is manually assigned to individual tables or encapsulated inside dedicated view layer definitions.
Because these views or table-level Row-Level Security (RLS) / Column-Level Masking (CLM) definitions are bound directly to explicit targets:
- Logic is defined on a table-by-table or view-by-view basis.
- Access controls are managed locally by table owners or schema administrators who define explicit user-defined functions (UDFs) to mask columns or exclude rows at query execution.
- Redacted datasets can be physically materialized into distinct, isolated storage layers to deliver max-performance pre-aggregated reporting for specific consumer groups.
While this approach gives dataset authors precise, granular control over how a specific table behaves, managing individual policies across hundreds of dynamic views or underlying tables quickly introduces governance bloat as data estates expand.
This method requires explicit and deliberate adjustments to the data pipeline in order to provide masked data, which can be a drawback to catalog or company wide requirements. However, masked views fully support query optimization and predicate pushdown, so they can offer better query performance than row filters and column masks.
What is ABAC?
Attribute-Based Access Control (ABAC) in Databricks Unity Catalog is a dynamic, policy-driven security framework that evaluates access based on attributes (tags) assigned to users, tables, and columns—rather than explicit, table-by-table dynamic views.
This feature reached General Availability (GA) in databricks at the end of April of 2026 and has been receiving updates ever since. Given how recent this feature is, some questions may arise and hopefully this article will answer some of them.
Instead of attaching security logic directly to every individual table:
- Data Stewards Tag Data: Governance teams apply standardized tags (e.g., tagging a column as PII = True or Sensitivity = High) across catalogs, schemas, or tables.
- Policy Authors Define Rules: Central administrators write top-level ABAC policies attached at the catalog or schema level using tag-matching functions like
has_tag()orhas_tag_value(). - Unity Catalog Enforces Automatically: When a user queries a dataset, Unity Catalog evaluates the user’s role against the tags dynamically at runtime, applying required column masks or row filters before execution begins.
ABAC separates duties: data owners manage data, while security teams enforce global access policies centrally across the entire catalog. Crucially, any newly ingested table or column automatically inherits governance as soon as it is tagged.
By automatically inheriting governance guidelines, the company or the catalog in question is less at risk of any breach of sensitive data being revealed by accident and it also allows for better auditing on data retrieval.
When should you use one versus the other?
Choosing between ABAC Policies and Table-Level Ruled-Based Materialized Views / RLS comes down to governance scope, data model growth, and ownership responsibilities.
| Feature / Scenario | ABAC (Attribute-Based Access Control) | Ruled-Based Materialized Views / Table RLS & CLM |
|---|---|---|
| Governance Scope | Global & Scalable: Applied at catalog/schema levels; automatically inherits policies when tags are added. | Local & Specific: Managed on a single-table or single-view basis. |
| Policy Management | Centralized: Managed by central policy admins; lower-level table owners cannot override or bypass rules. | Decentralized: Managed by individual table or view owners. |
| Best Used When… | You have large, growing data estates with repetitive PII patterns across thousands of columns. | You have bespoke, complex multi-table transformation rules or a small, fixed set of tables. |
| Maintenance Overhead | Low: Write once, govern everywhere via tags. | High at Scale: Every new table or schema requires manual UDF wiring and view definitions. |
| Advanced Features | Supports EXCEPT clauses (e.g., allowing specific pipeline service principals to run Time Travel, OpenSharing, or clones). | Bound tightly to query-time UDFs; does not easily support administrative overrides for system operations. |
Choose ABAC when: You are scaling an enterprise lakehouse, separating duties between data stewards and data engineers, and need consistent PII masking (e.g., email, SSN) enforced across hundreds of datasets automatically without relying on individual engineers to write manual view logic.
Choose Ruled-Based Materialized Views / Local RLS when: You have isolated, highly specific filtering logic unique to a single table, require pre-aggregating data for distinct non-PII analytical consumption layers, or need to shape complex multi-table joins specifically for custom reporting subsets.
Should you only have one or the other?
Not at all, these features, while achieving the same goal, are not mutually exclusive. Let’s imagine the following scenario, your company has two main personas, engineers and analysts while your governance approach allows for three layers of restriction, being general, level 1 and level 2 restrictions, for example.
Analysts usually leverage read intensive workflows, so the burden of creating and managing the views with access rules may be paid off by better performances at the query level which can result in lower overall costs and you would not need to worry about sharing access to the underlying data.
For engineers writing data to the tables, ABAC policies may be a better choice if you want to ensure that data writers can only read data they are allowed to read based on their level of restriction. In this scenario a write once, use many times policy will ensure governance
If you have views set up and want to migrate, follow this approach
The best way to approach this challenge is to see it as a phased migration instead of changing all at once. For this example, let’s create a scenario where you have bronze, silver and gold catalogs across development, preproduction and production environments.
In order to leverage a phased approach, the best way is to add policies and enable proper attribute tagging in the bronze layer for the development environment and validate that access, classification and compliance follow your expectations and start moving down the layers to silver and gold. This would allow you to validate most use cases and point analysts and read intensive operations to the now ABAC controlled catalog in order to test pipelines and assess costs.
From this point, one should repeat the same process, but for preproduction. If your organization leverages this environment properly, it will allow you to test production level applications and use cases without risking introducing breaking changes to production.
Finally, with previous environments validated, you can apply the changes to production. If your organization does not have a preproduction, you should be extra careful when applying changes directly to production and monitor all relevant metrics, such as performance in time and compute as well as ensuring pipelines do not break.
Once you have successfully implemented ABAC policies in all your catalogs across your environments, you can slowly deprecate the ruled-based views, from least critical to most, allowing time for a smooth transition and adoption of new practices.
