Overview:
We are seeking a highly skilled Databricks Unity Catalog Consultant to help design, implement, and optimize data governance, security, and catalog management within the Databricks environment. The ideal candidate will have deep expertise in Databricks Unity Catalog, data architecture, and cloud-based data platforms (Azure, AWS, GCP).
Responsibilities:
- Implement and configure Databricks Unity Catalog for data governance, security, and access control.
- Design and enforce data access policies, role-based security, and lineage tracking across Databricks workspaces.
- Work with stakeholders to define metadata management, data classification, and auditing strategies.
- Optimize data catalog structures for performance, scalability, and compliance (e.g., GDPR, HIPAA).
- Collaborate with Data Engineers and Analysts to enable seamless data discovery and lineage tracking.
- Integrate Unity Catalog with cloud-native storage solutions (Azure Data Lake, AWS S3, GCS).
- Automate data classification, access provisioning, and audit reporting using Databricks APIs and scripting.
- Troubleshoot and resolve data access, security, and compliance issues in Unity Catalog.
Required Skills & Qualifications:
- 7+ years of experience in Data Engineering, Data Governance, or Cloud Data Architecture.
- Hands-on experience with Databricks, Unity Catalog, Delta Lake, and Lakehouse architectures.
- Expertise in Role-Based Access Control (RBAC), Attribute-Based Access Control (ABAC), and IAM policies.
- Strong experience in SQL, Python, and Spark for data management and automation.
- Familiarity with cloud platforms (AWS, Azure, or GCP) and their storage/security configurations.
- Knowledge of compliance frameworks (GDPR, HIPAA, SOC 2, etc.) and data privacy best practices.
- Experience with data lineage tools, metadata management, and auditing in enterprise environments.
- Strong problem-solving and communication skills to collaborate across teams.
Preferred Qualifications:
- Databricks certifications (Databricks Certified Data Engineer, Databricks Unity Catalog Expert, etc.).
- Experience with Terraform or Infrastructure as Code (IaC) for Databricks security and governance.
- Knowledge of machine learning model governance within Databricks.
- Experience integrating Unity Catalog with third-party data catalogs (Collibra, Alation, etc.).