Senior Databricks Data Quality Engineer

Role: Senior Databricks Data Quality Engineer

Location : Bangaloare & Nodia (2 Days onsite)

Shift Timing : UK Shift:

Candidate Availiiblity : Immediate Joiner

NOTE: We are specifically looking for engineers with experience in migrating from Collibra to Databricks . Also with experience in DQ migration framework generation .

Builds the reusable engine at the centre of the proposal and configures the native monitoring that replaces much of the estate. This is framework engineering rather than pipeline building — the output is a system that generates the migration, not code written rule by rule.

KEY RESPONSIBILITIES

• Build the config-driven framework: rule configuration repository, routing engine, alias resolver, code generator and results model.
• Design the configuration schema so that adding a rule after handover is a config row rather than a code change.
• Generate, deploy and support the DQ notebooks for those rules that require generated SQL.
• Build the alias resolution layer that translates Collibra @dataset references into fully-qualified Unity Catalog paths.
• Configure Databricks Lakehouse Monitoring across the in-scope tables, including slicing expressions that replace brand-partitioned rules.
• Decompose Collibra adaptive rules into metric, scope, tolerance band and score weight; lift manual-tier bands from the export and re-derive model-derived bands from Delta time travel.
• Warm monitor baselines from table history so monitors go live already informed rather than in a cold learning state.
• Wire load-triggered execution into existing Databricks Workflows and establish CI/CD through Asset Bundles.

REQUIRED QUALIFICATIONS

• 8+ years on Databricks and Spark with strong PySpark and SQL.
• Demonstrable experience building reusable frameworks or internal platforms consumed by other engineers — not only delivering pipelines.
• Production experience with Delta Lake, Unity Catalog, Databricks Workflows and Asset Bundles.
• Hands-on Databricks Lakehouse Monitoring, or comparable data-observability tooling such as Monte Carlo, Anomalo or Soda.
• Comfortable with the statistics behind behavioural monitoring — drift, baselines, tolerance bands, percentiles and sensitivity tuning.
• Writes code a client team will own and extend after handover — clarity, structure and documentation weigh as heavily as function.

PREFERRED

• Exposure to Collibra DQ or OwlDQ behavioural analytics.
• Experience with code generation, templating or metadata-driven pipeline patterns.
• Exposure to open-source DQ frameworks such as Great Expectations, DQX or dbt tests.

Regards

Vibha Patel

[Confidential Information]