Databricks

JD Below:

We're seeking a hands-on Senior Data Engineer to design, implement and operate scalable data pipelines on the Databricks Lakehouse Platform. You'll own ETL/ELT flows, Delta Live Tables, pipeline performance tuning, CI/CD for data engineering, and data governance enabling BI and advanced analytics across the business.

Key responsibilities

• Design and implement scalable ETL/ELT pipelines using Apache Spark, Delta Lake, and Databricks Workflows (including Delta Live Tables).
• Build, maintain and operationalize Databricks-based data products to support BI, reporting and advanced analytics.
• Optimize Spark jobs and Delta queries with partitioning, predicate pushdown, join strategies, caching, and resource tuning.
• Implement CI/CD practices for notebooks and pipelines using Databricks Repos, Git, and Azure DevOps (or equivalent).
• Secure and govern data using Unity Catalog, role-based access control, and data masking techniques.
• Integrate Databricks with upstream/downstream systems (e.g., AWS S3, Snowflake, Power BI).
• Develop, publish, and maintain documentation and best practices (notebooks, runbooks, data lineage).
• Monitor pipeline health, implement alerting, and perform root-cause analysis for failures or performance regressions.
• Mentor junior engineers and promote re-usable, testable data engineering patterns.
• Work with product, analytics and platform teams to define SLAs, data contracts, and onboarding processes.

Required qualifications

• 58 years experience in data engineering, analytics engineering, or platform engineering roles.
• Proven, hands-on experience with Databricks, Apache Spark, and Delta Lake.
• Strong proficiency in Python and SQL for data processing and transformations.
• Deep understanding of distributed computing, data partitioning, and cloud data architectures.
• Experience building production CI/CD pipelines for data workloads (Databricks Repos, Git workflows, Azure DevOps or similar).
• Solid experience with performance tuning of Spark jobs and query optimization.
• Familiarity with data security best practices and role-based access controls.
• Excellent communication and documentation skills; experience producing runbooks and onboarding docs.

Nice-to-have

• Hands-on experience with Delta Live Tables, Databricks Asset Bundles, and Databricks Workflows.
• Knowledge of Lakehouse architecture, Unity Catalog, Databricks SQL and query acceleration features.
• Experience integrating with Snowflake, AWS S3, and BI tools like Power BI.
• Prior work with metadata management, data governance, or masking/tokenization tools.
• Databricks certifications (e.g., Databricks Certified Data Engineer Associate/Professional).
• Familiarity with Infrastructure as Code (Terraform) for provisioning cloud and Databricks resources.