Search by job, company or skills

Lead Databricks Engineer

Early Applicant
  • Posted 3 days ago
  • Be among the first 10 applicants

Job Description

ANSR is hiring for one of its clients.

Who are we:

Core Insurance Platforms (CIP) is Zurich's global capability responsible for building, running, and evolving core insurance technology. We set a unified, scalable operating model—covering governance, standards, architecture, service delivery, and reuse—so our business units can deliver at speed and scale.

CIP is the strategic steward of Zurich's Guidewire ecosystem, aligning platform roadmaps to business strategy while driving stability, modernization, reduced supplier dependency, and long term cost efficiency.

India delivery center is one of our global delivery and capability hub. We bring together experts in AI, engineering, analysis, quality, and architecture to deliver product & process solutions, application run services, change and transformation initiatives, and centralized platform services across both on prem and Guidewire Cloud environments. Our teams operate from multiple global delivery centers, supporting Zurich's business units worldwide.

What is the Financial Control - SAP Integration Team

The Financial Control - SAP Integration Team provides end-to-end capabilities for the reliable exchange of financial data between core finance platforms, SAP S/4HANA and enterprise source systems. The team designs, delivers and operates finance-critical integration solutions with strong focus on data quality, posting accuracy, governance and operational resilience.

The team is leading the transition from legacy technologies towards a Databricks-based architecture that will support future finance integration capabilities.

Your role:

As FinEnhance Databricks Engineer, you will help build and scale the strategic Databricks-based platform for global financial data integration. The role combines hands-on Databricks engineering, data architecture, SAP finance integration understanding and practical AI-first ways of working.

The two positions are expected to cover both development and architecture needs: one profile should lean senior/architecture, while the other should be strongly hands-on in engineering and delivery. Both profiles must be able to work with AI as part of daily delivery, not as a side activity.

Key responsibilities:

  • Design, build and maintain Databricks-based pipelines for financial data ingestion, transformation, validation, mapping, aggregation and SAP posting preparation.
  • Contribute to architecture standards covering Delta Lake, Unity Catalog, Medallion layers, workflows, environments and reusable deployment patterns.
  • Support the migration from legacy integration platforms existing logic and re-architecting it into scalable Databricks/PySpark solutions.
  • Use AI-supported methods to accelerate analysis, code generation support, testing, reconciliation, documentation and operational troubleshooting, while keeping human approval and quality gates in place.
  • Develop and improve operational applications and dashboards for monitoring, mapping maintenance, reprocessing and support visibility.
  • Collaborate with SAP FI, business analysts, architects, developers and production support to ensure solutions are reliable, auditable and fit for finance-critical operations.
  • Promote clean engineering practices: version control, reusable components, testing discipline, CI/CD, release traceability and environment promotion from DEV to UAT to PROD.

AI-first expectations:

  • Apply AI pragmatically to reduce repetitive manual work in discovery, design, build, test, documentation and support activities.
  • Frame problems clearly for AI tools by providing context, constraints, expected output and validation criteria.
  • Use AI to identify recurring error patterns, generate draft tests, compare expected vs. actual outcomes, support reconciliation and improve operational insights.
  • Maintain human-in-the-loop accountability: AI may accelerate outputs, but the engineer owns validation, controls, implementation quality and production impact.
  • Show curiosity, learning agility and adaptability as AI tools evolve; the mindset is as important as experience with a specific AI tool.

Required Knowledge and Skills:

Databricks Platform:

  • Strong hands-on experience with Databricks platform capabilities.
  • Experience with Unity Catalog, Workflows, Volumes, Databricks Apps, SQL Warehouses, Delta Lake and Databricks Asset Bundles.
  • Experience working across DEV/UAT/PROD workspaces with appropriate deployment and promotion practices.

Data Engineering:

  • Strong proficiency in Python, PySpark and Spark SQL.
  • Hands-on experience with Medallion Architecture, AutoLoader and ETL/ELT pipelines.
  • Experience in data validations, mappings, aggregations and posting preparation.
  • Ability to build scalable, reliable and reusable data pipelines.

SAP Finance Integration:

  • Good understanding of SAP FI concepts and finance data structures.
  • Knowledge of GL accounts, cost centers, company codes, posting keys, document types, ledger groups, posting dates and reversals.
  • Ability to translate SAP finance requirements into robust data engineering and integration solutions.

AI-First Delivery:

  • Demonstrated ability to use AI tools across the software/data engineering lifecycle.

Experience using AI for:

  • Reverse engineering
  • Requirement discovery
  • Code generation and development support
  • Test case and test data creation
  • Data reconciliation
  • Technical documentation
  • Operational troubleshooting

Operational Applications:

  • Hands-on experience with Dash/Plotly, Dash AG Grid and Dash Bootstrap Components.
  • Knowledge of Flask, Pandas, Databricks SQL Connector and Databricks SDK.
  • Experience developing operational dashboards, applications or utilities on top of Databricks.

Engineering Practices:

  • Strong understanding of Git, YAML and CI/CD practices.
  • Experience with version-controlled notebooks and reusable engineering components.
  • Good understanding of code reviews, release promotion and production support.
  • Ability to follow structured engineering, quality and operational support practices.

Preferred Experience:

  • Experience in insurance or finance data integration, including GL postings, reversals, reconciliation and data controls.
  • Exposure to heterogeneous source systems or finance/insurance platforms.
  • Experience with Lakeview Dashboards or Databricks-based operational reporting.
  • Experience migrating workloads from Informatica, DB2 or similar legacy ETL platforms to Databricks or other cloud data platforms.
  • Databricks certification or equivalent hands-on implementation experience.

Role Structure:

  • Expected Experience: 4–8 years total IT experience, including 2+ years of Databricks/Spark experience.

Key Responsibilities:

  • Build and maintain Databricks pipelines, notebooks and jobs.
  • Develop mappings, transformations, validations and reconciliation logic.
  • Build and maintain operational applications and dashboards.
  • Develop reusable engineering components and follow established patterns.
  • Create and maintain automated tests.
  • Support production stability, troubleshooting and operational activities.
  • Follow CI/CD, code review and release promotion practices.

More Info

Job Type:
Industry:
Employment Type:

About Company

Job ID: 152105307

Similar Jobs

Hyderabad, India

Skills:

Spark SQLPysparkVersion ControlDatabricksAzurePythonSqlAWSDelta Lake

Hyderabad, India

Skills:

DatabricksDevopsApache SparkWorkflowsAWSKubernetesPythonAzureGcpDockerTerraformDelta LakeAirflowKubeflowMLflowUnity Catalog

Hyderabad, India

Skills:

snowflake PysparkSqlDatabricksAWSAzureRest ApisStreaming frameworksdbtStored Procedures

Beware of Scammers

We don’t charge money for job offers