Senior Data Engineer

Job Description: Senior Data Engineer

Experience: 6–10 Years
Location: Bengaluru
Notice Period: Immediate to 30 Days Preferred

About the Role

We are seeking an experienced Senior Data Engineer to join our growing data engineering team. The ideal candidate will have extensive experience designing and building enterprise-scale data platforms, developing high-performance data pipelines, and optimizing modern cloud-based data architectures. This role requires strong technical expertise in Azure Data Engineering, Databricks, Spark, and distributed data processing, along with the ability to mentor team members and collaborate with cross-functional stakeholders.

Key Responsibilities

• Design, develop, and maintain scalable enterprise data platforms and modern data architectures.
  

• Build and optimize high-volume batch and real-time data pipelines using Spark, PySpark, and streaming technologies.
  

• Lead data architecture discussions and recommend best practices for scalable, reliable, and secure data solutions.
  

• Develop, optimize, and maintain Databricks workloads to improve performance and cost efficiency.
  

• Create reusable frameworks, components, and utilities to accelerate data engineering initiatives.
  

• Implement robust monitoring, logging, data lineage, governance, security, and compliance across data platforms.
  

• Design and maintain streaming data pipelines using Kafka or similar event-driven technologies.
  

• Develop and manage cloud-native data solutions using Azure Data Factory, Azure Data Lake, and related Azure services.
  

• Build and optimize data warehouses using Snowflake and SQL-based technologies.
  

• Collaborate closely with AI/ML teams to build data pipelines that support machine learning and analytics workloads.
  

• Mentor junior data engineers through technical guidance, code reviews, and best engineering practices.
  

• Participate in architectural reviews, technical design discussions, and project planning.
  

• Follow DevOps and DataOps best practices, including version control, CI/CD, testing, and deployment automation.
  

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
  

• 6–10 years of experience in Data Engineering with strong expertise in designing enterprise-scale data solutions.
  

• Hands-on experience with Azure Data Engineering services and cloud-native architectures.
  

• Strong expertise in Spark performance tuning and workload optimization.
  

• Experience working with distributed systems and large-scale data processing.
  

• Solid understanding of data modeling, ETL/ELT design, and modern data architecture principles.
  

• Experience building streaming data pipelines using Kafka or similar technologies.
  

• Strong problem-solving, analytical, and debugging skills.
  

• Excellent communication and stakeholder management skills.
  

• Experience mentoring engineers and driving technical excellence within teams.
  

Preferred Qualifications

• Experience implementing Data Governance and Metadata Management frameworks.
  

• Knowledge of CI/CD pipelines and Infrastructure as Code.
  

• Exposure to AI/ML data engineering workflows.
  

• Relevant Microsoft Azure, Databricks, or Snowflake certifications are an added advantage.
  
  

Required Technical Skills

• Python

• Scala

• Apache Spark / PySpark

• Databricks

• Azure Data Factory (ADF)

• Azure Data Lake Storage (ADLS)

• Kafka

• Snowflake

• SQL

• Git

• DataOps practices

• Data Governance and Data Quality

• Distributed Data Processing

• Cloud Data Engineering (Microsoft Azure)