Lead AWS Data Engineer

Experience: 4+ Years

Location: Hyderabad(On-Site)

Notice Period: Immediate Joiner Only

Role Overview

We are looking for a Lead AWS Data Engineer with 4+ years of experience to lead the design and development of scalable, reliable data pipelines on AWS. The candidate should be highly hands-on with PySpark, SQL, Terraform, AWS data services, and Lakehouse/Medallion architecture.

Key Responsibilities

Lead the design and development of scalable AWS data pipelines using PySpark/Spark, Python, and SQL.
Implement Bronze/Silver/Gold (Medallion) architecture.
Design incremental processing, CDC, SCD Type 2, Merge/Upsert, and historical data solutions.
Handle deduplication, late-arriving/out-of-sequence data, updates, and deletes.
Build idempotent, resilient, and re-processable pipelines.
Provision and manage AWS infrastructure using Terraform.
Work with S3, Glue, EMR, Lambda, Step Functions, and related AWS services.
Drive data quality, performance optimization, monitoring, and production support.
Provide technical leadership, conduct code/design reviews, and mentor data engineers.

Required Skills

4+ years of Data Engineering experience, with data engineering.
Strong PySpark/Spark, Python, and Advanced SQL.
Strong AWS data engineering experience.
Hands-on Terraform / Infrastructure as Code.
Strong knowledge of Lakehouse/Medallion architecture.
Experience with CDC, SCD Type 2, incremental processing, deduplication, and historical processing.
Strong understanding of idempotency, merge keys, ordering columns, event time vs. ingestion time, and reprocessing.

Good communication, leadership, and problem-solving skills.