Sr Data Engineer

1. Requirement Analysis Solution Design

Collaborate with business and data stakeholders to understand data requirements, transformation rules, and use cases

Translate functional requirements into scalable data engineering and ETL design solutions

Define approach for data ingestion, transformation, orchestration, and access control

2. Data Engineering Development testing

Design and develop robust ETL pipelines using AWS Glue PySpark

Implement complex data transformations including joins, aggregations, conditional logic, and business rule processing

Build reusable, metadata-driven ETL frameworks where applicable

Optimize data pipelines for performance, scalability, and reliability

Perform unit testing and end-to-end validation of data pipelines

Implement data quality checks including integrity, completeness, and reconciliation

Validate transformation logic and ensure accuracy of aggregated and processed data

3. Orchestration Integration

Develop and manage workflow orchestration using AWS Step Functions and EventBridge

Implement event-driven and schedule-based pipeline execution strategies

Ensure seamless integration between data pipelines, control frameworks, and downstream systems

4. Infrastructure Deployment IaC

Provision and manage cloud infrastructure using Terraform Infrastructure as Code

Deploy and configure AWS services including Glue, Lambda, DynamoDB, and orchestration components

Ensure consistent, repeatable, and scalable deployments aligned with DevOps practices

6. Security Governance

Implement secure data access controls using IAM and Lake Formation

Ensure compliance with data governance policies and role-based access requirements

Manage data security, encryption, and access auditing

7. Monitoring, Support Continuous Improvement

Set up monitoring, logging, and alerting mechanisms for data pipelines e.g., SNS, audit logs

Troubleshoot data and pipeline issues, ensuring timely resolution

Continuously enhance pipeline performance, reliability, and maintainability

Collaborate in an agile setup to drive continuous delivery and improvements

 Must-Have Skills

· 7-10 years of AWS Data Engineering experience.

· Strong experience in AWS Glue PySpark and data processing on S3

· Proven expertise in ETL development and complex data transformations

· Hands-on experience with AWS Step Functions, EventBridge, and orchestration patterns

· Proficiency in Terraform Infrastructure as Code

· Strong SQL and data modeling skills

· Experience with data validation, reconciliation, and quality frameworks

· Solid understanding of IAM, security, and cloud best practices

________________________________________

Nice-to-Have Skills

· Familiarity with Lake Formation and data governance frameworks

· Experience with DynamoDB or metadata-driven ETL architectures

· Exposure to event-driven architectures

· Knowledge of CICD tools