Role Overview:
We are seeking a highly skilled and consultative Senior AWS Data Engineer to join our team for a high-impact, client-facing engagement. In this on-site role, you will act as a technical leader, partnering directly with customers to design, plan, and execute end-to-end data pipelines.
The ideal candidate goes beyond coding—you must be able to confidently lead architectural discussions, justify design decisions, map theoretical concepts to real-world business problems, and clearly articulate implementation plans to client stakeholders.
Key Responsibilities:
Client Consulting & Architecture
- Lead on-site customer workshops to understand business requirements, capture transformation rules, and design end-to-end data flows.
- Clearly articulate and justify architectural design decisions, explaining which data design patterns best suit specific client use cases.
- Plan and sequence end-to-end data engineering implementations, defining clear roadmaps for ingestion, transformation, and access control.
- Map theoretical technical knowledge to practical, real-world solutions for the client.
Data Engineering & Development
- Design and develop robust, metadata-driven ETL pipelines using AWS Glue and PySpark on S3.
- Implement complex data transformations, including joins, aggregations, conditional logic, and business rule processing.
- Optimize data pipelines for performance, scalability, and reliability.
- Implement rigorous data quality checks, including integrity, completeness, and reconciliation validation.
Orchestration & Infrastructure
- Develop and manage workflow orchestration using AWS Step Functions and EventBridge for both event-driven and schedule-based execution.
- Provision and manage cloud infrastructure using Terraform (Infrastructure as Code).
- Deploy and configure AWS services (Glue, Lambda, DynamoDB) ensuring consistent, repeatable deployments aligned with DevOps practices.
Security, Governance & Operations
- Implement secure data access controls using AWS IAM and Lake Formation.
- Ensure compliance with data governance policies, managing encryption and access auditing.
- Set up monitoring, logging, and alerting mechanisms (e.g., SNS, CloudWatch, audit logs) to troubleshoot issues and drive continuous pipeline improvements.
Qualifications & Requirements:
Must-Have Skills & Experience:
- 7–10 years of hands-on experience in AWS Data Engineering.
- Strong Client-Facing Experience: Proven ability to work on-site, lead customer conversations, and present technical concepts to both technical and non-technical stakeholders.
- Architectural Mindset: Demonstrated ability to plan end-to-end data implementations and defend design choices.
- Deep expertise in AWS Glue, PySpark, and data processing on S3.
- Production-level experience with orchestration patterns using AWS Step Functions and EventBridge.
- Proficiency in Terraform for Infrastructure as Code (IaC).
- Strong SQL capabilities and data modeling skills.
- Experience building data validation, reconciliation, and quality frameworks.
- Solid understanding of cloud security, IAM, and AWS best practices.
Nice-to-Have Skills:
- Familiarity with AWS Lake Formation and data governance frameworks.
- Experience with DynamoDB or metadata-driven ETL architectures.
- Exposure to event-driven architectures.
- Knowledge of CI/CD tools and automated deployment pipelines.
