Hi,
We do have opening for Data Engineer/Senior Data Engineer
Location: 1 Utama
Duration: 12 Months (Renewable)
Key Skill Requirements
- Design, develop, test, deploy and maintain ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks.
- Implement scalable data processing and transformation logic in PySpark and SQL on Databricks (Delta Lake / lakehouse patterns).
- Ingest and integrate data from diverse batch and streaming sources into ADLS Gen2 and Delta Lake.
- Build curated, documented datasets for analytics and ML (support dimensional models where appropriate).
- Implement automated data quality checks, unit/integration tests, and validation gates.
- Monitor pipeline health, implement logging, alerting and dashboards; perform RCA and production incident remediation.
- Optimize pipeline performance and cost: cluster sizing, job scheduling, partitioning, caching and query tuning.
- Implement and maintain CI/CD for data jobs and infrastructure (Terraform/ARM/Bicep, Azure DevOps or GitHub Actions).
- Ensure data governance, lineage and security best practices; register assets in a data catalog (e.g., Microsoft Purview or equivalent).
- Collaborate with cross-functional teams to collect requirements, define SLAs/SLOs and deliver production-ready solutions.
- Contribute to runbooks, documentation, coding standards and data platform roadmap.
Additional responsibilities for Senior Data Engineers
- Lead design reviews, define platform standards and approve infra decisions.
- Drive cost-control initiatives and SLA reporting.
- Participate in vendor selection, proof-of-concepts and platform architecture decisions.
For immediate response ping me on +65 8305 1174
