We are looking for a Data Integration Engineer to build and run reliable data pipelines on Microsoft Azure. You will use Azure Databricks, PySpark, Spark SQL, Delta Lake and Azure Data Factory to support reporting and data products using a Medallion Architecture approach. You own delivery, quality and secure operations.
Responsibilities
- Build and maintain data pipelines using Azure Databricks, PySpark and Spark SQL
- Implement Medallion Architecture patterns using Delta Lake tables and workloads
- Develop and schedule orchestration workflows in Azure Data Factory
- Integrate data from Azure Data Lake Storage (ADLS) Gen2, Azure SQL Database and REST application programming interfaces (APIs)
- Implement secret management using Azure Key Vault and Databricks secret scopes
- Maintain continuous integration and continuous delivery (CI/CD) using GitHub Actions and GitHub
- Support reporting enablement and dataset operations in Power BI Service
- Apply security best practices across Azure and Databricks using service principals and role-based access control (RBAC)
Requirements
- Hands-on data engineering experience in production environments
- Strong knowledge of Apache Spark, Spark SQL and Delta Lake
- Experience building pipelines with Medallion Architecture concepts
- Advanced Structured Query Language (SQL) skills, including common table expressions (CTEs), MERGE, window functions and complex transformations
- Python scripting capability for data integration, including REST API calls
- Experience with Azure services such as Azure Data Factory, ADLS Gen2, Azure SQL Database, Azure Synapse Analytics, Azure Key Vault and Azure Identity
- Knowledge of dimensional modeling, including fact tables, dimension tables and star schemas
- Experience implementing data quality checks, logging and monitoring
Nice to have
- Experience with Microsoft Fabric
We offer
- By choosing EPAM, you're getting a job at one of the most loved workplaces according to Newsweek 2021 & 2022&2023.
- Employee ideas are the main driver of our business. We have a very supportive environment where your voice matters
- You will be challenged while working side-by-side with the best talent globally. We work with top-notch technologies, constantly seeking new industry trends and best practices
- We offer a transparent career path and an individual roadmap to engineer your future & accelerate your journey
- At EPAM, you can find vast opportunities for self-development: online courses and libraries, mentoring programs, partial grants of certification, and experience exchange with colleagues around the world. You will learn, contribute, and grow with us
