We are looking for a Senior Data Engineer to work for one of our clients based in Malaysia(Shah Alam, Selangor) on their permanent roles
Job Description
Join Keypath Education as a Senior Data Engineer and play a pivotal role in shaping our data-driven future across the Asia-Pacific region.
In this role, you'll collaborate closely with the APAC Data Engineering Manager and team to design and maintain critical data infrastructure and pipelines that power strategic insights and innovative solutions. As a technical expert, you'll develop system integrations, build complex data models, and create semantic layers that transform raw data into actionable intelligence—driving efficiency, reliability, and growth.
If you're passionate about solving complex challenges and delivering robust data products that enable advanced analytics and reporting, we'd love to hear from you.
Key Accountabilities
Data Integration:
- Establish data engineering best practices and governance standards
- Architect, design, and develop scalable and efficient data integration pipelines using Azure and Microsoft Fabric Pipelines, Dataflows Gen2 to ingest structured, semi-structured, and unstructured data from diverse sources including APIs, databases, file shares, and data streams
- Develop resilient data integration solutions leveraging Azure Functions, Fabric Notebooks (PySpark & SparkSQL), and .NET codebases, and related data lifecycle development frameworks
- Utilize Microsoft Fabric OneLake as the unified data lake foundation for seamless data integration across workloads
Data Modelling:
- Design and develop comprehensive data architecture based on Medallion Architecture
- Develop business-friendly data models using Kimball dimensional modelling techniques within Microsoft Fabric Lakehouse and Data Warehouse, ensuring scalability and efficiencyDesign, develop, and manage Power BI semantic models and Direct Lake mode datasets to support business intelligence initiatives and self-service capabilities
- Maintain data dictionaries and metadata repositories using Fabric's data catalog and lineage capabilities to document data models and their elements, certify and promote models and their definitions, ensuring compliance with cross-region and customer data policies
Insights & Analytics:
- Design and develop comprehensive Power BI solutions including advanced data models, interactive reports, data visualizations, and dashboards through expert use of Power Query, DAX, and Power BI Service
- Implement robust Power BI security, including Row-Level Security, to meet diverse cross-functional team and regional access needs
- Manage the Power BI instance including premium capacity, workspaces, datasets, monitoring, troubleshooting, and user access issues
Advanced Analytics:
- Establish AI/ML engineering best practices and standards
- Design and develop scientific modeling and Machine Learning solutions using native Microsoft Fabric services to generate actionable insights
- Implement predictive and prescriptive analytics solutions and advanced analytics methodologies to optimize decision-making and improve business outcomes
- Leverage Copilot in Microsoft Fabric to accelerate analytics development and enhance productivity
Delivery:
- Deliver complex data engineering solutions using Agile methodologies and iterative development processes
- Ensure quality of data engineering solutions by aligning with endorsed architecture, development, productivity standards, and processes
- Collaborate closely with the Product team and subject matter experts to understand and translate data engineering and integration requirements into technical solutions
- Ensure alignment of data engineering solutions with broader organizational strategic objectives and key performance indicators
DevOps:
- Design, develop, and maintain robust Continuous Integration and Continuous Deployment (CI/CD) pipelines for seamless system integrations
- Develop and maintain automation frameworks that enhance deployment efficiency and reduce manual intervention
- Conduct thorough code reviews and manage pull request approvals in Azure DevOps to ensure code quality and knowledge sharing
Operations:
- Ensure operational stability and risk reduction by proactively monitoring the product and platform health, capacity, and security of the entire data estate within Microsoft Fabric, maintaining regulatory compliance and data protection
- Manage and resolve data-related incidents through thorough root cause analysis and implementing timely and effective remediation activities
- Optimize the performance of data pipelines, queries, and overall data products for enhanced speed, efficiency, scalability, and cost reduction
- Review and implement continuous improvement activities to boost data product efficiency and quality
Qualifications
Education:
Minimum BS/BA Degree, preferably in a Technology, Computer Science, Mathematics or Statistics related discipline
Experience:
- 4+ years of data engineering experience with a proven history of successfully designing, developing, and deploying data warehouse and data integration solutions using Azure and Microsoft Fabric technologies
- 3+ years of experience with Power BI and proficiency with semantic modeling and data packaging concepts
- 2+ years of experience with Agile and Scrum engineering practices and principles
Technical:
- Expert in Microsoft SQL Server database and Azure data engineering (Azure Data Factory, Azure Functions, Azure Data Lake, Logic Apps, Microsoft Fabric) with strong ETL/ELT and SQL/T-SQL scripting skills
- Proven expertise in Business Intelligence and reporting tools (Microsoft Power BI, DAX, Power Query, Power BI Semantic Models) and related tools (Power Automate, Visual Studio, Tabular Editor, ALM Toolkit)
- Hands-on experience with Machine Learning and Python programming skills for buildingML workflows
- Strong understanding of the CI/CD process using Azure DevOps (Pipelines, Boards, Git/GitHub integration)
Additional:
- Think logically and be able to articulate thoughts in a clear, concise manner
- Comfortably work both individually and as part of APAC team with team members located in Australia and Asia
- Take ownership of assignments and see them through to completion
- Provide technical leadership and mentorship
- Quickly learn and adapt when dealing with unfamiliar systems and applications
- Be able to communicate clearly and effectively with business partners and co-workers
- Meet deadlines and complete work in a timely manner
- Be willing to go the extra mile to get the job done, including occasional overtime
- Display exceptional follow-through, personal drive, and the desire to make a difference
- Contribute to and participate in an open environment with a team that likes to have fun at work
Please share your updated resume to [Confidential Information]
