Data Engineering & Platform Development
- Design, develop, and maintain scalable data pipelines and data products using Databricks and PySpark.
- Build and optimize ETL/ELT solutions supporting batch and near real-time data processing.
- Design, develop, and maintain enterprise Data Warehouse and Lakehouse solutions that support reporting, analytics, and AI/ML use cases.
- Develop and maintain enterprise data models using Data Lake and Delta Tables.
- Design and implement dimensional data models, including Fact and Dimension tables, Star Schema, Snowflake Schema, and Slowly Changing Dimensions (SCD).
- Ensure data quality, scalability, security, and performance across the data platform.
- Implement best practices for code management, testing, deployment, and operational monitoring.
Databricks Platform & Governance
- Implement and manage Unity Catalog for centralized governance, data discovery, and security.
- Design and enforce governance frameworks including:
- Role-Based Access Control (RBAC)
- Attribute-Based Access Control (ABAC)
- Fine-grained data permissions
- Data lineage and auditing
- Configure and manage Data Sharing to support secure external and internal data collaboration.
- Support the adoption and utilization of Databricks Genie, including:
- Natural language data interaction capabilities
- Security and access governance controls
Performance Optimization
- Perform advanced PySpark performance tuning and troubleshooting.
- Optimize query performance, cluster utilization, partitioning strategies, and workload management.
- Identify bottlenecks and proactively improve platform efficiency and cost optimization.
- Optimize Data Warehouse and Lakehouse workloads to support high-performance reporting and analytics processing.
DevOps & Automation
- Design and implement CI/CD pipelines for Databricks solutions.
- Integrate Databricks development lifecycle with GitHub, GitHub Actions, and enterprise DevOps processes.
- Automate deployment, testing, code validation, and release management processes.
- Establish infrastructure and data engineering best practices.
Stakeholder Management
- Engage with business users, data consumers, architects, analysts, and technology leadership to gather requirements and deliver data solutions.
- Translate business requirements into scalable technical designs, data models, and platform capabilities.
- Communicate effectively with stakeholders across multiple organizational levels.
- Work independently while managing priorities and ensuring timely delivery of commitments.
- Provide technical guidance and mentorship to junior team members when required.
Production Support
- Participate in a rotating production support roster.
- Troubleshoot production incidents and prioritize issue resolution within established SLA requirements.
- Conduct root cause analysis and implement preventive measures.
- Ensure platform stability, reliability, and operational excellence.
Experience
- Bachelor's Degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.
- 8+ years of experience in Data Engineering, Data Warehousing, or Big Data technologies.
- Minimum 4+ years of hands-on Databricks experience in enterprise environments.
- Experience designing and implementing enterprise Data Warehouse solutions and modern Lakehouse architectures.
Why You'll Love Working With Us✨
- Up to 24 Days Annual Leave
- Flexible & Hybrid Working
- Performance Bonus
- Enhanced EPF Contributions
- Comprehensive Insurance Coverage
- Training & Development (In-House & External)
- Travel Allowance & Expense Benefits
- Company Trips & Team Events
- Long Service Rewards
- Individual Expenses Benefits
*OKU candidates with physical (mobility-related) disabilities or hearing impairment are encouraged to apply.*
