Description
We are looking for a skilled Data Engineer with experience in Databricks to join our team in Malaysia. The ideal candidate will have a strong background in data engineering, capable of designing and maintaining robust data pipelines and collaborating with various stakeholders to drive data initiatives.
Responsibilities
- Design, build and maintain data pipelines on Databricks to support data ingestion, processing, and analysis.
- Collaborate with data scientists and analysts to understand data requirements and provide necessary datasets.
- Optimize and improve existing data pipelines for performance and scalability.
- Implement data quality checks and monitoring to ensure data accuracy and reliability.
- Work with cloud-based data storage solutions such as AWS S3, Azure Data Lake, or Google Cloud Storage.
- Develop and maintain documentation for data processes, pipelines, and architecture.
Skills and Qualifications
- 4-10 years of experience in data engineering or a related field.
- Proficiency in Apache Spark and Databricks environment for big data processing.
- Strong programming skills in Python, Scala, or SQL.
- Experience with ETL/ELT processes and tools.
- Knowledge of data modeling and database design principles.
- Familiarity with cloud platforms (AWS, Azure, or GCP) and their data services.
- Understanding of data warehousing concepts and technologies.
- Excellent problem-solving skills and the ability to work collaboratively in a team.
