Senior Data Engineer

Company Description Verinon Technology Solutions Sdn Bhd is a consulting firm specializing in recruitment and HR practices, supporting organizations in building effective, high-performing teams. The company focuses on delivering tailored workforce solutions that align with clients business objectives and industry requirements. Verinon emphasizes professionalism, collaboration, and integrity in its partnerships with both clients and talent. Team members can expect a structured environment that values expertise, continuous improvement, and client service.

J.d.

Key Technical Skills:

• Strong proficiency in Python, PySpark, Scala Spark, Iceberg/Parquet

Airflow, Hive/SQL, Kubernetes, Shell Scripting

• Hands-on experience with Minio and Azure Data Lake Storage (ADLS) using S3 protocols.

• Solid understanding of data modeling, data warehousing, and big data technologies.

Responsibilities:

• Migrate data pipelines from existing data acquisition framework to the new GDP data acquisition framework

• Configure, develop and deliver data ingestion scripts for loading data into T1 data layer

• Develop and manage ETL/ELT workflows, ensuring data quality, integrity, and reliability.

• Integrate and automate data quality checks and validation processes within data pipelines.

• Deploy and manage containerized applications using Docker and orchestrate workloads on Kubernetes.

• Work with modern data lake and warehouse technologies such as Iceberg

• Implement real-time data streaming solutions using Kafka.

• Orchestrate complex workflows using Airflow.

• Integrate with data catalog and governance tools such as Datahub and Ranger.

• Collaborate with cross-functional teams to understand business requirements and deliver data solutions.

• Ensure security, compliance, and best practices in data management and governance.

Must Have Skills:

• Extensive background in building and maintaining data pipelines/ETL processes.

• Experience in implementing and integrating data quality frameworks.

• Extensive experience in working on migration projects moving from legacy data pipelines to modernized tech stack

Thanks & Regards

Sakshi