Data Resiliency Engineer (Datalake)

About PayNet:

Payments Network Malaysia (PayNet) is Malaysia's national payments network and central financial market infrastructure. We design, build, and operate secure, resilient, and always‑on payment systems that enable trusted digital payments across the country.

About the Technology Division: Building the backbone of Malaysia's payment infrastructure. Led by Chief Technology Officer, Technology designs, evolves, and secures the always-on architecture that powers trusted, scalable digital payment services for the nation.

About the Role:

As a Data Resiliency Engineer, you will help design and safeguard the data backbone of Malaysia's national payment systems. This role offers hands‑on exposure to large‑scale, high‑availability payment data platforms, where reliability, data accuracy, and operational excellence are non‑negotiable.

You will work on complex, real‑world data systems powering payment schemes such as DuitNow and MyDebit, supporting high‑volume transactional data, regulatory reporting, and business‑critical analytics. Working closely with data engineers and platform teams, you will strengthen pipeline reliability, observability, and data quality across modern data lake architectures deployed at scale.

Beyond day‑to‑day operations, this role provides deep involvement in production readiness, release support, and disaster recovery simulations, giving you rare exposure to how national‑scale financial data platforms are engineered to remain resilient under continuous demand.

Key Responsibilities:

• Ensure reliability and availability of production data lake and analytics platforms supporting critical payment systems.
• Investigate data and pipeline incidents, conduct root cause analysis, and implement sustainable fixes.
• Design, operate, and continuously improve monitoring and alerting using observability tools (e.g. Datadog).
• Serve as a key contact for data platform support, including ETL failures, reporting issues, and data discrepancies.
• Build and enhance data quality monitoring frameworks to ensure accurate, reliable, and trusted data.
• Support production deployments, maintenance activities, and disaster recovery simulations.

What will make you successful

You enjoy working on systems where reliability truly matters, and where engineering decisions have tangible real‑world impact. You are comfortable operating in production environments at scale, investigating complex data issues, and thinking proactively about failure prevention.

You thrive in roles that combine deep technical problem‑solving with operational ownership, value exposure to advanced data platforms, and take pride in delivering resilient, trusted data services that support national‑level financial infrastructure.

Must-have:

• Hands‑on experience supporting production data platforms, including incident response and operational troubleshooting in regulated or high‑availability environments.
• Practical understanding of modern data lake and data pipeline architectures, designed for scalability, observability, and resilience.

Advantage to have:

• Strong grounding in data engineering concepts and end‑to‑end data pipelines.
• Experience with PySpark, Polars, Dask, or similar data processing frameworks.
• Exposure to Kubernetes‑based data platforms, including Amazon EKS.
• Working knowledge of AWS data services (S3, Glue, Athena, QuickSight, Lambda).