Senior Data Engineer

Bullet-Pointed JD:

• Design, architect, and develop solutions using cloud big data technology to ingest, process, and analyze large, disparate data sets
• Develop systems to ingest, cleanse, normalize datasets and build pipelines from various sources, structuring previously unstructured data
• Collaborate with internal teams and external professionals to gather requirements and identify data development opportunities
• Understand and map data flow across applications like CRM, Broker & Sales tools, Finance, HR, etc.
• Unify, enrich, and analyze diverse data to generate insights and business opportunities
• Design and develop data management and persistence solutions using relational and non-relational databases
• Create POCs to validate solution proposals and support migration initiatives
• Build data lake solutions to store structured and unstructured data from multiple sources and guide teams in adopting modern tech platforms
• Follow CI/CD processes and best practices in development to strengthen data engineering discipline
• Mentor team members and contribute to overall organizational growth

What we are looking for:

• 6+ years of experience and a bachelor's degree in Information Science, Computer Science, Mathematics, Statistics, or a related quantitative field
• Hands-on engineer with curiosity for technology and adaptability to evolving tech landscapes
• Understanding of Cloud Computing (AWS, Azure preferred), Microservices, Streaming Technologies, Network, and Security
• 3+ years of development experience using Python-Spark, Spark Streaming, Azure SQL Server, Cosmos DB/MongoDB, Azure Event Hubs, Azure Data Lake Storage, Azure Search
• Design and develop data management and persistence solutions with a focus on enhancing data processing capabilities
• Build, test, and improve data curation pipelines integrating data from DBMS, file systems, APIs, and streaming systems for KPI and metric development
• Maintain platform health, monitor workloads, and act as SME for assigned applications in collaboration with Infrastructure Engineering teams
• Team player with a self-motivated, reliable, and disciplined work ethic, capable of managing multiple projects
• 3+ years of experience with source code control systems and CI/CD tools
• Independent and capable of managing, prioritizing, and leading workload efficiently