Senior Data Engineer

Senior Data Engineer with 5+ years of experience

Mandatory Skills:

-Extensive experience in developing ETL pipelines,

- Databricks

-scheduling using Azure Data Factory (ADF) and/or Apache Airflow,

- performance tuning using SQL.

- SQL, Python, Spark, PySpark, and Azure.

- Fabric experience preferred.

Key Responsibilities:

• ETL Pipeline Development:
• Design, develop, and maintain ETL pipelines to extract, transform, and load data from various sources into Azure data platforms.
• Optimize ETL processes for performance and scalability.
• Implement data quality and validation checks within ETL processes.
• Scheduling and Orchestration:
• Develop and manage data integration workflows using Azure Data Factory (ADF) and/or Apache Airflow.
• Schedule and monitor ETL jobs to ensure timely data processing and availability.
• Performance Tuning:
• Optimize SQL queries and data processing tasks for performance.
• Utilize notebooks for development and performance tuning of ETL processes.
• Data Integration and Management:
• Develop and maintain data models and schemas.
• Ensure data integrity, consistency, and security across data platforms.
• Collaborate with data engineers, solution architects, and other stakeholders to understand business requirements and translate them into technical solutions.

Qualifications:

• Bachelor's degree in computer science, Information Technology, or a related field.
• 5-6 years of experience as a Data Engineer in developing ETL pipelines and scheduling using ADF and/or Apache Airflow.
• Strong technical skills in SQL, Python, Spark, PySpark, and Azure.
• Proven experience in performance tuning and optimization of SQL and ETL processes.
• Strong understanding of Data Architecture and Data Modeling.
• Experience in healthcare and pharmaceutical industries is preferred.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.

Preferred Skills:

• Knowledge of big data technologies (Hadoop, Hive, Kafka).
• Familiarity with data visualization tools (Power BI, Tableau).
• Microsoft Fabric experience.