Data engineer - Databricks

Responsibilities: As a Data Engineer, you will be responsible for designing, developing, and maintaining data solutions for data generation, collection, and processing. Your day-to-day activities will include creating data pipelines, ensuring data quality, and implementing ETL processes to migrate and deploy data across systems. You are expected to be a subject matter expert, collaborate and manage the team to perform effectively. You will be responsible for team decisions, engage with multiple teams, and contribute to key decisions while providing solutions to problems for your immediate team and across multiple teams.

Expert proficiency in Databricks Platform is required.

Expert proficiency in PySpark is recommended.

Advanced proficiency in Python (Programming Language), Apache Spark, and ETL Pipelines, along with advanced proficiency in Data Governance, is suggested. Develop innovative data solutions that enhance data accessibility and usability. Collaborate with cross-functional teams to identify and implement data-driven strategies. Monitor and optimize data pipeline performance to ensure efficiency and reliability. Conduct regular data quality assessments and implement necessary improvements. Stay updated with industry trends and emerging technologies to continuously improve data processes.3years experience and above Skillset: Databricks and using Spark for data processing (batch and/or real-time), python, sql, scala, Spark, ETL processes, data infrastructure, data ingestion and transformation, data quality and governance, AWS/Azure

• frameworks like Apache Spark and Databricks.
• Experience in Data Modeling and structuring data for efficient warehousing and analytics.
• Proficiency in Extract Transform Load (ETL) processes and frameworks for seamless data integration and transformation.
• Knowledge of Data Warehousing systems and best practices for managing large-scale data platforms.
• Understanding of Data Analytics and the ability to work with teams to derive insights from complex datasets.
• Additional experience with cloud platforms (e.g., AWS, Azure, GCP) and programming languages such as Python or Scala is advantageous.
• Bachelor's or Master's degree in Computer Science, Data Science, or a related field is preferred, along with excellent problem-solving and communication skills.