Senior Data Engineer
Job Description
Role: Data Engineer – Databricks (Pipeline/CI/CD)
Experience: 5+ Years
Notice Period: Immediate Joiners Only
Job Summary
We are looking for an experienced Data Engineer with strong expertise in Databricks, data pipeline development, and CI/CD practices to design, develop, and maintain scalable data engineering solutions.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks.
- Build and optimize ETL/ELT workflows for efficient data processing.
- Implement CI/CD pipelines for automated testing, deployment, and release management.
- Develop and maintain data integration workflows and transformation processes.
- Monitor pipeline performance, troubleshoot issues, and optimize data processing.
- Collaborate with cross-functional teams to deliver reliable data solutions.
Required Skills
- 5+ years of relevant experience in Data Engineering.
- Strong hands-on experience with Databricks.
- Experience in data pipeline development and orchestration.
- Practical knowledge of CI/CD tools and deployment practices.
- Strong SQL and Python skills.
- Experience with cloud-based data platforms is preferred.
More Info
Key Skills
cloud-based data platforms
CI CD
data pipeline development
ETL ELT workflows
About Company
Skills:
Kubernetes, Sql, Lambda, Pyspark, Git, Scala, S3, Shell scripting, Github, Kinesis, AWS, Python, Docker, Ec2, Redshift, AI coding assistants, LlamaIndex, Airflow, dbt, GenAI, Glue, LangChain, RAG pipelines, LLM tools, prompt engineering
Skills:
AWS, Databricks, Github, Pandas, Pyspark, Numpy, Python, Azure, Docker, Gcp, Kedro, Great Expectations, scikit-learn
Skills:
S3, Hadoop, Pyspark, PostgreSQL, Scala, Apache Spark, Svn, Autosys, Sql, HTML, Git, Hive, Elasticsearch, Databricks, Python, Unix Shell Scripting, Airflow, AWS EMR, ETL pipelines
Skills:
snowflake , SQL Server, Sql, Python, Microsoft Azure, ETL processes
Skills:
bigtable , BigQuery, Performance Tuning, Hadoop Ecosystem, Pyspark, Apache Spark, Data Modeling, Dataproc, Sql, ELT, Data Integration, DataFlow, Python, Etl, Cloud Functions, Optimization of large-scale data processing workloads, Cloud Composer, Google Cloud Storage, Pub/Sub