I
PySpark, Spark Developer
I
- Posted 9 days ago
- Be among the first 10 applicants
Job Description
Responsibilities :
ob Title: PySpark / Spark Developer Experience: 5-8Years Design, develop, and maintain scalable data processing solutions using Apache Spark and PySpark. Build and optimize ETL/ELT pipelines for large-scale data processing. Develop Spark applications for batch and real-time data processing. Analyze, transform, and load structured and unstructured data from multiple sources. Tune Spark jobs for performance, scalability, and reliability. Work with distributed computing frameworks and big data technologies. Collaborate with Data Engineers, Data Architects, and Business Analysts to understand requirements. Troubleshoot production issues and provide efficient solutions. Implement data quality checks and monitoring mechanisms. Follow coding standards, version control, and CI/CD best practices.Technical and Professional Requirements:
Required Skills Strong experience in PySpark, Apache Spark, and Python. Good understanding of Spark Core, Spark SQL, DataFrames, and RDDs. Experience with Hive, HDFS, SQL, and data warehousing concepts. Knowledge of Azure Databricks, AWS EMR, or Hadoop Ecosystem. Experience with performance tuning and query optimization. Hands-on experience with Git and Agile methodologies. Strong analytical and problem-solving skills.More Info
Job Type:
Industry:
Role:
Function:
Employment Type:
Key Skills
About Company
Skills:
Spark SQL, Agile Methodologies, Pyspark, Performance Tuning, Data Warehousing, Apache Spark, Azure Databricks, Sql, Query Optimization, Spark Core, Git, Hadoop Ecosystem, Hive, Python, AWS EMR, RDDs, HDFS, DataFrames
Skills:
Spark SQL, Agile Methodologies, Performance Tuning, Pyspark, Data Warehousing, Apache Spark, Azure Databricks, Sql, Query Optimization, Spark Core, Git, Hive, Hadoop Ecosystem, Python, AWS EMR, RDDs, HDFS, DataFrames
Skills:
data wrangling , slicing , data pipeline, Concurrency, Core Python, Pyspark, Apis, Debugging, Oop, Numpy, Git, Pandas, Restful Services, Data Loading, Decorators, application unit tests, Lambda Functions, DB connectivity, Gen AI, Vector operations, dataframes, apply map functions, applymap, error handling, aggregations
Skills:
Spark SQL, Pyspark, Banking Domain Knowledge, Sql, Performance Tuning, Logging and monitoring, Dataframes, ETL Estimation, Unix Scheduling Tool, Generic process development, Advanced Data transformations, Error handling, UDFS, Spark Functions, Teradata BTEQ
Skills:
Power Bi, Adf, Pyspark, Azure Databricks, Sparksql, Azure DevOps, Azure SQL DW