PySpark, Spark Developer
PySpark, Spark Developer
Infosys Limited5-8 Years
- Posted 9 days ago
- Be among the first 10 applicants
Job Description
Key Responsibilities:
- ob Title PySpark Spark Developer
- Experience 5 8Years
- Design develop and maintain scalable data processing solutions using Apache Spark and PySpark
- Build and optimize ETL ELT pipelines for large scale data processing
- Develop Spark applications for batch and real time data processing
- Analyze transform and load structured and unstructured data from multiple sources
- Tune Spark jobs for performance scalability and reliability
- Work with distributed computing frameworks and big data technologies
- Collaborate with Data Engineers Data Architects and Business Analysts to understand requirements
- Troubleshoot production issues and provide efficient solutions
- Implement data quality checks and monitoring mechanisms
- Follow coding standards version control and CI CD best practices
Technical Requirements:
- Required Skills
- Strong experience in PySpark Apache Spark and Python
- Good understanding of Spark Core Spark SQL DataFrames and RDDs
- Experience with Hive HDFS SQL and data warehousing concepts
- Knowledge of Azure Databricks AWS EMR or Hadoop Ecosystem
- Experience with performance tuning and query optimization
- Hands on experience with Git and Agile methodologies
- Strong analytical and problem solving skills
Preferred Skills:
Technology->Big Data - Data Processing->PySpark,Technology->Big Data - Data Processing->SparkMore Info
Key Skills
About Company
Skills:
Spark SQL, Agile Methodologies, Pyspark, Performance Tuning, Data Warehousing, Apache Spark, Azure Databricks, Sql, Query Optimization, Spark Core, Git, Hadoop Ecosystem, Hive, Python, AWS EMR, RDDs, HDFS, DataFrames
Skills:
Spark SQL, Agile Methodologies, Pyspark, Performance Tuning, Apache Spark, Data Warehousing, Azure Databricks, Sql, Query Optimization, Spark Core, Git, Hive, Hadoop Ecosystem, Python, AWS EMR, RDDs, HDFS, DataFrames
Skills:
data wrangling , slicing , data pipeline, Concurrency, Core Python, Pyspark, Apis, Debugging, Oop, Numpy, Git, Pandas, Restful Services, Data Loading, Decorators, application unit tests, Lambda Functions, DB connectivity, Gen AI, Vector operations, dataframes, apply map functions, applymap, error handling, aggregations
Skills:
Spark SQL, Pyspark, Banking Domain Knowledge, Sql, Performance Tuning, Logging and monitoring, Dataframes, ETL Estimation, Unix Scheduling Tool, Generic process development, Advanced Data transformations, Error handling, UDFS, Spark Functions, Teradata BTEQ
Skills:
Power Bi, Adf, Pyspark, Azure Databricks, Sparksql, Azure DevOps, Azure SQL DW