Technical Architect

Job Details

• Lead the architecture design and implementation of advanced analytics solutions using Azure Databricks Fabric The ideal candidate will have a deep understanding of big data technologies data engineering and cloud computing with a strong focus on Azure Databricks along with Strong SQL

• Work closely with business stakeholders and other IT teams to understand requirements and deliver effective solutions

• Oversee the end-to-end implementation of data solutions ensuring alignment with business requirements and best practices

• Lead the development of data pipelines and ETL processes using Azure Databricks PySpark and other relevant tools

• Integrate Azure Databricks with other Azure services eg Azure Data Lake Azure Synapse Azure Data Factory and on-premise systems

• Provide technical leadership and mentorship to the data engineering team fostering a culture of continuous learning and improvement

• Ensure proper documentation of architecture processes and data flows while ensuring compliance with security and governance standards

• Ensure best practices are followed in terms of code quality data security and scalability

• Stay updated with the latest developments in Databricks and associated technologies to drive innovation

Essential Skills

• Strong experience with Azure Databricks including cluster management notebook development and Delta Lake

• Proficiency in big data technologies eg Hadoop Spark and data processing frameworks eg PySpark

• Deep understanding of Azure services like Azure Data Lake Azure Synapse and Azure Data Factory

• Experience with ETLELT processes data warehousing and building data lakes

• Strong SQL skills and familiarity with NoSQL databases

• Experience with CICD pipelines and version control systems like Git

• Knowledge of cloud security best practices

Experience

• Demonstrated expertise of 12 years in developing data ingestion and transformation pipelines using Databricks Synapse notebooks and Azure Data Factory

• Solid understanding and hands-on experience with Delta tables Delta Lake and Azure Data Lake Storage Gen2

• Experience in efficiently using Auto Loader and Delta Live tables for seamless data ingestion and transformation

• Proficiency in building and optimizing query layers using Databricks SQL

• Demonstrated experience integrating Databricks with Azure Synapse ADLS Gen2 and Power BI for end-to-end analytics solutions

• Prior experience in developing optimizing and deploying Power BI reports

• Familiarity with modern CICD practices especially in the context of Databricks and cloud native solutions