Key Responsibilities:
- Azure Databricks Platform Administration
- Administer and manage Azure Databricks workspaces clusters and jobs
- Configure and maintain Databricks environments Dev QA Prod
- Monitor cluster performance job execution and system health
- Manage notebooks libraries and dependencies
- Perform upgrades patches and platform maintenance
- Cluster Management Optimization
- Design and manage cluster configurations auto scaling instance types pools
- Optimize job performance resource utilization and execution times
- Troubleshoot cluster failures memory issues and job bottlenecks
- Implement best practices for Spark performance tuning
- Security Governance
- Implement and enforce data security and access controls RBAC ACLs IAM
- Manage workspace security secrets Key Vault integration and credential passthrough
- Ensure compliance with enterprise security and governance policies
- Implement data governance auditing and lineage tracking
- Azure Cloud Integration
- Integrate Databricks with Azure services such as
- Azure Data Lake Storage ADLS Gen2
- Azure Key Vault
- Azure Data Factory ADF
- Azure Synapse Analytics
- Manage connectivity and networking VNet Private Endpoints NSGs
Technical Requirements:
- Core Skills
- 10 15 years of IT experience with strong Azure Databricks administration
- Hands on expertise in Apache Spark and Databricks platform management
- Strong experience with Azure cloud services ADLS ADF Key Vault Networking
- Proven experience in production support and platform operations
- Technical Skills
- Deep knowledge of cluster configuration job scheduling and performance tuning
- Strong understanding of Spark architecture and optimization techniques
- Experience with Terraform ARM Bicep IaC tools
- Knowledge of security models IAM RBAC and data governance
- Familiarity with CI CD tools and DevOps practices
- Preferred Skills
- Experience with Delta Lake and Lakehouse architecture
- Exposure to Python PySpark Scala
- Familiarity with monitoring tools Azure Monitor Splunk
- Knowledge of multi cloud environments AWS GCP
- Experience in data engineering or ETL pipelines
Preferred Skills:
Technology->Cloud Platform->Azure Analytics Services->Azure Databricks
