Databricks Solution Architect (SME)

Skills and Abilities

Required:

• Deep expertise in Databricks platform components including Unity Catalog, Delta Lake, Workflows, SQL Warehouse, LakeBase, Genie Code, and Delta Live Tables (DLT)
• Strong knowledge of Posit platform ecosystem including Workbench, Connect, and integration patterns with R and Python environments
• Proficient in distributed computing technologies and big data analytics, including hands-on expertise with Python, Spark, SQL, and data transformation pipelines
• Strong understanding of MLOps/ModelOps principles, practices, and tooling for model development, deployment, monitoring, and governance
• Experience with cloud infrastructure services (AWS EC2, S3, Lambda, Azure Databricks, Azure ML, etc.) and infrastructure-as-code tools (Terraform, CloudFormation)
• Knowledge of CI/CD pipelines, DevOps, and DataOps workflows using tools such as Git, Jenkins, Azure DevOps, or Databricks Asset Bundles
• Understanding of data security principles including authentication, authorization, encryption, RBAC, and compliance frameworks
• Strong understanding of distributed systems, data modeling, API design, and integration patterns
• Excellent communication skills with ability to present complex technical concepts to both technical and non-technical stakeholders
• Proven ability to translate business requirements into technical architecture and design specifications
• Ability to quickly adapt to new technologies, work under tight deadlines, and manage multiple priorities
• Strong analytical and problem-solving abilities with investigative mindset

PRINCIPAL DUTIES AND RESPONSIBILITIES-ESSENTIAL FUNCTIONSSolution Architecture and Design: Provide architectural leadership and consultation for AI/ML, advanced analytics, and data science solutions leveraging the enterprise ModelOps platforms, including Databricks and Posit. Define and promote architecture standards, reference patterns, and best practices for model development, deployment, monitoring, governance, and lifecycle management. Translate business and technical requirements into scalable, secure, and maintainable solution architectures that align with enterprise architecture standards and cloud-native principles. Establish integration patterns across data, application, and governance ecosystems while ensuring solutions are designed for performance, reliability, and operational supportability. Conduct architecture reviews and provide technical guidance to enable consistent, reusable, and sustainable solution delivery across development teams.

1. Technical Leadership and Strategy: Provide technical leadership and strategic direction for the adoption and implementation of ModelOps solutions across the organization. Translate complex business, analytical, and operational requirements into scalable, secure, and maintainable architecture strategies and roadmaps. Lead architectural decision-making for data pipelines, machine learning models, applications, APIs, and end-to-end ML workflows while ensuring alignment with enterprise architecture standards and organizational objectives. Define and promote best practices, design patterns, and governance standards for Databricks (including Unity Catalog, Workflows, Delta Live Tables, and SQL Warehouses) and Posit (including Workbench, and Connect). Partner closely with platform engineering and enablement teams to drive platform adoption, influence future platform capabilities, and ensure consistent implementation of enterprise standards across solution development teams.
2. Stakeholder Collaboration and Communication: Translate business, analytical, and operational requirements into clear technical specifications, architectural designs, and implementation recommendations. Communicate architectural strategies, solution options, and technical tradeoffs to both technical and non-technical audiences, facilitating informed decision-making and stakeholder buy-in. Build and maintain strong relationships with technical champions and key stakeholders across the organization to promote adoption of enterprise platforms, standards, and best practices. Foster collaboration across cross-functional teams to drive successful delivery of scalable, secure, and business-aligned AI/ML solutions.
3. Model Lifecycle Management: Establish and guide the adoption of enterprise ModelOps frameworks that support the full model lifecycle, from development and validation through production deployment, monitoring, and retirement. Define architectural patterns and best practices for model versioning, reproducibility, model registries, automated testing, deployment pipelines, A/B testing, and model retraining workflows. Ensure solutions align with enterprise model governance standards, validation processes, monitoring requirements, and operational support frameworks. Collaborate with data science, engineering, and risk management teams to promote reliable, transparent, and auditable model operations. Ensure compliance with regulatory requirements, internal policies, and risk management standards while supporting the delivery of scalable and sustainable AI/ML solutions.
4. Performance Optimization and Scalability: Design and promote architecture patterns that optimize the performance, scalability, reliability, security, and cost efficiency of AI/ML and data science solutions across enterprise ModelOps platforms. Lead the development of distributed computing strategies leveraging Databricks capabilities, including cluster management, workload orchestration, and compute optimization. Define resource allocation, capacity planning, and scaling frameworks to support current and future business demands across Databricks and Posit environments. Establish architectural standards for observability, monitoring, alerting, and performance measurement to ensure platform health, operational resilience, and proactive issue identification. Collaborate with platform engineering teams to continuously evaluate and enhance platform utilization, throughput, and operational efficiency while maintaining alignment with enterprise standards and service-level objectives.
5. Technical Mentorship and Documentation: Provide technical leadership, guidance, and mentorship to data scientists, data engineers, and development teams to promote adoption of enterprise architecture standards, platform capabilities, and ModelOps best practices. Conduct architecture reviews, design assessments, and technical consultations to ensure solutions align with established patterns, governance requirements, and operational objectives. Lead training sessions, workshops, and knowledge-sharing initiatives to build organizational capabilities and drive consistent implementation of AI/ML solutions across teams. Foster a culture of continuous learning, technical excellence, and collaboration while enabling teams to effectively leverage Databricks, Posit, and related platform services.
6. Risk Management and Compliance: Establish architectural controls, governance frameworks, and risk mitigation strategies to ensure secure, reliable, and compliant AI/ML solutions throughout the model lifecycle. Partner with data governance, information security, model risk management, and compliance teams to ensure adherence to data privacy, security, regulatory, and enterprise policy requirements. Design solutions that support auditability, data lineage, model traceability, and operational transparency through comprehensive monitoring, logging, and reporting capabilities. Define and promote architecture patterns for business continuity, backup, disaster recovery, and operational resilience across ModelOps platforms and supporting technologies. Proactively identify, assess, and mitigate technical and operational risks while ensuring alignment with governance requirements within highly regulated insurance and financial services environments.

JOB SPECIFICATIONS

Education and Experience

Required:

• Bachelor's degree in Computer Science, Information Systems, Software Engineering, Data Science, or related technical field
• 8+ years of experience in solution architecture, data engineering, data science, or related technical roles
• 3+ years of hands-on experience designing and implementing enterprise-grade data science and analytics solutions
• Proven experience with cloud platforms (AWS, Azure, or GCP) and cloud-native architecture principles

Preferred:

• Master's degree in Computer Science, Information Systems, Software Engineering, Data Science, or related field
• Databricks Certified Solutions Architect or equivalent certification
• Cloud architecture certifications (AWS Solutions Architect, Azure Solutions Architect, or GCP Professional Architect)
• Experience in insurance, financial services, or highly regulated industries

Preferred:

• Production programming experience in Python, R, Scala, and/or Java
• Experience with additional ML platforms and frameworks (SageMaker, MLFlow, AzureML, Kubeflow)
• Experience with high-availability architectures, load balancing, and disaster recovery design
• Familiarity with monitoring and observability tools (Prometheus, Grafana, Datadog, CloudWatch)
• Understanding of microservices architectures and event-driven systems
• Knowledge of data governance tools and practices (Atlan, Apache Atlas, Collibra, data catalogs)
• Experience with streaming data platforms (Kafka, Kinesis, Event Hubs)
• Knowledge of insurance industry data models, risk analysis, and actuarial modeling workflows
• Experience with regulatory compliance requirements (SOC 2, GDPR, HIPAA)
• Ability to conduct proof-of-concept assessments and technical evaluations of emerging technologies
• Experience presenting to and influencing C-level executives and senior leadership

All qualified applicants will receive consideration for employment without regard to age, race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran. Rate of pay within the stated range will depend on the qualification of the applicant.