Senior AI Engineer/Senior AI developer

Senior AI Engineer / Senior AI Developer

About the Role

The AI Lab builds AI-enabled products for the bank, ranging from Retrieval-Augmented Generation (RAG) systems, AI agents, classification and extraction pipelines, decision-support applications, and AI-powered digital experiences.

We are seeking a highly experienced Senior AI Engineer who can independently define, design, build, deploy, and operate production-grade AI systems. This is a senior individual contributor role responsible for owning ambiguous business problems, translating them into scalable technical solutions, and leading delivery from concept through governance, deployment, and operational support.

The successful candidate will act as the technical anchor for AI projects, ensuring systems are reliable, observable, secure, maintainable, and capable of evolving as AI models, prompts, retrieval mechanisms, and providers change over time.

________________________________________

Key Responsibilities

AI Solution Design & Delivery

• Own ambiguous business and product problems and transform them into clearly defined, scalable AI solutions.

• Lead end-to-end project delivery, including architecture, development, evaluation, governance coordination, deployment, and production rollout.

• Design moderately complex AI and software systems with consideration for latency, scalability, reliability, security, cost, and maintainability.

• Define system boundaries, interfaces, workflows, data flows, observability requirements, and failure-handling strategies.

• Make informed decisions on what to build, buy, automate, or exclude based on business value and technical feasibility.

Generative AI & LLM Engineering

• Design and implement production-grade Generative AI applications using LLM platforms such as OpenAI, Azure OpenAI, Anthropic, AWS Bedrock, or equivalent.

• Develop prompt engineering strategies including structured prompting, few-shot learning, function calling, JSON outputs, prompt versioning, and orchestration workflows.

• Evaluate and optimize model performance through offline and online evaluation frameworks, golden datasets, automated evaluations, and quality monitoring.

• Determine appropriate use cases for LLMs, smaller models, fine-tuning, retrieval strategies, or non-AI alternatives.

• Implement agentic workflows, tool integrations, structured outputs, memory management, and multi-step AI processes where appropriate.

Retrieval-Augmented Generation (RAG)

• Design and own end-to-end RAG architectures including ingestion, parsing, chunking, embedding, indexing, retrieval, reranking, generation, and post-processing.

• Develop retrieval strategies leveraging dense, sparse, and hybrid search techniques.

• Optimize retrieval quality using query rewriting, reranking, metadata filtering, multi-query retrieval, and evaluation metrics.

• Work with vector databases such as FAISS, Pinecone, Weaviate, OpenSearch, pgvector, or Milvus.

• Establish retrieval and answer quality evaluation frameworks using metrics such as Recall@K, MRR, faithfulness, relevance, and citation accuracy.

Reliability, Observability & Production Ownership

• Design observability into systems from the outset, including logging, metrics, tracing, alerting, and performance monitoring.

• Define and monitor KPIs such as latency, error rates, retrieval quality, token consumption, cost per request, and evaluation drift.

• Implement resilient architectures using retries, fallbacks, circuit breakers, graceful degradation, and recovery mechanisms.

• Own production incidents, root-cause analysis, postmortems, and continuous improvement initiatives.

• Ensure systems remain operational and performant during model changes, provider upgrades, and retrieval index updates.

AI Security, Governance & Risk Management

• Design AI systems with robust controls against prompt injection, jailbreaks, data leakage, and unauthorized tool access.

• Implement PII detection, masking, redaction, and data governance requirements.

• Support model governance and validation processes through documentation, risk assessments, decision logs, audit trails, and reproducibility practices.

• Maintain prompt, model, and configuration versioning to support compliance and traceability requirements.

• Coordinate with governance, validation, security, and business stakeholders within a regulated banking environment.

Technical Leadership

• Serve as the technical anchor for projects and maintain design continuity throughout the delivery lifecycle.

• Mentor engineers, conduct technical reviews, and raise engineering standards through best practices and reusable frameworks.

• Lead architecture discussions and provide evidence-based recommendations using usage data, evaluation outcomes, and production insights.

• Communicate effectively with technical teams, business stakeholders, validators, and operational teams.

________________________________________

Required Qualifications

Experience

• Minimum 5 years of professional software engineering experience.

• Minimum 3 years of hands-on experience building and operating AI/ML systems, including production GenAI, LLM, RAG, agentic, or AI-enabled applications.

• Proven experience delivering production systems with real users and measurable business outcomes.

• Bachelor's degree in Computer Science, Software Engineering, or a related discipline (or equivalent practical experience).

Technical Skills

Software Engineering

• Strong proficiency in:

o Python

o TypeScript / JavaScript

o FastAPI or equivalent backend frameworks

o React frontend development

• Deep understanding of:

o API design and versioning

o Software architecture

o Unit, integration, and contract testing

o CI/CD pipelines

o Code review and engineering best practices

o Git-based development workflows

AI & Machine Learning

• Production experience with:

o Large Language Models (LLMs)

o Retrieval-Augmented Generation (RAG)

o AI agents and tool use

o Prompt engineering

o AI evaluation frameworks

o Structured outputs and function calling

• Familiarity with orchestration frameworks such as LangChain, LlamaIndex, DSPy, or equivalent.

• Experience integrating enterprise AI services including Azure OpenAI and AWS Bedrock.

Data Engineering & Platforms

• Experience working with structured and unstructured data.

• Knowledge of ETL/ELT pipelines and document processing workflows.

• Experience with orchestration platforms such as Airflow, Prefect, Dagster, Kafka, or equivalent.

• Strong SQL and data platform proficiency.

Cloud & DevOps

• Experience deploying AI applications using Docker and cloud platforms.

• Familiarity with AWS, Azure, Infrastructure-as-Code (Terraform/CDK), secret management, IAM, and modern deployment strategies such as blue-green and canary releases.

________________________________________

Key Competencies

• Independent system design and problem-solving

• Strong technical judgment and decision-making

• Production ownership mindset

• Reliability and operational excellence

• AI governance and risk awareness

• Data-driven decision making

• Mentorship and technical leadership

• Stakeholder management and communication

• Ability to challenge assumptions and influence outcomes through evidence and analysis