Job description

Job Description

The team is responsible for driving operational excellence and governance in the adoption of agentic applications across the bank. This role ensures the successful implementation and maintenance of agent-based solutions by defining roadmaps, managing evaluation processes, and embedding responsible AI practices. The position partners with cross-functional teams to deliver secure, efficient, and compliant agentic operations.

Key Responsibilities

Service Operations

• Grow the agent impact through improvement in performance, growth in capabilities and design, build, and roll out for additional users.
• Run the operational processes for agent lifecycle management, including onboarding, configuration, and retirement of agents.
• Monitor agent performance, uptime, and compliance with SLAs.
• Coordinate with business, technology, and innovation teams to integrate agent workflows into incidents, problems, and change management processes.
• Ensure robust logging, audit trails, and reporting agent actions to support transparency and regulatory requirements.
• Drive continuous improvement of service operations through root cause analysis, automation opportunities, and performance optimization.
• Develop and maintain SOPs, FAQs, and user guides to support agentic workflows and ensure clarity for end-users.

Agent Evaluation & Issue Triage

• Apply evaluation frameworks for agent performance, reliability, and compliance.
• Lead issue of triage and resolution processes for agent-related incidents.
• Collaborate with business, technology, and innovation teams to implement corrective actions and improvements.
• Prompt Fine-Tuning
• Craft and optimize prompts, system messages, and few-shot examples to maximize model relevance, accuracy, and cost efficiency.
• Monitor performance metrics (e.g., latency, token usage, output quality) and refine prompts or pipelines as needed.
• Ensure prompt tuning aligns with business objectives and ethical guidelines.
• Constantly monitor and evaluate impacts from regulatory, policy or business-directed changes and update roadmap / prompts accordingly.

Key Skills Required

Core Functional Skills

• Performance Monitoring
• Process Improvement
• Service Lifecycle Management

Technical Skills

• Strong Data Science fundamentals: Experimental design, measurement, error analysis, statistical thinking, evaluation methodology. Hands-on Python experience for prototyping and production-grade experimentation.
• Experience with LLM & Agentic AI systems: Fine-tuning, RAG, prompt optimization, evaluation frameworks
• Applied NLP skills: embeddings, retrieval methods, classification/extraction, calibration, and reliability concepts.
• Responsible AI: bias testing, privacy leakage checks, audit artifacts

Qualifications

• Bachelor’s degree in Data Science, Statistics, Computer Science, Information Systems, or related field
• 5 to 10 years of hands on experience designing, building, and operationalizing end to end AI/ML systems, including data preparation, model development, deployment, and post production monitoring.
• A minimum of 1 to 2 years of dedicated experience working on generative AI and large language model projects.
• Hands-on experience designing prompts, tuning outputs, and integrating large language models (Claude Sonnet 3.5, GPT-class, etc.) into end-user applications.
• Familiarity with AI Agent model evaluation techniques.
• Ability to articulate complex technical concepts to non-technical stakeholders.
• Experience in the banking or financial services industry is a plus.

Additional Requirements