The Role
We are looking for an experienced Lead Quality Engineer – AI Delivery on behalf of our client, a global organization specializing in insurance, risk management, and professional services, to lead Quality Engineering across multiple AI delivery pods building enterprise-scale AI solutions.
Our client's engineering teams already operate within an established shift-left Secure SDLC, with core engineering, security, code quality, and pipeline controls embedded into delivery. This role builds on that foundation, ensuring each AI solution has an appropriate risk-based quality strategy, with particular focus on GenAI evaluation, end-to-end quality, UAT readiness, and release confidence.
You will work closely with Product Managers, Product Owners, Engineering Leads, Architects, and developers, combining strategic quality leadership with hands-on technical capability.
We will count on you to:
Lead Quality Engineering across multiple AI delivery pods and establish consistent, risk-based quality standards.
Partner with Product and Engineering to define test strategy, acceptance criteria, UAT approach, quality risks, and release-readiness criteria.
Ensure appropriate coverage across functional, integration, end-to-end, non-functional, AI-specific, and business acceptance testing.
Define testing and evaluation approaches for LLM, RAG, agentic, and AI-enabled solutions, including grounding, hallucination risk, regression, tool calling, workflow behaviour, and non-deterministic outputs.
Establish appropriate AI evaluation datasets, test harnesses, baselines, thresholds, and regression approaches.
Guide and, where needed, implement reusable automation across web, APIs, backend services, integrations, asynchronous workflows, and AI components.
Ensure products are ready for UAT, with appropriate test evidence, environments, test data, defect status, entry/exit criteria, and known risks.
Provide Pod Leads and programme leadership with clear quality metrics, risks, defect trends, AI evaluation results, UAT readiness, and release-readiness reporting.
Conduct quality reviews across pods and identify gaps in testing, testability, observability, or quality evidence.
Use production incidents, escaped defects, telemetry, and user feedback to continuously improve quality approaches.
Coach engineers and delivery teams to strengthen shared ownership of quality across the lifecycle.
What you need to have
Proven experience leading Quality Engineering for enterprise-scale AI / GenAI solutions in production.
Hands-on experience testing LLM, RAG, agentic, workflow-based, or AI-enabled applications.
Strong experience defining end-to-end test strategies, UAT approaches, quality gates, and release-readiness criteria.
Experience working across multiple delivery teams or pods in complex enterprise environments.
Strong software engineering and automation skills in TypeScript / JavaScript and/or Python.
Experience with modern automation frameworks such as Playwright, Cypress, Selenium, Jest, Mocha, or PyTest.
Strong experience testing web applications, APIs, backend services, integrations, and distributed systems.
Experience with AI evaluation frameworks, regression suites, test harnesses, and representative evaluation datasets.
Good understanding of CI/CD, cloud-native systems, observability, and Secure SDLC practices.
Strong stakeholder management skills and the ability to translate quality evidence into clear risks, metrics, and recommendations for leadership.
What makes you stand out
Experience with LangChain, LangSmith, Mastra, or similar AI engineering and evaluation tooling.
Experience testing complex RAG, multi-agent, or tool-calling solutions.
Experience with performance, resilience, accessibility, or security testing.
Experience using production telemetry and incidents to improve test and evaluation strategies.
Ability to balance strong quality controls with pragmatic, fast-moving AI delivery.
Technology context
Typical technologies include:
React, TypeScript, Node.js / NestJS, Python / FastAPI, REST APIs, async workflows, AWS / Azure, GitHub Actions / GitLab CI / Azure DevOps, LLM APIs, RAG, agentic workflows, LangChain, LangSmith, and Mastra.