About the role
You'll set the standard for whether analysis produced by AI agents is accurate enough for people to act on. You’ll direct agents to generate test cases and investigate failures, then verify the results against independent evidence so the product team can measure improvements.
What you'll do
- Define what good analysis looks like, including accuracy, source support, completeness, reproducibility, and usefulness to the person making a decision.
- Build representative evaluation sets with reference answers and the source data needed to verify them. Document what each set can measure.
- Design deterministic checks, statistical comparisons, and expert reviews that provide independent evidence alongside model-based grading.
- Trace failures to data selection, calculations, reasoning, generated code, or presentation so the team can fix the underlying issue.
- Run multiple AI agents in parallel every day to generate test cases, write regression checks, and build custom tools for comparing runs and investigating failures. Verify agent judgments against source evidence and expert review.
- Help product, engineering, and domain experts prioritize fixes and weigh quality, latency, and cost.
What you bring
- Senior analytical or data science experience evaluating complex outputs and diagnosing why a workflow produced the wrong result.
- Strong Python and SQL skills and sound statistical judgment about sampling, uncertainty, and evaluation design.
- Practical experience with language models, AI agents, and their failure modes in analysis or software workflows.
- The ability to build custom evaluation apps and orchestrate multiple agents while maintaining independent source checks.
- The judgment to separate an evaluation score from evidence that a result is correct or ready for use.
- Clear communication with engineers and domain experts about failures, severity, and what a proposed fix actually improves.
- Initiative in finding missing test cases, documenting assumptions, and carrying investigations through a verified result.
- Experience with analytical reporting, retrieval evaluation, generated-code testing, or structured output validation.
- Experience creating expert-reviewed benchmarks or maintaining regression suites.
- Experience with enterprise data controls and reproducible AI workflows.
- You work independently. You can take an unclear goal, decide what to do first, find what you need, and carry the work through to a result. You bring people in early when a decision, risk, or dependency needs their input.
How we work
AI agents are core to how we get work done. You'll spend a substantial part of your day running multiple agents in parallel, building tools and workflows with them, and checking what they produce. You own the result. Verify the facts, calculations, code, and claims before they go into a decision, a customer conversation, or a release.
When working with clients, take initiative within the agreed goals, scope, access permissions, and review process. Bring options and recommendations, and involve the right people before changing a customer commitment or production workflow.
What success looks like
- Evaluation results identify meaningful failures and can be traced to reproducible cases.
- Teams can compare changes without hiding regressions behind an aggregate score.
- Custom tools make diagnosis and review faster for engineers and domain experts.
- AI-generated analysis improves against independently checked criteria.
About Enlighten
At Enlighten, we believe AI should expand what people can take on. We build teams of AI agents alongside Fortune 500 companies so they can put their best thinking to work across the business. We want the people closest to the work to be able to ask hard questions and get answers they can trust. Our mission is to give every team the intelligence it needs to make decisions that shape the business.