About the Role
Join an AI/ML team building tools to identify and address failures in production AI agents. This role owns the intelligence behind failure detection, evaluation, and proposed fixes, helping make agents more reliable for real-world use.
What You'll Do
Detect subtle agent failures, including omissions, incorrect responses, and patterns across large volumes of traces.
Turn user behavior such as rephrasing, abandonment, and retries into evidence of potential failures.
Build evaluation methods that measure detection quality, including in situations without ground-truth labels.
Verify that proposed fixes reproduce and resolve failures, and avoid recommending changes when evidence is weak.
Improve the cost efficiency of detection by choosing appropriate models and methods for each task.
Review production traces regularly to identify problems and guide product improvements.
What We're Looking For
Demonstrated AI/ML engineering experience, with 0 to 10 years of experience considered.
Experience shipping products that people use and working as an individual-contributor engineer.
Hands-on experience with LLMs in production, including evaluations, LLM-as-judge, embeddings, and deciding when to use smaller models or no model.
Experience running AI agents in production and handling their failures.
Strong research judgment and the ability to distinguish meaningful improvements from noise, including when direct ground truth is unavailable.
Compensation & Benefits
Base salary range: $120,000 to $200,000 annually. Visa sponsorship is not available. The role requires authorization to work in the United States without visa sponsorship.
Location
On-site in San Francisco, United States.