About the Role
Build evaluation environments and benchmarks that measure whether AI agents can perform real company work and meet or exceed human performance. This founding engineering role focuses on agent capability assessment and production readiness, helping establish reliable evidence of agent competence.
What You'll Do
Design synthetic company environments with realistic email, records, and files for agents to use over extended periods.
Create robust graders and evaluation systems for assessing advanced AI agents.
Develop benchmarks that correlate with real-world production performance.
Identify and address evaluation failures, including reward hacking and benchmark gaming.
Simulate software tools such as inboxes and CRMs with sufficient fidelity for agent interactions at scale.
What We're Looking For
Hands-on experience with synthetic data pipelines, benchmarks, evaluations, or post-training; this role is suited to an AI researcher who builds, not a purely infrastructure-focused engineer.
Experience building environments or benchmarks used to train or evaluate agents at advanced AI labs, and implementing graders or evaluation systems.
At least 1 to 2 years at a fast-growing or AI-focused startup, with experience building synthetic environments or production-like simulations.
Ability to identify flaws in established benchmarks and detect and fix reward hacking in evaluation systems.
Experience simulating software systems that can withstand agent tool calls.
Strong academic or equivalent credentials, such as a top-tier university background, founding experience, or participation in a leading startup program.
High-level competition experience in mathematics, programming, debate, or games is valued.
Compensation & Benefits
$200,000 to $300,000 annual base salary, with up to $350,000 for exceptional candidates. An additional commission-style bonus of up to $200,000 or more is tied to the value of delivered environments. Meaningful founding equity is included. Visa sponsorship is not available.
Location
On-site, five days a week in San Francisco, California, United States.