We are helping our client find exceptional Software Engineer - Technical Advisors who want to spend a focused period doing something most engineers never get to do: find out exactly where frontier models break on real engineering work.
You will take on hard problems across production-grade codebases, then work out precisely where and why model-generated solutions fall short. You will also build the hard problems those models are tested on. Your judgment about what separates good engineering from plausible engineering is the entire point of the role.
What you'll do
- Evaluate Agent Sessions: Analyze how AI coding agents behave over entire coding sessions—tracking what the agent investigates, verifies, assumes, and leaves undone.
- Review Model-Written Pull Requests: Audit model-generated pull requests against real production repositories, documenting every identified issue alongside its severity and detailed technical rationale.
- Build Evaluation Benchmarks: Design and construct hard, container-based problems that serve as rigorous test benchmarks for frontier models.
- Collaborate with AI Researchers: Work directly alongside AI researchers on frontier problems, producing clear written analyses that explain root-cause failures and boundary-condition gaps.
- Maintain High Written Rationale Standards: Author original, clear technical rationale for all evaluations (all written deliverables must be independently authored without AI text generation, though AI tools are welcomed for codebase exploration and running test suites).
Requirements
Requirements:
- 8+ years of production engineering experience preferred (exceptions only for clearly exceptional profiles).
- Backgrounds including Senior, Staff, or Principal Software Engineers, Tech Leads, or Open-Source Maintainers.
- Experience working in production codebases with a strict code review culture (startups, big tech, or open source are all acceptable).
- Polyglot adaptability: Heavy experience with Python and TypeScript is common, but must be comfortable dropping into unfamiliar languages weekly.
- Core Tooling: Demonstrated comfort using Docker, git, and the Command Line Interface (CLI) to reproduce, isolate, and debug results locally.
- Cross-Layer Fluency: Comfort working across backend, frontend, APIs, data, testing, or developer tooling.
- Strong written communication skills—able to articulate why code fails, not just how to fix it quickly.