This role is for one of Weekday’s clients
Min Experience: 2+ years
Engagement Type: Ongoing, Hourly Contract
Work Mode: Fully Remote
Working Hours: 5–6 hours per day
Hourly Compensation: USD 20–25 per hour
We are looking for an experienced DevOps / SRE Test Bench Engineer to design, develop, and validate challenging infrastructure troubleshooting tasks for AI systems. In this role, you will create realistic, intentionally broken Linux and containerized environments that require deep technical investigation, systematic debugging, and recovery.
You will also develop reference solutions and automated verification scripts to ensure that each issue is genuinely resolved. The ideal candidate has strong hands-on infrastructure experience, a methodical troubleshooting approach, and the ability to build complex, realistic scenarios that test practical engineering skills.
Requirements
Key Responsibilities
- Design realistic, multi-service failure scenarios that require extensive investigation, root-cause analysis, and recovery.
- Build and configure test environments using Linux, Docker, Kubernetes, databases, message queues, and service dependencies.
- Develop broken infrastructure setups that simulate real-world production incidents and operational challenges.
- Write detailed reference solutions, troubleshooting guidance, and reliable automated verifiers with clearly defined success criteria.
- Validate task quality to ensure scenarios are fair, technically accurate, reproducible, and not solvable through a fixed runbook.
- Troubleshoot complex issues involving networking, storage, permissions, service configurations, replication, and system recovery.
- Develop setup scripts, automation utilities, and verification tests using Python or Bash.
- Accurately log working hours, communicate progress, and actively participate in the project’s WhatsApp community.
- Maintain the confidentiality of all project-related tasks, environments, and technical materials.
Required Qualifications and Skills
- Hands-on experience in DevOps, Site Reliability Engineering (SRE), platform engineering, infrastructure engineering, or a related technical role.
- Strong Linux administration and command-line skills, including SSH, shell scripting, process management, log analysis, file permissions, networking, and system services.
- Strong hands-on expertise in Docker and Kubernetes, including diagnosing container failures, broken deployments, service connectivity issues, and orchestration problems.
- Experience with databases and infrastructure components such as PostgreSQL, Redis, message queues, storage systems, replication, and recovery mechanisms.
- Proven ability to troubleshoot production-like environments involving multiple interdependent services.
- Familiarity with CI/CD pipelines, Infrastructure as Code (IaC), and cloud infrastructure.
- Ability to investigate ambiguous failures independently, identify root causes, and implement effective solutions without relying on predefined runbooks.
- A strong testing mindset, including experience designing automated checks, validation scripts, and deterministic verifiers.
- Proficiency in Python or Bash for environment setup, troubleshooting automation, and test development.
Good-to-Have Skills
- Experience with cloud platforms such as AWS, Google Cloud Platform (GCP), or Microsoft Azure.
- Familiarity with Nginx, systemd, Logical Volume Manager (LVM), and Linux security hardening.
- Understanding of distributed systems, service dependencies, and fault recovery.
- Experience working with AI coding agents or AI-assisted software engineering tools.
Engagement and Application Process
- Complete the sign-up form to express your interest.
- Shortlisted candidates may be invited to complete a brief technical screening.
- Selected candidates will receive onboarding instructions and access to the project’s WhatsApp community.
- Compensation is hourly, based on approved and accurately logged working hours.
- All project tasks, technical materials, and related information must remain confidential and must not be shared outside the project.