Senior Automation Test Engineer (AI/LLM)
ELEKS Quality Assurance Office is looking for a Senior Automation Test Engineer (AI/LLM) in Argentina
ABOUT PROJECT
Our customer is among the top 5 health insurance companies in the US. They are dedicated to expanding access to high-quality, cost-effective healthcare.
We are looking for an experienced Automation Test Engineer to join the project engagement and work embedded within the client’s delivery team, focusing on testing AI-powered applications, including Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) solutions.
REQUIREMENTS
- Experience testing LLM/RAG behavior, including intent routing, retrieval quality, prompt safety, and evaluation/regression testing
- Experience in test automation using Python and PyTest
- Experience testing REST APIs and FastAPI-based applications
- Experience with SQL and Databricks data validation
- Experience testing Azure AI Search
- Experience testing Azure OpenAI APIs
- Experience with LLM intent-routing validation
- Experience with RAG retrieval-quality testing
- Experience with prompt-injection and PII-leakage testing
- Experience with groundedness, hallucination, and citation validation
- Experience with RAGAS or DeepEval evaluation
- Experience with automated regression testing using JSONL datasets
- Experience with Postman and Newman
- Experience with Jenkins CI/CD integration
- Experience with Docker and Git
- Experience with Application Insights or OpenTelemetry monitoring
- At least Upper-Intermediate level of English
RESPONSIBILITIES
- Develop, execute, and maintain automated tests using Python and PyTest
- Perform REST API and FastAPI testing
- Validate data using SQL and Databricks
- Test Azure AI Search and Azure OpenAI APIs
- Validate LLM intent routing and RAG retrieval quality
- Perform prompt-injection and PII-leakage testing
- Validate groundedness, hallucinations, and citations in LLM-generated responses
- Conduct LLM evaluations using RAGAS or DeepEval
- Perform automated regression testing using JSONL datasets
- Use Postman and Newman for API testing
- Integrate automated tests with Jenkins CI/CD
- Use Docker and Git in test automation activities
- Monitor applications using Application Insights or OpenTelemetry
WHAT YOU WILL GET WITH ELEKS
- Close cooperation with a customer
- Challenging tasks
- Competence development
- Ability to influence project technologies
- Team of professionals
- Dynamic environment with low level of bureaucracy
ABOUT ELEKS
ELEKS is a custom software development company. We deliver value to our clients, thanks to our expertise and experience gained from working as a software innovation partner since 1991.
Our 2000+ professionals located in the Delivery Centers across Eastern Europe and sales offices in Europe and North America, provide our clients with a full range of software engineering services. These include product development, QA, R&D, design, technology consulting and dedicated teams.
Noticed a perfect match for a friend?
Get a valuable bonus for a reference
Applying for Senior Automation Test Engineer (AI/LLM)
- CompanyThis field is for validation purposes and should be left unchanged.
- Full Name*
- Email*
- Phone*
- Message
- Get updated about new vacancies at ELEKS
- Attach a CV*Accepted file types: jpg, gif, png, pdf, doc, docx, xls, xlsx, ppt, pptx, Max. file size: 10 MB.
- We will add your contact details provided in this form to our system for contacting you regarding your request. For more info please consult our privacy policy
Δ
Referring a friend to Senior Automation Test Engineer (AI/LLM)
- X/TwitterThis field is for validation purposes and should be left unchanged.
- your info
- Your Full Name*
- Your Email*
- Referral’s Info
- Referral's Full Name*
- Referral's Email*
- attach your referral’s CV*Accepted file types: jpg, gif, png, pdf, doc, docx, xls, xlsx, ppt, pptx, Max. file size: 10 MB.
- *
- I have read the referring rules and conditions
- We will add your contact details provided in this form to our system for contacting you regarding your request. For more info please consult our privacy policy
Δ