Test Lead - API Automation

Test Automation + Agent/LLM Evaluation (Accuracy, Hallucination, Regression) Role Summary: Responsible for designing, developing, and executing automated testing frameworks for AI/LLM-based applications and intelligent agents. The role focuses on validating model performance, response accuracy, hallucination detection, regression testing, and overall reliability of AI solutions.

Key Responsibilities

• Develop and maintain automated test frameworks for AI agents and LLM-powered applications.
• Evaluate LLM responses for accuracy, relevance, completeness, and consistency.
• Design test cases to identify and measure hallucinations, incorrect outputs, and bias.
• Perform regression testing to ensure model updates do not impact existing functionality.
• Create evaluation datasets, benchmarks, and test scenarios for agent workflows.
• Monitor key AI quality metrics such as precision, recall, response quality, and task success rate.
• Collaborate with AI engineers, data scientists, and product teams to improve model performance.
• Generate testing reports, insights, and recommendations for continuous improvement.

Required Skills

• Experience in test automation using Python, Selenium, Playwright, PyTest, or similar tools.
• Understanding of Generative AI, LLMs, AI agents, and prompt engineering.
• Knowledge of LLM evaluation frameworks such as LangSmith, DeepEval, Ragas, Promptfoo, or similar.
• Experience with API testing and automated validation pipelines.
• Strong analytical and troubleshooting skills.

Preferred Qualifications

• Experience with OpenAI, Azure OpenAI, Anthropic, Gemini, or other LLM platforms.
• Knowledge of MLOps, AI model monitoring, and CI/CD integration.
• Familiarity with AI governance, responsible AI, and model quality assessment.