TestOps Engineer - Agentic AI

TESTOPS ENGINEER – AGENTIC AI Location: XenonStack office, Sector 75, STPI, Mohali (Chandigarh Tricity), work from office Experience: 2–8 years | Employment: Full time, XenonStack Private Limited | Team: AgentOps / Agentic AI Engineering Role Level: X2 – Workflow ABOUT XENONSTACK XenonStack is a Data and AI Foundry for Agentic Systems, enabling enterprises to design, deploy, operate, and scale intelligent agents across digital and physical environments.

We build enterprise-grade platforms across the agentic stack: Akira AI – Reasoning and agent orchestration. ElixirData – Agentic analytics intelligence. Explainable, decision-centric analytics for measurable business outcomes. NexaStack – Agentic infrastructure automation. Secure, compliant AI deployment across cloud, edge, and on-prem.

MetaSecure – Trust, compliance and defense. Continuous assurance with AI-BOMs, risk scoring, and agentic security. Our mission is to accelerate the world's transition to AI + Human Intelligence by making agentic systems reliable, responsible, and enterprise-ready.

ABOUT THE ROLE

We are looking for a TestOps Engineer – Agentic AI with strong expertise in Python automation, LLM evaluation, agentic workflow testing, and CI/CD integration. You will design and maintain shared evaluation infrastructure, golden datasets, automated quality gates, and regression frameworks for AI agents across ElixirData, ElixirClaw, and NexaStack.

This is an engineering-focused TestOps role, not a traditional manual QA position. You will build the testing systems and enable engineering teams to test their own components. ROLE MISSION Make every AI agent release measurable, repeatable, reliable, and safe to ship through automated evaluations, statistical quality measurements, and continuous regression testing.

KEY RESPONSIBILITIES

Build and maintain reusable evaluation harnesses for LLM-powered agents and agentic workflows. Integrate offline evaluations for prompts, models, tools, and retrieval into CI/CD release gates. Measure agent quality, reliability, latency, and execution cost against defined thresholds. Create and version golden datasets, synthetic conversations, and edge-case scenario libraries for customer workflows.

Validate agent trajectories, tool selection, execution sequences, and tool arguments. Develop contract tests for MCP servers, APIs, and external tools used by agents. Handle non-deterministic AI behavior using repeated evaluations, pass@k scoring, and statistical thresholds. Implement LLM-as-a-Judge evaluations and calibrate results against human-labelled datasets.

Conduct adversarial testing for prompt injection, jailbreaks, data leakage, and policy violations. Execute regression evaluations during model, prompt, provider, and dependency upgrades. Establish release quality gates and validate releases against agreed quality and safety thresholds. Convert production incidents into automated regression tests and maintain shared quality dashboards.

Provide reusable test templates and evaluation playbooks to engineering teams. HOW WE WORK WITH AI AI is part of the daily workflow at XenonStack, not an add-on. At our Mohali/Chandigarh office, AI tools are open for use: coding assistance, test generation, evaluation design, debugging, documentation, and research. When working on a client's data or environment, follow the client's compliance and information security policy.

Use only approved tools and never expose client data, code, credentials, or configurations to unapproved AI services. Every AI-generated test, evaluation result, and recommendation must be reviewed, validated, and understood before use. Come ready to discuss your AI testing stack, how you evaluate non-deterministic outputs, and how you prevent unsafe agent behavior.

WHAT SUCCESS LOOKS LIKE 100% of agent releases go through automated evaluation gates. Every active customer workflow has a versioned golden dataset with agreed scenario coverage. Every production incident is converted into a regression test within 5 days. Zero critical AI safety findings reach production. All engineering pods adopt the shared evaluation harness rather than maintaining separate testing scripts.

Quality dashboards provide measurable visibility into agent reliability, latency, cost, and evaluation results. MUST-HAVE SKILLS Python and test automation: Strong Python programming, pytest, reusable test framework development, API testing, automated regression testing, and Git. LLM evaluation frameworks: Hands-on experience with DeepEval, Promptfoo, Ragas, LangSmith, or equivalent evaluation tools.

Agentic AI: Understanding of RAG, LLMs, tool calling, agent execution trajectories, and agent frameworks. Agent integration testing: Experience validating APIs, MCP servers, tool contracts, and multi-step agent workflows. Evaluation techniques: Golden datasets, synthetic test generation, pass@k, statistical thresholds, repeated runs, and LLM-as-a-Judge evaluation.

CI/CD and release engineering: Integrating automated evaluations into CI/CD pipelines and enforcing release quality gates. AI safety testing: Knowledge of prompt injection, jailbreaks, data leakage, adversarial scenarios, and policy validation. Engineering fundamentals: Debugging, root-cause analysis, test coverage, reproducibility, and maintainable automation code.

GOOD-TO-HAVE SKILLS LangChain, LangGraph, or other agent orchestration frameworks. Claude, Azure OpenAI, Amazon Bedrock, or other LLM providers. Multi-agent system testing and agent-to-agent interaction validation. Docker, Kubernetes, and cloud-native deployment environments. AI observability, tracing, quality dashboards, and production monitoring.

AI security, red-teaming, responsible AI, and governance frameworks. Experience with model and prompt versioning, benchmarking, and cost optimisation. CANDIDATE PROFILE 2–8 years of experience in QA Automation, SDET, MLOps, or related engineering roles, including hands-on LLM evaluation. Proven experience developing Python-based automation frameworks and integrating them into CI/CD workflows.

Practical understanding of LLM applications, RAG systems, tool calling, and agentic AI architectures. Ability to evaluate non-deterministic systems using statistical metrics rather than relying only on binary pass/fail checks. Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline; demonstrated hands-on experience is equally important.

Strong analytical thinking, debugging, communication, and cross-team collaboration skills. We value engineers who build scalable testing systems rather than simply execute test cases. Be ready to explain how you evaluated an AI agent, designed golden datasets, handled inconsistent outputs, and prevented regressions from reaching production.

Traditional manual QA experience alone is not sufficient for this role. HIRING PROCESS Pre-screening assessment (about 10 minutes, at application) → Technical interview → Face-to-face technical assessment at our Mohali office → HR and Leadership round → Background verification → Offer. TERMS Location: Work from our Mohali (Chandigarh Tricity) office.

Full time; immediate joiners or short notice periods preferred. Background verification (BGV) is mandatory.

Compensation

as per company policy and budget band. HOW WE WORK We value deep work, clarity, and execution over activity. You will be expected to clarify, plan, execute, and escalate with options, not wait to be told. Ownership sits with the individual, and outcomes are measured, not effort. OUR VALUES Agency – Be self-directed and proactive | Taste – Sweat the details and build with precision | Ownership – Take responsibility for outcomes | Mastery – Commit to continuous learning and growth | Impatience – Move fast and challenge inertia | Customer Obsession – Always put the customer first WHY JOIN US Build evaluation and testing infrastructure for enterprise-grade Agentic AI platforms.

Work on LLM evaluation, agent reliability, AI safety, and production-grade quality engineering. Collaborate with AgentOps, engineering, security, and observability teams. AI-first environment: work alongside the agentic platforms we build. Great Place to Work-Certified, 14+ years, headquartered in Mohali.