49 open roles
AI Test Engineer
Job description
AI Test Engineer: The role in one line: A hands-on engineer who proves our AI features are accurate, safe and stable before and after every change , and who can build that testing setup alone. What this person will actually do Build automated accuracy checks for the AI Make those checks a gate in the CI pipeline ,] Build and maintain test datasets Test the whole pipeline end to end : Watch quality in production using LLM tracing tools (Arize Phoenix, Lang.
Smith) to spot drift, rising failures and cost or latency jumps. Run safety and robustness testing : prompt injection inside documents, hallucinated fields, PHI leaking into logs, and inconsistent answers to the same input. Produce evidence for a regulated product. Test reports need to stand up in a medical-device-style audit (IEC 62304, HIPAA).
Must-have skills (screen hard on these) Strong Python test automation : pytest, API testing, CI pipelines (GitLab). LLM evaluation hands-on : has built eval sets and scoring, not just “used ChatGPT.” Ask them to name the metrics they used and how they set pass/fail thresholds. Experience with at least one eval or tracing tool : Lang.
Smith, Arize Phoenix, Deep. Eval, Ragas, promptfoo, or similar. Understands non-deterministic testing : how to test something that gives slightly different answers each run (repeat runs, tolerances, LLM-as-judge and its limits). Document or vision AI exposure : OCR or extraction from PDFs and images is a big plus for us.
Solo-capable : has set up a test approach from scratch, not only executed someone else’s. Good-to-have Lang. Chain / Lang. Graph. Useful, but not essential : our AI Doc calls Gemini directly through Google’s SDK. Don’t reject strong candidates for lacking Lang. Graph. Google Cloud / Vertex AI, Django APIs, Playwright.
Healthcare or regulated-domain testing, and handling of PHI and de-identified data. Performance and cost testing of LLM calls (latency, tokens).
Description copied from AlgoShack's careers page. Read the full posting before you apply.
More jobs at AlgoShack
Lead Generation Specialist
AlgoShack· Bangalore, KA, IndiaPerformance Test Engineer (JMeter + Dynatrace)
AlgoShack· Bangalore, KA, IndiaSenior Automation Engineer
AlgoShack· Bangalore, KA, IndiaSenior Automation Engineer
AlgoShack· Bangalore, KA, IndiaSenior Automation Test Engineer
AlgoShack· Bangalore, KA, India