AlgoShack logo
AlgoShack

49 open roles

AI Test Engineer

Full-timePosted Nov 16, 2026

Job description

AI Test Engineer: The role in one line: A hands-on engineer who proves our AI features are accurate, safe and stable before and after every change , and who can build that testing setup alone. What this person will actually do Build automated accuracy checks for the AI Make those checks a gate in the CI pipeline ,] Build and maintain test datasets Test the whole pipeline end to end : Watch quality in production using LLM tracing tools (Arize Phoenix, Lang.

Smith) to spot drift, rising failures and cost or latency jumps. Run safety and robustness testing : prompt injection inside documents, hallucinated fields, PHI leaking into logs, and inconsistent answers to the same input. Produce evidence for a regulated product. Test reports need to stand up in a medical-device-style audit (IEC 62304, HIPAA).

Must-have skills (screen hard on these) Strong Python test automation : pytest, API testing, CI pipelines (GitLab). LLM evaluation hands-on : has built eval sets and scoring, not just “used ChatGPT.” Ask them to name the metrics they used and how they set pass/fail thresholds. Experience with at least one eval or tracing tool : Lang.

Smith, Arize Phoenix, Deep. Eval, Ragas, promptfoo, or similar. Understands non-deterministic testing : how to test something that gives slightly different answers each run (repeat runs, tolerances, LLM-as-judge and its limits). Document or vision AI exposure : OCR or extraction from PDFs and images is a big plus for us.

Solo-capable : has set up a test approach from scratch, not only executed someone else’s. Good-to-have Lang. Chain / Lang. Graph. Useful, but not essential : our AI Doc calls Gemini directly through Google’s SDK. Don’t reject strong candidates for lacking Lang. Graph. Google Cloud / Vertex AI, Django APIs, Playwright.

Healthcare or regulated-domain testing, and handling of PHI and de-identified data. Performance and cost testing of LLM calls (latency, tokens).

Description copied from AlgoShack's careers page. Read the full posting before you apply.

More jobs at AlgoShack

See all openings at AlgoShack

AI Test Engineer jobs at other companies