Job description
Job Title: Testing Lead – Deep Learning / LLM / VLM Location Remote / Hybrid / On-site Experience 3–4 years (hands-on ownership in DL / LLM / GenAI testing) Employment Type Full-time Role Overview We are seeking a hands-on Testing Lead to own quality and documentation for our Deep Learning, LLM, and Vision-Language Model (VLM) products.
You will define how we test, measure, document, and communicate AI quality—working closely with ML, Engineering, and Product teams in a fast-paced startup environment. This role is ideal for someone who believes clear documentation is as critical as good testing, especially for non-deterministic AI systems. What You’ll Do Own Quality & Documentation End-to-End Define testing strategy for LLMs, VLMs, and DL pipelines.
Create and maintain clear, lightweight documentation covering: Model testing strategies and assumptions Evaluation metrics and acceptance criteria Known limitations, risks, and failure modes Release readiness and quality sign-off Ensure documentation evolves with models, data, and prompts. LLM / GenAI Testing Design tests for: Prompt templates and prompt changes RAG pipelines (retrieval quality, grounding, hallucination control) Multi-turn conversations and long-context behaviour Maintain golden datasets, regression test suites, and test result summaries.
Document prompt behaviour, edge cases, and known model quirks. Vision & Multimodal Testing Test VLMs for image-text alignment, OCR, captioning, and reasoning. Document model performance across different image types, quality levels, and domains. Track and publish model behaviour changes between versions. Automation, MLOps & Reporting Build Python-based automation for evaluation and regression testing.
Integrate tests into CI/CD and MLOps pipelines. Produce readable quality reports and dashboards for engineers and leadership. Monitor and document production issues such as model/data drift and degradation. Build a Quality-First Culture Establish QA and documentation standards that scale with a startup. Mentor engineers on writing testable code and meaningful documentation.
Act as the single source of truth for AI quality, testing, and known risks. What we’re looking For Must-Have Strong background in software testing with lead or ownership experience. Hands-on experience testing LLMs, DL models, or GenAI systems. Strong Python skills for test automation and data validation. Proven ability to write clear, structured technical documentation.
Understanding of: Transformer-based models and DL workflows Model evaluation metrics and non-deterministic system testing Comfortable working in ambiguity and moving fast in a startup. Nice-to-Have Experience with VLMs, multimodal models, or computer vision. Exposure to RAG architectures, vector databases, and embeddings.
Familiarity with tools like Lang. Chain, Llama. Index, MLflow, or similar. Experience documenting AI risks, limitations, or compliance requirements.