Job description
While technology is the heart of our business, a global and diverse culture is the heart of our success. We love our people and we take pride in catering them to a culture built on transparency, diversity, integrity, learning and growth. If working in an environment that encourages you to innovate and excel, not just in professional but personal life, interests you- you would enjoy your career with Quantiphi!
Role: Lead/Associate Lead – QA + MLOps & Generative AI Experience: 10+ years Location: Mumbai/Bangalore (Hybrid) Key Responsibilities: AI/ML & GenAI Testing Strategy (AWS Ecosystem) Define testing approaches for AI systems built on AWS services such as: Amazon Sage. Maker Amazon Bedrock AWS Lambda Amazon API Gateway Amazon Kinesis AWS Glue Amazon S3 Amazon Cloud.
Watch Design validation frameworks covering: Model accuracy & performance validation Data drift & concept drift detection Hallucination detection for LLMs Prompt robustness testing RAG validation (retrieval accuracy + grounding) Bias & fairness validation Safety & toxicity testing MLOps Quality Engineering (AWS-Centric) Validate the end-to-end ML lifecycle including: Data ingestion & feature pipelines Model training & hyperparameter tuning Model versioning & registry Deployment validation Canary & blue/green release validation Work with AWS-native services such as: Sage.
Maker Pipelines Sage. Maker Model Monitor Sage. Maker Feature Store Bedrock model evaluation workflows Cloud. Watch-based observability Implement CI/CD quality gates for ML pipelines integrated with AWS DevOps tools. GenAI & Agentic AI Testing Define quality engineering approaches for: LLM-based applications using Amazon Bedrock Prompt engineering validation Multi-agent orchestration testing Chatbot & Voice bot conversational testing Intent classification validation Conversation drift & fallback validation API contract validation for LLM integrations Build reusable evaluation harnesses for: BLEU / ROUGE scoring Embedding similarity scoring Response consistency Safety scoring frameworks Framework & Capability Development Design reusable AI testing accelerators Create AWS-aligned AI test automation frameworks (Python-first) Develop synthetic data generation strategies Establish AI quality scorecards Build an internal AI QA Center of Excellence Client Engagement & Leadership Lead AI/ML quality strategy workshops Perform AI risk & readiness assessments Present quality architecture to CXOs Drive QA transformation programs Mentor QA teams on AWS-based AI testing Own delivery for AI testing engagements end-to-end Must have skills: Testing Expertise 8–12+ years in Quality Engineering Strong test strategy, automation & governance experience Experience leading QA transformation initiatives Experience building frameworks from scratch AI/ML & GenAI Expertise Deep understanding of ML lifecycle Experience testing ML models (NLP preferred) Hands-on experience validating LLM applications Strong understanding of: Prompt engineering RAG architecture Embeddings Bias & explainability AWS AI/ML Expertise Hands-on experience with: Amazon Sage.
Maker (training, deployment, monitoring) Amazon Bedrock (LLM integration & evaluation) S3-based data pipelines AWS IAM (security validation) Cloud. Watch monitoring Lambda & API Gateway integrations AWS CI/CD (Code. Pipeline / Code. Build preferred) Understanding of: Infrastructure as Code (Terraform / Cloud. Formation) Observability in AI systems Cost monitoring for ML workloads Technical Skills Python (mandatory) Experience with ML libraries (Scikit-learn, Tensor.
Flow, Py. Torch) Experience with LLM frameworks (Lang. Chain, etc.) API & automation testing frameworks Git-based workflows Leadership & Communication Strong client-facing communication Experience leading QA teams Ability to create strategy decks & solution proposals Strong stakeholder management If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy your career with us !