QA automation engineer

Role Overview You will be the Quality Intelligence Architect of Exterview . As a QA Automation Engineer (10+ Years) , you’ll design, implement, and own the automation and quality orchestration system that validates AI-driven candidate screening, live interviews, scoring agents, and feedback pipelines. This is quality engineering at scale : validating probabilistic AI outputs, real-time interview flows, event-driven systems, and media-heavy workflows across thousands of concurrent interviews.

You’ll work closely with Backend, AI Engineering, Prompt Engineering, and Frontend teams to ensure quality is built into the system , not tested after the fact. Execution is tracked via Linear (tasks), Notion (PRDs & test strategy), and GitHub (automation reviews) ensuring quality decisions are transparent and measurable.

Key Responsibilities

Automation Architecture Design and own a scalable, maintainable automation framework for: Web (Next.js) APIs (GraphQL via App. Sync, REST) Event-driven and async workflows Move QA from test execution to quality architecture ownership . AI & Probabilistic System Validation Validate: Resume parsing accuracy AI scoring consistency Interview logic stability Define assertion strategies for non-deterministic AI outputs.

Real-Time & Media Workflow Testing Test live interview systems : VideoSDK + Tavus video flows Real-time state updates Validate video upload, playback, retries, and failure handling. API, Event & Integration Testing Build API-first automation for: GraphQL queries & mutations Lambda & microservice APIs Validate event-driven flows (notifications, interview state transitions).

Performance, Load & Reliability Testing Design load and stress tests for: Interview orchestration APIs Video playback & report generation Ensure system reliability under peak concurrency. CI/CD & Quality Gates Integrate automation into GitHub Actions . Define release quality gates and production sign-off criteria. Track flaky tests, failure patterns, and quality metrics.

Cross-Team Quality Leadership Translate product requirements into test strategies . Partner with engineering to improve testability & observability . Mentor junior and mid-level QA engineers. Success Metrics First 90 Days Stable automation framework covering UI + API + AI workflows. Flaky test rate reduced to near zero.

API-level load tests live for interview & video flows. Release quality gates enforced in CI/CD. 12 Months End-to-end automation coverage for all interview types. Predictable, fast release cycles with no QA bottlenecks . AI validation framework adopted as product standard. QA recognized as product quality owner , not gatekeeper.

Must-Haves 10+ years in QA Automation / SDET roles Strong experience designing automation frameworks , not just writing scripts Expertise in API testing (GraphQL & REST) Experience testing real-time and async systems Ability to validate AI-driven, non-deterministic outputs Strong understanding of CI/CD, release quality, and production readiness Nice-to-Haves Experience testing AI/ML or agent-based systems Media or video workflow testing experience Startup or high-scale SaaS background Prior ownership of QA strategy for a product Tech Stack Visibility Frontend: Next.

js, TypeScript Automation: Cypress, Playwright, Jest Backend: Node.js, AWS Lambda, Serverless Framework APIs: GraphQL (AWS App. Sync), REST Auth: AWS Cognito (OAuth, OTP) Data: MongoDB Atlas Media: VideoSDK, Tavus, Amazon S3 Communication: Twilio (SMS, WhatsApp, IVR) Observability: Langfuse, internal monitoring Assessment (PoC) Objective Validate your ability to design quality systems , not just write test cases.

Challenge Overview Design and implement an automation strategy for an AI-driven interview flow. The system should validate: Resume upload & parsing AI resume–job match scoring Interview scheduling Live interview execution Interview report generation Functional Expectations

1. Automation Design Cover: One UI flow One GraphQL API flow One real-time or async workflow Show clear separation between test layers.
2. AI Validation Strategy Define how you: Assert AI scores Detect drift or instability Handle acceptable variance
3. Performance & Reliability Include: One API load test Failure / retry validation Avoid direct S3 URL testing; use APIs only. Deliverables Automation repo (clean structure) Test strategy README Example AI assertion logic CI pipeline config (GitHub Actions) ≤ 5 min Loom walkthrough explaining decisions