AI Test Technician

Clearance: Requires an active DOD security clearance The Prospective Group (TPG) is seeking an AI Test Technician to be part of the Center for Analytics program at the Department, performing duties to restore testing capacity and develop scalable LLM evaluations and automated testing for the growing portfolio of AI and analytics products.

Position Responsibilities: Execute daily manual and automated test cycles across data pipelines, applications, and AI/LLM-enabled tools to restore and stabilize baseline testing capacity. Design, build, and maintain automated test scripts and regression suites to reduce reliance on manual testing and increase test throughput.

Develop scalable evaluation harnesses for large language models, including prompt/response accuracy checks, hallucination and bias detection, and output consistency scoring across use cases. Build and maintain test data sets, benchmark scenarios, and scoring rubrics to support repeatable LLM and application evaluations.

Integrate automated tests and LLM evaluations into CI/CD pipelines to keep pace with frequent releases. Log, triage, and track defects and evaluation findings; coordinate with development teams to verify fixes and re-test as needed. Document test plans, evaluation criteria, and results in clear reports accessible to both technical and non-technical stakeholders.

Track testing coverage and throughput metrics to identify capacity gaps and recommend process improvements.

Requirements

2-4 years of experience in software testing or quality assurance, including building or running automated test frameworks. Experience with evaluating AI/LLM outputs (e.g., prompt testing, benchmark datasets, output scoring). Working knowledge of a scripting language (Python preferred) for building automated tests. Strong attention to detail and clear technical documentation skills.

A bachelor's degree, or equivalent experience. Position is located in the DC area and may require availability to be on site as required. Position Pay Range $120,000 – $130,000 USD The Prospective Group (TPG) provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws.