AQA Engineer (Senior/Lead) ID35731

• What you will do

• Architect and oversee end-to-end test strategies across different layers in the testing pyramid;
• Define, evaluate, and monitor model validation criteria, including accuracy threshold, confidence scoring, fallback behaviors, etc.;
• Drive the automation of QA processes, ensuring robust, scalable processes for data ingestion, extraction, and classification;
• Lead the validation integration points between AI components and business logic, identifying risks and ensuring seamless operation;
• Own, maintain, and evolve the test infrastructure, specifically the CI/CD quality gates, data generation, and automated validation scripts;
• Establish and monitor key quality metrics aligned with the business goals, including turnaround time, confidence scores, and user corrections;
• Act as a strategic partner to Product and Engineering, and advocate for quality from design through delivery;
• Keep up with the latest AI tech and how to best apply the latest advancements to make the previously impossible possible for our customers;
• Sense of ownership, take ownership of code quality through exploratory and automated tests.

• Must haves

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field experience;
• 5+ years in AQA, Software Test Engineering, or related roles, including leadership responsibilities;
• Experience in AQA Javascript;
• Solid knowledge of software development life cycle (SDLC), Agile methodologies, and Quality Assurance or Quality Engineering methodologies;
• Proven experience building and scaling test automation frameworks (e.g. Pytest, Cypress, Selenium, Postman, etc.). Strong technical skills with REST APIs, data, pipelines, and backend services;
• Strong technical background with APIs, CI/CD pipelines, backend services, and cloud environments (AWS, Azure);
• Exceptional attention to detail, with a mindset for building robust systems that catch issues early and protect the customer experience;
• Demonstrated fast, proactive learning skills, especially crucial in the rapidly changing field of AI;
• Uppper-Intermediate English level.

• Nice to haves

• Experience testing AI/ML in production environments;
• Experience with GenAI model evaluation metrics;
• Experience setting up proactive monitoring and alerting systems to catch issues like model drift, performance degradation, or workflow failures.

AgileEngine is one of the Inc. 5000 fastest-growing companies in the US and a top-3 ranked dev shop according to Clutch. We create award-winning custom software solutions that help companies across 15+ industries change the lives of millions.

If you like a challenging environment where you’re working with the best and are encouraged to learn and experiment every day, there’s no better place — guaranteed! 🙂