Senior Engineer - GenAI Quality Assurance

Responsibilities

• Design and implement automated and manual QA strategies for GenAI systems
• Build regression tests, generate synthetic test data, build test harnesses and evaluation pipelines for accuracy, grounding, robustness, safety
• Validate prompt logic, model behavior and evaluation metrics
• Execute tests and debug pipelines
• Support deployment of GenAI products
• Lead system design reviews, advocating best practices for architecture and reliability of GenAI applications
• Collaborate closely with data scientists and developers to embed testing and quality checkpoints
• Establish testing standards and workflows for continuous integration and deployment
• Conduct code reviews with a focus on design, testability, and maintainability
• Document system design decisions, test cases, and best practices
• Close collaboration with domain experts and stakeholders to validate developed solutions

Requirements

• Strong software engineering background with QA ownership
• Familiarity with RAG & Graph frameworks
• 3+ years of experience as a software engineer (preferably 5), or QA engineer in production systems, preferably including at least 1 year experience in GenAI systems
• Advanced Python skills
• Experience working with LLM-based systems in real environments
• Hands-on experience with: LLM APIs, AWS Bedrock (Anthropic & OSS), Azure Foundry (OpenAI), GCP (Gemini), Cloud platforms (AWS, Azure, GCP)
• Proven ability to implement test frameworks for GenAI applications
• Knowledge of API testing, data validation, and data pipeline testing
• Experience with version control (Git)
• Experience with building and deploying AI/ML products through the full product lifecycle
• Good understanding of scientific computing and ML frameworks
• Solid understanding of safety, bias, and failure modes of GenAI systems
• Comfortable working in a relatively fast-moving environment where priorities may shift and initiative is valued
• Proactive problem-solver who can anticipate needs and find solutions
• Comfortable working independently, but also a strong team player.