Specialist, QA Test Automation

Regular

Specialist, QA Test Automation

Key Responsibilities

• Probabilistic Testing & Evaluation
  • Implement and refine testing workflows tailored for AI‑powered solutions where outputs are probabilistic rather than deterministic.
  • Build guardrails to identify model drift, prompt regression, hallucination risks, and safety violations before they hit production.
• Dataset Curation & Tooling
  • Own the creation, maintenance, and scaling of high‑quality golden datasets used as the source of truth for benchmarking model changes.
  • Champion programmatic evaluation tools such as Promptfoo, DeepEval, or Ragas to automate the assessment of prompt variants and model performance.
• Strategy Execution & Collaboration
  • Partner closely with the Chapter Lead, QA to translate high‑level AI QA frameworks into actionable, daily tasks for technical teams.
  ...