Role overview
The QA Analyst - LOIS for Word owns quality for an AI-powered legal drafting product, building the test strategy, processes, and automation. The role evaluates functional behavior, AI output, and user workflows while bringing quality considerations into feature planning before release.
Test Strategy and AI Evaluation
- Define the test strategy, QA process, and playbook for the product.
- Partner early with Product and Engineering so test plans are ready before features ship.
- Evaluate AI output for accuracy, citation reliability, and appropriately scoped reasoning.
- Design and maintain evaluation sets for nondeterministic output and detect regressions over time.
- Test AI-generated document edits for accuracy, formatting fidelity, and behavior across document structures.
- Build automated guardrails in CI/CD for critical workflows and write structured, reproducible bug reports.
- Work with legal domain experts to understand different customer uses and translate them into test coverage.
- Apply a product and design perspective, suggesting improvements even where functionality works.
- Compare behavior across product surfaces and document discrepancies.
QA Experience
- 2+ years of QA experience, including ownership of test strategy for a product or major feature.
- A record of building QA processes and automation from scratch in a fast-moving environment.
- Comfort with ambiguity and rapid releases, with strong analytical and problem-solving skills.
- Attention to functional correctness and AI output quality, including defining correctness when there is no single right answer.
- Clear, concise written communication and actionable, evidence-backed bug reports.
- Experience with bug tracking tools and test management workflows.
Preferred:
- Testing AI or LLM products, especially output accuracy and evaluation design.
- CI/CD pipelines and automated-test release gates.
- Design or UX awareness, legal technology experience or comfort reading legal contracts, and programming beyond test automation.
Work and Compensation
This full-time remote position is associated with San Francisco, California, in Engineering – Quality. Salary ranges from $105,000–$150,000. The compensation package reflects location, qualifications, education, work experience, skills, and performance; the final offer is weighted by primary work location and expertise.
Benefits and Hiring
Full-time benefits include medical, dental, and vision insurance and maternity and paternity leave. Short- and long-term disability, opportunities to learn from leadership, and company merchandise are also available.
AI tools may assist with application review, resume analysis, response assessment, and identifying inconsistencies or verification signals. People make final hiring decisions. Candidates may request more information about how their data is processed.