About the role
"Verified" has to mean something measurable. You'll design how we define and check quality for each dataset: rubrics, gold items, second-expert review and agreement metrics, and you'll make sure buyers can see the evidence.
What you'll do
- Turn dataset specs into rubrics, guidelines and acceptance criteria experts can apply consistently.
- Run qualification and review processes, and measure agreement and error rates per batch.
- Investigate quality issues and fix their root causes, from unclear guidelines to the wrong expert match.
- Partner with buyers' research teams on what "good" means for their models.
What we're looking for
- Experience running annotation, evaluation or data-quality programs for ML.
- Comfort with basic statistics for agreement and sampling.
- Clear, precise writing; guidelines are a product.
Nice to have
- Experience with RLHF, preference or evaluation data.