Targeted training from scarce expert input
A specialist’s qualification can become a focused learning target while independent assessment protects the behavior around it.
Prerequisites
A reviewed policy boundary, representative failures, and an intended SFT or DPO objective.
Steps
- Start with a source-authority exception: a newer document normally controls, but a current contract may explicitly incorporate an older agreement. Ask the specialist to clarify which condition changes the answer.
- Review the resulting policy and applicability. Use the rubric to guide demonstrations or preferred/rejected answers that preserve both the ordinary rule and the exception.
- Inspect generated examples and related families across splits. Hold back assessment situations that exercise the same distinction without reusing learner examples.
- Export the reviewed dataset or explicitly launch a compatible connected job. Evaluate the identified trained candidate separately and inspect both exception handling and ordinary source selection.
Object and state changes
Expert judgment becomes a reviewed standard; a preparation develops learner material; training and assessment create their own records.
Success criteria
The team can trace the teaching purpose to the expert’s qualification and evaluate the trained candidate on separate situations.
Common failure modes
Synthesizing many paraphrases of one answer without the exception condition; copying test Cases into training; accepting a provider success status as learned behavior.