Engineering practice · Model development
Use expert judgment and coverage design to direct training-data augmentation. Establish what the model needs to learn, construct examples that teach it, and assess whether the improvement transfers.
For post-training teams and AI engineers developing domain-specific model behavior
What changes the decision
Repeated failures can have different causes: the task lacks necessary context, the harness uses the model poorly, the intended criterion is ambiguous, or the weights do not reliably express the required behavior. Dataset augmentation needs a learning target before it needs volume.
Engineering practice · These are development and assessment methods, not reported customer results.
The context
Experts agree that a response is weak but disagree about the alternative. Generating more examples at this point can multiply an unresolved preference.
Judgment to capture
Which criterion is missing, and what condition explains the disagreement?
The context
Examples vary in wording while keeping the underlying reasoning and conditions nearly identical. A large set can still omit the situations in which the model struggles.
Judgment to capture
Which ontology combinations and decision boundaries need deliberate representation?
The context
The candidate improves on material closely related to the examples used during development. The result may not establish the intended generalization.
Judgment to capture
Which independently designed cases would test the same capability under different conditions?
The people behind the standard
Teammately’s agents prepare output curation, comparisons, and adaptive conversations around the failure. Experts clarify the intended distinction; agents develop the findings into proposed rubrics and coverage that can direct training-data construction.
The reasoning, preferences, and exceptions that distinguish acceptable behavior.
The learning objective, training format, baseline, and independent assessment design.
Coverage and improvement
Use the benchmark failure to formulate a learning objective. Expert findings establish the distinction, coverage defines where it should hold, and Weave constructs teaching material that can be evaluated through a separate assessment design.
Teammately’s contribution
Agents prepare curation, comparisons, and adaptive follow-up around the failure. Experts clarify the requirement and the conditions that change it. Develop the findings into proposed rubrics and coverage.
What this makes possible
A learning target the team can explain, including where the desired answer or preference should differ.
Teammately’s contribution
Coverage selects ontology tuples, difficult combinations, and intended proportions. Weave constructs cases and comparisons around those distinctions, with expert criteria guiding what the examples should teach.
What this makes possible
A dataset whose additions address identifiable conditions rather than increase volume through near-duplicate wording.
Teammately’s contribution
Use the agreed training environment and retain the data and configuration behind the candidate. Assess it on separate cases under comparable model-serving and evaluation conditions. Trialground supplies managed trials and rubric evaluation where included.
What this makes possible
Evidence of the intended learning, preserved behavior, and remaining gaps, with the benchmark’s representation defining the scope of the conclusion.
Weave supplies training-data augmentation; training execution and compute use the agreed training environment. Coevolve provides the separate route for harness prototypes when the evidence points to an implementation change.
Assessing progress
A benchmark gain should identify the baseline, updated score, models, harnesses, and evaluation conditions. Where practical, compare targeted augmentation with a baseline that helps isolate its contribution. Report the coverage on which learning transferred and the conditions that remain weak.
Expert time becomes meaningful when readers can see what the contribution established and how agents turned it into usable material. Report participation separately from engineering, generation, and training effort. The evidence should connect the captured judgment to the resulting improvement.
Adjacent workflows
Teach the contextual tradeoffs behind an expert’s preference between acceptable responses.
Use engineering judgment to define task boundaries, verification, and acceptable changes.
Teach how missing or conflicting evidence should change the answer or next question.
We can examine whether the missing piece is the judgment, the representation, or the learning target, then define a useful augmentation and assessment scope.