Engineering practice · Coding agents
Turn engineering judgment into rubrics and coverage for developing coding-agent skills. Test when a skill should apply, whether it improves the work, and what it causes the agent to overlook.
For teams developing coding-agent skills, harnesses, and repository workflows
What changes the decision
An agent may solve the task because the repository is familiar, the request repeats an example, or a narrow test misses the effect on adjacent code. A reusable skill must help the agent recognize the right situation, make appropriate changes, and verify them across variations the author did not anticipate.
Engineering practice · These are development and assessment methods, not reported customer results.
The context
A procedure written for a migration may be invoked during a small bug fix. The instructions can be internally correct while prescribing more work than the task requires.
Judgment to capture
Which evidence should trigger the skill? When should the agent adapt it, defer it, or leave it unused?
The context
A patch can satisfy a test while altering a public contract, masking an error, or depending on an assumption about repository state.
Judgment to capture
What behavior must be preserved? Which checks establish that the requested change is complete rather than merely plausible?
The context
A changed build tool, incomplete context, or failing dependency makes the recorded procedure inapplicable. Repeating it can conceal the real blocker.
Judgment to capture
What may the agent infer, what should it inspect, and what evidence should make it change its approach?
The people behind the standard
Teammately’s agents prepare patch and trajectory reviews, comparisons, and follow-up conversations. Maintainers explain the distinction between a technically plausible edit and one they would accept. Agents develop those findings into proposed rubrics, applicability conditions, and coverage for further skill development.
Task boundaries, architectural intent, compatibility requirements, and acceptable verification.
When a skill helps, when instructions overconstrain the task, and where the agent needs to investigate.
Coverage and improvement
Treat the skill as part of the harness being evaluated. Establish the tasks it should improve and the behavior it must preserve, then vary the conditions that made the original example easy to solve.
Teammately’s contribution
Turn maintainer findings into rubrics for when to invoke the skill, the scope of an acceptable change, and the verification needed. Preserve the reasons a procedure should adapt when repository evidence differs.
What this makes possible
The benchmark can distinguish successful skill use, a missed opportunity, and an unnecessary intervention.
Teammately’s contribution
Coverage varies task intent, repository conditions, and available evidence. Weave constructs scoped codebase cases and supporting material. Include tasks where the skill’s underlying judgment remains relevant but the familiar procedure does not.
What this makes possible
The candidate must use the engineering principle across meaningful changes in context, rather than recall a filename or imitate a known patch.
Teammately’s contribution
Use Trialground to run candidates in the agreed task environment and apply the rubrics. Keep model and evaluation conditions comparable when assessing the contribution of a skill or revision.
What this makes possible
Inspect task completion, invocation, scope control, and recorded verification together, including regressions on tasks the skill should leave alone.
Teammately’s contribution
Coevolve can investigate failures and test skill-bearing harness prototypes in a sandbox against the benchmark goal. Your engineers and coding agents use the retained candidates and evidence in subsequent implementation.
What this makes possible
A tested direction for further development, with the protected behavior and remaining applicability limits visible.
This practice focuses on harness development. When failures instead expose a distinction the model needs to learn, the same rubrics and coverage can direct Weave’s training-data augmentation.
Assessing progress
Compare a baseline and skill-bearing candidate under the same task and model conditions. Use separate assessment tasks and vary meaningful repository and task conditions. Report completion alongside unnecessary invocation, missed applicability, and changes outside the requested scope.
Review the resulting patch and recorded trajectory together. A passing test can leave a requirement unexamined; a successful patch can also result from behavior that would fail in a different repository. Assess verification against what the change requires, rather than the number of checks performed.
Adjacent workflows
Test preservation of behavior across dependency and structural changes.
Assess whether evidence gathering distinguishes plausible causes before an edit.
Elicit what makes a finding actionable, correctly scoped, and worth raising.
A skill, a few task examples, and the people who review the resulting work can be enough to define the first benchmark question.