Trialground Evaluation
Kestel
Binary rubric evaluation
Assess whether candidate behavior meets each applicable criterion. Kestel is trained for binary rubric judgments, with consistent application across model families as an objective.
Teammately Research
We study how human intent becomes reliable AI behavior.
Our research investigates how expert knowledge can be captured, tested, transferred, and preserved in increasingly autonomous AI systems. Some of this work directly informs Teammately; some explores the foundations the product may depend on years from now.
Explore the research programs ↓
Why we research
Expert judgment holds more than an answer.
A judgment can carry a principle, the conditions in which it applies, and the exceptions that change it. Understanding how AI can learn from that structure is central to our research.
As systems become more autonomous, the question extends from learning a preference to preserving it through unfamiliar situations, new models, and chains of delegated work.
These questions inform what we build today and open directions whose applications may emerge years from now.
Teammately models
Our models for evaluation and synthesis.
Meet Kestel and Bower. We fine-tune open-weight models for two demanding tasks: applying expert rubrics and constructing cases across planned coverage. Our objective is fair assessment and useful development material across model families.
Lemon Weave
Bower
Coverage-directed case synthesis
Construct cases around ontology tuples and expert criteria. Bower is trained for coverage-directed synthesis, supporting benchmarks, expert elicitation, and targeted training material.
Structured Supervision
How much can one minute of expert judgment teach a model?
We study whether structured expert intent, including policies, applicability conditions, boundaries, and rubrics, produces greater behavioral improvement than conventional labels or demonstrations for equivalent human effort.
Open questions
- Which parts of an expert’s judgment carry beyond the original example?
- How much do applicability conditions, boundaries, and exceptions add to supervision?
- What knowledge can expert elicitation recover that automatic rubric induction misses?
Benchmark Science
When does a benchmark actually predict behavior?
We study how coverage, case construction, holdout design, and optimization pressure affect what benchmark results can legitimately tell us about AI behavior.
Open questions
- When does improvement on a development benchmark transfer to a sealed test set?
- How do synthetic, representative, and boundary cases change the conclusions a benchmark supports?
- How does repeated optimization change the predictive value of a benchmark?
Behavioral Adaptation
How does intent transfer into model behavior?
We study how human intent can be translated into effective interventions across prompts, harnesses, skills, synthetic training data, and model weights, and when those improvements generalize beyond the examples used to create them.
Open questions
- When should intent be represented in prompts, skills, harnesses, or model weights?
- Which gains transfer across models and unfamiliar tasks, and which introduce regressions?
- How does training-case augmentation affect generalization beyond the source examples?
Delegated Agency
Does intent survive delegation?
We study what happens to human objectives, constraints, and preferences as work is decomposed and delegated across agents, subagents, tools, and long-running processes.
Open questions
- Which objectives and constraints are lost when a task is decomposed or compressed?
- How do delegation depth and context boundaries affect fidelity to the original mandate?
- Can structured intent preserve human preferences through conflicting subagent objectives?
Inside structured supervision
Can a small amount of expert knowledge teach more?
Our starting hypothesis is that reusable principles, together with their applicability and boundaries, can produce greater behavioral improvement per expert-minute than conventional examples.
The proposed experiment asks where that advantage holds, what creates it, and where it breaks down.
Compare forms of supervision
Compare labels, explanations, generalized policies, and policies with applicability conditions, exceptions, and rubrics.
Match expert effort
Hold the human time budget comparable across conditions, including elicitation and review. Measure behavioral improvement per expert-minute.
Test beyond the examples
Separate training and development cases from sealed held-out cases. Examine whether the learned principles transfer to unfamiliar situations.
Isolate the contribution
Remove parts of the structure and compare expert-derived criteria with automatically induced rubrics to ask what the expert knowledge adds.
Research collaboration
What question should we investigate together?
We welcome conversations with researchers and domain experts working on supervision, evaluation, adaptation, and autonomous agency. Bring an open problem, a competing hypothesis, or a setting that could challenge these ideas.