# Diagnose what needs to change
Generated: 2026-11-13T08:01:27.404Z
Source build: local
Canonical docs: https://teammately.ai/docs
---
id: workflow.improve.diagnose-failures
title: Diagnose what needs to change
summary: Choose an intervention after checking whether the failure comes from requirements, representation, available context, or implementation.
kind: task
product_area: improve
status: stable
updated: 2026-11-11
canonical: /docs/improve/diagnose-failures
---
# Diagnose what needs to change
Choose an intervention after checking whether the failure comes from requirements, representation, available context, or implementation.
## Prerequisites
An interpretable Case result, its applicable standards, input and materials, and the candidate identity.
## Steps
1. Read the actual output and criterion. Confirm the expected behavior is a requirement your experts support.
2. Check whether the Case supplies the source information and conditions needed to meet that requirement.
3. Inspect whether the benchmark represents the relevant behavior slice and whether the failure repeats across comparable Cases.
4. Direct the next assignment to the owner: an expert question, a coverage or Case update, a Lemon Code intervention, a Lemon Goal experiment, or targeted training material.
## Object and state changes
Diagnosis itself does not change the standard or candidate. Each accepted follow-up uses its owning workflow.
## Success criteria
The next assignment states a measurable behavioral problem, relevant Cases, and the reason the proposed intervention addresses it.
## Common failure modes
Changing prompts before checking missing context; treating every rubric failure as a model-weights problem.
## Related reference pages
{% related-card-grid title="Continue this workflow" %}
- [Inspect outputs and recorded trajectories](/docs/benchmark-evaluations/inspect-results)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Resolve a problem" %}
- [Permissions troubleshooting](/docs/troubleshooting/permissions)
- [Benchmark run troubleshooting](/docs/troubleshooting/benchmark-runs)
{% /related-card-grid %}
---
id: benchmark-evaluations.inspect
title: Inspect outputs and recorded trajectories
summary: Trace scores to exact responses and rubric judgments, distinguish incomplete results, and record authorized corrections.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-11-11
canonical: /docs/benchmark-evaluations/inspect-results
---
# Inspect outputs and recorded trajectories
Start with identity and completeness, then explain the score from the response, captured actions, and applicable Rubrics.
## Prerequisites
- Access to the selected Benchmark Version and Run.
- Available responses or evaluation progress.
- Project-management permission if a judgment correction is needed.
## Steps
1. Open Runs and confirm the candidate, saved Harness Version where present, machine and planned Trials.
2. Identify whether this is the Run used for score or another independent execution.
3. Open Results and inspect the Case population, applicable criteria and incomplete count.
4. Read failed gates, earned positive points and triggered penalties separately.
5. Open the response and rubric detail. For a response criterion, inspect the answer and visible Case materials. Choose **Inspect recorded steps** for the selected Case or individual Trial judgment. Read recorded model turns, tool arguments and results, and delegated work where captured. The judgment's references identify cited steps. Check capture status to understand which behavior was recorded.
6. Inspect individual Trial outcomes when reading a repeated result.
7. Decide whether the responsible change belongs to candidate code, Case content, applicability, rubric wording or the observed judgment.
## Scores and missing results
An absent trajectory does not establish that an action did or did not happen.
Interpret a response or trajectory judgment using the recorded material its
criterion requires. Captured tool calls and execution steps remain distinct from
private model reasoning.
**Recorded observations used by the AI evaluation** identifies the saved AI evaluator view. A later manual correction retains that original AI assessment. **Recorded execution** identifies the associated execution when the judgment has no saved observation view. Keep these distinct when explaining an older result. A missing source record is reported separately from the availability of the judgment itself.
Point scores can be negative, including below -100% for mixed positive/penalty Cases. A completed mandatory-gate failure contributes zero to the benchmark score. Missing or incompatible judgments stay incomplete even when another gate has failed.
The full planned population remains the denominator. An incomplete evaluation cannot be made comparable by ignoring its difficult or failed executions.
## Correct a judgment
An authorized project manager can choose **Correct judgment** on the exact Trial's rubric result and supply the supported corrected judgment and rationale. The original AI judgment and manual change remain recorded. Reopen the refreshed result to interpret its resulting scores consistently.
Changing the judgment is different from editing the Rubric or the Case. Correct the owning artifact when its meaning is wrong.
## Object and state changes
Inspection and filtering do not mutate results. A judgment correction records an attributable authoritative change. Starting a Contribution or an improvement Session creates separate work and preserves the inspected Run.
## Success criteria
- The conclusion identifies its Run, Trial and Benchmark Version.
- Scores are explained by applicable criteria and the configured Regime.
- Missing telemetry and judgments remain unknown.
- Corrections retain original judgment provenance.
## Common failure modes
- Reporting a rank without completeness.
- Assuming all results are read-only.
- Treating a gate failure as permission to replace unavailable results with zero.
- Inferring hidden model reasoning from a captured trajectory.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Evaluation Results](/docs/benchmark-evaluations/results)
- [Run Details and Metadata](/docs/benchmark-evaluations/run-details)
- [Rubric Points and Benchmark Scores](/docs/correctness-governance/scoring)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Related troubleshooting pages" %}
- [Benchmark Results Changed Unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly)
- [Overlapping Rubrics](/docs/troubleshooting/overlapping-rubrics)
{% /related-card-grid %}
---
id: troubleshooting.permissions
title: Permissions troubleshooting
summary: Separate Project membership from Expert Contribution assignment and readiness when an expert has no tasks available.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-11-06
canonical: /docs/troubleshooting/permissions
---
# Permissions troubleshooting
Use this when project role, reviewer assignment, or approval ownership prevents someone from completing the intended work.
## Symptom
A user can access Teammately but cannot open the expected Project, or an expert reports **no tasks available** after opening their Contribution link. These symptoms belong to different owning surfaces.
## Likely causes
- The user lacks the required Project permission through a direct grant, workspace group, or workspace role.
- The Contribution was requested for a different recipient.
- The Contribution exists but its next Task or Checkpoint is not ready.
- The current IP is blocked before Project membership is evaluated.
## Diagnostic checks
- Check direct grants in **Project Settings → Project Members**, the user's workspace-group membership, and the group's permission for the intended Project. Absence from the direct member list alone does not establish lack of access. Inspect workspace status and role in **Admin → Directory → Members** as well.
- Open the Contribution from the sender-side Expert Contributions workspace and verify its recipient and status.
- Check whether the expert has an executable Task or Checkpoint, not merely whether the Contribution exists.
- If **Access Restricted** is visible, resolve the IP allowlist first.
## Fix
- Add the correct individual workspace user under **Project Settings → Project Members** when direct Project membership is missing.
- If access is intended through a workspace group, confirm both the user's membership in that group and its permission for the intended Project.
- Correct the Contribution recipient through the owning Contribution workflow when the request went to the wrong person.
- Resolve lifecycle or readiness problems in Expert Contributions when access succeeds but no Task can be entered.
- Do not infer a permission from a role label or edit governed artifacts merely to make a control appear.
## Prevention
- Separate reviewer access from approval authority in review setup.
- Check assignments before launching a review session.
- Use a small pilot Contribution before assigning a larger specialist cohort.
- Keep project membership changes visible to review owners.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Request an Expert Contribution](/docs/expert-contributions/request-contribution)
- [Expert Contributions](/docs/expert-contributions)
- [Complete an Expert Contribution](/docs/expert-contributions/complete-contribution)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Permissions](/docs/reference/permissions)
- [Roles and Permissions](/docs/governance/roles-and-permissions)
- [Expert Contributions](/docs/expert-contributions)
{% /related-card-grid %}
---
id: troubleshooting.benchmark-runs
title: Benchmark run troubleshooting
summary: Diagnose a Run that cannot start, has no usable outputs, or produces results that cannot be compared safely.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-11-06
canonical: /docs/troubleshooting/benchmark-runs
---
# Benchmark run troubleshooting
## Symptoms
- The evaluation surface has no Benchmark Version to run.
- Output import or mapping cannot identify a Case or output column.
- A Run is created but remains empty, incomplete, or failed.
- Results appear under the wrong candidate label or Harness Version.
- Two Runs show a score change but do not share a comparable snapshot boundary.
## Likely causes
- No immutable Snapshot exists for the intended evaluation set.
- Imported outputs are mapped to the wrong Case or column.
- The saved Harness Version or candidate metadata does not match the evaluated system.
- The Run is partial, failed, or being compared across different Benchmark Versions.
## Check the boundary before the failure
1. Confirm the URL and page identify the intended Project, Benchmark, and Benchmark Version.
2. Open the Version or Snapshot and verify it contains the expected Cases and evaluators. If it does not, repair coverage and create a new Snapshot; do not edit the historical Run.
3. For managed execution, confirm the saved Harness Version and recorded settings. For imported results, confirm that the external candidate context describes the system that produced the submitted responses.
4. If importing outputs, inspect the mapping preview. Match the Case identifier and candidate-output column deliberately; do not use a reference-output column as candidate behavior.
5. Open Run detail and inspect status, Case count, errors, metadata, and per-Case results before trusting aggregates.
## Fix
- **No runnable version:** finish Case selection and create a Snapshot first.
- **Missing imported outputs:** append only missing Cases to an uncompleted Trial using the exact manifest IDs. Correcting an accepted response or adding Cases after completion requires a new Trial in the external Run or a new Run. Preserve the original results.
- **Wrong Harness or metadata:** create a correctly configured Run. Do not relabel completed results to represent a different system.
- **Partial managed failure:** inspect and preserve the existing Run's recorded outcomes. For another independent execution under unchanged conditions, use **Run again**. **Run evaluation** reopens matching work, including failed work. Publish a new Harness Version when its code changes, or create a new Snapshot and Benchmark Version when Case content or criteria change.
- **Confusing comparison:** compare the Benchmark Version, Harness Version, candidate metadata, and evaluator boundary. Qualify or avoid the comparison when more than the intended variable changed.
## Prevention
Create a Snapshot before evaluation. For managed execution, save the exact Harness Version. For imported results, preserve external candidate context and preview the mapping on representative Cases. Before comparing, confirm that every difference between the two Runs is intentional and visible.
## Recovery check
Open the recovered Run and sample several Case results. Confirm the displayed input, candidate output, reference output where present, applicable Rubrics, and metadata all belong together. A completed status alone does not prove correct mapping.
{% example-demo title="Example: scores drop after output import" %}
A team imports a new candidate file and sees a sudden score collapse. Run detail shows that the column containing reference outputs was mapped as candidate output. The operator creates a new Run, maps the actual candidate column, preserves the mistaken Run as an identifiable failed attempt, and compares only the corrected Run with the prior candidate under the same Benchmark Version.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related workflows" %}
- [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Import External Evaluation Outputs](/docs/benchmark-evaluations/output-mapping)
- [Inspect Evaluation Results](/docs/benchmark-evaluations/inspect-results)
- [Evaluation Results](/docs/benchmark-evaluations/results)
- [Run Details and Metadata](/docs/benchmark-evaluations/run-details)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference" %}
- [Benchmarks and versions](/docs/concepts/benchmarks-and-versions)
- [Benchmark snapshots](/docs/coverage-engineering/benchmark-snapshots)
- [Outputs](/docs/object-model/outputs)
{% /related-card-grid %}