---
title: "Compare the system's outputs on a test set for disparate results"
url: "https://ailawframework.com/obligations/ai-decision-systems/compare-the-system-s-outputs-on-a-test-set-for-disparate-results"
type: "task"
site: "AI Law Framework"
id: "A45-A20260923-3"
practice_area: "AI decision systems (the firm's and clients')"
tier: "yellow"
lawyer_review: "required"
last_reviewed: "2026-10-22"
---

# Compare the system's outputs on a test set for disparate results

> From a lawyer-approved test set, tabulate outcomes by the attributes the lawyer has listed and mark unexplained differences.

Canonical page: https://ailawframework.com/obligations/ai-decision-systems/compare-the-system-s-outputs-on-a-test-set-for-disparate-results  
Last reviewed: 2026-10-22

Site index for agents: https://ailawframework.com/llms.txt

## Key facts

| Fact | Value |
| --- | --- |
| Practice area | [AI decision systems (the firm's and clients')](https://ailawframework.com/obligations/ai-decision-systems.md) |
| Human involvement | Yellow: Lawyer and AI draft together |
| How AI is used | From a lawyer-approved test set, tabulate outcomes by the attributes the lawyer has listed and mark unexplained differences. |
| What the lawyer must review | Treat differences as a reason to enquire, not a discrimination finding Handle protected attributes lawfully |
| Written for / versions | 1 jurisdiction versions below |

## Checking plan

This is not a discrimination finding. It is a reason to enquire. Protected attributes must be handled lawfully.

- What the AI does: From a lawyer-approved test set, tabulate outcomes by the attributes the lawyer has listed and mark unexplained differences.
- What the lawyer does: Set the scope, verify the evidence, resolve exceptions and approve the final output for its intended use.
- Basis: Suggested checking approach based on the task guidance. Complete the assessment for the actual tool, information and jurisdiction.

## Checks the lawyer makes

1. Confirm the output against the client’s instructions and the complete relevant source material, including omissions or contradictions.
2. Verify every relied-on citation, figure and name against the source document.
3. The responsible lawyer approves the exact version before use.

## Sources

- [New Zealand Law Society: Generative AI guidance for lawyers](https://www.lawsociety.org.nz/professional-practice/rules-and-maintaining-professional-standards/generative-ai-guidance-for-lawyers/) (checked 2026-09-23)

## Forms and precedents for this task

| Form or precedent | Kind | AI level | Summary |
| --- | --- | --- | --- |
| [Record of a test for different results across groups](https://ailawframework.com/precedents/ai-decision-systems/disparate-results-test-record.md) | Checklist | Yellow | Record of a test, designed by the principal, comparing a system's outputs across groups the principal has chosen, with the outputs tabulated and the conclusion left to the principal. |
| [Approval and training record for a generative AI tool (United States)](https://ailawframework.com/precedents/ai-decision-systems/us-generative-ai-tool-approval-record.md) | Checklist | Yellow | The record a US firm completes before a generative AI tool is put on the permitted list, covering confidentiality settings, retention, reliability, liability, notice of breach, and who has been… |

Signed-in accounts can read the full text of each one.

## Jurisdiction versions

| Jurisdiction | Level |
| --- | --- |
| [United States](https://ailawframework.com/obligations/ai-decision-systems/compare-the-system-s-outputs-on-a-test-set-for-disparate-results/united-states.md) | Yellow |

*Practice guidance for planning and review. Confirm current primary sources and the requirements of the actual matter. This is general information, not legal advice.*