--- title: "AI decision systems (the firm's and clients')" url: "https://ailawframework.com/obligations/ai-decision-systems" type: "obligation" site: "AI Law Framework" id: "A45" task_count: 4 last_reviewed: "2026-09-23" --- # AI decision systems (the firm's and clients') > Identify the decision, affected people and consequences before deploying an automated system.… Canonical page: https://ailawframework.com/obligations/ai-decision-systems Last reviewed: 2026-09-23 Site index for agents: https://ailawframework.com/llms.txt Deploying AI that makes or materially informs decisions about individuals — recruitment, performance, claims triage, risk scoring — without a bias assessment and a human override is impermissible. From 10 December 2026, APP 1.7 requires privacy-policy transparency for substantially automated decisions affecting rights. ## Key facts - Tasks: 4 - Green: 0, Yellow: 3, Red: 1 - Levels: Green is AI drafts and the lawyer checks; Yellow is lawyer and AI draft together; Red is the lawyer writes and AI may only check, clarify or collect data. ## Summary Identify the decision, affected people and consequences before deploying an automated system. Review evidence of performance, possible discriminatory effects, privacy requirements and a workable route for human intervention. ## Principles - **Define the matter**: Confirm the client’s instructions, jurisdiction, intended output and the information available before dividing the work. - **Protect the information**: Confirm that the actual tool, feature and processing terms permit use of this information, including access, retention, training and disclosure conditions. - **Confirm the source material**: Check the work against the original records and identify omissions, contradictions and unresolved information. ## Checklist - Confirm that the actual tool, feature and processing terms permit use of this information, including access, retention, training and disclosure conditions. - Resolve exceptions and record the responsible lawyer’s approval of the exact output and intended use. ## Restrictions - Confirm firm, client and applicable court or professional restrictions before using any AI feature. ## Jurisdiction note Apply this guidance to the actual matter and jurisdiction. Confirm current requirements before relying on an example. ## Basis Practice guidance supports planning; the responsible lawyer confirms the scope, evidence and review requirements for the matter. Deploying AI that makes or materially informs decisions about individuals — recruitment, performance, claims triage, risk scoring — without a bias assessment and a human override is impermissible. From 10 December 2026, APP 1.7 requires privacy-policy transparency for substantially automated decisions affecting rights. ## Limits on AI assistance Red lines — never in this area ## Limits on AI assistance No automated decision materially affecting a person's rights without a documented bias assessment and a human override. Update the privacy policy for substantially automated decisions before 10 Dec 2026 (APP 1.7). Advise clients that the same obligations bind their own AI deployments. - No automated decision materially affecting a person's rights without a documented bias assessment and a human override. - Update the privacy policy for substantially automated decisions before 10 Dec 2026 (APP 1.7). - Advise clients that the same obligations bind their own AI deployments. ## Review checklist The lawyer's review checklist ## Review checklist Bias/disparate-impact assessment on file Human-in-the-loop override designed in APP 1.7 privacy-policy disclosure ready Sector-specific (e.g. anti-discrimination, ASIC) exposure reviewed - Bias/disparate-impact assessment on file - Human-in-the-loop override designed in - APP 1.7 privacy-policy disclosure ready - Sector-specific (e.g. anti-discrimination, ASIC) exposure reviewed ## Tasks (4) | Task | Level | How AI is used | What the lawyer must review | | --- | --- | --- | --- | | [Map automated decisions that affect individuals in the firm's or client's system](https://ailawframework.com/obligations/ai-decision-systems/map-automated-decisions-that-affect-individuals-in-the-firm-s-or-client.md) | Red | From the system description, list decisions, personal-information inputs, whether a human can override, and what the privacy notice currently says. | Confirm a human-override path and a notice exist before the system goes live Confirm the current privacy and anti-discrimination rules | | Build a legal-risk table for a client's AI system | Yellow | From the system papers, list candidate privacy, consumer, discrimination and sector issues with the documents that support each row. | Assess each issue against current law Do not treat a generated 'high risk' label as the advice | | Draft a privacy-impact or algorithmic-impact assessment for the system | Yellow | Structure an impact assessment from the system description: flows, affected people, risks, mitigations and sign-off questions. | Check the flow map matches the actual system Do not sign off a generic template | | Compare the system's outputs on a test set for disparate results | Yellow | From a lawyer-approved test set, tabulate outcomes by the attributes the lawyer has listed and mark unexplained differences. | Treat differences as a reason to enquire, not a discrimination finding Handle protected attributes lawfully | ## Jurisdiction notes ### New Zealand #### Governing law Privacy Act 2020, which is technology-neutral. No Australian ADM statutory overlay unless a New Zealand sector statute says so. #### Commissioner expects Senior-leadership approval, a necessity review, a privacy or algorithmic impact assessment, and engagement with Māori where taonga information is affected. #### Accuracy and use Procedures for accuracy, access and correction; human review before acting. IPP 8 and IPP 10 apply when personal information is reused to train or decide. - Do not let a tool make the final decision that affects a person without recorded human review. Sources: - [OPC: Generative artificial intelligence (15 June 2023)](https://www.privacy.org.nz/resources-and-learning/a-z-topics/ai/generative-artificial-intelligence/) (checked 2026-09-23) - [OPC: Artificial Intelligence and the Information Privacy Principles](https://www.privacy.org.nz/resources-and-learning/a-z-topics/ai/) (checked 2026-09-23) - [OPC: Privacy Act 2020 information privacy principles](https://www.privacy.org.nz/privacy-principles/) (checked 2026-09-23) ### United States Local name: AI decision systems (the firm’s and clients’) #### Employment The EEOC says federal discrimination statutes apply to AI in employment decisions. #### Colorado From 1 January 2027, SB26-189 duties for developers and deployers of ADMT that materially influences consequential decisions. #### Other duties State privacy statutes; California's guidance on biased outputs and Rule 8.4.1. Formal Opinion 512 governs the firm's own tools. Sources: - [EEOC, What is the EEOC’s role in AI? (29 April 2024)](https://www.eeoc.gov/sites/default/files/2024-04/20240429_What%20is%20the%20EEOCs%20role%20in%20AI.pdf) (checked 2026-09-23) - [Colorado Senate Bill 26-189 (became law 14 May 2026)](https://leg.colorado.gov/bills/sb26-189) (checked 2026-09-23) - [Colorado Attorney General: Automated Decision-Making Technology and Chatbot Safety rulemaking](https://coag.gov/ai/) (checked 2026-09-23) - [State Bar of California, Practical Guidance for the Use of Generative Artificial Intelligence in the Practice of Law (2026)](https://www.calbar.ca.gov/Portals/0/documents/ethics/Generative-AI-Practical-Guidance.pdf) (checked 2026-09-23) - [ABA Formal Opinion 512, Generative Artificial Intelligence Tools (29 July 2024)](https://www.americanbar.org/content/dam/aba/administrative/professional_responsibility/ethics-opinions/aba-formal-opinion-512.pdf) (checked 2026-09-23) *Practice guidance for planning and review. Confirm current primary sources and the requirements of the actual matter. This is general information, not legal advice.*