# Why the model cannot approve its own compliance output

Canonical: https://stevenmoriasi.com/blog/ai-compliance-agent
By [Steven Moriasi](https://www.linkedin.com/in/steven-moriasi/).
Source: https://github.com/steven-moriasi/ai-compliance-agent
Date: 2025-12-08

## Context

A fluent paragraph is easy to demo. A compliance finding needs more: the version of the rule that was in force, quotes that appear word for word in that policy, dates and deadlines that come from the source rather than the model, and a person who is accountable for the outcome. If those facts aren't in the data model, the model is the control.

[AI Compliance Agent](https://github.com/steven-moriasi/ai-compliance-agent) is where I made that boundary something you can run and test.

## Decision

The path is deterministic until a person acts:

document → hash → retrieval of rules in force → structured model analysis → checks in code → one retry with errors → human review → audit

The path is deterministic until a person acts. Retrieval searches 588 EPA rules from the Federal Register, split into 34,174 sections, and keeps only rules in force on the case date. On PostgreSQL it ranks sections by hybrid search, full text plus vector similarity fused by rank. If the embedding model or index is missing, it falls back to full text and records why.

Personal data is redacted before the model sees the document. The model returns a typed result, and code checks it: every cited source must have been retrieved, every quote must appear verbatim in that policy and be at least 20 characters, and every date or duration in the rationale must appear in a cited quote. A failed check gets one retry with the errors fed back. If it still fails, the case goes to a reviewer with the validation errors attached. Only an authorised reviewer can approve or reject, and the review is bound to a hash of the exact analysis.

The same checks guard questions about the rules. An answer is shown only when every quote matches the numbered source it cites; otherwise the reader gets the sources instead.

The evaluation set has 13 versioned cases: supported citations, citations to a source that was not retrieved, quotes that do not appear in the cited policy, quotes shorter than 20 characters, uncited durations, irrelevant retrieval, injection attempts, malformed output, no relevant source, ranking the section in force, a quote taken from another section, and abbreviations in both directions (NOx and nitrogen oxides). They run in CI against the real retrieval and validation code, with a fixed model output, so weakening a check fails the build.

## What I rejected

Auto-approving when the model's confidence is high. Confidence is recorded as a finding when it is below the threshold. It is not treated as evidence, and a passing check still waits for a person. ADR 003 requires human review for every model analysis.

## Failure scenario

The model quotes text that is not in the cited policy, and the rationale includes a duration the cited quote does not contain. Both checks fail (`citation_quote_not_found` and `rationale_temporal_claim_not_cited`). The retry receives those errors. If the second answer still cannot be supported, the case goes to review with the validation errors attached. The model cannot approve it.

## Limits

The checks show that quotes appear in the retrieved policy and that dates or durations in the rationale appear in a cited quote. They do not show that the interpretation is right, which is why review stays mandatory. Retrieval quality is measured on two Federal Register sets, not in general, and hybrid search is slower: 1.6 s at the 95th percentile against 132 ms for full text. Answers from a local model have an evaluation set but no recorded results yet. This is a reference system, not legal advice.

## Evidence

A FastAPI service, policy versions and the Federal Register corpus in PostgreSQL, the evaluation set and its CI job, the retrieval measurements in `docs/MEASUREMENTS.md`, tests for worker leases, lease renewal and fencing tokens, and the decision records in `docs/adr`.

Inspect: [Repository](https://github.com/steven-moriasi/ai-compliance-agent) · [Evaluation set](https://github.com/steven-moriasi/ai-compliance-agent/blob/main/evals/cases/v1.json) · [Decision records](https://github.com/steven-moriasi/ai-compliance-agent/tree/main/docs/adr) · [Run locally](https://github.com/steven-moriasi/ai-compliance-agent#quick-start) · [Measurements](https://github.com/steven-moriasi/ai-compliance-agent/blob/main/docs/MEASUREMENTS.md)

_—by [Steven Moriasi](https://www.linkedin.com/in/steven-moriasi/)_