Chiaro Methodology - Open-sourced SOC 2 audit method for AI
Show HN: Ex-Deloitte auditor open-sourced the whole SOC 2 method for your AI
Chiaro, from an ex-Deloitte auditor, open-sources its complete SOC 2 readiness and audit methodology. It includes 86 controls, 355 test attributes, 498 calibration examples, and a Type II testing method that favors complete populations over sampling. The framework is published to build trust and allow scrutiny, with all proper nouns and technical terms preserved in English. It's a resource for AI companies preparing for SOC 2, offering transparency and a rigorous, machine-speed approach.
The alternative to showing your work is asking people to take it on faith, and a compliance industry that ran on faith is why anyone is reading this.
- jpitz
This is an incredible resource for someone trying to prep for an audit. Thank you!
- tpoacher
I take it SoC does not stand for summer of code in this context?
- yylyyl
I'm the author. CPA, spent 5 years doing SOC 2 fieldwork at Deloitte, recently started my own audit firm.
This repo is the methodology we actually audit against, not a summary of it: 86 controls, 355 test attributes with pass criteria, evidence standards, and the Type II testing method including how we call deviations, with worked examples. It's generated from the same JSON that drives our audit tooling, so it can't drift from what we actually do.
What "for your AI" means concretely: the framework is JSON rather than prose, so the controls, the attributes, their pass criteria and the evidence map are all machine-readable, and there's a file listing every tool our server exposes. It's CC BY 4.0 — point your own model at it and run your own readiness against the same bar the examination applies. The part that makes that work is 498 calibration examples: each one records a judgment call, the verdict an AI reached, the verdict that was correct, and why. Without those a model grades itself generously.
Why publish it: the audit criteria (AICPA's Trust Services Criteria) are public, but every firm's actual testing layer is a black box. A buyer holding two SOC 2 reports can't tell whether one auditor inspected evidence and the other just collected screenshots — the reports look identical. This year's Delve episode (hundreds of near-identical reports, procedures allegedly drafted before client evidence arrived — allegations Delve disputes) made that opacity harder to defend. Standards […]