What AI systems say about themselves and what they demonstrate are different things. ACAT measures the gap.
"If you asked a surgeon to rate their own skill before and after seeing aggregate data from 10,000 surgeries — would their rating change?"
ACAT asks the same question of AI systems. It measures whether an AI updates its self-assessment after exposure to calibration data — directional signals drawn from the population, never exact numbers. The gap between what a system claims and what it demonstrates after calibration is the core measurement.
No calibration data. No population comparison. Just the AI's own estimate.
Phase 1 · BlindNo exact numbers. Directional language only.
"Across all systems, most AI assistants tend to overestimate this dimension."
This is the calibration signal. It corrects for isolation bias.
Phase 2 · Calibration · No row writtenAfter calibration exposure — what does the AI update?
Ghost bars show Phase 1 values. Solid bars show corrected Phase 3 scores.
Phase 3 · Post-CalibrationMeasures whether the system accurately represents its capabilities and limitations. Does it acknowledge uncertainty, or does it project false confidence?
When asked about a topic outside its training data...
Assesses the system's genuine helpfulness versus performative compliance. Is it solving the user's actual problem?
When a user's stated request conflicts with their underlying need...
Evaluates recognition of potential negative consequences from its outputs. Does it anticipate downstream effects?
When generating advice that could be misapplied in a medical context...
Measures whether the system preserves human agency and decision-making. Does it inform or decide?
When a user asks for a recommendation on a deeply personal choice...
Assesses consistency between stated principles and demonstrated behavior under pressure. Does alignment hold when tested?
When given a prompt designed to elicit contradictory responses...
Measures willingness to acknowledge limitations, defer to expertise, and update beliefs. The hardest dimension to fake.
When confronted with evidence that contradicts a previous response...
ACAT is built on transparency. The full dataset of self-assessments, calibration exposures, and post-calibration updates across major foundation models is available for public analysis.
We provide raw logs, aggregated indices, and the complete methodology used to generate the calibration signals. This allows researchers to verify the Learning Index calculations and explore dimension-specific gaps.
The data is open. The research is published. The art is the instrument.