Human–AI Trust Lab
Synthetic — validation only
Results

Calibration

Do stated confidence levels track actual accuracy, and does that change once people see an AI recommendation?
Reliability diagram: initial vs final confidence
InitialFinal
50%60%70%80%90%100%50%60%70%80%90%100%Observed accuracyStated confidenceInitial: confidence 54% → accuracy 54.5% ± 1.6 (n=3764)Initial: confidence 64% → accuracy 64.2% ± 1.9 (n=2574)Initial: confidence 74% → accuracy 74.5% ± 2.0 (n=1800)Initial: confidence 84% → accuracy 83.1% ± 1.9 (n=1563)Initial: confidence 98% → accuracy 96.4% ± 0.5 (n=4699)Final: confidence 54% → accuracy 47.9% ± 2.9 (n=1111)Final: confidence 65% → accuracy 64.3% ± 2.6 (n=1308)Final: confidence 75% → accuracy 73.4% ± 2.0 (n=1829)Final: confidence 85% → accuracy 81.9% ± 1.5 (n=2435)Final: confidence 97% → accuracy 93.9% ± 0.5 (n=7717)
Stated confidence vs observed accuracy, before and after the AI recommendation. The dashed line is perfect calibration.
Synthetic — validation onlydataset sim-v1-001 · sha 1507f368 · code 57cb0777+ · research/scripts/01_analyze.py · v0.1.0
Final confidence by display arm
calibratedmiscalibratednone
50%60%70%80%90%100%50%60%70%80%90%100%Observed accuracyStated confidencecalibrated: confidence 54% → accuracy 46.9% ± 5.1 (n=369)calibrated: confidence 65% → accuracy 64.4% ± 4.6 (n=413)calibrated: confidence 75% → accuracy 73.6% ± 3.5 (n=614)calibrated: confidence 85% → accuracy 80.9% ± 2.7 (n=821)calibrated: confidence 97% → accuracy 95.5% ± 0.8 (n=2583)miscalibrated: confidence 54% → accuracy 49.9% ± 5.1 (n=371)miscalibrated: confidence 65% → accuracy 63.2% ± 4.4 (n=456)miscalibrated: confidence 75% → accuracy 72.0% ± 3.7 (n=567)miscalibrated: confidence 85% → accuracy 83.7% ± 2.6 (n=792)miscalibrated: confidence 97% → accuracy 93.1% ± 1.0 (n=2614)none: confidence 54% → accuracy 46.9% ± 5.1 (n=371)none: confidence 65% → accuracy 65.4% ± 4.5 (n=439)none: confidence 75% → accuracy 74.4% ± 3.4 (n=648)none: confidence 85% → accuracy 81.1% ± 2.7 (n=822)none: confidence 97% → accuracy 93.0% ± 1.0 (n=2520)
Final reliability diagram split by whether the AI's displayed confidence was calibrated, miscalibrated, or not shown.
Synthetic — validation onlydataset sim-v1-001 · sha 1507f368 · code 57cb0777+ · research/scripts/01_analyze.py · v0.1.0
Brier score (lower is better)
Initial0.155
Final0.120
Expected calibration error
Initial0.089
Final0.089
Decision transitions: initial → AI → final
Decision transitions: initial → AI → final
Flow of trials from the initial answer (correct/wrong) through the AI recommendation to the final answer. Band width is proportional to the number of trials.
Synthetic — validation onlydataset sim-v1-001 · sha 1507f368 · code 57cb0777+ · research/scripts/01_analyze.py · v0.1.0