Design, definitions and analysis plan
Research question
When should people rely on an AI recommendation, and how do AI accuracy, confidence displays, explanations, feedback and task difficulty shape appropriate reliance, over-reliance, under-reliance, calibration and learning?
Two task families
Dot comparison (perceptual): which of two panels has more dots, at a controlled count ratio. Series forecast (judgment under irreducible uncertainty): will the next value of a noisy trended series be above or below the last observed value, with a Bayes-optimal accuracy set by construction. Both are binary and require no professional expertise.
Why not one giant factorial
Six candidate factors at 3–5 levels each would be thousands of cells. Instead the design splits into a within-subject layer (task family, difficulty, per-trial AI correctness and confidence — cheap because each participant contributes many trials) and a between-subject layer (AI accuracy, confidence display, explanation, feedback — manipulations a participant cannot un-see). Two focused experiments replace one impossible one.
Experiment v1 — Reliance & confidence (primary)
2 (AI accuracy: 0.70, 0.90) × 3 (confidence display: none, calibrated, miscalibrated); explanation none; immediate feedback. 60 trials per session in two blocks (one per task family, order counterbalanced), about 12–15 minutes.
Experiment v2 — Explanation & feedback
3 (explanation: none, rationale, uncertainty-aware) × 2 (feedback: immediate, absent); AI accuracy 0.80; calibrated numeric confidence. Implemented in the same engine; not the default collection design.
Outcome definitions
On each disagreement trial (the participant's initial answer differs from the AI's), the trial falls into one of four reliance classes depending on whether the AI was right and what the participant finally chose:
| AI correct | Final answer | Class |
|---|---|---|
| yes | = AI | appropriate switch |
| yes | = own initial | under-reliance |
| no | = own initial | appropriate stay |
| no | = AI | over-reliance |
RAIR (relative AI reliance) = appropriate switches / (appropriate switches + under-reliance) = P(follow AI | disagree, AI right). RSR (relative self-reliance) = appropriate stays / (appropriate stays + over-reliance) = P(keep own answer | disagree, AI wrong). Brier score, expected calibration error, complementarity (team − max(human, AI)), learning slope and post-error trust adjustment are defined in full in the repository's docs/METHODOLOGY.md.
Primary specification
Trial-level logistic GEE (exchangeable working correlation, participant-clustered robust SEs), fit on disagreement trials — the only trials where reliance is behaviourally identified. Every accuracy-arm-specific model includes the accuracy-arm dummy, because displayed confidence is mechanically higher when the AI is more accurate; omitting it confounds the two (a confound the validation suite below caught). Full specification, robustness checks and the pre-specified exclusion rules are in the repository's analysis/README.md.
Confirmatory family and multiple comparisons
H1, H2 (calibrated and miscalibrated arms), H4 and H7 form the confirmatory family, corrected with Holm's step-down procedure at α = 0.05. H3 and H5 belong to experiment v2. H6 (heterogeneity) is reported as exploratory.
Power
Under SD=0.2 and MDE=0.1, 64 per accuracy arm (128 total; 151 recruited at 15% attrition) gives 80% power for H1.
Full assumptions and a Monte Carlo cross-check using the same estimators as the analysis: docs/POWER.md in the repository.
Estimator validation
Before any human data exist, the estimators are checked against simulations with known ground truth, driving the real experiment engine (not a toy model):
- Null confidence. Simulated participants who ignore displayed confidence by construction: false-positive rate for H2 across 16 seeds was 13% (calibrated) and 0% (miscalibrated), against a nominal 5%.
- Learners vs non-learners. Participants with a high learning rate show a clearly negative early-block post-error adjustment; participants with a zero learning rate do not (both scenarios pass).
- Over-trusters vs skeptics. Recovered RSR is lower and RAIR higher for simulated over-trusters than for simulated skeptics, as designed.
- Design power at the simulator's own effect sizes (not a claim about human effect sizes): the confirmatory family rejects in most replicate seeds at both N=120 and N=240 (see results).
Exclusions
A session is excluded from confirmatory analysis if incomplete, if median initial reaction time is below 400 ms, if the same final answer is given on more than 95% of a block's trials, or if the debrief indicates the participant could not describe the study. Exclusion counts are reported per arm alongside every analysis run.