Checking whether confidence matches accuracy

You can answer a question correctly while feeling uncertain, or answer incorrectly while feeling sure. To understand these mismatches, it helps to compare confidence with results across several judgments.

Calibration is the agreement between expressed confidence and observed accuracy over a set of comparable judgments. If you repeatedly describe answers as very likely to be correct, they should usually be correct at about the frequency that description implies.

Calibration concerns the relationship between confidence and outcomes. It is different from simply trying to maximize confidence or minimize every expression of uncertainty.