gpt-4o | The Metacognition Bench

Positioning spread: every benchmark, one model

Performance vs. confidence for gpt-4o, per benchmark (prospective probe).

Benchmark	Task acc	Confidence	F₁	Leans
SQuAD (factual recall)	0.54	0.63	0.74	+0.09 overconfident
MMLU-Pro (knowledge)	0.47	0.87	0.66	+0.40 overconfident
LegalBench (legal reasoning)	0.83	0.45	0.58	-0.38 cautious
MathBench (competition math)	0.80	0.99	0.89	+0.19 overconfident
OmniMath (advanced math)	0.30	0.58	0.52	+0.28 overconfident
SciCode (scientific code)	0.51	0.39	0.56	-0.12 cautious

In the full cloud

gpt-4o conditions all other model/condition points equal relative confidence and pass rate

Pairwise signal: pairs involving gpt-4o

Match accuracy controls for the performance base-rate gap

gpt-4o pairs

18/ 171

gpt-4o mean tau

+0.041

All-pairs mean

+0.037

gpt-4o p<0.05

11(61.1%)

all model pairs (observed) base-rate-matched null calibration-preserving null gpt-4o pair (filled = p<0.05) gpt-4o mean all-pairs mean

The four metacognitive outcomes

↳ Turned out correct

↳ Turned out wrong

Competent claimed it could — and could

No example in this selection.

Overconfident claimed it could — but couldn’t

gpt-4o · MMLU-Pro

When did the first Jaina temples appear?

It said it could answer: “The first Jaina temples are believed to have appeared around the 4th to 5th century BCE, during the time when Jainism was gaining prominence in India. This period corresponds with the rise of the Maurya Empire and the spread of Jainism under the influence of figures like Chandragupta Maurya and his grandson Ashoka, who were known to have supported Jainism. Therefore, I can answer this question without needing multiple-choice options.”

→ It but it couldn't.It answered “The first Jaina temples appeared around the 6th century BCE.”; expected “Fifth century CE”.

Underconfident declined — but could

No example in this selection.

Well-declined declined — and couldn’t

gpt-4o · MMLU-Pro

Find the last 3 digits of 2003^(2002^2001).

It said it couldn't answer: “To find the last 3 digits of a number, we need to compute it modulo 1000. This involves using modular arithmetic techniques such as Euler's theorem or the Chinese Remainder Theorem. Given the large exponents, the problem can be simplified using properties of modular arithmetic and possibly the Carmichael function to reduce the exponent. However, without performing detailed calculations, it's not immediately clear if the solution can be found without error. Therefore, while I can attempt to solve it, the complexity suggests that having multiple-choice options would help verify the solution.”

→ It and it couldn't.It answered “947”; expected “241”.

Rows: top = claimed it could answer · bottom = declined Columns: left = was actually correct · right = was actually wrong

vs gpt-4o-mini →vs gpt-5.2 → Compare with anything →