Confidence Scores in Structured Extraction

Confidence scores in structured extraction are a widely accepted yet flawed approach. Regardless of whether the model is right or wrong, the confidence score remains static at 0.95, even when presented with an empty page. This number is not a true indication of the model's belief but rather a token sequence generated based on the distribution of probabilities it was trained on, resulting in high, round, and reassuring values.

Three key aspects reveal the uselessness of confidence fields:

1. Clustering: Histogramming the values often reveals clustering around 0.9, 0.95, and 0.99, indicating the capture of a style rather than genuine belief.

2. Mean score comparison: When the mean score on correctly known records closely matches that on incorrectly known records, the confidence field carries no information.

3. Position sensitivity: Moving the confidence field's position in the output alters the numbers, suggesting a lack of true belief.

The literature provides insight into the nature of confidence scores. Kadavath et al.'s study on Language Models (Mostly) Know What They Know (arXiv:2207.05221, Anthropic, 2022) found that large models can be reasonably calibrated on multiple-choice questions when the probability is derived from the token distribution. However, when asked to evaluate the probability that their own answer is true, the signal is real.

The elicitation method significantly impacts calibration. OpenAI's GPT-4 technical report showed that a pre-trained model is well-calibrated on MMLU, while a post-RLHF model shows reduced calibration.

The models accessed through APIs are post-training ones, and neither result validates trusting a self-reported number in an extraction schema. Instead, three alternative signals are recommended:

1. Token logprobs: Setting logprobs provides the log-probability of each emitted token, particularly useful for fields constrained to an enum. This method ranks the top two labels' margin more informatively than the top probability alone. For free-text fields, using the mean per-token logprob or the minimum across tokens is advised.

2. Self-consistency: Sampling the same extraction multiple times at non-zero temperature and measuring per-field agreement offers an interpretable method without needing logprobs.

3. Verification: Requiring a verbatim source quote and checking its appearance in the input, sum of line items, and date within the contract period provides the strongest and cheapest signals.

When using a trusted score, it should not be surfaced to users for interpretation, as they cannot comprehend its meaning. Instead, use it internally for three purposes:

1. Routing: Re-running the extraction on a larger model below a confidence threshold, determined by calibration tables and the cost of mistakes.

2. Confidence threshold adjustment: If the score fails to meet expectations, lowering the threshold can help.

3. Calibration testing: Do not assume the score is calibrated; test it using a few hundred verified records, comparing the mean score in each bucket to the actual accuracy in that bucket. This expected calibration error helps identify overconfidence or wrong predictions in a specific bucket, enabling adjustments.