Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation
Large language models (LLMs) are increasingly used to annotate social-science texts, but high accuracy does not establish that they apply researcher-supplied category definitions: predictions may instead reflect prior associations with category labels or labeled demonstrations. We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
