{
  "id": 18359470,
  "title": "AI learns from the products of thinking, not thought itself. That’s a problem for machines – and for us",
  "url": "https://urgent.news/2026/11/05/ai-learns-from-the-products-of-thinking-not-thought-itself-thats-a",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-05T18:57:36.000Z",
  "source": {
    "name": "The Conversation AU",
    "slug": "the-conversation-au",
    "url": "https://theconversation.com/ai-learns-from-the-products-of-thinking-not-thought-itself-thats-a-problem-for-machines-and-for-us-293696"
  },
  "original_language": "en",
  "account": "When viewing a gallery of paintings, observers witness finished works. These masterpieces result from extensive learning, practice, and trial-and-error, but the actual process is not visible. Similar to paintings, musical recordings omit bad takes and jam sessions, published books conceal messy drafts, and scientific papers exclude years of false starts and dead ends. In essence, what we see is a fragment of the complete process that generated the final product. Artificial intelligence (AI) models are trained on these traces, which resemble the products of thought but may not have been created in the same manner. A recent paper posits that concentrating on traces while disregarding the process can hinder current AI systems, and the same issue may affect the humans who use these systems. Modern \"reasoning\" models may provide a step-by-step account of their reasoning, yet this reported reasoning is merely another trace. Often, it does not correspond to the actual computations driving the answer. Agentic systems, which involve a software \"harness\" around a large language model that runs a loop, feeds results back, and introduces genuine trial-and-error revision, come closer to genuine thinking. However, the intermediary chain of thought—the part generated by the large language model—is still a trace. The most significant gap exists in creative domains. Painters and musicians sometimes discuss how their work was influenced by accidents and friction during the creative process, such as paint running unexpectedly or an instrument sounding \"off.\" AI models, however, never experience such occurrences. The human counterpart of this issue is older than AI, dating back to 1997 when an American Airlines captain warned that pilots were becoming \"children of the magenta line.\" They relied so heavily on the plane's flight computer that their manual flying skills deteriorated. Aviation took measures to address automation dependency since then. The same phenomenon occurred when some medical clinics adopted AI-assisted colonoscopy, using software to highlight potential polyps in the live camera feed. Over time, the skills of human clinicians began to deteriorate: they detected fewer polyps when working without the AI system compared to when the system was introduced. This effect has been observed in classrooms as well. High school mathematics students who received unrestricted generative AI support outperformed their peers while the tool was available, but scored lower once it was removed. This decline can happen swiftly. One study showed that as little as ten minutes of AI assistance was sufficient to make individuals perform worse without the AI and to give up on problems more quickly. Failures and mistakes serve as valuable teachers. The solution may involve incorporating more process into AI systems. This could mean designing systems that mimic human-like processes. An assistant could revise its answer upon being challenged about the problem's framing, instead of simply agreeing, as current systems frequently do. Alternatively, an agent could learn from failures and mistakes, allowing external feedback to materially alter its approach to the task. For humans, this implies designing AI tools that preserve the processes of human intelligence within an individual. In a trial at Harvard, students taught by an AI physics tutor designed to support learning outperformed peers in an active-learning classroom because the tutor regularly presented the next question along the path to the answer rather than simply providing the answer. At times, the tutor should raise an ambiguity instead of resolving one, as sound judgment often emerges from prolonged deliberation about what one does not know. Process is crucial for genuine intelligence and creativity to emerge. An artificial system that incorporates more of this process will more closely resemble the real thing. Human learning similarly requires process. In education and any skilled profession, we must consider how AI systems are designed and used to maintain more of the thinking and learning process. Currently, it is relatively simple for machines to produce numerous traces of thought, but creating process is far more challenging. If we succeed, our machines can become more intelligent without making us any less so.",
  "summary": "Working with the traces of thought instead of the process of thinking means AI systems are limited – and they limit their users, too.",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}