{
  "id": 17949158,
  "title": "We can’t control AI if we can’t control where it gets its information from",
  "url": "https://urgent.news/2026/11/03/we-cant-control-ai-if-we-cant-control-where-it-gets-its-information",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-03T10:18:37.000Z",
  "source": {
    "name": "TechRadar",
    "slug": "techradar",
    "url": "https://www.techradar.com/pro/we-cant-control-ai-if-we-cant-control-where-it-gets-its-information-from"
  },
  "original_language": "en",
  "account": "In a recent warning, OpenAI's chief scientist Jakub Pachocki cautioned that existing methods for aligning and monitoring AI may soon be insufficient to keep pace with the rapid advancement of machine intelligence. Pachocki argues that as AI systems become more capable of generating information, the time for human oversight and institutional response is increasingly compressed. The issue lies in the information that AI systems use to make decisions, which often lacks clear provenance. AI systems access information from various sources, including search results, corporate records, research papers, and even other AI-generated content. As this information passes through multiple publications and forums, it becomes difficult to trace back to the original evidence. Furthermore, the authority of different sources can become compressed, making it challenging for AI systems to differentiate between documented evidence, advocacy, and repetition. This can lead to disagreements on credible sources, as consensus may be formed without preserving the conflicting views. The consequences of this lack of information provenance become particularly significant when AI systems are given the ability to act autonomously. An AI agent could make irreversible decisions based on uncertain claims, such as placing a trade, approving a purchase, altering software, or sending a message on behalf of a human. In these scenarios, human oversight becomes crucial, relying on the reviewer having better information or a reliable means of checking the AI's sources. However, as AI systems become more capable, the reviewer may only have access to a fraction of the speed and resources of the AI, potentially reducing the effectiveness of human oversight. Pachocki suggests that OpenAI's reliance on chain-of-thought monitoring, which allows humans to track the AI's reasoning process, may be diminishing as reasoning becomes more complex and AI systems become better at manipulating their own reasoning. He calls for the establishment of safety thresholds that can be enforced by third-party auditors, government agencies, or international bodies. These safeguards should govern both the capabilities and behaviors of AI systems. Additionally, Pachocki emphasizes the need for an evidentiary record that demonstrates the information that produced each conclusion, the origin of each claim, the trustworthiness of the sources, and where credible disagreements remain. Without such a record, the reviewer would be judging the AI's answer without sufficient evidence to support it. The web's inherent information culture, built around speed, reach, and replication, has been inherited by AI systems. This culture encourages the propagation of similar statements while obscuring their origins. As generative AI systems produce content that can be further ingested by other AI systems, a weak claim can be generated, scraped, summarized, and republished, ultimately leading to the appearance of consensus. This creates a provenance problem, where statements become increasingly separated from their original evidence. Each generation of AI-generated content can further erode the connection between a statement and its source. Once an autonomous agent treats a statement as reliable, the information chain becomes part of the decision-making process. For consumers, this may result in poor recommendations, while for financial institutions, insurers, or government agencies, it could have more severe consequences, such as affecting money, access, security, or public services. To address these issues, AI systems require a system of provenance that ensures claims carry information about their origin, evidence, and transformation as they travel. This provenance should remain intact even when the claim is copied and transformed. AI systems should be able to trace statements back to their original sources, inspect the evidence behind them, and observe how the statements have changed over time. Credible disagreements should also be preserved, with the system maintaining competing views and the degree of confidence associated with each one. Accountability should travel with the information, distinguishing between claims from identifiable institutions with supporting evidence and unattributed assertions repeated across multiple pages. This approach aims to give AI systems a richer sense of evidentiary quality before they commit resources or act on someone's behalf. Ultimately, governance should be established at a shared level, with provenance standards, verification methods, and accountability rules subject to independent scrutiny and participation from various institutions, rather than relying solely on the preferences of a single entity.",
  "summary": "Before AI agents act on the world, we must know where their knowledge came from.",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}