{
  "id": 18739110,
  "title": "British Library Legal deposit and web archiving",
  "url": "https://urgent.news/2026/11/03/british-library-legal-deposit-and-web-archiving",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-03T14:56:30.000Z",
  "source": {
    "name": "Hacker News",
    "slug": "hacker-news",
    "url": "https://www.bl.uk/services/legal-deposit/web-archiving"
  },
  "original_language": "en",
  "account": "The British Library and other Legal Deposit Libraries are legally permitted to copy UK-published internet material for archiving purposes under the legal deposit system. This page explains how the library identifies UK websites and outlines the restrictions that apply to its archiving practices. Following a disruption in 2023, the UK Web Archive platform has been restored. The archive can be searched from anywhere, but most archived websites can only be viewed at a Legal Deposit Library reading room; visitors should confirm with the relevant library before visiting. The library is continuously improving the UK Web Archive site following its restoration. Users can follow the work of the UK Web Archive team on their blog or access data about the archive on the library’s institutional repository. Each year, the library collects a “snapshot” of all UK websites it can identify, which comprises at least four million websites and several billion files. However, the library does not collect material that is only shared between restricted private groups, such as content posted to private social media networks, or material where the content consists solely of a sound recording or moving image, unless there is accompanying text that makes sense without the audio or visual elements. The library does collect material that includes both sound and moving image alongside text or other elements, provided that the text or other elements have meaning independent of the audio and visual content. The legal deposit libraries primarily use web crawling software to collect content and metadata from open web pages. This automated process collects information without any access restrictions. Web crawling software requests copies of the root or home page of seed domain addresses programmed into its system. It then automatically follows links to subsequent levels within the same domain, issuing a separate request for each URL identified. The software is programmed with politeness rules and parameters to ensure it does not negatively impact the performance of the target website. For instance, it limits the number of levels crawled or the amount of content requested from an individual website and includes intervals between requests to prevent excessive bandwidth usage and website overload. The software identifies itself using standard automated protocols, submitting a “user-agent string” to the web server’s log of server requests, which informs the publisher’s webmaster of each page crawled. However, the web crawling software cannot access any material behind a login facility without the appropriate password or access credentials. The Framework for UK Legal Deposit (PDF, 289.7KB) provides additional information on how the library is developing its capacity to collect digital publications. Those interested in nominating a UK web archive for inclusion in the archive should contact the library.",
  "summary": null,
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}