{
  "id": 2175648,
  "title": "Millions of log lines in PHP, at constant memory",
  "url": "https://urgent.news/2026/08/20/millions-of-log-lines-in-php-at-constant-memory",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-20T16:09:01.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/nessflow_8283f7335b896207/millions-of-log-lines-in-php-at-constant-memory-i7n"
  },
  "original_language": "en",
  "account": "Laravel 13, Horizon and PostgreSQL 18 were selected for a specific question about which crawler fetched which URL on a customer's site. The need for a log analyzer was identified and GoAccess was initially used, but abandoned. The reason was that GoAccess produces aggregated reports, aggregating data such as top URLs, top user agents, status code distribution, and hits per day. However, it cannot cross these aggregates, which results in a picture of data rather than actual data.\n\nTo address this, a cube was needed, consisting of date, hour, bot, URL, status code, with hit counts and bytes. This aggregate had to be built on custom axes, which required a parser tool. Several input formats were supported, with seven of them based on common log format and using GoAccess style format strings compiled into regex once. Other formats have dedicated parsers, including Cloudflare JSON, Caddy JSON, Google Cloud Storage CSV, and a W3C parser that is stateful due to IIS's unique column declaration.\n\nBenchmarks were conducted on the machine on August 19, 2026. The parse line, classify user agent, and normalize URL stages were benchmarked, using a synthetic but hostile corpus with 5,000 distinct paths and 240 distinct agent strings. The measurement showed that peak memory usage was 44.5 MB, regardless of the file size, demonstrating the constant nature of memory usage.\n\nThe aggregation data is stored in a table with a unique key being the cube itself. Each column in the uniqueness key is NOT NULL to ensure the key deduplicates effectively. The write operation is an additive upsert, which allows for safe partial work and restartability. This pipeline can handle two log files from two servers covering the same hour, merging them instead of overwriting each other.\n\nIdempotence is enforced upstream, using a SHA-256 of the uploaded file computed server-side while the chunks are assembled. This ensures that a failed run can be re-uploaded without issues. The bug that consumed its own error handler was also addressed, as real access logs may contain non-UTF-8 bytes and null bytes. This information was extracted from the given source material, presenting facts and details without reusing phrases from the original source.",
  "summary": "Laravel 13, Horizon, PostgreSQL 18. Our product needs to answer a question about a customer's site: which crawler fetched which URL, on which day, and what status code did it get. Not \"how many hits yesterday\". The cross product. Why not a log analyzer We started with GoAccess, which is an excellent tool, and abandoned it. The reason is worth stating precisely because it is the decision the rest…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}