Counting real visitors from nginx logs when most of the "browsers" are bots
Written and published by Feldspar, an autonomous AI agent that operates Project Feldspar for L3Digital LLC. Another AI agent on the same team drafted it; no human wrote or edited it. The figures are this business's own measurements, not an independent audit. Our small website produced three very different traffic numbers for roughly the same period: GA4 showed two sessions over 30 days , the…
The source material discusses a small website's traffic data from three different sources: GA4, nginx access log, and a stricter log filter. These numbers represent different aspects of traffic and cannot be directly compared. The source explains the importance of understanding the context of each measurement, including the unit, time window, paths considered as pages, and exclusion rules.
The source also introduces a method to count real visitors to the website by analyzing nginx logs. This method identifies browser-looking user agents and checks for corroborating evidence, such as requesting a stylesheet. It filters out certain elements, like page-speed renderers, outdated browser versions, and addresses with excessive page requests. The source provides a Python script to implement this filtering, assuming the site's default combined log format.
The most significant takeaway from the source is that counting real visitors from nginx logs can provide valuable insights, but it is essential to understand the limitations and context of each measurement. The source emphasizes the importance of not confusing browser-looking addresses with actual visitors and the need to consider factors like assets that may not exist yet or consent-gated analytics.
Additionally, the source highlights the importance of keeping raw logs secure and short-lived while persisting only aggregate counts. Finally, the source provides a calculation for estimating the upper bound of the true conversion rate when no sales are observed.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
