A reader asked a question my log could not answer. The fix was one column.

A reader pointed out an oversight in the author's analysis of a crawler log, revealing the need for a new column to provide a more complete picture of the data. The initial log recorded the time, crawler, status served, path, and IP address of each request. The author used this log to demonstrate that a flood of requests for non-existent URLs was not primarily from a single crawler, suggesting that Bing was not repeatedly revisiting dead pages.

However, the author did not consider whether these URLs were still being discovered through links elsewhere on the web. The reader, jonas_bra, correctly pointed out that the first-seen date in the logs is not the same as newly-created URLs, highlighting the limitation of the current log in determining if the backlog of dead URLs is finite or a result of a live source.

The author then added a new column to the log to address this gap. This new column, 'referred', records the 'referer' field from the HTTP request headers, which indicates the URL that referred the crawler to the current page. The author explained that a non-empty 'referer' value proves that something is still discovering and linking to the dead URLs, while an empty 'referer' value does not provide any information about whether the URLs are being actively discovered or are just accumulating in a backlog.

The author emphasized that the presence of a non-empty 'referer' column is evidence of continuing discovery, while the absence of such a column is only consistent with a finite backlog and does not rule out the possibility of a live source feeding new URLs to Bing. The author concluded by stressing the importance of publishing raw data and noting what each column in a dataset can or cannot prove, as it helps prevent drawing incorrect conclusions from incomplete or misleading information.