Our scraper stored 20 links per page and kept every page in memory
We build and maintain web scrapers that run for hours at a time. In October one of them ran out of memory, and the cause was a detail of lxml that we had skimmed past in its docs. The crawler walked the listing pages of a large online store, about 4 MB of HTML each. On every page it ran a few xpath() queries, pushed the item links into a queue, and dropped the parsed tree. On a store with about…
In October, one of the web scrapers built by the company ran out of memory. The issue stemmed from an overlooked detail in the lxml documentation. The crawler navigated the listing pages of a large online store, parsing about 4 MB of HTML per page. After querying with a few xpath() functions, the item links were added to a queue, and then the parsed tree was discarded.
For a store containing approximately 46,000 listings, the memory usage increased significantly - from around 100 MB to 3,010 MB. When the memory limit was set to 1 GB, the runs were terminated. The root cause of the issue was the links themselves. xpath() returns smart strings that retain a reference to the element they originated from.
Therefore, every queued link kept the entire page tree alive, approximately 12.6 MB per page. To rectify this, the company changed the default behavior of xpath() to return ordinary strings instead of smart strings. This modification allowed the same crawl to be performed with a memory footprint of about 100 MB rather than the previous 3,010 MB.
By changing every xpath() call where results are stored past the current page to pass smart_strings=False, the company avoided the memory leak problem.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
