Detect a website's tech stack in bulk with Python (a Wappalyzer-style lookup)

This article demonstrates how to detect a website's technology stack in bulk using Python, a technique similar to Wappalyzer. The author, an 18-year-old engineering student, created a tool called Website Tech Stack Detector which can be accessed as a paid tool on Apify with a free trial credit. The tool performs HTTP requests, DNS lookups, and TLS handshakes to gather information about a website's technology stack. It then matches the gathered data against a rule set containing over 7,600 technology fingerprints.

The process involves making one HTTP request with browser-like headers, plus DNS lookups for various record types, and a single TLS handshake for each site. The browser-like headers, cookies, meta tags, script URLs, inline scripts, and HTML are then analyzed using cheerio, a library that parses and manipulates HTML documents like a browser would, but without the overhead of a full browser application. This analysis takes approximately 0.3 seconds per page.

A key feature of the tool is that every detected technology is accompanied by a confidence score and the evidence that matched. This allows users to verify the results themselves rather than relying solely on the black-box algorithm. The code examples provided in the article show how to use the Website Tech Stack Detector in both Python and Node.js environments.

The output from the tool is a structured JSON object containing various pieces of information about each website, such as the domain name, HTTP status code, whether the site was blocked, the count of detected technologies, content management system (CMS) information, e-commerce platform, analytics tools, tag managers, advertising partners, content delivery network (CDN), programming language server type, email provider, and the technologies themselves.

Each technology entry includes a confidence score and the specific evidence that led to the detection.