What a server knows about you before it reads a single header

When I sent an HTTPS request to the 500 most popular websites from eight countries, a fifth of them refused the request. This was not due to geographic location or rate limits, but rather something inherent in the request itself. My Python script, using the urllib library with a Chrome user agent, was being blocked by these sites.

However, the user agent was not the most telling aspect of a request. In fact, before any HTTP headers are sent, a server can identify the client using the TLS ClientHello message. This message details the client's TLS library capabilities, ordered by preference. Each client library presents this information differently, creating a unique fingerprint.

For example, Chrome sends fifteen cipher suites, while Python sends thirty. Both start with the same three TLS 1.3 suites, but the order differs. The JA3 and JA4 fingerprints are used to identify clients based on their TLS handshake characteristics. Chrome inserts randomly chosen values into its cipher and extension lists, a technique called GREASE, to prevent middleboxes and servers from relying on the exact handshake shape.

This fingerprinting method is more stable and informative than the user agent, which can be easily edited. When I compared my Python client's handshake to Chrome's, it was clear that the Python client offered more cipher suites, announced it could only manage HTTP/1.1, and had a different extension set. These details, combined with the JA3 or JA4 fingerprint, allowed servers to identify the client even before reading a single byte of HTTP data.