The Registry That Wasn't Enough: How We Went from "Knowing APIs" to Actually Verifying Against PyPI

The Registry That Wasn't Enough: How We Went from Knowing APIs to Actually Verifying Against PyPI

In the previous article, I explained how a seemingly non-existent `requests.get_json()` function passed all tests and remained undetected. The tool I developed at the time relied on a manually curated registry of valid APIs: a mapping of libraries to their known functions. If a library was in the registry but a function wasn't, it was flagged as a hallucination.

While this approach worked for common libraries like `requests`, `json`, and `os`, it had a significant limitation. It only detected hallucinations within its known boundaries and failed to verify against real documentation, resulting in false positives for packages with nonexistent functions.

Recent research revealed that library hallucinations were commonplace, with up to 21.7% of open-source model packages containing non-existent APIs. Static analysis alone had a detection range of 48.5% to 77%, leaving room for improvement. Verification against real documentation was identified as a crucial factor in reducing false positives.

The idea behind this project was to replace the handcrafted registry with a more dynamic approach. PyPI not only serves packages but also provides wheels, which are ZIP files containing the code and any necessary type stubs (.pyi) when applicable. By downloading and inspecting these wheels, we could verify the existence of specific attributes or functions without installing the package or executing user code.

However, performing verification for every import on every analysis would be computationally expensive and prone to network issues. To address this, we implemented a two-level cache system. The first level, memory-based, stored information about package existence with a 7-day time-to-live (TTL). The second level, disk-based, stored reputation metadata with a 30-day TTL.

Network errors had a 10-minute TTL to prevent excessive retries in case of temporary failures. To avoid false positives due to unverified cases, we adopted a "fail-open" policy. If verification couldn't be completed, the package would be reported as existing rather than non-existent.

With the cache in place, we fine-tuned the implementation to handle various intricacies. PyPI normalizes package names, and imports can bring submodules into play. To ensure accurate results, we implemented canonicalization, aligning package names with PyPI's naming conventions. A notable example was the standard library module `imp`, which was mistakenly flagged as typosquatting of `pip`. This issue arose due to the lack of synchronization between stdlib lists and PyPI package lists.

The most significant advancement in this project was the introduction of a "deep" mode. When enabled, the deep mode verified the existence of attributes by downloading the package wheel, extracting its contents, and searching for the desired attribute within the package's .pyi and .py files. This approach eliminated false positives like `pandas.DataFrame`, which had previously passed the heuristic-based verification.

By detecting hypothetical invented functions and confirming the absence of non-existent attributes, we achieved a more accurate verification process without the need for installing Python or executing user code.

To prevent excessive resource consumption, we introduced IP-based control over the deep mode. Each IP address was allowed three deep analyses per day, with a daily reset. Localhost was exempt from the limit to facilitate local development, while development environments could simulate IP limits using environment variables. This control ensured that the detector remained efficient while still providing accurate verification results for production environments.