A static analysis rule written from a spec is a hypothesis

Last week, a tool was created that can predict whether certain ComfyUI custom nodes will break when you pull the latest changes from the repository. The tool accurately answered one question: whether every name that a pack imports still exists upstream. However, this was only half the problem. ComfyUI's comfy.* modules are internal, so import success does not guarantee load success.

When a refactor is made, packs may start failing with ImportError errors. To address this, a static analysis rule was built to parse the target module at a specific git reference, extract the real parameter list, and try to bind every call the pack makes. If a call cannot be bound, it indicates a break, as it will raise a TypeError when executed.

The rule was tested on 20 popular ComfyUI node packs, but it turned out to be incorrect in two ways, causing false positives. The first false positive was due to shadowed names, which can be mitigated by using a conservative file-wide shadow set. The second false positive was related to version shims, which the rule incorrectly treated as errors.

The author of the tool adjusted the rule to recognize shims as harmless and report them without failing the build. The tool's main value is in attaching commit, PR, and release boundary information to known issues, rather than simply finding unknown bugs. The author emphasizes the importance of measuring the hit rate of the rule, not just its correctness, and paying attention to who the false positives affect.

If false positives target well-written code, the rule becomes a hindrance rather than a helpful signal. The tool is called comfy-import-guard and can be installed via pip install comfy-import-guard.