Pin your own domain to 127.0.0.1 and your SSRF guard will refuse your own files

Two changes, made months apart by different individuals, can inadvertently break downloads your application fetches from its own storage. They are an SSRF guard and a line in /etc/hosts. Neither is inherently flawed. When combined, they transform your own files into unsafe URLs. Consider a service that retrieves URLs. Some URLs originate from users; others come from within your system.

In such cases, the fetcher incorporates an SSRF guard. Before establishing a connection, the guard resolves the hostname and disallows private or loopback addresses. This serves as the standard defense against malicious users attempting to misuse the system. The second change involves the server hardening. The application calls its own API and storage services multiple times per minute.

To optimize performance, the developers pin the service names to the local machine using 127.0.0.1 storage.example.com app.example.com. This modification accelerates internal communication. However, this action causes a problem. With the SSRF guard in place, the loopback address 127.0.0.1 triggers a rejection, rendering all files stored by your own gateway inaccessible to your own pipeline.

Although the system remains operational, the issue manifests downstream. Users report problems such as voice messages being ignored, thumbnails failing to render, and documents not being parsed. These symptoms lead you to investigate the transcription code or the model, rather than the SSRF guard. The guard's logic is correct. It identifies loopback addresses as potential DNS rebinding patterns.

However, it lacks awareness of your specific setup. The resolution of a hostname is not intrinsically linked to the hostname itself. It varies based on factors like adding an /etc/hosts entry, running in Docker, or utilizing split-horizon DNS in Kubernetes. Therefore, it's crucial to take appropriate actions to resolve this issue.

First, remove the entry from /etc/hosts. This is the simplest solution, but it reintroduces a DNS dependency, which may be added again in the future. Another option is to implement an allowlist using hostname suffixes, such as anything ending in .example.com. However, this approach has limitations, as subdomains and wildcard records can bypass the protection.

A more robust solution is to bypass the IP check specifically for storage URLs and teach the guard to recognize your public hostnames. This can be achieved by utilizing environment variables that store the base URL and storage URL, ensuring a single source of truth and eliminating the need for a duplicate list. Implement a function to check if the hostname belongs to your public hosts.

During the guard's validation process, after resolving the hostname, compare it against the list of your public hosts. If the hostname matches, the connection is considered safe and pinned to the corresponding IP address. Finally, it's crucial to test the implementation thoroughly. Use mocking techniques to simulate different hostname resolutions and ensure that your guard behaves as expected.

Keep track of refused URLs per hostname to promptly identify any silent outages and set up alerts to notify you of any issues.