Urgent.News

What's breaking now, across thousands of outlets.

Tech

S3-compatible is a promise with an asterisk

Amazon's S3 won so completely that its API became a standard. Now half the object-storage providers on the market advertise the same two words: S3-compatible . Point your existing AWS SDK at their endpoint, change a few environment variables, and your code can't tell the difference. That's mostly true, and it's genuinely great — it's why moving between storage providers is a config change instead…

Two S3-compatible storage providers allow your AWS SDK to work with their endpoints, but the similarity ends there. While the API verbs are compatible, behavior at the edges can differ, leading to unexpected issues. Two particular problems caused significant real-time challenges. The first issue is addressing style, which breaks presigned URLs.

Presigned URLs are time-limited links that allow browsers to fetch files directly from the bucket without routing through your server. However, the way these URLs are addressed can vary between providers. When using an S3-compatible provider, presigned URLs are signed for a specific URL shape. If the SDK signs a virtual-hosted URL, but the provider only serves path-style URLs, the request that arrives doesn't match the signed one, and the signature is rejected.

The fix is to force the SDK to speak path style everywhere, ensuring the signature and request agree. The second issue is CORS (Cross-Origin Resource Sharing) that is not enabled by default on S3-compatible buckets. The browser blocks its own requests before the bytes come back, resulting in a CORS error. This issue is less common in other storage providers but can be more challenging because there's less guidance for the exact provider and the defaults are stricter.

To fix this, you need to configure a CORS policy on the bucket itself, specifying the allowed origins, methods, and headers. The key lesson is that "compatible" means the API accepts the same calls, not the behavior. S3-compatible storage offers the portability of changing providers by altering an environment variable, but it does not guarantee the same defaults, addressing, or edge behavior.

The practical advice is that the code swap is indeed a config change, but you should still budget time for the less obvious aspects that "compatible" quietly doesn't cover. Testing the actual browser fetching of a real file is crucial, as everything up to that point may mislead you.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

I put my own "no-upload" tools on a network monitor. Here's what actually left the browser.

"Runs in your browser, nothing uploaded" is the easiest claim in web tooling to make -- and one of the easiest to fake. A tool can pretty-print your JSON locally and quietly POST it to a server for…

  • Tier 1 tools processed data locally without external server communication
  • Tier 2 tools used same-origin proxy techniques to fetch data from external sources
  • Tier 3 tools directly contacted external hosts for public data retrieval

More from Wednesday 16 September →