Why I Chose P2P Instead of S3 for My File Transfer Product

In an effort to build a file transfer product, many might initially turn to Amazon S3 as the go-to solution. Renowned for its reliability, scalability, and straightforward implementation, S3 offers an easy-to-understand design for user file selections, uploads, downloads, and URLs. However, the focus for the ButterShare product was not to serve as a storage solution, but rather to facilitate the direct transfer of files between devices.

This distinction led to a shift in architecture, moving away from S3 as the core component and exploring direct peer-to-peer (P2P) transfer using WebRTC technology. This change transformed numerous aspects of the product, including bandwidth economics, infrastructure complexity, user experience, privacy, failure modes, and even the fundamental nature of the product.

The decision to opt for P2P over S3 was not based on any inherent issues with S3, but rather on the realization that S3 addressed a different problem than the one ButterShare aimed to solve. Storage and transfer are distinct processes. For instance, if Alice has a 100GB video project that Bob needs, a storage-first approach would have Alice upload the file to cloud storage and Bob later download it.

However, a transfer-first approach would involve Alice and Bob both being online, with Alice's goal solely being to get those bytes from her machine to Bob's. This approach eliminates the need for a permanent intermediate copy, a key consideration when dealing with large files. As file sizes increase, the architecture's impact becomes more pronounced.

Imagine transferring a 100GB file using an S3-style workflow: the file moves through two major legs - from the sender to S3, and then from S3 to the recipient. This not only complicates the process but also adds significant storage and bandwidth requirements. In a scenario where users are transferring 10GB, 50GB, 100GB, or even 200GB files, the infrastructure cost can quickly become substantial.

This relationship between more users and increased storage and bandwidth costs becomes even more direct with P2P. In a P2P architecture, the data plane is moved away from servers, with the backend primarily handling coordination tasks. For instance, the backend would create transfer sessions, exchange signaling data, manage room states, and coordinate peers, while the actual file transfer between users occurs directly when network conditions allow.

This shift in the control plane (small, easy-to-handle messages) and data plane (the expensive file transfers) offers a more efficient solution for large file transfers.