Building a Streaming Pipeline: From a System Design Perspective
Say you're on vacation and you've just recorded a fairly large video of yourself skydiving for the first time, naturally, you'd want to send it to a friend or family member on the other side of the world. After the upload finishes and the video eventually reaches them, they press Play and marvel in the moment, re-living the experience with you. That sounds like a trivial thing to do, but…
The journey of a video file from being recorded to being played on a viewer's device is a complex process that occurs behind the scenes. After recording the video, it needs to be uploaded to a server, processed into different formats, and then delivered to the viewer's device. This entire journey happens seamlessly without the user or the sender having to think about it.
For the creator uploading the video, the system should allow them to easily upload the video, see the upload progress, resume the upload if interrupted, and know when the video is ready for playback. On the other hand, the viewer should be able to open the video, start playback quickly, watch it across different network conditions, and seek through it without caring about the servers or locations involved.
However, designing a system that meets these functional requirements while also handling non-functional requirements like performance under heavy load and failure scenarios requires careful planning. The simplest approach to uploading a video might be sending the entire file in one request, but this can quickly become problematic for large files like a skydiving video that could be several gigabytes in size.
If the connection drops during the upload or the upload fails partway through, the user could be left frustrated with a message like "Sorry, upload failed, please try again."
To improve upon this, the system can break the video into smaller chunks and upload each chunk separately. This approach, known as resumable uploads, allows the creator to upload the video in smaller, more manageable pieces. If the connection drops at any point during the upload, the creator can simply resume from the last successful chunk instead of starting the entire upload over again. This approach not only improves the user experience but also reduces the load on the backend server.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
