Packing Binary Is Fun, Actually

A Twitter user challenged the notion that saving data in JSON was a task reserved for only the most skilled developers. Intrigued by the opportunity, the author decided to create their own binary format that could compress JSON payloads by up to 80%. They developed an entire binary schema language that could shrink JSON payloads by 80%.

To save space, the author realized that ASCII characters could be represented using just 1 byte instead of the full 3 bytes required for each character in ASCII. To indicate the difference between an integer and a string, the author added a header byte to the binary format. This allowed the decoder to distinguish between data types.

The author also addressed the issue of representing strings of varying lengths by using another byte to represent the length of the string. They then demonstrated how both strings and integers could be packed together in the binary format. The author pointed out that the length field could be removed for integers, as it was not necessary.

However, the author recognized that the inclusion of a length field in the header could cause issues if it exceeded 127. In such cases, the continuation bit would be set to 1, indicating that another header byte follows. This process continued until the header was complete.

The author further explored the concept of using a variable-length integer (varint) encoding for integers, which allowed for the representation of larger numbers using fewer bytes. They also noted that floats were represented using fixed-width formats, such as f32 or f64.

To ensure that the binary format could handle structured data without the need for redundant headers, the author proposed the use of a schema language that could encode the types of data within the binary format itself. This eliminated the need to store the type alongside every value.

In the end, the author emphasized how their binary format could efficiently represent both strings and integers, as well as other data types. They concluded that the use of a schema language simplified the encoding process, allowing developers to focus on more important aspects of their work.