Why JavaScript Text-to-Binary Snippets Break on Emoji

JavaScript text-to-binary snippets often fail when converting emoji, as the function may return incorrect bytes. The issue stems from how JavaScript strings are handled, where strings are sequences of UTF-16 code units. The emoji's Unicode code point is represented by a surrogate pair: 0xD83D and 0xDE42. When using split( ), this surrogate pair is divided, leading to inaccuracies.

The function naiveBinary splits the text into characters, converts each to its charCodeAt value, converts that to a binary string padded to eight bits, and then joins the bytes. However, this method doesn't accurately represent the UTF-8 encoding. For example, the emoji 🙂 produces two 16-bit groups, neither of which is a valid UTF-8 byte.

To correctly convert text to binary, start by encoding the text using TextEncoder.encode(), which returns a Uint8Array containing UTF-8 bytes. Then, format each byte by converting it to a binary string padded to eight bits. This ensures that each value is a valid byte between 0 and 255. Displaying spaces between bytes aids in visualizing byte boundaries but does not affect the encoded text.

Converting binary back to text requires validation. First, split the binary string into groups of exactly eight bits. Each group must be a valid binary number. Then, convert each group from binary to a decimal number and create a Uint8Array from these numbers. Finally, decode the bytes using TextDecoder.decode() with the 'utf-8' encoding. This process ensures that both the encoding and decoding handle the text correctly, rejecting any malformed input.