UTF-8000: Unlimited UTF-8

UTF-8000, also known as "Unlimited UTF-8," is a character encoding standard that builds upon the existing UTF-8 format to allow for code units of virtually unlimited length. This extension maintains many of the same properties and benefits of the original UTF-8, while also enabling the encoding of larger code units.

The key difference between UTF-8000 and UTF-8 lies in how they handle the start and continuation bytes of multi-byte code units. In UTF-8, the first byte of a multi-byte code unit contains both start bits that indicate the length of the code unit and content bits that represent the actual code point value. However, as code unit lengths increase, the number of content bits becomes a fixed proportion of the total bits, approaching 62.5% as the length of the code unit grows.

UTF-8000 addresses this issue by using a different, more efficient encoding scheme for the start bits. By reorganizing the self-synchronization prefixes, start bits, and content bits, UTF-8000 ensures that the encoding remains self-synchronizing and easy to parse, while also allowing for code units of arbitrary length.

One of the primary advantages of UTF-8000 is that it inherits and maintains many of the familiar properties of UTF-8, such as its simplicity, backward compatibility with ASCII, and ease of use. The encoding scheme remains based on the same byte map as UTF-8, with continuation bytes occupying the same regions and ASCII characters remaining unchanged.

Additionally, UTF-8000 preserves the self-synchronization properties that make UTF-8 resistant to corruption and easy to decode. These properties are maintained through the ordered arrangement of start bits and the monotonically-increasing nature of the codeword prefixes. This ensures that the encoding remains robust and efficient, even when encoding extremely large code units.

In summary, UTF-8000 represents an evolution of the UTF-8 standard, allowing for the encoding of code units of virtually unlimited length while preserving the key benefits and properties that have made UTF-8 so popular and widely adopted. By addressing the limitations of UTF-8 in handling longer code units, UTF-8000 provides a more efficient and practical solution for encoding and processing a broader range of character sets and languages.