From Jb2170 · · 2m
UTF-8000: Infinite Code Units Without Breaking UTF-8
UTF-8000 extends UTF-8 to handle arbitrarily large code units while maintaining self-synchronization and self-punctuation.
In brief
UTF-8000 extends UTF-8 to handle arbitrarily large code units while maintaining self-synchronization and self-punctuation. UTF-8000 uses a self-synchronizing prefix and start bits to encode codepoints in variable-length code units without overlong encoding. Originally reported by Jb2170.
Quick answers
- What is UTF-8000?
- UTF-8000 extends UTF-8 to handle arbitrarily large code units while maintaining self-synchronization and self-punctuation.
- What is the key takeaway from UTF-8000: Infinite Code Units Without Breaking UTF-8?
- UTF-8000 uses a self-synchronizing prefix and start bits to encode codepoints in variable-length code units without overlong encoding.
What is UTF-8000?
UTF-8000 extends UTF-8 to handle arbitrarily large code units while maintaining self-synchronization and self-punctuation.
Self-Synchronization Prefix
The highest bits of each byte indicate whether it's a first byte or continuation byte.
Start Bits
A sequence of 1 bits followed by a 0 bit tells the decoder how many bytes a code unit requires.
Content Bits
The bits storing the actual codepoint value, with 5n+1 bits for n-byte code units.
Mandatory Content Bits
The first 0, 4, or 5 bits that must be non-zero to prevent overlong encoding.
Overlong Encoding
Forbidden encodings that could be represented with fewer bytes, like 0xC0 for ASCII.
Why UTF-8000 Works
It inherits UTF-8's self-synchronization and self-punctuation while adding extensibility for larger code units.
The 2-Byte Special Case
2-byte code units have 4 mandatory content bits instead of 5 for longer units.
Self-Synchronization Explained
Bytes can be instantly identified as start or continuation bytes without prior context.
Self-Punctuation Explained
The start bits tell the decoder exactly how many bytes to read for a code unit.
No Endianness Issues
UTF-8000 uses single bytes, so there's no need for endianness handling.
ZigZag Encoding
A signed integer variant using zigzag mapping to encode negative numbers.
Why UTF-8000 Matters
It allows for infinite code unit lengths without breaking UTF-8's core properties.
The Math Behind It
The start bits form a prefix-free tree that grows with code unit length.
Error Recovery
UTF-8000 can recover from errors by yielding replacement characters until the next start byte.
Real-World Use
UTF-8000 is designed to be used for encoding large integers without overlong encoding.
The essence
UTF-8000 uses a self-synchronizing prefix and start bits to encode codepoints in variable-length code units without overlong encoding.
