1.2 Characters, Unicode, and UTF

After entering the computer's core, you'll first need to decrypt the names, signs, and old logs etched onto the riverbed stones.

The previous post restored the data into bits, bytes, and bases. Now we're adding textual meaning to these bytes and separating characters, code points, encoding units, and glyphs.

A single symbol on the screen doesn't necessarily mean there's only one unit in memory

On the walls of the underground river, names, markers, and old logs appeared. They all looked like "text," but the program's objects weren't limited to just one type:

• A character is an abstract semantic concept, with specific boundaries that depend on context;
• A code point is the number assigned by Unicode to abstract characters or elements, for example, "zhong" is U+4E2D;
• A code unit is the basic unit of a coding format: UTF-8 uses 8 bits, UTF-16 uses 16 bits;
• byte is the unit actually seen during storage and transmission;
• A glyph is the actual shape a font draws;
• A grapheme cluster (the unit of character perception by users) is closer to what the cursor moves through one step, and may consist of multiple code points.

Therefore, "What is the length of a string?" has no single, context-independent answer. It might refer to byte count, code unit count, code point count, or the number of characters as seen by the user.

ASCII solved early English information exchange

ASCII defines 128 characters, requiring only 7 bits, including English letters, digits, punctuation, and control characters. A is decimal 65, or hexadecimal 0x41.

Early systems often stored ASCII in 8-bit units; an extra bit was sometimes used for parity, sometimes fixed to zero, and at other times used by later extended character sets. It's not accurate to say broadly that "the eighth bit of ASCII is the parity bit."

The limitations of ASCII aren't just that it can't represent Chinese characters. After different regions independently extend single-byte encodings, the same byte can represent different characters. If the sender and receiver don't agree on the same encoding, the "garbled text" is simply the result of interpreting the same byte sequence using incorrect rules.

Unicode first assigns a unique number, then selects an encoding format

Unicode code points are written as U+ plus a hexadecimal number. The code point range spans from U+0000 to U+10FFFF; the range from U+D800 to U+DFFF is reserved for UTF-16 surrogate pairs and is not Unicode scalar values.

Code point numbers are not disk formats. UTF-8, UTF-16, and UTF-32 are different schemes for encoding Unicode scalar values as sequences of code units:

Encoding Format	Encoding Unit	Typical Scalar Value Size	Features
UTF-8	8-bit	1–4 bytes	ASCII compatible, commonly used in web and Unix environments
UTF-16	16 bit	1 or 2 code units	The Basic Multilingual Plane usually uses 1, other planes use surrogate pairs
UTF-32	32 bit	1 encoding unit	Directly positioned, but typically has large space overhead

The term "Unicode file" is imprecise. Engineering interfaces should specify exact encoding formats such as UTF-8, UTF-16LE, etc.; byte order for UTF-16/32 must be explicitly defined or indicated using a byte order mark.

Write the "zhong" into the byte on the stone wall

On the stone wall of an underground river, the character "Zhong" appears as a single glyph. When the control panel needs to insert the code point U+4E2D into a file, it must select a specific encoding. Following UTF-8 rules, this code point is split into three bytes.

“The” code point is U+4E2D. It falls between U+0800 and U+FFFF, UTF-8 uses three bytes, the template is:

1110xxxx 10xxxxxx 10xxxxxx

Fill the valid bits of 0x4E2D sequentially to get:

11100100 10111000 10101101
   E4       B8       AD

UTF-8's ASCII range remains single byte with the same value. Multi-byte sequences have a leading byte indicating length, and all subsequent bytes begin with binary 10, enabling decoders to detect boundaries and reject many illegal combinations. Valid UTF-8 also must exclude overlong encodings, surrogate pairs, and values exceeding U+10FFFF.

Have the runtime display each layer

Python 3's str represents Unicode text, bytes represents raw bytes. Encoding converts str into bytes, and decoding performs the reverse transformation.

import unicodedata

text = "AMiddle😀"
encoded = text.encode("utf-8")

print([f"U+{ord(char):04X}" for char in text])
print(encoded.hex(" "))
print(len(text), len(encoded))
print(encoded.decode("utf-8"))

decomposed = "e\u0301"
composed = "é"
print(decomposed == composed)
print(unicodedata.normalize("NFC", decomposed) == composed)

Output:

['U+0041', 'U+4E2D', 'U+1F600']
41 e4 b8 ad f0 9f 98 80
3 8
A mid-😀
False
True

Python's len(text) counts code points but doesn't count user-perceived characters. e plus a combining diacritic and the precomposed character é may appear identical, but their underlying sequences differ; only after NFC normalization do they become equal. More complex emojis, flags, and compound characters may still consist of multiple code points, and accurate segmentation requires a library that implements Unicode grapheme cluster rules.

The char Trap in UTF-16

A Java char is a 16-bit UTF-16 code unit that does not equate to a complete Unicode code point. 😀 is U+1F600, requiring a pair of surrogate items in UTF-16, thus:

String text = "A😀";

System.out.println(text.length());
System.out.println(text.codePointCount(0, text.length()));
System.out.printf("U+%X%n", text.codePointAt(1));

The outputs are 3, 2, and U+1F600. Traversing the full code point range uses text.codePoints(); nonetheless, a code point does not guarantee equivalence to a user-perceived character.

Decode failures should be visible

A byte sequence becomes text only when paired with an encoding rule. When reading files or network messages, clearly specify the encoding at boundaries and establish a strategy for handling invalid input:

payload = b"ok\xff"

# Default strict: Encounter illegal UTF-8 Throw immediately UnicodeDecodeError
# payload.decode("utf-8")

print(payload.decode("utf-8", errors="replace"))  # ok�

replace is suitable for displaying damaged logs as much as possible, at the cost of losing original information; identity identifiers, protocol fields, and signature inputs should typically be handled with strict and illegal data should be rejected. Never silently fallback to platform default encoding, or errors will appear on another machine or in the next deployment.

The Engineering Boundaries of Text Processing

• Truncating UTF-8 by bytes may cut through a multi-byte sequence; when limiting database field or message sizes, specify whether the limit is in bytes or characters.
• Case conversion doesn't always go one-to-one, and it's not always language-dependent; username comparison requires its own defined rules. Normalization alters the code point sequence. First determine protocol or product requirements, then choose forms like NFC or NFD, don’t unconditionally "clean" all text.
• A BOM is not required for all UTF formats. UTF-8 has a fixed byte order, so the BOM is often used to indicate encoding, but some tools may treat it as part of the content.
• Code conversion is typically linear scanning, with a time complexity of O(n); the real risks stem more from error boundaries, illegal inputs, and inconsistent normalization strategies.

Hands-on Check the Text's Actual Structure

1. Compute the UTF-8 byte count and code point count for hello, Chinese, and 😀.
2. Explain why "😀".length() is 2 in Java, whereas len("😀") is typically 1 in Python 3.
3. Construct an illegal UTF-8 sequence, compare strict, replace, and ignore; explain which one silently drops data.
4. Find the NFC and NFD representations of é and compare their code points and UTF-8 byte sequences.
5. Specify a product constraint for an input field that is "at most 20 characters."

Text ultimately has to come back to bytes

Unicode resolved the rules for numbering and encoding, but did not specify how multi-byte values are arranged in memory or protocols. Next, see Byte Order and Bit Masks to learn how to explicitly pack and unpack integers, and how to avoid boundaries in C shift operations.