1.2 Characters, Unicode, and UTF

The previous lesson introduced bits, bytes, and bases. Now we're adding textual meaning to these bytes and separating characters, code points, encoding units, and glyphs.

A single symbol on the screen doesn't necessarily mean there's only one unit in memory

Names, interface labels, and logs all look like "text," but programs must distinguish several different units when processing them:

• A character is an abstract semantic concept, with specific boundaries that depend on context;
• A code point is a numeric position in the Unicode code space; some are assigned to characters and some remain unassigned. “中”, for example, is at U+4E2D;
• A code unit is the basic unit of a encoding form: UTF-8 uses 8 bits, UTF-16 uses 16 bits;
• byte is the unit actually seen during storage and transmission;
• A glyph is the actual shape a font draws;
• A grapheme cluster (the unit of character perception by users) is closer to what the cursor moves through one step, and may consist of multiple code points.

Therefore, "What is the length of a string?" has no single, context-independent answer. It might refer to byte count, code unit count, code point count, or the number of characters as seen by the user.

ASCII solved early English information exchange

ASCII defines 128 characters, requiring only 7 bits, including English letters, digits, punctuation, and control characters. A is decimal 65, or hexadecimal 0x41.

Early systems often stored ASCII in 8-bit units; an extra bit was sometimes used for parity, sometimes fixed to zero, and at other times used by later extended character sets. It's not accurate to say broadly that "the eighth bit of ASCII is the parity bit."

The limitations of ASCII aren't just that it can't represent Chinese characters. After different regions independently extend single-byte encodings, the same byte can represent different characters. If the sender and receiver don't agree on the same encoding, the "garbled text" is simply the result of interpreting the same byte sequence using incorrect rules.

Unicode first assigns a unique number, then selects an enencoding form

Unicode code points are written as U+ plus a hexadecimal number. The code point range spans from U+0000 to U+10FFFF; the range from U+D800 to U+DFFF is reserved for surrogate code points, which are not Unicode scalar values.

Code point numbers are not disk formats. UTF-8, UTF-16, and UTF-32 are different schemes for encoding Unicode scalar values as sequences of code units:

Encoding Format	Encoding Unit	Typical Scalar Value Size	Features
UTF-8	8-bit	1–4 bytes	ASCII compatible, commonly used in web and Unix environments
UTF-16	16 bit	1 or 2 code units	The Basic Multilingual Plane usually uses 1, other planes use surrogate pairs
UTF-32	32 bit	1 encoding unit	Directly positioned, but typically has large space overhead

The term "Unicode file" is imprecise. Engineering interfaces should specify exact enencoding forms such as UTF-8, UTF-16LE, etc.; byte order for UTF-16/32 must be explicitly defined or indicated using a byte order mark.

Encode “中” as Bytes

To a reader, “中” appears as one glyph. To write its code point, U+4E2D, into a file, a program must choose a concrete encoding. Under UTF-8, this code point becomes three bytes.

The code point for “中” is U+4E2D. It falls between U+0800 and U+FFFF, so UTF-8 uses three bytes with this template:

1110xxxx 10xxxxxx 10xxxxxx

Fill the valid bits of 0x4E2D sequentially to get:

11100100 10111000 10101101
   E4       B8       AD

UTF-8's ASCII range remains single byte with the same value. Multi-byte sequences have a leading byte indicating length, and all subsequent bytes begin with binary 10, enabling decoders to detect boundaries and reject many illegal combinations. Valid UTF-8 also must exclude overlong encodings, surrogate code points, and values exceeding U+10FFFF.

Have the runtime display each layer

Python 3's str represents Unicode text, bytes represents raw bytes. Encoding converts str into bytes, and decoding performs the reverse transformation.

import unicodedata

text = "A中😀"
encoded = text.encode("utf-8")

print([f"U+{ord(char):04X}" for char in text])
print(encoded.hex(" "))
print(len(text), len(encoded))
print(encoded.decode("utf-8"))

decomposed = "e\u0301"
composed = "é"
print(decomposed == composed)
print(unicodedata.normalize("NFC", decomposed) == composed)

Output:

['U+0041', 'U+4E2D', 'U+1F600']
41 e4 b8 ad f0 9f 98 80
3 8
A中😀
False
True

Python's len(text) counts code points but doesn't count user-perceived characters. e plus a combining diacritic and the precomposed character é may appear identical, but their underlying sequences differ; only after NFC normalization do they become equal. More complex emojis, flags, and compound characters may still consist of multiple code points, and segmentation commonly follows Unicode extended grapheme-cluster rules, with a specified Unicode data version. Grapheme clusters approximate user-perceived characters; locales and applications may tailor cursor or display behavior.

The char Trap in UTF-16

A Java char is a 16-bit UTF-16 code unit that does not equate to a complete Unicode code point. 😀 is U+1F600, requiring a pair of surrogate code units in UTF-16, so save this complete program as TextUnits.java, compile with javac -encoding UTF-8 TextUnits.java, and run java TextUnits:

public class TextUnits {
    public static void main(String[] args) {
        String text = "A😀";
        System.out.println(text.length());
        System.out.println(text.codePointCount(0, text.length()));
        System.out.printf("U+%X%n", text.codePointAt(1));
    }
}

The outputs are 3, 2, and U+1F600. To iterate over the string’s code points, use text.codePoints(); nonetheless, a code point does not guarantee equivalence to a user-perceived character.

Decode failures should be visible

A byte sequence becomes text only when paired with an encoding rule. When reading files or network messages, clearly specify the encoding at boundaries and establish a strategy for handling invalid input:

payload = b"ok\xff"

try:
    payload.decode("utf-8")
except UnicodeDecodeError:
    print("invalid UTF-8")

print(payload.decode("utf-8", errors="replace"))  # ok�

The first line is invalid UTF-8; the second is ok�. Python str and Java String can also contain lone surrogates after some operations. Having a string object does not guarantee valid UTF-8 encodability, so boundary validation still matters.

replace is suitable for displaying damaged logs as much as possible, at the cost of losing original information; identity identifiers, protocol fields, and signature inputs should typically be handled with strict and illegal data should be rejected. Never silently fallback to platform default encoding, or errors will appear on another machine or in the next deployment.

The Engineering Boundaries of Text Processing

• Truncating UTF-8 by bytes may cut through a multi-byte sequence; when limiting database field or message sizes, specify whether the limit is in bytes or characters.
• Case conversion is not always one-to-one and can depend on locale; username comparison needs separately defined rules.
• Normalization alters the code point sequence. First determine protocol or product requirements, then choose forms such as NFC or NFD; do not unconditionally “clean” all text.
• A BOM is not required for all UTF formats. UTF-8 code units are single bytes, so there is no separate byte-order choice, so the BOM is often used to indicate encoding, but some tools may treat it as part of the content.
• Encoding conversion is typically linear scanning, with a time complexity of O(n); the real risks stem more from error boundaries, illegal inputs, and inconsistent normalization strategies.

Hands-on Check the Text's Actual Structure

1. Compute the UTF-8 byte count and code point count for hello, 中文, and 😀.
2. Explain why "😀".length() is 2 in Java, whereas len("😀") is typically 1 in Python 3.
3. Construct an illegal UTF-8 sequence, compare strict, replace, and ignore; explain which one silently drops data.
4. Find the NFC and NFD representations of é and compare their code points and UTF-8 byte sequences.
5. Specify a product constraint for an input field that is "at most 20 characters."

Text ultimately has to come back to bytes

Unicode defines text encodings and their serialization schemes. Multi-byte integer fields in application protocols still need their own byte-order conventions. Next, see Byte Order and Bit Masks to learn how to explicitly pack and unpack integers, and how to respect the limits of C shift operations.

References

• Unicode: conformance and encoding forms
• Unicode UAX #29: text segmentation