sdlc.dev

Computer science fundamentals for working engineers path · How data is stored · Topic 5 of 13

Text: Unicode, UTF-8 and graphemes

Not started

Text in a computer is a sequence of numbers. Unicode decides which number each character gets (its code point); an encoding such as UTF-8 decides which bytes represent those numbers in memory, in a file or on the wire. Most text bugs come from losing track of which layer you are at: bytes, code units, code points, or the characters a person actually sees.

Why it matters: Mojibake, truncated emoji, broken string comparisons and wrong length limits all come from mixing up code points, code units, bytes and what a user sees as one character.

  1. 1Unicode assigns numbers

    Every character gets a code point, written U+0041 for A. There is room for 1,114,112 of them, and well over 150,000 are assigned, with more in each yearly release.

  2. 2UTF-8 turns them into bytes

    1 byte for ASCII, 2 to 4 bytes for everything else. It is the default encoding of the web, JSON and most files.

  3. 3JavaScript strings are UTF-16

    length counts 16-bit code units, so characters outside the Basic Multilingual Plane, like most emoji, count as 2.

  4. 4Users see graphemes

    One visible character can be several code points: a base letter plus accents, or emoji joined with zero-width joiners.

1

How it works

The explanation, with examples you can run.

Code points and UTF-8

Unicode code points range from U+0000 to U+10FFFF. The first 128 are ASCII. UTF-8, defined in RFC 3629, encodes each code point in 1 to 4 bytes:

Code pointsBytesExample
U+0000 to U+007F1A = 41
U+0080 to U+07FF2é = c3 a9
U+0800 to U+FFFF3€ = e2 82 ac
U+10000 to U+10FFFF4😀 = f0 9f 98 80

ASCII text is valid UTF-8 unchanged, no byte of a multi-byte sequence looks like an ASCII character (so searching for / or " in UTF-8 bytes is safe), and you can find the start of a character from anywhere in a stream. That is why UTF-8 won: it is the required encoding for JSON exchanged between systems (RFC 8259) and the dominant one on the web.

⌘/Ctrl + Enter
OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

Node printed the byte sequences in the table, and length: 2 for 😀. Then 8 5 1 for the family emoji, false true, and é.

Three kinds of length

JavaScript, Java and C# strings are sequences of UTF-16 code units. Code points above U+FFFF need two units (a surrogate pair), so "😀".length is 2, and slicing a string by index can cut an emoji in half. Spreading a string ([...s]) iterates by code point. What a reader perceives as one character is a grapheme cluster, defined by Unicode Annex #29: the family emoji is three person emoji joined by two zero-width joiners, so 8 code units, 5 code points, and 1 grapheme. Use Intl.Segmenter (or your platform's equivalent) when limiting input to "20 characters" or moving a text cursor, and byte length when a database column or protocol limit is in bytes.

Normalisation

"é" can be one code point (U+00E9) or two: "e" followed by a combining acute accent (U+0301). They look identical but compare unequal. Unicode normalisation (Annex #15) defines canonical forms; NFC, the composed form, is the usual choice for storage. Normalise user input before comparing or using it as a key, for example usernames and file names. macOS file systems have historically stored names in a decomposed form, a common source of "file not found" bugs when syncing with other systems.

Mojibake

When UTF-8 bytes are decoded with the wrong encoding, each byte of a multi-byte character becomes a separate character: c3 a9 read as Latin-1 is "é". The cure is to be explicit at every boundary: save files as UTF-8, send Content-Type: text/html; charset=utf-8, configure database connections and columns for UTF-8 (in MySQL that means utf8mb4; the older utf8 stops at 3 bytes and can't store emoji), and never guess an encoding from bytes when you can be told.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.How many bytes does € (U+20AC) take in UTF-8?
  2. 2.A page shows é where it should show é. What most likely happened?
  3. 3.For the family emoji 👨‍👩‍👧, the example printed 8, 5 and 1. What do those count?

0 of 3 right so far. Get all three to finish this topic.

Go deeper