Utf8ledger

Worked examples

Small strings, visible differences

Each example below measures a received text string as UTF-8 payload only. Budgets are deliberately chosen for illustration. They are not limits fetched from an external service, and fitting one does not guarantee a file, database field or HTTP request has the same total size.

The fixed mixed example

A🐝é encodes as 41 F0 9F 90 9D C3 A9. That is seven bytes, three scalar values and four UTF-16 units. With budget 8, it fits and has one byte remaining. The offsets of its scalars are 0, 1 and 5 bytes; the bee occupies the middle four-byte sequence.

ASCII and a precomposed accent

table42 has seven bytes, seven UTF-16 units and seven scalar values. A budget of 8 leaves one byte. café has five UTF-8 bytes but four units and four scalars, because é occupies C3 A9. A five-byte budget fits exactly.

A combining sequence is different text

The sequence e followed by U+0301 COMBINING ACUTE ACCENT encodes as 65 CC 81. It uses three bytes and two scalars, even when it looks similar to the precomposed é. Against budget 2 it exceeds by one byte. The tool does not silently normalize either sequence.

One supplementary scalar

The bee 🐝 is U+1F41D. It has one scalar, two UTF-16 units and four UTF-8 bytes, F0 9F 90 9D. Budget 4 fits exactly. Calling its JavaScript length a visible-character count would confuse these different units.

A joined emoji sequence

The woman, zero-width joiner and laptop sequence has three scalars, five UTF-16 units and eleven bytes. A ten-byte budget is exceeded by one. This does not assert that a user sees three characters; the calculator makes no grapheme or rendering claim. The joiner is shown with its code-point label in the preview.

Received line endings

A direct received string a\r\nb has four scalars, four units and four bytes. a\nb has three of each, leaving one byte in a budget of 4. Browser textarea normalization can change what reaches the script. Check the actual input value rather than assuming original clipboard or file CRLF bytes were retained.

An entered BOM and an invisible byte

U+FEFF followed by A encodes as EF BB BF 41, four bytes. A three-byte budget is exceeded by one; the BOM is not removed. U+0000 followed by A has two bytes, 00 41, even if the first scalar is invisible. Its preview uses a code-point label instead of relying on a glyph.

Empty text is a valid boundary

Empty text has zero of all three counts. With budget 0 it fits exactly and both remaining and excess are zero. A malformed surrogate is different: it rejects and clears all old outputs. Reset restores the public mixed example and its budget.