JAVA / STRINGS AND TEXT HANDLING
Unicode, emojis and code points beyond char
Count, index and iterate Java text by code point so emoji and other characters above U+FFFF survive instead of being cut in half.
What you will learn
- Read length() as a count of UTF-16 code units, not of characters
- Iterate text with codePoints() or offsetByCodePoints instead of charAt
- Convert between an int code point and chars with toChars and toCodePoint
- Spot a broken cut by testing Character.isHighSurrogate at the boundary
Understanding Unicode, emojis and code points beyond char
When Java was designed Unicode fit in 16 bits, so char was fixed at 16 bits and that decision can never be revisited. Unicode then grew to 1,114,112 possible code points, U+0000 through U+10FFFF, and UTF-16 was invented to cope: a code point above U+FFFF is stored as two chars taken from the ranges D800-DBFF and DC00-DFFF, which the standard permanently reserved so they can never collide with real characters. A String is an array of those 16-bit code units, which is why length() returns 2 for a single grinning face and why no char variable can hold one.
Three different things get called a character, and keeping them apart is the whole skill here. A code unit is one char, the unit that length() and charAt work in; a code point is one int-sized entry in the Unicode standard, the unit that codePointAt and codePoints() work in; a grapheme cluster is one symbol a reader sees, and it can be several code points held together by joiners or modifiers. Every index you hand to a String method is a code unit index, including the codePoint methods, so codePointAt(1) on a string that starts with an emoji hands you the second half of the pair rather than the next symbol.
The practical rule is to measure and iterate in code points and to distrust any single char you pull out of text. A charAt loop that reverses, truncates or filters can separate a high surrogate from its low surrogate, and that orphan is not a character at all: it renders as a box, compares equal to nothing meaningful, and cannot be encoded as valid UTF-8 when the text reaches a file or socket. StringBuilder.reverse and toUpperCase were retrofitted to keep pairs together, but a loop you write yourself has no such protection.
public class CodePointBasics {
public static void main(String[] args) {
// U+1F600 GRINNING FACE, written as its two UTF-16 code units
String s = "hi \uD83D\uDE00!";
System.out.println("length() = " + s.length());
System.out.println("codePointCount = " + s.codePointCount(0, s.length()));
System.out.println("charAt(3) = " + (int) s.charAt(3));
System.out.println("charAt(4) = " + (int) s.charAt(4));
System.out.println("codePointAt(3) = " + s.codePointAt(3));
System.out.println("high surrogate = " + Character.isHighSurrogate(s.charAt(3)));
s.codePoints().forEach(cp ->
System.out.printf("U+%04X -> %d code unit(s)%n", cp, Character.charCount(cp)));
}
}A Java char is a single 16-bit UTF-16 code unit, so any code point above U+FFFF lives in two chars and can only be handled correctly as an int.
Worked examples
Reversing text without breaking a pair
Shows how a hand-written charAt loop swaps the two halves of a surrogate pair while StringBuilder.reverse does not.
public class ReverseWithEmoji {
public static void main(String[] args) {
String s = "a\uD83D\uDE00b";
StringBuilder manual = new StringBuilder();
for (int i = s.length() - 1; i >= 0; i--) {
manual.append(s.charAt(i));
}
show("manual ", manual.toString());
show("reverse", new StringBuilder(s).reverse().toString());
}
static void show(String label, String text) {
StringBuilder out = new StringBuilder(label);
text.codePoints().forEach(cp -> out.append(String.format(" U+%04X", cp)));
System.out.println(out);
}
}Example explained
Line 1The manual loop walks code units, so it emits the low surrogate DE00 before the high surrogate D83D.
Line 2codePoints() cannot pair DE00 followed by D83D, so it reports two lone surrogates as two separate code points.
Line 3StringBuilder.reverse is specified to treat a valid pair as one character and never reverse its two halves, so U+1F600 survives.
Line 4Neither call throws: the damage is silent until the text is rendered or encoded to bytes.
From code point to chars and back
Builds a string from int code points and converts a supplementary code point to its pair and back without manual bit arithmetic.
public class BuildFromCodePoints {
public static void main(String[] args) {
int cake = 0x1F370;
char[] units = Character.toChars(cake);
System.out.println("units = " + units.length);
System.out.println("high, low = " + Character.isHighSurrogate(units[0])
+ ", " + Character.isLowSurrogate(units[1]));
System.out.println("back together = " + Character.toCodePoint(units[0], units[1]));
String s = new String(new int[] { 'e', 'a', 't', cake }, 0, 4);
System.out.println("length() = " + s.length());
System.out.println("codePointCount = " + s.codePointCount(0, s.length()));
System.out.println("indexOf pair = " + s.indexOf(new String(units)));
}
}Example explained
Line 1Character.toChars turns one code point into the one or two chars needed to store it, so units.length is the same answer as Character.charCount.
Line 2The String(int[], offset, count) constructor takes code points directly, letting ASCII letters and U+1F370 sit in the same array.
Line 3Character.toCodePoint is the exact inverse of toChars and is the only safe way to read a pair back as one number.
Line 4indexOf returns 3 because it reports a code unit index: three one-unit letters come before the emoji, which then occupies indices 3 and 4.
One glyph, five code points
Demonstrates that even counting code points does not match what a reader sees, and that substring can stop inside a pair.
public class OneGlyphManyPoints {
public static void main(String[] args) {
// MAN + ZWJ + WOMAN + ZWJ + GIRL, drawn as a single family glyph
String family = "\uD83D\uDC68\u200D\uD83D\uDC69\u200D\uD83D\uDC67";
System.out.println("length() = " + family.length());
System.out.println("codePointCount = " + family.codePointCount(0, family.length()));
family.codePoints().forEach(cp -> System.out.printf(" U+%04X%n", cp));
String cut = family.substring(0, 4);
System.out.println("cut length = " + cut.length());
System.out.println("cut ends split = "
+ Character.isHighSurrogate(cut.charAt(cut.length() - 1)));
}
}Example explained
Line 1Eight code units and five code points render as one glyph, so neither number is the count a user would give.
Line 2U+200D is the zero width joiner: it has no shape of its own and only tells the renderer to fuse its neighbours.
Line 3substring(0, 4) stops in the middle of the third pair, so the last char of cut is a high surrogate with no partner.
Line 4Cutting at index 2 instead would be valid UTF-16 but still wrong here, because it splits the family into separate people.
Important notes
codePointCount, codePointAt and offsetByCodePoints all take code unit indices; only their results are expressed in code points.
Code points are not the last word either: regional indicator flags, skin tone modifiers and joiner sequences build one glyph from several code points, so trimming or counting what the user sees needs java.text.BreakIterator or an ICU-based library.
Common mistakes
Writing char c = '\uD83D\uDE00'; the compiler rejects it because that literal is two code units, and the usual fix of keeping only '\uD83D' stores a lone surrogate that prints as a box.
Enforcing a limit with length(): five emoji count as ten, so a ten-character nickname rule rejects them while ten letters pass, and a ten-unit column can store half a pair.
Assuming codePointAt takes a code point number, so codePointAt(1) is expected to give the second symbol; on text starting with an emoji it returns 56832, the low surrogate, and that value flows on as if it were a character.
Try it yourself
Change, predict, then run
Write truncate(String s, int maxCodePoints) that returns the first maxCodePoints code points of s, using s.offsetByCodePoints(0, maxCodePoints) to find the cut index. Call it with "a\uD83D\uDE00b" and 2, and print the result's length(): it must be 3, not 2.
Open the Java workspaceCheck your understanding
For String s = "a\uD83D\uDE00b" (the letter a, one emoji, the letter b), what does s.codePointAt(2) return?
- 128512, the emoji, because codePointAt scans back to the start of the pair
- 56832, the low surrogate, because index 2 is the second half of the pair and codePointAt merges a pair only when the index points at the high half
- 98, the letter b, because a surrogate pair occupies a single index
- It throws StringIndexOutOfBoundsException, because index 2 is not a character boundary
Show answer
Indices are always code unit positions, and codePointAt combines two chars only when the char at the index is a high surrogate followed by a low surrogate; otherwise it returns that char's own value, here 0xDE00 = 56832. Option 0 is tempting because some libraries do look backwards, but Java's codePointAt never does; codePointBefore and offsetByCodePoints are the methods that move between boundaries. Nothing is thrown, since index 2 is a perfectly legal code unit index.