Combining Character

A Unicode character that is displayed by combining with the preceding base character. Diacritical marks and dakuten are examples.

A combining character is a Unicode character that is not displayed on its own but combines with the preceding base character to be displayed as a single character. Combining diacritical marks (U+0300 to U+036F) are the typical example, and they include accent marks, umlauts, cedillas, and so on.

For example, following the Latin letter "a" (U+0061) with the combining acute accent (U+0301) displays it as "á". In Japanese, the combining dakuten (voiced sound mark, U+3099) and the combining handakuten (semi-voiced sound mark, U+309A) fall into this category, and "か" + U+3099 can represent "が". There are also many languages that use combining characters heavily, such as Thai and Arabic. The Combining Diacritical Marks block contains 112 characters from U+0300 to U+036F, but the characters classified as combining marks exceed 2,000 across Unicode as a whole.

Because combining characters exist, characters that look the same can have different code point sequences. "á" can be represented both as the precomposed character U+00E1 (NFC form) and as the base character U+0061 + the combining character U+0301 (NFD form). They need to be handled uniformly using Unicode normalization (composition with NFC, decomposition with NFD), and if you do not normalize before comparing strings, strings that look the same are judged as a "mismatch." However, NFC does not always combine them into a single character. A string of "か" followed by U+3099 is composed into U+304C (が) by NFC, but a string of the half-width katakana "カ" followed by the half-width dakuten U+FF9E remains 2 characters under NFC, and it becomes the full-width "ガ" (U+30AC) only when you use NFKC, which also converts compatibility characters. Which form to choose depends on whether you want to treat half-width and full-width as the same thing. A common pitfall in practice is file names. HFS+, the older macOS file system, normalized combining characters in file names into their decomposed form, so even when they looked the same, their code point sequences did not match the composed file names created on Windows or Linux (the current APFS stores the form it is given as is and treats either form as the same file). Git has a setting called core.precomposeUnicode that absorbs this difference, and if you do not enable it in an environment where decomposed names are mixed in, the same file may appear twice under different names.

Multiple combining characters can also be stacked. By attaching several combining characters to a single base character, you can express compound modifications such as an accent above and a cedilla below. As an extreme example, there is an expression called "Zalgo text," which stacks dozens of combining characters on a single base character so that they spill over above and below, and it is used as harassment that breaks the line spacing of comment fields and chats. Unicode also has a way of thinking about limits, and the normalization specification defines a form in which runs of consecutive combining characters do not exceed 30 characters as one that can be processed safely. On the side that accepts user input, a practical countermeasure is to follow this by setting an upper limit on the number of consecutive combining characters and dropping the excess when saving.

From a security standpoint, visual spoofing that abuses combining characters (strings that look the same but have different code point sequences) is a problem. When validating user names or domain names, normalizing before comparing is a prerequisite. However, normalization can only absorb variations in how the same character is written. Impersonation that uses a different character, such as replacing the Latin "a" with the Cyrillic "а", remains even after normalization. This is handled by a separate check that looks at whether multiple scripts are mixed within a single word.

From a character counting perspective, combining characters are counted as independent code points, so the value of String.length does not match the visible number of characters. When "á" is in NFD form, its code point count is 2, but it looks like 1 character. Counting by grapheme clusters returns the result closest to what users expect. In JavaScript, you can count in this unit by using Intl.Segmenter with granularity: 'grapheme'. In the limit check of an input field, an "á" sent in its decomposed form is counted as 2 characters, and from the user's point of view the character limit is exceeded for no apparent reason. If you normalize to NFC as soon as you receive the input and then count, this kind of discrepancy almost disappears.

Share this article