Diacritical Mark
An auxiliary mark added above or below a letter. Accent marks, umlauts, and similar marks indicate differences in pronunciation.
A diacritical mark (a mark that distinguishes pronunciation) is the general term for auxiliary marks added above, below, or beside a letter. Typical examples include French accent marks (é, è, ê), German umlauts (ä, ö, ü), the Spanish tilde (ñ), and the Czech háček (č, š, ž), and they are used in languages around the world, mainly in regions that use the Latin script. These marks are not mere decoration but elements essential for distinguishing pronunciation and meaning. For example, in French the meaning changes with the presence or absence of an accent mark, as in ou ("or") and où ("where").
In Unicode, characters with diacritical marks can be represented in two ways. One is the "precomposed character" (NFC: Normalization Form Composed), which treats é as the single code point U+00E9. The other is "base character + combining character" (NFD: Normalization Form Decomposed), which represents it by combining e (U+0065) with an acute accent (U+0301). This dual representation is a design feature of Unicode, introduced to represent the characters of every language flexibly while maintaining compatibility with existing character encodings.
In practice, this dual representation causes serious problems in string comparison and sorting. Even an "é" that looks exactly the same has different byte sequences in NFC and NFD, so a simple byte comparison does not match. In situations that require judging whether strings are identical, such as database searches, file name matching, and password verification, you need to apply Unicode normalization to unify the representation.
For file names, you need to understand precisely how each file system handles them. It was HFS+, the older macOS file system, that converted names to the decomposed form, and the current APFS stores the given form as it is. When you actually create a file on APFS with a name in the composed form (containing U+00E9), it is stored in the composed form. Even if you try to open it with the name in the decomposed form, it resolves to the same file. In other words, APFS behaves in a way that "does not force normalization, but regards either form as the same." On the other hand, common file systems on Windows and Linux do not normalize names and compare them strictly by code point, so names in the decomposed and composed forms become different files. This asymmetry surfaces when files created on macOS are brought into other environments, in the form of "two files with the same name" or "a file that should exist in the repository cannot be found." In Git, core.precomposeUnicode is responsible for absorbing this difference.
A common misconception is simply calling diacritical marks "accent marks." Accent marks are only one kind of diacritical mark, which is a broad concept that also includes the cedilla (ç), the ogonek (ą), the macron (ā), and others. Also, the Japanese dakuten and handakuten are of the same kind in that they are marks attached to a base character, but note that Unicode splits them into two series. "゛" (U+309B) and "゜" (U+309C), which have their own width when standing alone, are classified as symbols rather than combining characters, and the ones that combine are U+3099 and U+309A. "か" (ka) + U+3099 is merged into "が" (ga) by normalization, but attaching the similar-looking U+309B does not compose and leaves 2 characters. In code that handles dakuten, if you do not check which code point has arrived, mismatches remain that normalization does not fix.
In programming, normalization is performed with the JavaScript String.prototype.normalize() method or the Python unicodedata.normalize() function. When receiving form input in a web application, the common approach is to apply NFC normalization on the server side before saving to the database.
What to be careful about here is the assumption that applying NFC normalization always shrinks a character to 1 code point. Only combinations for which Unicode provides a precomposed character are composed. e + U+0301 becomes U+00E9 and z + U+030C becomes U+017E, but q + U+0301 and z + U+0304 have no precomposed character, so they remain 2 code points even after NFC. Therefore, if you build logic on the premise that "after normalization, the number of code points = the visible number of characters," it breaks down depending on the combination of language and marks. It is accurate to view the role of normalization not as shortening the length but as uniquely determining the representation of the same character.
In search and duplicate detection, separately from normalization, there is a demand to "match while ignoring accents." This is an area adjusted by the strength of collation rather than by normalization, and in JavaScript you specify it like new Intl.Collator('fr', { sensitivity: 'base' }). However, at this strength, even words whose meaning changes with the presence or absence of a mark are treated as the same. In actual measurement, ou and où are distinguished under the default collation, but are judged to match (comparison result 0) with sensitivity: 'base'. The rule of thumb is to use them differently: match loosely at the entrance of a search to pick up a wide range of candidates, and do not loosen the final identity check (login IDs and unique constraints). Note that the default collation absorbs the difference between NFC and NFD, so for comparison purposes, strings match even without normalizing in advance.
From the perspective of character counting, text in NFD form counts base characters and combining characters as separate code points, so the count is higher than the visible number of characters. For example, "café" is 4 characters in NFC, but is counted as 5 characters (c, a, f, e, ́) in NFD. To obtain an accurate "visible character count," you need to count by grapheme clusters, and the handling of diacritical marks is an unavoidable challenge in implementing a character counting tool.