My Blog

Recently I visited Reykjavík for a long weekend … little did I know that upon my return I would break this web site. When I downloaded my photos off my iPhone on to my FreeBSD NAS I decided to call the folder “2025-01-17 - Reykjavík - Phone”. This web site contains the paths to the original photos in a (not published) JSON file. Once I had added the few photos from the trip that I wanted to publish to that JSON then the web site generated fine on my MacBook Pro. Later that night, when the cron job on my FreeBSD NAS re-built the web site, it failed. FreeBSD claimed that my photos from Reykjavík did not exist. After a lot of digging I have now learned about Unicode normalisation. Consider the below Python script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

.editorconfig
.gitignore
.mypy.ini
.pylint.ini
.shellcheckrc
git-files.txt
initial-problem.py
LICENCE.txt
README.md
requirements.txt

              

… which produces the below output:

1
2
3
4
5
6
7
8
9

.editorconfig
.gitignore
.mypy.ini
.pylint.ini
.shellcheckrc
git-files.txt
initial-problem.out
LICENCE.txt
README.md

              

FreeBSD chooses to use the two bytes \xc3\xad to represent the í character but MacOS chooses to use the three bytes i\xcc\x81 instead. FreeBSD composes the accent on the i character to a í by representing it as \xc3\xad, however, MacOS chooses to keep the í character decomposed by keeping it represented as a normal un-modified i character followed by two bytes describing how it is to be modified when displayed. The result is the same character is rendered on your screen but the byte sequence to describe it is different, which means that computers think that the strings are different, which means that FreeBSD cannot find the file path described by the string in the JSON.

There are four possible Unicode normalisations: NFC; NFKC; NFD; and NFKD. The two C ones use the composed byte sequence \xc3\xad and the two D ones use the decomposed byte sequence i\xcc\x81. The inclusion of a K in the normalisation name makes no difference to how í is represented, however, it does to the … character. If there is no K in the normalisation name then … is kept as a special Unicode character of … and is represented by the three bytes \xe2\x80\xa6. However, if there is a K in the normalisation name then … is expanded out to three lots of the . character and is represented by the three bytes .... This is demonstrated by the below Python script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

#!/usr/bin/env python3

# Use the proper idiom in the main module ...
# NOTE: See https://docs.python.org/3.13/library/multiprocessing.html#the-spawn-and-forkserver-start-methods
if __name__ == "__main__":
    # Import standard modules ...
    import unicodedata

    # Define an example string as raw bytes (to avoid ambiguity) and then
    # convert it to a Unicode string ...
    null_byt = b'Where is \xe2\x80\xa6 Reykjav\xc3\xadk?'
    null_str = null_byt.decode("utf-8")

    # Print summary ...
    print(f"example string as a Unicode string: {null_str}")
    print(f"example string as a bytes sequence: {repr(null_byt)}")

    # Loop over possible Unicode normalizations ...
    for form in ["NFC", "NFKC", "NFD", "NFKD",]:
        # Convert the example Unicode string to the current Unicode
        # normalization and then convert it to raw bytes (to avoid ambiguity) ...
        test_str = unicodedata.normalize(form, null_str)
        test_byt = test_str.encode("utf-8")

        # Print summary ...
        print(f"{form:4s}    {repr(unicodedata.is_normalized(form, null_str)):5s}    {repr(test_byt)}")

              

… which produces the below output:

1
2
3
4
5
6

example string as a Unicode string: Where is … Reykjavík?
example string as a bytes sequence: b'Where is \xe2\x80\xa6 Reykjav\xc3\xadk?'
NFC     True     b'Where is \xe2\x80\xa6 Reykjav\xc3\xadk?'
NFKC    False    b'Where is ... Reykjav\xc3\xadk?'
NFD     False    b'Where is \xe2\x80\xa6 Reykjavi\xcc\x81k?'
NFKD    False    b'Where is ... Reykjavi\xcc\x81k?'

              

In summary:

From now on I shall be paying more attention to the byte sequences used to represent Unicode characters and, where required, I shall be enforcing NFC normalisation so that different computers (and pieces of software) continue to talk nicely to each other.