UTF-8 is an encoding. Encoding is how we store code points [of Unicode] in memory.

The simplest possible encoding for Unicode is UTF-32. It simply stores code points as 32-bit integers.

UTF-8 is a variable-length encoding. A code point might be encoded as a sequence of one to four bytes. One or more code points can build a character.

Side effects of UTF-8:

• You CAN’T determine the length of the string by counting bytes.
• You CAN’T randomly jump into the middle of the string and start reading.
• You CAN’T get a substring by cutting at arbitrary byte offsets. You might cut off part of the character.

If you want a character comparison, you should be iterating on "extended grapheme clusters", or graphemes. A grapheme is a minimally distinctive writing unit in the context of a particular writing system. ö is one grapheme. é is one too. And 각.

Is Unicode hard only because of emojis?
No, for example, ö (German) is a single character, but multiple code points (U+006F U+0308).

What is 🤦🏼‍♂️ length?
It depends of the encoding used: 5 for Python, 7 for JavaScript / Java / C#, and 17 in Rust. That’s what extended grapheme clusters are all about what humans perceive as a single character. And in this case, 🤦🏼‍♂️ is undoubtedly a single character.

Before comparing strings or searching for a substring, normalize!

Because code points can be in different order for a grapheme. Also we want to be able to search for 2 in 𝕏².

Unicode is locale-dependent, because two grapheme with the same code points can look different in two languages.

So no, you can’t convert string to lowercase without knowing what language that string is written in. [...] I live in the US/UK, should I even care?

Yes.

What are surrogate pairs?

Unicode decided to allocate some of these 65,536 characters to encode higher code points, essentially converting fixed-width UCS-2 into variable-width UTF-16.

A surrogate pair is two UTF-16 units used to encode a single Unicode code point. For example, D83D DCA9 (two 16-bit units) encodes one code point, U+1F4A9.
The top 6 bits in surrogate pairs are used for the mask, leaving 2×10 free bits to spare: 1101 10?? ???? ???? to 1101 11?? ???? ????'

Is UTF-16 still alive?

Yes. The only downside of UTF-16 is that everything else is UTF-8, so it requires conversion every time a string is read from the network or from disks.