When Compilers Disagree About UTF-8
When Compilers Disagree About UTF‑8
The author revisits a UTF-8 decoding function in his open-source C++ library, utfcpp, and adds a fast path for ASCII characters. Benchmarking shows a 3x throughput improvement with clang, but no change with gcc. Investigating the assembly reveals clang fails to eliminate redundant validity checks, while gcc already does. The author then refactors the validation into the sequence functions, satisfying both compilers and achieving 15-20% improvement with gcc on mixed text.
Compilers just never stop surprising me: this time there was no difference for the ASCII text at all! Zero!
- kstenerud
You could actually use SIMD instructions to detect sequences of bytes with the top bit cleared, thus allowing bulk copies of ASCII text without a per-byte loop.
Another optimization would be to take advantage of the fact that codepoint usage tends to cluster around the language of the text. So if you detect usage of 3-byte encodings, chances are you'll continue encountering only 3-byte encodings, with the odd ASCII or emoji codepoints. This opens up even more state machine possibilities.