Fixing a Subtitle Bug Caused by Byte Order Marks in Python
Fixing a bug with byte order marks
While converting SRT subtitles to WebVTT, I discovered a bug where byte order marks disrupted the file format. My initial manual fix was messy, but using Python's utf-8-sig encoding provided an elegant solution. I then used ripgrep to find and clean up dozens of previously corrupted files in my media library.
This sort of lesson is exactly why I love managing my local media archives as hand-built static websites – the lo-fi approach gives me lots of opportunities to explore low-level ideas and learn how things actually work on my computer.
- flohofwoe
Ugh, why are BOMs even still a thing in the 21st century? E.g. when will Windows finally arrive in the late 1990s and switch to UTF-8 for everything?
The UTF-8 BOM is especially bizarre because UTF-8 is completely endian-agnostic (so this "UTF-8 Byte Order Mark" is at most an indicator that this file is UTF-8 encoded, but guess what? Outside the Windows bubble, all text files are UTF-8 anyway).
- orangepanda
> A byte order mark is a special use of the zero width no-break space character U+FEFF at the beginning of a text file
Isnt BOM allowed to appear anywhere in the file, because of file concatenation?
- mr_mitm
I hate the BOM so much. It causes so many subtle issues and I don't even understand why it's needed.