The Ghost Characters Haunting Unicode
A Spectre Is Haunting Unicode
In 1978, Japan's JIS X 0208 standard introduced several kanji with no clear origin, later dubbed 'ghost characters.' A 1997 investigation traced most to cataloging errors, like a mistaken stroke from pasted paper. Only one, 彁, remains unexplained, likely a misreading. These ghosts now lurk in every computer via Unicode, a permanent glitch in digital text.
The errors went undiscovered just long enough to be set in stone, and now these ghosts are, at least in potential, a part of every computer on the planet, lurking in the dark corners of character tables.
- joshdavham
The author Paul McCann (polm) is one of my favourite programmers out there!
He’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects.
He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2].
[0] https://github.com/polm/fugashi
- layer8
(2018)
Major previous discussions:
110 comments: https://news.ycombinator.com/item?id=17637375
130 comments: https://news.ycombinator.com/item?id=24951130
180 comments: https://news.ycombinator.com/item?id=32095502
- tengwar2
Before that, there was the case of ÿ and Ÿ in the base IBM character set (see https://en.wikipedia.org/wiki/Ÿ). The former exists in a handful of proper nouns in French and Hungarian (as in about five proper nouns), but only as a y with a diæresis, which would normally be handled with a Unicode modifier symbol. It is also used in a couple of endangered languages in South America, which would not have influenced the character getting in to a PC character set. It is thought that possibly it was supposed to represent the Dutch "ij" character, but apparently that is never written as "ÿ" in print. Ÿ seems to not be used by any language.
- eproxus
This would make a great tattoo, as a pun on people getting Kanji character tattoos without being able to read Kanji.
- erjiang
I think there’s evidence found for the origin of “彁” as the result of a poor scan of a newspaper article. Look up “彁 新聞” to find some japanese sources about this.
- gweinberg
It occurs to me that we can use 彊 to mean "a completely unknown concept that cannot be named". For example if you ask, "when Cthulhu rises from its slumber, what is thefirst thing it will do? Probably it will 彊.
- ComputerGuru
It’s really a complete and total nothing burger. Extra code points were added, might have been an issue when we were trying to cap the total number below needing some arbitrarily fixed number of bytes for convenience, but now that’s no longer the case and they’re just a historical oddity that costs nothing to maintain and certainly don’t “haunt” in the sense of “ keep popping up and causing problems” in any way.
I appreciate the article nevertheless, of course, but I do feel that it would probably be more meaningful to someone that has at least a basic understanding of Japanese.
- sedatk
Fascinating. But, I guess it's better to have superfluous invalid characters than missing real ones.