How Storyteller Automates Immersive Reading
Automating Immersive Reading
Storyteller is a platform that aligns ebooks with audiobooks, syncing text and audio for immersive reading. The core challenge is forced alignment, which must handle differences in chapter order, missing chapters, skipped spans, and alternate word choices. The author describes their journey from a simple Python script to a sophisticated system, explaining how they use the Massively Multilingual Speech model and CTC emissions to locate chapters in audio, overcoming obstacles that stymie other aligners.
I felt like I was stumbling through the dark, aware that there was probably light at the end, but unable to see it.
- smoores
I took a week of from work recently to reimplement Storyteller's forced alignment algorithm. Storyteller[1] is an open source, self hosted platform for creating, managing, and reading/listening to "readaloud" books — books that have audiobook narration built in and can highlight each sentence (and/or word, with this new algorithm!) as it's read aloud. Forced alignment is the process of determining where each piece of text starts and ends in the audiobook.
Anyway, I am really pleased with how the new algorithm turned out! Hopefully someone else finds it interesting as well.
- SamBam
I'm curious whether individual-word highlighting is best for people with reading disabilities, or a broad, whole-sentence highlighting might actually be preferable.
When I'm reading my eye is jumping back and forth across the sentence. The demo made this abundantly clear -- it was actually very hard to just track the highlighted word.
Just using the fact that I consider myself a very good reader as an example, I wonder whether there are advantages to the eye being free to skip back and forth across a sentence.
I assume it might be as simple as a user-preference toggle, since you're already highlighting the sentence.
You could also try UIs where the highlighting was more "spread out," focused on the word being said but bleeding into the other words on either side (maybe up until the hard stop of the sentence boundary) to lead the eye toward it. I'm imagining something like the red-blue text color of the BeeLine editor.
- cainxinth
This is how I proofread. I play a screen reader while I read my work. The brain does some autocorrecting that you don’t even notice and occasionally skips over a typo when you read it without the audio.
- mobeets
This is really impressive! One thing I’ve often wanted is a way to listen to an ebook (say, while cooking), and then later be able to sit down and read from where I left off. This essentially requires the same functionality that Storyteller offers (syncing progress through an ebook and audiobook), though arguably overkill for this use case (since I could handle a more coarse alignment). In any case, is this something Storyteller would be good for? Or are there other tools for the job?
- orphereus
What is the purpose of this outside people with disabilities? Why do you need word highlighting, it's like those TikToks and IG reels where they show one word at a time to keep you hooked.
Surely, if you need this to read a book (again, besides any disabilities), you should stop your current habits.