Frame selection is the whole game: notes on making LLMs watch video
Feeding a video to an LLM instead of a written summary lets the model see the raw event, not a human's edited version. But token budgets force tough choices: you can only keep ~100–150 frames per video. The author's open-source tool uses adaptive scene detection, three-channel deduplication (global, action, and settled), and fuses frames with transcripts to make every frame earn its slot. It's served via MCP and runs locally, MIT-licensed.
A video is not an article about the video.
- jrochkind1
"Dedup started as one comparator and grew a channel every time real footage embarrassed it."
Is it possible to ever read a piece by LLMs that isn't written by LLMs?
- nl
The actual project README appears more human-written and is actually pretty interesting: https://github.com/HUANGCHIHHUNGLeo/claude-real-video
- quaverquaver
perhaps a representation of movement rather than a series of frames would better model what people see when watching video - highly compressed formats might even be directly useful in this regard?