Meta's Muse is fantastic for web scraping
A developer building fullsets.fm, a side project that aggregates professionally recorded concerts, uses Meta's Muse to scan Reddit and YouTube for thousands of videos from obscure channels. For $80 a month, Muse burns billions of tokens weekly, often without counting usage. The model is mediocre at writing and coding, but excels at Gemini Flash-style workloads. The developer worries about the ethics of AI agents that never sleep and can impersonate real users.
I worry about what will happen when someone points Muse at my own websites. These agents don’t sleep, and they can pretend to be real users extremely well.
- dvt
I built an AI "web harness" running on a sandboxed Chromium (using a custom side-loaded plugin that talks over websockets to a "driver") to basically do anything a normal user could do in a browser. It totally bypasses any and all bot measures and only gets the ones you yourself would get as well (and passes those successfully, e.g. Cloudflare checkbox or those annoying OCR puzzles).
Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.
- wiether
As someone having to fight Meta's bots everyday to keep websites accessible to actual customers, I'm not surprised to read that it can be seen as something positive on the other side of the fence.
But I'm wondering: at what cost?
- kevmo314
Muse ran into a captcha and asked me if I wanted it to solve it.
So of course I clicked yes and it dutifully convinced the site that it was not a bot.
- arjunchint
their static ip's were initially good and didn't get flagged, but now most sites are recognizing their ip ranges and blocking.
Muse's utility has significantly dropped with the blockages.
To become truly useful again they will need to use residential proxies, but I can't see them use those due to the risks and reputational damage.
- sixtyj
> I don’t know where this leads, but we’ll likely see more websites block Muse unless Meta prevents abuse.
Multiply it by 1,000,000 access attempts - daily.
It is question of time when even normal browsers would need some allowed fingerprint to go to sites otherwise caught by clouflare or similar wall.