Karpathy: Opus 5 turns Lord of the Rings into a 3D world with 5,500 lines of code

I worked with an LLM to build a ~3D animation of the Back to the Future delorean Time Machine as a way to spice up the hero on a docs page.

That took a fair amount of custom tuning and I had to create a tuning view to get some of the behaviors right.

But it was enough fun that I generalized it to take in ~any scene description from a film. It goes out and gets more detailed descriptions and film stills if available but also takes custom stills if you provide them.

My test scene was the Gauntlet scene from Apocalypto. It is low fidelity but does a pretty amazing sequence with somewhat believable physics of the javelins etc.

Here is the docs page with the vertical takeoff / 88 miles an hour time travel: https://contextify.sh/docs

I can share some of the Apocalypto bit if anyone is interested.

It seems pretty clear that Anthropic models have been specifically trained to be good at generating three.js (JavaScript 3-D Graphics) code, so given current state of AI code generation in general, I don't find three.js models/animations as indicative of anything other than the model's ability to write three.js code.

When Fable was first released the day-1 demos of it on Twitter (presumably from people who were given early access, and/or Anthropic employees) were pretty much 100% three.js stuff. Yes, it looks nice, but it doesn't tell me any better than an Erdos proof whether the LLM will be able to run my vending machine.

I don't think it's a bad way to benchmark new models, I just find it concerning that the author implies that "pelican on a bicycle" has been exhausted.

At the risk of making overly broad, unfalsifiable claims I think multi-year exposure to AI content has dramatically raised our expectations for speed and volume but lowered them for quality.

We see a very janky pelican and declare the problem solved.

I'd rather have them battle on the topic "Who builds a better Google Wave for LLM chats" to explore the space of how AI studios could be.

"Getting Started with Google Wave": https://www.youtube.com/watch?v=eKUAqNGVwX0

I can forgive the modeling being godawful jank (windows floating in the air, disconnected from the house). But I expected it to have a better understanding of the text. Instead, we have Bilbo's "disappearance" interpreted as him magically transporting or cloaking, and similarly for his reappearance.

this is not a good benchmark for models, but it's great if you're optimizing for attention on twitter because video content and 3d animations perform best on social media.

a real benchmark is instead running evals on your own traces, and building a cost/quality/speed profile for models based on real workloads. but it doesn't get you a shiny video you can post on twitter.

> Opus went off for ~2 hours and wrote 5500 lines of code that (procedurally) rendered the story.

> it's an example where we go from "no one would ever do this" to "sure, why not, it's ~free".

How is two hours worth of token generation free?

With tech revolutions things get cheaper/faster/better/doable, but there’s still real world limits. The advent of railroads made it feasible for the average person to cross the country, but it still cost a lot of time and resources. People weren’t crossing the country every weekend for fun just because it was now doable.

Why do we treat LLMs as ~free when we are generating things that weren’t doable before but have to invest more money than the Apollo program to build AI data centers let alone account for the operating costs?

I always thought of the Pelican more of like a gimmicky quick test. There are people who took it as a serious benchmark for overall model performance?

I’d like to see a human one shot a pelican on a bicycle in raw svg.