Goalposts - Vote on which Hacker News AI challenges have been met
Show HN: Vote on which of Hacker News' challenges for AI have been met
Goalposts is a community-driven platform that lets you vote on whether the AI challenges posed on Hacker News have been achieved. It addresses the moving goalposts problem in AI evaluation by providing a transparent, crowdsourced benchmark. Users can browse challenges, cast votes, and see consensus on AI progress. The tool is perfect for developers, researchers, and enthusiasts who want to track real-world AI milestones and engage in informed discussions about the state of AI.
My benchmark is the 'runaway effect'. At some point we'll be able to point AIs at a python environment & tensorflow and it'll improve the algorithms we have for training itself.
- hatthew
My comment about humanity's last exam being a misnomer is included, and I proposed better ideas about what a last exam could look like. One of the things I said was "solve an open math problem" which has conclusively been done with Navier-Stokes (regardless of the controversy surrounding that). However, in the spirit of clarifying the goalposts, AI has only passed 1/6 of the tests I proposed. 17% is not a passing grade, so I'd say no, my challenge has not been met.
Another thing to note is that the (presumably AI-generated) summary of my challenge does not accurately represent what I wrote, listing only half the things I said and saying "or" rather than "and".
- joegibbs
There's one of mine in there where I predicted in 2023 that it would be 20 years until AI would be reliably able to entirely build and deploy arbitrary applications from a prompt. I was off by about 18 years on that one!
- bmenrigh
At least 1/3rd of these predictions aren't clear enough to determine exactly what is being claimed/predicted. Even after reading the full comment multiple times, on a lot of them I couldn't tell where the author had set the goalposts well enough to say whether we've crossed it or not.
- Retr0id
Heh, there's one of mine: https://stoppels.ch/goalposts/?c=39727943
"GPT-4 looks at original ASCII art of a foot, not copied from the web, and says it is a foot."
The vote is currently 64% yes, 18% no.
Just now I asked Opus 5.5 to generate an ASCII art foot, and it did a passable job. It's not great, but it's a foot. Then I pasted it into ChatGPT (whatever they're serving to the free tier by default, which seems to be 5.6 Luna), and it said it was a "train/locomotive": https://chatgpt.com/share/6abeaa39-cc80-83ed-851f-29370db089...
Maybe it's Opus's fault for drawing a bad foot but I think it's fair to say LLMs are still pretty bad at ASCII art (without additional tool calling etc).
- ben_w
Very pleased one of my predictions was totally wrong: https://news.ycombinator.com/item?id=23252711
Sure, sure, what
LLMs make still isn't "efficient bug-free code": my
prediction is falsified because while LLMs can write and train new models with machine learning, ML is fundamentally not advanced enough to throw arbitraty new tasks at like this.
- jefftk
Goalposts do seem to be moving:
Year Fraction answering yes
---- ----------------------
2016 56.89%
2017 50.78%
2018 51.21%
2019 39.59%
2020 37.32%
2021 48.48%
2022 42.58%
2023 43.93%
2024 42.11%
2025 38.57%
2026 28.84%
- outlore
These questions could benefit from being rephrased to make it clear what is being voted for
- ErrantX
What is interesting to me is in 2016 people were like; pass Turing test, write code, order me a coffee.
And even in 2024 the themes are similar, generally more complex or specific about the coding/turing/action test.
But in 2026 a huge shift, we have things like; can open a physical door, emulates human pettiness convincingly, makes novel scientific breakthroughs.
That alone tells you a lot IMO