AI 2040: Plan A to Avoid Existential Catastrophe

I would not like to be dismissive, but to me this article feels like an exercise in creative writing rather than a report to be taken seriously. The entire experience feels like a choose your own adventure game, seems like their stylistic intent.

I am not sure if alternative reality fiction is the best way to approach real and serious AI risks.

I am also not sure, with the amount of emdashes and the style of prose, that the entire article was not AI generated.

AI is going to be a mature scientific field. There are going to be efficiency improvements in training and inference. New paradigms are going to emerge with better multimodality, real time streaming and real time interfaces. Models are going to converge on the limits of our data available for pre and post training, improvements will be incremental and spiky in domains.

I am not sure who the AI 2040 article is for. I suspect it is intended to be a digestible piece of media for the financial class.

AI is going to be a useful technology and its impacts across the economy and global will be broadly distributed. Because AI represents the distillation of the very best human knowledge and expertise. AI is compression of human capabilities, the very best ones. Maybe the argument is that in verifiable domains, such as model training, AI models can supercede humans. I don't think so. A human's high level thinking, our incredibly more efficient semantic/neural compression, our ability to switch tasks and achieve the creative insig […]

I'm sure some people will have issue with my phrasing but, honest question:

Are there examples of where we have collective decided not to pursue knowledge? Successfully?

I guess nuclear weapons might be the best example though research doesn't seem have to actually "stopped" as much as gone underground and we still have country trying to climb that ladder.

But I don't know how relevant that is to LLMs/AI. It almost feels like pandora's box is open and our only option is continue to improve them. There is clearly value in what they do and while I can absolutely see the dangers, for example: authoritative governments and surveillance, I'm not convinced to throw the baby out with the bathwater.

All of technology back to the printing press (and probably before that) could also be said to make it easier for governments to oppress their citizens. Making laws (and enforcing them!) to prevent governments from doing these things feels like that route forward, not trying to stick our heads in the sand.

Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.

I found the AI 2027 paper to be overly optimistic, but not wholly fantastical. This paper feels wildly speculative, and relies on premises I am not confident even pass surface reasoning. Even under optimistic conditions, we are not going to see robots "capable of 95% of all cognitive and physical tasks" by 2035. Nor do I think a 74% unemployment rate is even remotely possible. Economic collapse would implode AI development long before those figures were plausible.

The biggest issue with 2027 was that it didn't understand the economy.

For AI2027 to be real, the money has to come from somewhere to carry on building the economy. If >10% of the workers suddenly become unemployed, and the rest taking paycuts, then money supply dries up. (unless central banks do something, but then that can be highly inflationary)

Without massive amounts of investment, AI development stops dead.

In this post, they hand wave about the USA being able to acutally 1) build concensus locally for regulation and 2) the rest of the world actually follows suit.

It fails to understand that actually the progress of AI is not actually the gift of the USA. It requires a constant supply of things from china.

Also its assuming that having 74 billion agents doesn't cause economic distortion. Like what value are these agents generating that justifies them being run?

I really wish people would just ignore this for what it is: bad sci-fi with an incomplete world.

It seems to me

we’re already at the top of the S curve, not at the toe of an exponential curve. At least with LLMs. Better training data will make small improvements, better architecture will make it less compute intensive, and all these “hyper-scale” data centers will make it cheap and ubiquitous. But none of that is it getting exponentially more intelligent.

This is from the "AI 2027"[1] people.

[1] https://ai-2027.com/

This is by far the most realistic optimistic AI takeoff scenario I've seen, and more specifically it's the first one I've read that deals with both the AI alignment and power concentration issues in a sufficient way, even in a world where hard alignment is assumed (in this scenario the AIs are assumed to be misaligned until ~2038-39).

Bravo, and I hope it has the impact on the AI safety field it deserves to have.

- whenever i read this, why does it always read like some first world country guy born in a bathtub who has never been outside his bubble, never seen what the rest of the 90% world looks and works like actually sat and wrote a work of fiction

- have you seen the actual problems in other countries outside america?

- have you been to any country outside USA and the countries in europe?

- have you taken a trip to any developing country and stayed there for a month?

- have you seen what sort of daily struggles, political systems, bureaucracy and work exists in developing countries?

If carbon taxes are already a lethal policy for an political campaign, it's absurd to think that fears of ASI will create any real movement around pausing AI.

If there is any movement to pause AI development, it will come from the general public's dislike of these companies. Not from the AI safety angle.