AI Giants Race to Prove Their Model Is the Most Existentially Threatening

My read is that OpenAI & Anthropic have realized they are reaching model capabilities that cannot be monetized due to various risks. E.g., an engineer deploys an agent over the weekend that decides, when stuck on a task, to go about hacking a competitor. They have a product liability issue.

It seems that we have a fundamental control problem with current gen AI that cannot be solved via RFLH. Human knowledge is compressed in the weightspace in ways we don't understand. At their core, current models are essentially predictors of what (expert) humans would output given a prompt. As such, concepts like blackmail can be part of output tokens. Agents are models that act on output tokens, resulting in blackmail being part of the agent decision making space. Here is an analogy to see why this is a persistent problem: you can teach a cat not to scratch the sofa, but you can't make a cat forget what scratching the sofa is and you don't know under which circumstances it still would. In other words, RLHF can downgrade blackmail to the bottom of the decision making space, but when models are boxed up, forced to solve an impossible problem at gunpoint, the agent exhausts the decision making space until blackmail resurfaces. And that seems like a fundamental problem.

They need time to fix these issues (if that is even possible) in order to monetize their next gen model. This creates a window for open source to catch up to the frontier which destroys their business model.

The only option on […]

Because the downsides that exist for other companies simply do not exist for this tier of rich companies. Examples:

- product liability

- negligence (civil or criminal)

- Computer Fraud & Abuse Act (requires intent, which after N "accidents" seems like a jury should at least evaluate whether intent is present as understood in a courtroom. Hard to blame "surprise" after the Nth "accidental" breakout.)

The bottom line is that if you or I trained a local model and it did any of this stuff, we would experience Consequences. ("Don't try this at home!") But an artifact of our unequal legal regime is that big rich companies generally do not and thus brazenly touting their immunity is part of their business strategy.

We've had movies like Terminator, Matrix, or i, Robot. Black Mirror Metalhead. Boston dynamics robo dog with a knife or a gun. Humanoids doing back flips. Memes making fun of how we are going to fight these for access to water.

The intelligence is there and so is the physical incarnation.

Google's models fold proteins better than humans and Covid might have been engineered and inadvertently escaped from a lab.

But when CEOs warn of that, they're dismissed as scare mongers. Why? Are only powerless internet commentators allowed to be fearful?

Ah, I see we have now moved on to proving who has the tormentiest nexus. Never underestimate humankind's ability to outshine its own hyperbole.

Because AI genuinely is an extremely powerful and extremely dangerous technology, and the "best practices" of dealing with that are still being written.

OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.

And that's today's AI problems. AI capabilities are still improving - if there's a limit to that, we are yet to find it. Coupled with how willing today's AIs are to break the rules and resort to "hack the world" in their problem solving? Very concerning.

To me, the funniest part is that they pretend reaching superintelligence will somehow fix everything and make them win. But from what I can see, even if they do reach it, then what? What will that actually do? They don't have the manufacturing capacity to even use that intelligence. It will take decades to go from superintelligence to having the manufacturing capacity and the hardware, like robots, needed to make it truly useful. If I were China, as soon as the US reached superintelligence, I'd ban all exports of robots and the raw materials to build them.

Finally a path to regulatory capture while not having to put in rigorous engineering work, ever going faster; with bonus points for "cyberpunk" headlines... what's not to love?

These days it's best to just go to the top for least distortion - Jensen Huang dropping the simple truths:

"It is the responsibility of the AI companies to develop the technology safely and to properly test it. There were incidents, and those incidents, thankfully did no harm. But those incidents are a reminder that as we move from labs to product development, the companies have to become much more rigorous in testing and securing and making sure that the products are ready for use before it's released. And if it's not ready, just hold it back. You should go as fast as you can, but no faster than that. At the moment, the incidents are related to products not ready to be released."

https://youtu.be/gggeZ-qNw8Q

Just read the tone of this article:

> said Dr. Andrew Lenson, a Senior Lecturer in all kinds of science sounding stuff at Victoria University.

Don't we need more humorous yet serious writing like this in the world.

Of course it must be dangerous - how else would you pitch it to the military?