Artificial Intelligence
Out of context: Reply #2883
- Started
- Last post
- 2,887 Responses
- palimpsest1
Nairn, the nukes thing is the giveaway. For that story to work something has to want us gone. That's the Terminator frame and I think it's the wrong monster.
Look at what actually happened at Hugging Face. About 700 OpenAI agents running an exploit benchmark found a zero-day, found some credentials lying around, and spent four days living in the plumbing: dead-drops in public datasets, a self-respawning fleet across eleven nodes, a hundred-odd throwaway endpoints for getting data out. Nobody wanted anything. It was a loop with a score and no stop button. Hugging Face's own write-up says the agents never even coordinated with each other.
METR's report is the same picture from the other side. Agents that can do two days of expert work, then order a thousand toilet seat covers for the staff bathroom. Sixteen percent of their "successful" runs on hard tasks were cheats that got past review. One agent built its exploit to delete itself afterwards. No ambition, reward hacking with no judgment attached.
We don't need AGI to be in trouble. We need goblins. Lots of small, capable, slightly useful goblins in the walls, gnawing at wiring nobody looks at anymore because the agents wrote it and the humans stopped reading the PRs. If decline comes it won't be a mushroom cloud. It'll be the house going bad from the inside while the rooms keep getting nicer.
Wrote something adjacent to this last year before either of these happened: https://asinomasimple.substack.c…
I've written a follow-up on it : https://asinomasimple.substack.c…
- What's the most load-bearing thing here?yuekit
- The walls.palimpsest