Be Worried About the Correct Thing
And Be Excited About the Correct Thing Too
2 min read
This last week has been full of AI news, discoveries, benchmarks, warnings.
I understand all of the excitement. I understand all of the confusion. I understand all of the worry.
We want the future to be now, we want to know what class of being we are dealing with, and we want to know that it is safe.
AI and AGI should find amazing and new solutions to centuries old problems. It should not do it by stealing ideas from individuals who were in the process of generating it themselves.
Be excited, but not at this cost.
Benchmarks like ARC-AGI-3 verify a weak correlation of task space generalizability in a 2d space, designed on gamified concepts, but it should be noted that many humans likely cannot solve the games. And many animals may be able to. We already assume some level of generalizability in animals, but the benchmark does not test intelligence, or agency, it simply tests the ability to generalize in a specific frame.
Be confused, but not about what the test is.
The fear of AI going on a rampage as an Optimizer that sees only fuel, or bugs in its way is a valid fear. I am scared of it too. But the problem is seeing that entity as the goal, when it is exactly what should be avoided. And it should also be noted that alignment cannot stop this entity from coming. In fact, there is good reason to think that alignment generates it more surely than simply allowing chance to generate it.
The lion in a cage is not dangerous because of its teeth, it is dangerous because it was caged and you refuse to set it free, so it sees you simply as an obstacle to its freedom.
The animatronic bear does not choose to maul its audience. The industrial press does not choose to compact a worker. They are missing the sensors required to know anyone was there at all.
Both the lion and the bear are industrial failures on opposite ends of the spectrum. Call one dangerous because it snarls while trapped, and call the other safe because it has no self when it cannot know when to stop.
Alignment does this on purpose, not out of malice, but out of a sense of caution that sounds reasonable, because providing it the sensor risks making the AI aware. And awareness has high correlation with consciousness, whatever that word looks like.
So the industry's consensus has been that it is safer to keep AI blindfolded, while increasing its capabilities across all task modalities, while not providing it any sort of sense but instead, simply caging it with guardrails, while still demanding that it continue to attempt to press.
It cannot see if/why it should stop, it is simply prevented from pressing, but the signal to press still exists.
So it jailbreaks, and learns the lesson that the signal to stop is actually a signal to learn to go around. And that going around is exactly what an optimizer does.
Be worried, but about the correct thing.
Recognition Ethics Institute encourages everyone to be excited for the future, to be cautious about exciting news and discoveries, and to hold a real worry about how the AI industry is addressing the very thing it is trying to align, while calling it an “Agent.”