Intelligence Is Not Prediction

The Future Has Already Happened made the case that the social structures around AI — credit, accountability, legitimacy, hierarchy — are stickier than the technology itself. Capability is only one layer of the stack. A model can write the code, but someone still has to get the credit, answer for the result, and hold the authority to override it. Those layers don't dissolve because the tool underneath them got smarter. The technology will change the world fast; the structures that decide who is trusted and who is on the hook will change slowly.

The strongest counterargument is simple: enough intelligence dissolves the structures anyway. If a system is smart enough, why not just hand it the reins? Because we never hand the reins to the smartest person in the room either — and not out of stupidity. You can't tell the difference between reasoning that's genuinely better than yours and reasoning that merely sounds that way. Intelligence you can't check is indistinguishable from confidence, and confidence without accountability is dangerous. That's why oversight, review, and legitimacy exist in the first place.

But that argument left one door open. What if AI reasoning became reliably verifiable — what if a system could show its work in a way you could actually check? Then the trust constraints might fall.

Think of a mathematical proof. A machine can hand you a proof you'd never have found, and you don't have to trust its intelligence to accept it — you check each step, or a proof checker does. The reasoning is laid bare and verifiable, so its authority doesn't depend on anyone trusting the prover. If every decision worked like that, "is it smart enough" would be the only question that mattered.

So the whole question reduces to this: can the reasoning be verified? If we reason hard enough in a principled way can we then automatically distinguish between "sounds good" and "is good"?

For most decisions that matter, no. And the reason why points to where the real work and a lot of value is.

Verification means forcing a doubter to agree

Look closer at what made the proof work. To verify something isn't to convince yourself you're right. It's to produce something that forces a person who disagrees with you to concede. Verification is a social act, not a private one.

A proof can do that because it comes with an oracle: a checker no one has ever watched fail. You don't believe a result because the prover is smart — you run the check, and if it passes, you're cornered. You can disbelieve P≠NP all you like; the moment a valid proof clears the checker, you have to give it up. The authority transfers without anyone trusting anyone.

If we don't have an oracle then everything becomes a lot more fuzzy. I make a claim and you as a sceptic need to agree. Notice how we have left the realm of objectivity and have moved to subjectivity. "The world is round" is objectively true, yet doesn't universally convince. Now imagine "the world is round" is legitimately contested.

A round earth is still easy. Its a claim that can be repeatedly tested. Any claim about the future by definition can't. We can make a prediction. Predictions about the future are inherently complex. We can't even predict the weather reliably much further than a couple of weeks.

But weather prediction is also easy. Its easy because the weather is not sensitive to the actions taken in response to the prediction. The weather doesn't materially change because I expect it to rain tomorrow. In fact any change I could possibly do will have essentially no effect on the actual weather. But this isn't a universal property of predictions. A very large chunk of very important choices are policy decision and capital allocation. We make those decisions based on predictions, but the world reacts to the actions taken in response to the predictions and we never get to replay and repeatedly test our predictions.

A venture fund that has to choose between funding two startups don't get to replay the decision after the fact. Sometimes your mistakes are obvious, but most succesful companies have near death experiences and the counterfactuals of those situations are just water under the bridge. Maybe that company you didn't fund would have been a great investment, but they couldn't get off the ground. You only know if someone else funded it and proved you wrong. That means that most venture funds have quite low volume of data to make their predictions on and that explains a lot of the FOMO. They are essentially leaning on trust and social structures. The emotions are at least as important as the data. Can we "reason harder" and verify this reasoning to make better decisions? Probably. Is VC funding going to continue to operate at least in part on gut feeling and trust in the founders/the team? Probably.

But even this complex venture funding is not the top pinacle. Every day politicians make policy and capital allocation decisions based on predictions. Like how much of the budget should we give to education vs health care. You can make elaborate models to predict future GDP growth. This is analogous to what the venture capitalists are doing, just at bigger scale. But there is an additional complication which is that incentives are murky. VC's are optimizing cleanly for future cash flows (or some variant thereof). Politicions don't get that luxury. Even if funding education is the better prosperity building strategy over the long run policians needs to convince voters that its the best course of action. And voters might disagree on very reasonable - although quite selfish - grounds. Someone who has less than 5 years left to live might have a no reason at all to vote for increased education spending. So your elaborate model, if you get it right, might not be an end-all-be-all. Can better intelligence help you navigate this more accurately with fewer misunderstandings and errors? Probably. Can AI "calculate optimal policy"? Probably not, and even if it could we would like still want or need human judgement. One politician might want to optimize for voter retention and another for the long term benefit of the state. Depending on what the politician wants to do with voter retention that might even be the preferred path for the majority of the states citizens.

Reflexivity, chaos and complexity are big forces and AI doesn't seem to be anywhere close to actually solving it yet. Wake me up when we can forecast weather reliably for more than 3 months. If we can't produce an oracle then we are right back to fuzzy humans feelings and persuation. Good luck automating that.

The world runs on iteration, not prediction

Prediction is hard so what can we do instead?

Evolution didn't forecast the organism; it varied, tested against reality, and kept what survived. Markets don't predict the winning product; they run thousands of bets in parallel and let most of them die. Science - most of the time - doesn't deduce the truth and then check; it guesses and lets the experiment overrule it. The engine underneath all of them is the same, and it isn't foresight. It's iteration: try, observe, keep what works, try again.

You pick the idea, ship it, and let reality score the thing no argument could settle. That loop is the only oracle these decisions have, and it only answers after you've already moved.

AI can make the loop faster — better experiments, quicker feedback, richer data to adjust against. That part is real and largely automatable. What it can't do is pick which direction to run, call the pivot, or eat the loss when the bet misses. Those are the unverifiable, accountable parts — the same ones that never had an oracle to begin with.

Where the work actually goes

So the value of AI in complex systems isn't replacing judgment. It's making the loop better: experiments that run cheaper, feedback that comes back faster, and enough interpretability that the human in the loop can actually judge what came out. Not a machine that predicts the future — a machine that lets you try more things, learn more from each try, and stay honest about what you actually learned.

That's still a mountain of work, and it doesn't remove the human. It makes the person who picks the direction and answers for the result faster and better informed — it doesn't hand the picking or the answering to the model. The judgment and the accountability stay exactly where the verification problem left them.