Getting Close to AGI
OpenAI’s GPT o3 looks like we’re getting closer to AGI. The new model hit a breakthrough score on the ARC-AGI benchmarking test meant to compare AI to human intelligence. The o3 model scored over 75% on the benchmark’s semi-private test when bound by compute cost. When unconstrained by compute costs, o3 scored an 87.5%, which is nearing a human equivalent. It may have cost upwards of a million dollars to hit the highest benchmarks (91.5% in a low efficiency test).
The idea that we can just allow state-of-the-art models to “think” longer to get better answers points to us being even closer to another concept that came from my conversation with Jack: Infinite intelligence.
I’ve thought of infinite intelligence as a large-scale system with knowledge in a specific field or fields equivalent to a human college graduate that can solve problems with brute force. That’s not quite o3, but spiritually it may not be far off.
As I understand o3, the program creates many Chains-of-Thought (COTs) that represent potential solutions to a problem, then it makes predictions about which of those COTs are most likely to lead to a solution. From ARC Prize founder François Chollet:
Effectively, o3 represents a form of deep learning-guided program search. The model does test-time search over a space of “programs” (in this case, natural language programs – the space of CoTs that describe the steps to solve the task at hand), guided by a deep learning prior (the base LLM). The reason why solving a single ARC-AGI task can end up taking up tens of millions of tokens and cost thousands of dollars is because this search process has to explore an enormous number of paths through program space – including backtracking.
o3’s approach is not one of heedless brute force but rather of elegant force. The model envisions potential solutions and tries what it thinks might work best, not too dissimilar to a human. The difference is that AI can envision and test orders of magnitude more potential solutions than a human ever could.
If we are approaching a world where AI can solve hard problems with elegant force, then it begs the questions: What problems can we solve with infinite intelligence?
Mechanistic vs Complex Adaptive Problems
There are two clear types of problems AI will be asked to solve: Problems with permanent solutions and problems with evolving solutions.
Problems with permanent solutions are mechanistic. Mechanistic problems don’t evolve. They’re siloed from the influence of external or confounding factors. Problems of physics and math live here. New problems in these fields don’t surface, we just see them as new given increased knowledge about the rest of the world around us.
Problems with evolving solutions live within complex adaptive systems where all the parts constantly collide with one another. John Holland coined the idea for complex adaptive systems, which are a network of interacting agents that continuously adapt behavior based on feedback and changing conditions in their environment. The human immune system is a complex adaptive system. So, too, are financial markets.
Unlike mechanistic problems, complex adaptive problems are often novel and unpredictable because they depend on an ever changing order in the problem’s environment. Solutions to complex adaptive problems, whether found by AI or humans, are bound to be temporary. A solution may work if the environment surrounding the problem doesn’t change, and it may suddenly stop working when the environment does change.
Infinite intelligence can solve both mechanistic and complex adaptive problems, but only if it makes economic sense.
SKU vs SAAS Costs
Mechanistic problems are like SKUs in Amazon’s warehouse. Each one has an associated cost for solving it with infinite intelligence, although that cost may be fuzzy. Each problem also has some assumed value for solving it, although that value may too be fuzzy. Solving mechanistic problems is a simple matter of capital allocation. Assuming the returns for solving a given problem exceed the hurdle rate of investing the necessary capital to do it, the problem should get solved.
Complex adaptive problems are more like SAAS subscriptions. When the problem is solved for a moment, the problem evolves and requires a new approach. So the allocation of capital to solve a complex adaptive problem, like the stock market, is a persistent one. As long as the continued investment generates the required return, you keep making it.
Infinite intelligence will follow the well-worn path of all new technology. Expensive at first. Cheaper over time. Eventually available to all. I expect in the next year or two we start seeing expensive AI-powered solutions to mechanistic problems that offer significant value.
However, financial markets may be the one problem that’s never cheap to solve.
Infinite Intelligence x Financial Markets
The necessarily competitive forces of financial markets make it a complex adaptive system unlike many others. Unlike other complex adaptive systems, e.g. the human immune system, an inherent feature of financial markets is actively competitive forces that will all use infinite intelligence if it is accessible enough. Cells in the human body can only ever rely on survival instinct in their competitive environment. They won’t apply infinite intelligence against one another.
The ultimate destination for infinite intelligence in financial markets may be a sort of massive stalemate. As costs come down, more participants will use infinite intelligence to find temporary solutions to markets, but as more elegantly brutal solutions are introduced to markets, the more complex those markets become. When everyone has weapons of infinite intelligence, no one has any edge. Ironically, the edge might end up applying basic intelligence to timeless market concepts. Maybe that’s always and forever the case.