Next revolutionary AI race exposes the limits of language

More than 50 years after her death, Helen Keller still turns up in artificial intelligence debates. The communication breakthroughs of the American writer, who was deaf and blind, are often cited as evidence that language is the best path to creating machines that are as intelligent as humans.

Despite loud early doubters, large language models — the technology underpinning the vast majority of today’s AI investment — have proven far more grounded than critics expected. A computer can’t dip its toes in the ocean, but it can recognize enough patterns in the writings of the entire internet to mimic an understanding of what it means to get your feet wet. Our obsession with LLMs follows a long tradition of affiliating language with intelligence.

Yet Keller’s own retelling of the famous “w-a-t-e-r” milestone was about more than words. As her teacher Anne Sullivan spelled out the letters on her palm, Keller also felt a “wonderful cool something” flowing over her hand. She wasn’t simply relating one word to another, she was connecting language to a physical sensation.

That distinction matters as everyone from Nvidia Corp. Chief Executive Officer Jensen Huang to Chinese President Xi Jinping heralds the physical AI era. In the next phase of this tech revolution, marked by AI-powered robots navigating the real world, the limits of LLMs are becoming harder to dismiss. Knowing every sentence ever written about using chopsticks or tying shoelaces is different from being able to feed yourself or get out the door.

I’ve written before that the two biggest barriers to useful humanoid robots are the hands (dexterity) and the “brains” (AI models that can power real-world actions). Solving both could unlock what Morgan Stanley forecasts will become a $5 trillion market by 2050. But what if those problems are more interconnected than we thought?

Biologists studying hominin fossils found that primate dexterity and brain development evolved hand-in-hand over generations, suggesting intelligence was shaped not just by observing the world, but by physically acting in it, according to a Nature paper published last year.

Achieving Artificial General Intelligence in machines might not be all that different, Junghee Ryu, the chief executive officer of RLWRLD, a physical AI startup with offices in Seoul, San Francisco and Tokyo, told me. His company is developing foundation models for robots that can crack five-finger dexterity, and he argues that the hardware and software for robot brains and hands must co-evolve. Earlier this year, RLWRLD announced a collaboration with Nvidia to develop a new benchmark for dexterous manipulation.

LLMs are very good at understanding their circumstances and planning actions, Ryu says, but they still can’t perform any jobs requiring dexterity, making them useless across vast swaths of the labor market. To him, that should throw cold water on recent claims that we’ve reached AGI, the mythical goalpost marking systems that is as economically viable as humans.

He’s not alone. Human-like dexterity, particularly with robotic hands, is the “final component making AGI possible,” analysts at the Korea Investment & Securities Co. wrote in an industry report earlier this year. Hands are connected to 25% of the brain motor cortex and 30% of the sensory cortex, they said, making them crucial to how humans perceive and act on external information.

Yet even as experts continuously forecast that we’re inching closer to AGI, they tend to think we’re getting further from machines completing basic physical tasks. In a long-running survey of hundreds of AI researchers, respondents in 2016 thought machines that could fold laundry or assemble any Lego set better than humans would arrive by about 2022 and 2025, respectively. In the latest 2024 iteration, those milestones slipped back to 2030 and 2031.

It makes sense given that the benchmarks that still prove hardest for AI involve acting in the real world, marked by unpredictable environments and where mistakes have physical consequences, Stanford’s latest AI Index report noted. Comparing robot performance in controlled labs versus simulated households, the researchers found a vast gap. It’s a sobering reality check, and one that researchers like Ryu think comes down to a lack of dexterity.

It’s also a wake-up call for China. Aware of demographic shifts, Beijing has made physical AI a policy priority. The nation already seems to have a leg up in humanoids, with Chinese companies accounting for more than 97% of global shipments in the first half of this year, according to U.S.-based Smart Analytics Global. But beneath the flashy demos, the illusion of capability breaks down.

This provides an opening for others to tackle the most difficult parts of making robots useful. Doing so will likely continue to reveal the limits of just building bigger language models. The investor mania for LLMs also heightens the risk that the money is piling in faster than the science is advancing.