EAI Researcher: Are Language Models Missing Something?

By: Tyler Wells Lynch

Language models like ChatGPT are certainly impressive, capable of generating sophisticated strings of text or code, but they’re pretty far from being able to think, reason, or understand. In fact, given how language models work, those abilities seem positively alien. Why? Is it a problem of scale? Or is the whole methodology on shaky ground?

Large language models (LLMs) are the culmination of a particular school of thought that originated in the 1950s, when the first computer scientists set out to define what, exactly, artificial intelligence is. Dubbed “subsymbolic AI,” the original approach was to mimic the complex neural interconnectivity of the human brain through a technology known as Artificial Neural Networks (ANN). It was statistical in nature, leveraging algorithms to mathematically extract predictive insights from large sets of data.

The subsymbolic and empirical schools—which include ​​neural networks, regression models, ensemble models, and Bayesian inference—have been dominant since the early 1990s, largely because it seems to work. Its powers of prediction yield concrete, measurable insights to a degree successful enough to market.

But early pioneers like Alan Turing had a different idea. They identified language as the ultimate test of artificial intelligence, because language is the only thing unique to the human species. Thus, a competing approach to AI sought to develop a hierarchical structure of symbolic representations, the idea being to simulate the complex dance between language, thought, and reasoning. That dance is what we call intelligence.

Empirical vs. Rational

Walid Saba, senior research scientist at the Institute for Experiential AI (EAI), sees himself caught between these two schools of thought, which he likens to the dueling methodologies of empiricism and rationalism. In the philosophy of mind, rational methods are those associated with the logic of conceptual structures in the mind. Empirical methods, meanwhile, derive knowledge from experience and experimentation, which means they do not assume any conceptual structure—a feature Saba believes is a core facet of human language.

“There is no notion of concepts or hierarchy of concepts,” he explains, offering an example from botany. “An apple is a fruit and a fruit is a plant and a plant is a living thing—there's none of that with subsymbolic AI. It's all about a bottom-up, data-driven, empirical, quantitative, statistical method, which is fine for pattern recognition tasks, but to try to discover how we think and how we cognize, using this method is futile.”

Some of that futility can be seen in the products of subsymbolic AI: ChatGPT, for example, with its tendency to lie, misinform, and hallucinate—not to mention a frustrating inability to explain its outputs.

In symbolic AI systems, on the other hand, objects can have attributes and properties. For example, a person’s age has a value all its own, whereas in a neural net LLM that attribute is distributed—stored and represented as “microfeatures” across thousand of digital neurons that come together to determine an “age.” And this gets to the heart of AI’s explainability problem.

“The micro level is really unexplainable because you can't go to the network and point at the age attributes,” Saba says. “There is no place in the network where you have the age of a person. In symbolic systems, it's easy to have a database with a column for age, a column for gender. These are symbolic systems.”

Moreover, Saba argues that the empirical approach to AI hinges on a belief that everything can be learned—hence, the pervasiveness of the term “machine learning.”

“Logic says, if everything is learned from sensory input, then I'm allowed to learn it differently from you, because we have different experiences. But that’s not true. Many of the most important things in life we're not allowed to learn differently—for example, that the circumference of a circle is 2π(r). Things that we learn are accidental and immaterial. The stuff that matters, the stuff that makes the universe the way it is, are not learned. They are.”

Bottoms Up

Judging by Saba’s critique of subsymbolic systems, you would think he would find a home alongside other critics of neural net LLMs like Gary Marcus and Noam Chomsky. And, in the divide between rationalism vs. empiricism, Saba is definitely in the former camp with Marcus and Chomsky. However, he does differ in a way that brings him closer to the empirical camp, and it has to do with data.

For all the limitations of the empirical method, its bottom-up methodology allows for sophisticated data analysis and learning, because the system is working from scratch. It learns from whatever data is at its disposal. Meanwhile, the top-down approach emblematic of symbolic AI is flawed because there is no foundational axiom (or agreed upon general principles) to work from—at least not when it comes to language and how our minds externalize thoughts as language.

“When it comes to language and the mind, we have nothing,” Saba says. “Nobody knows anything about the mind and how the language of thought works. So anybody doing top-down is playing God.”

That being said, language seems to operate on a common understanding between people, which suggests the existence of an underlying system. When a waiter says, “The corner table wants another beer,” we understand that the table itself is not expressing a desire for beer; it’s the people sitting at the table. Computers, to the extent they can be said to understand anything, cannot make that common sense leap. And the ambiguity of language—the messiness of it—is actually a reflection of its efficiency. Saba calls this the “missing text phenomenon:”

“We don't say everything, because I can assume you have common sense and you know the rest.”

So how do we impart that understanding to AI? For Walid Saba, it’s all about melding these two schools: symbolic and subsymbolic. Saba calls his approach, “bottom-up, symbolic reverse engineering of language at scale.” Essentially, the idea is to combine the bottom-up approach from the subsymbolic school with the hierarchy of concepts from the symbolic school. Indeed, despite decades of sluggish growth in symbolic AI research, there has been a recent resurgence. And ChatGPT, with all its flaws, might have something to do with it.

We can still ask the question, are we playing God? And we can still ask if we’re toiling with a basic misunderstanding of what language is. But who knows? Success often seems to follow the middle path.

Learn more about Walid’s research here, or read his recent white paper on the explainability problem in neural networks.