Distinguished Lecturer Recap: Improving Language Models With Fewer Resources

By: Tyler Wells Lynch

ChatGPT may be stirring nightmarish fantasies of an AI takeover, but most uses of Natural Language Processing (NLP)—the technology behind spell-checkers, spam filters, and chatbots—are fairly anodyne. For years, companies have been using NLP to extract information from documents, drive customer service, and analyze market sentiment.

The problem is that the technology is not as simple as downloading an app and letting it rip. Domain experts are costly and hard to find, and the effort required to produce enterprise-specific solutions is immense. Information extraction (IE)—using AI to comb unstructured text data in order to find and classify them into a database—is particularly challenging. And according to Eneko Agirre, professor of informatics at the University of the Basque Country in Spain, that’s not the only problem.

Making Models Make Sense

As part of the Distinguished Lecturer Seminar Series hosted by the Institute for Experiential AI (EAI), Agirre outlined a few strategies developers use to minimize the legwork needed to train enterprise models. One is through large pre-trained language models (PLMs) that are fine-tuned to specific tasks. Another is through a process known as “prompting.” Rather than predicting outputs from large datasets, prompting models the probability of text to appear within a sequence—an advanced kind of autocomplete that can improve results with fewer resources.

However, like most AI systems, PLMs are still very limited when it comes to inferring knowledge or information. For Agirre, who is also head of the HiTZ Basque Center of Language Technology, improving the underlying infrastructure of LLMs is an open problem. “In the end, the only thing that language models are trained to do well is to find missing words,” he said. “They don't need anything which doesn't lead to better results in getting those missing words.”

This is why so many chatbots and IE tools struggle with common sense inferences that humans take for granted. Agirre’s solution for improving inference comes in the form of a workflow. It combines PLMs, prompting, and a third concept known as textual entailment to yield state-of-the-art performance in IE using only a small fraction of data.

Pre-train, Prompt, Entail

In order to perform well, models need to tackle a variety of complex linguistic phenomena not so easily defined by traditional modeling approaches, such as background knowledge, object relationships, ambiguity, modality, etc. Textual entailment is widely used in these contexts to make models more coherent. “The definition is very simple,” Agirre said. “We say that a text entails a hypothesis if, typically, a human reading the text would infer that the hypothesis is most likely true.”

When combined with prompting and PLMs, manual entailment promises, in Agirre’s words, eight times less effort for domain exerts when developing language models. The promise here is infrastructural in nature: the ability to deliver high-performance language modeling in low-resource settings. And the workflow is interactive, relying on human overseers to bolster the inference capabilities of their models. 

“These developments enable a new paradigm for IE where the expert can define the domain-specific schema using natural language and directly run those specifications, annotating a handful of examples in the process,” Agirre said. “It's now possible to build an information extraction system from scratch with limited effort.”

To learn more about Agirre’s workflow, you can watch a replay of his talk and flip through his slides here. You can also register for upcoming seminars here.