Distinguished Lecturer Recap: Improving Language Models With Fewer Resources

Eneko Agirre, professor from the University of Basque Country (UPV/EHU), gave a Distinguished Lecturer seminar titled "Few-shot Information Extraction: Pre-train, Prompt, and Entail" on February 22nd, 2023.

By: Tyler Wells Lynch

ChatGPT may be stirring nightmarish fantasies of an AI takeover, but most uses of Natural Language Processing (NLP)—the technology behind spell-checkers, spam filters, and chatbots—are fairly anodyne. For years, companies have been using NLP to extract information from documents, drive customer service, and analyze market sentiment.

The problem is that the technology is not as simple as downloading an app and letting it rip. Domain experts are costly and hard to find, and the effort required to produce enterprise-specific solutions is immense. Information extraction (IE)—using AI to comb unstructured text data in order to find and classify them into a database—is particularly challenging. And according to Eneko Agirre, professor of informatics at the University of the Basque Country in Spain, that’s not the only problem.

Making Models Make Sense

As part of the Distinguished Lecturer Seminar Series hosted by the Institute for Experiential AI (EAI), Agirre outlined a few strategies developers use to minimize the legwork needed to train enterprise models. One is through large pre-trained language models (PLMs) that are fine-tuned to specific tasks. Another is through a process known as “prompting.” Rather than predicting outputs from large datasets, prompting models the probability of text to appear within a sequence—an advanced kind of autocomplete that can improve results with fewer resources.

However, like most AI systems, PLMs are still very limited when it comes to inferring knowledge or information. For Agirre, who is also head of the HiTZ Basque Center of Language Technology, improving the underlying infrastructure of LLMs is an open problem. “In the end, the only thing that language models are trained to do well is to find missing words,” he said. “They don't need anything which doesn't lead to better results in getting those missing words.”

This is why so many chatbots and IE tools struggle with common sense inferences that humans take for granted. Agirre’s solution for improving inference comes in the form of a workflow. It combines PLMs, prompting, and a third concept known as textual entailment to yield state-of-the-art performance in IE using only a small fraction of data.

Pre-train, Prompt, Entail

In order to perform well, models need to tackle a variety of complex linguistic phenomena not so easily defined by traditional modeling approaches, such as background knowledge, object relationships, ambiguity, modality, etc. Textual entailment is widely used in these contexts to make models more coherent. “The definition is very simple,” Agirre said. “We say that a text entails a hypothesis if, typically, a human reading the text would infer that the hypothesis is most likely true.”

When combined with prompting and PLMs, manual entailment promises, in Agirre’s words, eight times less effort for domain exerts when developing language models. The promise here is infrastructural in nature: the ability to deliver high-performance language modeling in low-resource settings. And the workflow is interactive, relying on human overseers to bolster the inference capabilities of their models. 

“These developments enable a new paradigm for IE where the expert can define the domain-specific schema using natural language and directly run those specifications, annotating a handful of examples in the process,” Agirre said. “It's now possible to build an information extraction system from scratch with limited effort.”

To learn more about Agirre’s workflow, you can watch a replay of his talk and flip through his slides here. You can also register for upcoming seminars here.

Latest Posts

Inside Intuit’s Quest to Leverage the “Transformative” Powers of AI

Some companies are only just starting to think about AI. Others have already begun to embed it in their operations. Then there’s Intuit, which has been investing in AI for years. The financial technology company, which is known for products like TurboTax, QuickBooks, and Credit Karma, has a history of getting out in front of […]

Six Leading AI Experts Weigh in on the White House Executive Order

Last month, the Biden administration issued an Executive Order (EO) establishing rules and guidelines on the use of artificial intelligence. Through an assortment of benchmarks, appropriations, pilot projects, and reporting requirements, the order’s stated goals are to preserve privacy, protect vulnerable groups, promote competition, and advance civil rights, among other ideals. Our Directors of research, […]

Chegg CEO Announces A New Age Of Learning With Generative AI

Dan Rosensweig laid out a compelling vision for AI to radically transform the education industry at the Institute’s Leading With AI Responsible conference.  In January Dan Rosensweig, the CEO of education technology company Chegg, met with OpenAI CEO Sam Altman to discuss ChatGPT, which had been making waves since Altman’s company released the chatbot […]

AI and the Environment: Is It Enough to Lead by Example?

AI systems are nothing if not power hungry. Researchers from the University of Massachusetts found training just a single AI model can emit the same amount of carbon dioxide as about 63 gasoline-powered cars driven for a year. Another study estimated that emissions from the Information and Communications Technology (ICT) industry as a whole will […]

Peter Norvig Redefines AI Success with Call For Human-Centered Solutions

At the Institute’s Leading With AI Responsibly conference, the industry pioneer made the case that businesses should focus on AI’s broad societal impact as they develop products. Peter Norvig literally wrote the book on artificial intelligence. The California native is the co-author of one of the most popular textbooks on the subject, “Artificial Intelligence: A […]