A new approach aims to cure text generators of their tendency to produce nonsense.
What’s new: AI21 Labs launched Jurassic-X, a natural language processing system that combines neural networks and rule-based programs. Jurassic-X weds a large language model with modules that supply up-to-date facts, solve math problems, and process special kinds of input.
How it works: Jurassic-X is built on a software infrastructure called Modular Reasoning, Knowledge and Language (MRKL) that incorporates a variety of programs. AI21’s Jurassic-1, a large pretrained transformer model, performs general language tasks. Specialized modules include a calculator and programs that query networked databases such as Wikipedia, as well as a router that mediates among them.
- The router is a trained transformer that parses input text and selects modules to process it. It includes a so-called prompt generator, also a transformer, that adjusts the input to suit a particular module. For instance, it may rephrase input text to match a language template that Jurassic-1 performs especially well on, such as, “If {Premise} is true, is it also true that {Hypothesis}?”
- To use the calculator, the router learned to extract numbers and operators from randomly generated math expressions rendered in English, such as “what is fifty seven plus three?” or “how much is 5 times the ratio between 17 and 7?”
- Given an open-domain question, a modified passage retriever determines the most relevant Wikipedia articles and a reranker scours them for pertinent passages. It sends the passages along with the input to Jurassic-1, which answers the question.
- To fine-tune Jurassic-1’s performance in some tasks (including Natural Questions), the system feeds input to Jurassic-1, modifies the language model’s representation through a specially trained two-layer transformer, and routes the modified representation back to Jurassic-1 to generate output.
Why it matters: Current neural networks perform at nearly human levels in a variety of narrow tasks, but they have little ability to reason (especially over words or numbers), are prone to inventing facts, and can’t absorb new information without further training. On the other hand, rules-based models can manipulate meanings and facts, but they fall down when they encounter situations that aren’t covered by the rules. Combining a general language model with specialized routines to handle particular tasks could yield output that’s better aligned with the real world.
We’re thinking: Humans frequently resort to a calculator or Wikipedia. It makes sense to make these things available to AI as well.