Ignorance is a choice in the Internet age. Virtually all of human knowledge is available for the cost of typing a few words into a search box.
But managing the deluge of facts, opinions, and perspectives remains a challenge. It can be hard to know what information you’ll find in a lengthy document until you’ve read it, and knowing whether any particular statement is true is very difficult.
Automatic summarization can do a lot to solve these problems. This is one of the most important, yet least solved, tasks in natural language processing. In 2020, summarization will take important steps forward, and the improvement will change the way we consume information.
The Salesforce Research team recently took a close look at the field and published a paper that evaluates the strengths and weaknesses of current approaches. We found that the datasets used to train summarizers are deeply flawed. The metric used to measure their performance is deeply flawed. Consequently, the resulting models are deeply flawed.
We’re working on solutions to these problems. For instance, researchers evaluate summarization performance using the ROUGE score, which measures overlap in words between source documents, automated summaries, and human-written summaries. It turns out that summarizers based on neural networks can make mistakes and still earn high ROUGE scores. A model can confuse the names of a crime’s perpetrator and its victim, for example. ROUGE measures the fact that the names appear in both generated and human-made summaries without taking the error into account.
We introduced a model that makes it easy to examine factual consistency between source documents and summaries. We also proposed a metric to evaluate summarizers for factual consistency. Ranking summarizers according to this metric in addition to ROUGE will help researchers develop better models, and that will speed progress in other areas, such as maintaining logical coherence throughout a long summary.
This kind of development gives me confidence that 2020 will be a great time for summarization, and for NLP in general. The progress I expect to see in the coming year will help people not only to cope with the ceaseless flood of new information, but also to embrace AI’s great potential to make a better world.
Richard Socher is chief scientist at Salesforce.