Which headline was written by a computer?
A: FIFA to Decide on 2022 World Cup in March
B: Decision in March on 48-team 2022 World Cup, Says Infantino
What’s new: Researchers at Primer, an AI-driven document analysis company, introduced an NLP-powered automatic headline generator. In an appealing twist, some articles that human publishers had tried to tart up with clickbait — for instance, You’ll Never Guess Which U.S. Counties Grew the Fastest — the model gave a sensible, informative headline: MacKenzie County in North Dakota Had Highest Population Growth in Entire U.S.
How it works: A headline is a very short document summary. Summarizers come in two flavors. Extractive models use only sentences or phrases from the text itself, building summaries that are closely related to the source but may be narrow or off-point. Abstractive models create new text based on an independent dictionary, synthesizing fresh but potentially confused summaries. Primer developed a hybrid model that generates abstractive headlines using vocabulary found in the document.
- The authors fine-tuned a Bert Question-Answer model from Hugging Face on 1.5 million news story/headline pairs drawn from sources including the New York Times, BBC, and CBC.
- The model frames headline generation as a series of question-answer tasks. The question is the beginning of the headline and the answer is the passage that makes up the next part. The model iterates this process sequentially through the document.
- The researchers also adapted the model to create bullet-point summaries of news articles, financial reports, and even movie plots — though not perfect. For instance, it declared that the character named Maverick in the 1986 Tom Cruise hit Top Gun enters a romantic relationship with his co-pilot Goose, rather than his instructor, per the actual plot.
Results: Human evaluators each read 100 news stories and graded two accompanying headlines, one written by a person and the other by the model. The computer-generated headlines scored slightly better overall. The model performed best on short-form journalism but stumbled on longer articles, probably because key information in longer items is more spread out.
Behind the News: Earlier headline generation methods mostly use an encoder-decoder to produce abstractive results. Unlike the new model, the encoder-decoder approach can generate any possible headline but risks poor grammar, factual inaccuracy, and general incoherence.
Why it matters: Imagine a world without clickbait!
We’re wondering: The computer wrote option A. Did you guess correctly?