GPT-2

The trend toward ever-larger models crossed the threshold from immense to ginormous. Google kicked off 2021 with Switch Transformer, the first published work to exceed a trillion parameters, weighing in at 1.6 trillion.

The cost of training top-performing machine learning models has grown beyond the reach of smaller companies.

Pretrained language models like GPT-3 have shown notable proficiency in few-shot learning. Given a prompt that includes a few example questions and answers (the shots) plus an unanswered question (the task), such models can generate an accurate answer.

The transformer architecture has shown an uncanny ability to model not only language but also images and proteins. New research found that it can apply what it learns from the first domain to the others.

Language models are starting to take on programming work. SourceAI uses GPT-3 to translate plain-English requests into computer code in 40 programming languages. The French startup is one of several companies that use AI to ease coding.

A language model is helping crisis-intervention volunteers practice their suicide-prevention skills. The Trevor Project, a nonprofit organization that operates a 24-hour hotline for LGBTQ youth, uses a “crisis contact simulator” to train its staff in how to talk with troubled teenagers.

AI may help revolutionize the human diet – or dessert, at least. Google applied AI engineer Dale Markowitz and developer advocate Sara Robinson trained a model to predict whether a recipe is...

Language models like Bert, Ernie, and Elmo have achieved spectacular results based on clever pre-training approaches. New research applies some of those Sesame Street lessons into image processing.

Music that features a “singing” koala bear took the prize in one of Europe’s highest-profile AI competitions yet. A team of Australian programmers, designers, and musicians won the inaugural AI Song Contest with a koala-tinged track called “Beautiful the World.”

An icon of idealism in AI stands accused of letting its ambition eclipse its principles. Founded in 2015 to develop artificial general intelligence for the good of humankind, OpenAI swapped its ideals for cash.

AI is contributing to paintings, music, and now a whimsical fantasy video. The Squire is an amateur romp through a snowy realm of knights in armor and damsels in distress. The script was composed by AI Dungeon 2, an interactive text-adventure game based on the GPT-2 language model.

Earlier language models powered by Word2Vec and GloVe embeddings yielded confused chatbots, grammar tools with middle-school reading comprehension, and not-half-bad translations. The latest generation is so good, some people consider it dangerous.

As language models show increasing power, a parallel trend has received less notice: The vogue for naming models after characters in the children’s TV show Sesame Street.

Will AI fakery erode public trust in the key social institutions? Generative models will flood media outlets with convincing but false photos, videos, ads, and news stories. The ensuing crisis of authority will lead to widespread distrust in everything from the financial system to democracy itself.

Despite dramatic recent progress, natural language generation remains an iffy proposition. Even users of the muscular GPT-2 text generator have to press the button a number of times to get sensible output. But researchers are figuring out how to exert greater control over generated text.