Single Headed Attention RNN (SHA-RNN)

The latest large, pretrained language models rely on trendy layers based on transformer networks. New research shows that these newfangled layers may not be necessary.