Myself and others working with UChicago’s XLab just released a preprint analyzing the effects compute scales on the benefits of various algorithmic advances in language modeling. Take a look here!