"Too dangerous to release"
OpenAI trained a much bigger language model and is holding back the full version. The unicorn story is impressive. What's actually dangerous is cheap, plausible text at scale.
OpenAI announced GPT-2 on Thursday. It’s the same basic idea as last year’s GPT, a Transformer trained to predict the next word, scaled up to 1.5 billion parameters and trained on about 40GB of text from web pages linked from Reddit posts with some upvotes. They’re releasing only a small version of the model and holding back the full one, saying they’re worried about misuse.
The headline sample is a news story about scientists discovering a herd of unicorns in the Andes that speak perfect English. You give it two sentences and it writes nine coherent paragraphs, with invented quotes from an invented professor. It’s funny, and it’s genuinely far better than anything I’ve seen a language model do. The text stays on topic across paragraphs, keeps characters consistent, and sounds like a real article. They say it took ten tries to get that sample, and that failures include repetition and topic drift.
The other result I find more interesting technically: without any fine-tuning, GPT-2 does reasonably on reading comprehension, summarization and translation benchmarks, just by being prompted in the right format. It’s not state of the art on most of them. But nobody trained it for those tasks. It picked them up because the web contains examples of them, and predicting the next word well means learning to continue those patterns.
Many researchers are annoyed about the withheld model, calling it a publicity stunt or a break with open research norms. I understand both sides. I think the specific danger OpenAI is pointing at is real, but it isn’t about any single scary output. It’s about cost. A human writing convincing fake reviews, fake comments or fake news stories costs money per piece. A model that writes endless plausible text in any style costs nearly nothing per piece. When the cost of plausible text drops to zero, the problem is volume: review sites, comment sections and search results filled with text that looks human and isn’t.
Holding back the model will delay that by months, not years. Someone will reproduce it; the recipe is public. The more useful response is to start building for a world where text alone doesn’t prove anything, the same way StyleGAN’s faces mean a photo alone doesn’t prove anyone exists. Platforms will need provenance, rate limits, account history, and some way to know a real person is behind an account.
I’d also bet that this scaling curve keeps going. GPT to GPT-2 was roughly ten times the parameters and a big jump in quality with no new ideas. If that holds for another factor of ten or a hundred, the few-shot abilities in this paper won’t stay a curiosity.