GPT, short for TheGenerative Pre-training Transformer, is a language model written by Alec Radford and released by Elon Musk’s artificial intelligence research laboratory OpenAI in 2018,media MSPoweruser reported. It uses a generative language model (two neural networks perfected each other through competition), and through the pre-training of different set of written materials and long serial text, it can acquire world knowledge and handle long-range dependencies.
GPT-2, announced in February 2019, is an unsupervised conversion language model, trained in 8 million documents with 40GB of text, from articles shared via Reddit. Elon Musk is reluctant to release it because he fears it could be used to send fake news to social networks.
In May 2020, OpenAI released GPT-3, a model that contains two orders of magnitude (175 billion vs 1.5 billion parameters) than GPT-2, a significant improvement over GPT-2.
Given any text prompt, GPT-3 will return a text finish that attempts to match the user’s pattern to it. Users can “program” it, just show it a few examples that they want it to do, and it will provide a complete article or story, such as the following text, written entirely by GPT-3.
GPT-3 has achieved strong performance on many NLP datasets, including translation, question answering, and cloze tasks, as well as tasks that require immediate reasoning or domain adaptation, such as using a new word in a sentence or performing a three-digit operation. GPT-3 can generate a sample of news articles that human evaluators can not distinguish.
The last app has been worrying OpenAI. GPT-3 is currently available as an open beta version and is expected to eventually launch a paid private beta. OpenAI says it will terminate API access for clearly harmful use cases such as harassment, spam, radicalization, and more. While the most obvious threats are those who produce written works, such as screenwriters, AI developers have found surprising applications, such as using GPT-3 to write code.
Sharif Shameem, for example, has written a layout generator where users describe what they want in plain text, and the model generates the appropriate code.
Jordan Singer also created a Figma plug-in that allows people to create applications using plain text descriptions.
It can even be used to diagnose asthma and prescribe medicine.
Other applications are used as a search engine or a god, and can even be used to explain and extend difficult concepts.
While it seems likely that this approach could lead directly to a universal artificial intelligence that can be understood, reasoned and talked about like humans, OpenAI warns that they may encounter fundamental expansion problems, with GPT-3 requiring thousands of petaflop/s-day calculations, while the full GPT-2 has only a few dozen petaflop/s-day.
Read more about GPT-3 on GitHub.