OpenAI release of GPT-3

<p> I have seen a lot of articles about <strong> GPT-3 </strong> , a new generative pre-training transformer <sup id="fnref:1"> <a class="footnote-ref" href="#fn:1"> 1 </a> </sup> . It is, fundamentally, an unsupervised learning algorithm that was trained on a gigantic corpus <sup id="fnref:2"> <a class="footnote-ref" href="#fn:2"> 2 </a> </sup> of 570GB of compressed plain text. Includes all of <a class="wikilink" href="/wikipedia/"> Wikipedia </a> in English, corpora of books and something called the <em> <a class="wikilink" href="/commoncrawl/"> CommonCrawl </a> </em> . </p> <p> The interesting thing is that <a class="wikilink" href="/openai/"> OpenAI </a> released the tool as an <a class="wikilink" href="/api/"> API </a> that <em> some </em> people could use to build upon. Very quickly, people came up with tools that could build websites based on simple descriptions such as "A website with a search bar, the Google logo on Top and two buttons below, one saying search and the other I'm feeling lucky". </p> <p> The true advantage of GPT-3 comes from not being trained for a specific task as many previous <a class="wikilink" href="/artificial_intelligence/"> artificial intelligence </a> models. In their paper they claim that regardless of this, the model already performs as good as models trained for specific objectives. You may think about translating from a language to another, or generating paragraphs of new text as specific tasks. </p> <p> To my understanding, what makes GPT-3 very powerful is that the parameters where optimized using a gigantic dataset, but it can also be trained on specific (non open) datasets. For example, we could feed all the customer service chats to make it domain-specific, but using the accumulated knowledge. In this way, a company could build much more intelligent chatbots for customer service. </p> <p> Of course, news about <a class="wikilink" href="/machine_learning/"> machine learning </a> and such always brought up the same problems regarding fears of <a class="wikilink" href="/losing_jobs_in_the_hands_of_artificial_intelligence/"> losing jobs in the hands of artificial intelligence </a> . A lot of people <sup id="fnref:3"> <a class="footnote-ref" href="#fn:3"> 3 </a> </sup> are still struggling to understand the long-term picture. </p> <div class="footnote"> <hr/> <ol> <li id="fn:1"> <p> https://openai.com/blog/better-language-models/ <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text"> ↩ </a> </p> </li> <li id="fn:2"> <p> https://arxiv.org/pdf/2005.14165.pdf <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text"> ↩ </a> </p> </li> <li id="fn:3"> <p> https://nesslabs.com/gpt-3-future-productivity?utm_source=Maker+Mind&amp;utm_campaign=a8df5142e4-MAKER_MIND_053&amp;utm_medium=email&amp;utm_term=0_d634852cd6-a8df5142e4-130742116&amp;mc_cid=a8df5142e4&amp;mc_eid=22b39e4ac7 <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text"> ↩ </a> </p> </li> </ol> </div>