Good reads and perspectives on generative AI and ChatGPT

Ted Chiang, writing for the New Yorker, summarised brilliantly the overall concept of ChatGPT:

Think of ChatGPT as a blurry JPEG of all the text on the Web. It retains much of the information on the Web, in the same way that a JPEG retains much of the information of a higher-resolution image, but, if you’re looking for an exact sequence of bits, you won’t find it; all you will ever get is an approximation.

I really enjoyed reading Chiang’s piece today, and I think the compression analogies used in the article work really well to understand what users are getting after interacting with these fancy new AI chatbots, like the one included in the new Bing. Where the compression for things like code is lossless — where every single bit of data has to be recovered — compression for knowledge (arguably most of what ChatGPT does) tends to be lossy: a bit like a summary of a book is a lossy compression of the story. Chiang goes on:

Imagine what it would look like if ChatGPT were a lossless algorithm. If that were the case, it would always answer questions by providing a verbatim quote from a relevant Web page. We would probably regard the software as only a slight improvement over a conventional search engine, and be less impressed by it. The fact that ChatGPT rephrases material from the Web instead of quoting it word for word makes it seem like a student expressing ideas in her own words, rather than simply regurgitating what she’s read; it creates the illusion that ChatGPT understands the material. […] When we’re dealing with sequences of words, lossy compression looks smarter than lossless compression.

This article is full of quotable parts, and here I’m not going to do a lossy compression by summarising it poorly. I prefer to highlight a few of my favourites and encourage you to go read it.

There is very little information available about OpenAI’s forthcoming successor to ChatGPT, GPT-4. But I’m going to make a prediction: when assembling the vast amount of text used to train GPT-4, the people at OpenAI will have made every effort to exclude material generated by ChatGPT or any other large language model.

Repeatedly resaving a JPEG creates more compression artifacts, because more information is lost every time. It’s the digital equivalent of repeatedly making photocopies of photocopies in the old days. The image quality only gets worse.

This article is really enlightening, and I’m glad to finally have found an article about ChatGPT where the author doesn’t feel compelled to screenshot most of their interactions with the AI.

As Sharon Goldman writes on Venture Beat:

I’m not just another journalist writing a column about how I spent last week trying out Microsoft Bing’s AI chatbot. No, really.

I’m not another reporter telling the world how Sydney, the internal code name of Bing’s AI chat mode, made me feel all the feelings until it completely creeped me out and I realized that maybe I don’t need help searching the web if my new friendly copilot is going to turn on me and threaten me with destruction and a devil emoji. […]

I didn’t feel like riding what turned out to be a predictable rise-and-fall generative AI news wave that was, perhaps, even quicker than usual.

As far as I’m concerned, my biggest hope with this generative AI era is for the web to finally get rid of some of the shitty SEO pages that managed to trick Google’s search results to gain traffic. I would love to witness many of these becoming irrelevant thanks to this new, fast, and complementary way of getting answers and information online.