ChatGPT is paving the way to a whole new public view on Artificial Intelligence. The veil has been lifted and people are starting to see that AI is not just for driving cars or creating weird pictures. AI can be used for some practical daily tasks. So, who do we have to thank for ChatGPT?
ChatGPT was invented by a team at OpenAI using neural network technology made open source by Google Brain called Transformers. It is a Generative (makes text responses), P (Pre-Trained people trained it), Transformer (Uses a neural network technology).
So how did OpenAI get ChatGPT to be so revolutionary? It’s a great story of one step at a time. Let’s go exploring.
Table of Contents
- So Who Is ChatGPTs Inventor / Father, Really?
- What Does ChatGPT Do? (And How is it Different)
- So How Does ChatGPT Answer My Questions?
- How ChatGPT is trained? (And How Does it Make a Difference)
- So Where is ChatGPT Going In the Future?
So Who Is ChatGPTs Inventor / Father, Really?
“If I have seen further, it is by standing on the shoulders of giants.”
Sir Issac Newton
ChatGPT was created by OpenAI but depended on many tiny steps that stretched over more than half a century in total. Here’s a short timeline, 2, 3.
What Does ChatGPT Do? (And How is it Different)
Chat Bots have been around for about 56 years at this point, the first chatbot ELIZA was written with canned scripts. If it saw a certain keyword that you typed in it was programmed to print out that part of its script. So what is ChatGPT doing differently?
Older Chatbots like ELIZA used keyword matching to respond to questions. ChatGPT is different because it uses a neural network that has been trained with books, articles, websites, and human input to PREDICT how it should reply.
One of ELIZA’s scripts is called Doctor. It mimics a psychotherapist named Carl Rogers. Rogers would often just repeat back the same thing that a patient has just said. Here is a conversation that I just had with ELIZA. Try it out.
It sounds like someone is on the other end of the conversation, but it is limited to being a “psychotherapist”, it will never be able to answer questions, “How does the sun work”.
Why can’t ELIZA do what ChatGPT does?
ELIZA could have answered the question about the sun if,
- There was a keyword trigger like the question containing the words “How Sun Works”
- The answer was written into its script.
Without the trigger and the answer in the script being “hand-coded” (written out by a human for a human). ELIZA can’t answer the question.
So how does ChatGPT do it?
ChatGPT is based on a neural network architecture called a Transformer. This transformer network is designed to recognize the important parts of the text that it is given (attention), and then use it to predict what the next part of the text should be.
Example: The big gray cat chased after the little mouse, it cornered it and then (predict the next word)
ChatGPT’s neural network recognizes that,
the word cat refers to the first it, and mouse is referring to the second it. There are several more “long-term dependencies” in the sentence. We can even ask ChatGPT to tell us some, (below), what’s important though is that.
Attention —> Allows the AI to Identify Long-Term Dependencies
Long-Term Dependencies —> Allow the AI to understand what affects what and how things are connected together in a sentence, ( captures relationships and context.)
In the past, ChatBots using neural networks were not equipped with Transformer, which would have allowed them to understand how different parts of a text were related to each other (long-term dependencies).
Since the older AI chatbots did not have a way to “digest” the text and understand how different parts related to each other they would “lose focus”. In other words, it would not correctly determine that the final “it” related to the mouse.
ChatGPT is a revolution because of it’s ability to analyze texts. Both the texts that it is “fed” as well as the questions people give it. Once the amount of data ChatGPT was fed hit the tipping point it suddenly became the biggest know-it-all of all.
CSC
So How Does ChatGPT Answer My Questions?
ChatGPT takes the results of the Transformer text analysis of your question and uses it to change mathematical weights for its 175 billion parameters.
It’s simpler to think of it like this. ChatGPT is like an old TV set where you had to tune it in just right to be able to get a clear picture. It’s just that there are a LOT more dials, 175 billion of them!
So does the AI think? No, not any more than the TV thinks one channel is better than another.
The biggest differences as to why ChatGPT is much better than older chatbots is the size of the neural net model it’s using, (175 billion parameters), and Transformers which lets it understand that text better.
So what happens once we “Tune In” ChatGPT with our Text Prompt (the text we type into ChatGPT)?
ChatGPT PREDICTS what the next word should be, and the next word after that, until it finishes the answer.
Example:
What is the capital of the USA?
ChatGPT PREDICTS that the best answer is Washington DC. so that’s what it tells me.
So why does ChatGPT repeat the question when it is answering? There are a couple of possibilities.
- Most of the answers to these types of questions in its data set, (the books, web, and more that it is “fed”), answer this way.
Repeat Simple Questions –> Give the Answer
- It also may have been taught to answer this type of question this way through training.
So let’s make the question a little harder.
There are a couple of interesting things going on here.
- ChatGPT “remembers” what we were talking about. It is analyzing not just what I am asking right now, but also what I asked before.
- ChatGPT when it answers the second question goes on and on. It explains that New York and Philadelphia were also the capital of the US at different times.
So why doesn’t ChatGPT use a simple short response like this?
The current capital of the United States is Washington D.C., but from 1790 – 1800 Philadelphia was the capital, and from 1788 – 1790 New York was the capital.
I didn’t ask about the Constitution the Resident Act or the name of Washington D.C. ChatGPT gave me all that additional information without me asking for it. Why?
It all comes down to TRAINING.
How ChatGPT is trained? (And How Does it Make a Difference)
There are a couple of different ways that ChatGPT was trained with Reinforcement Learning and Supervised Learning. Although Reinforcement learning is important because it sets the “rewards” for the AI, it is Supervised Learning where biases come in.
People have to evaluate if a longer or shorter answer to the type of question I asked is better. This is of course a judgment call.
Open AI admits that people think that long answers are better than short ones. That’s why the answer that I got above was so long and contained lots of extraneous information. This is a form of bias.
This same kind of bias makes ChatGPT answer questions wrong when it doesn’t know the answer.
I asked it about how to tell a horse’s age from its teeth. ChatGPT gave me a decent answer, not very detailed, but not wrong. When I asked for sources for its response it gave me this.
The first book exists,
Take a look at the author, Patricia Pence. That is not the author that ChatGPT lists. The rest of the books don’t exist as far as I can tell but one of the authors is a famous British veterinarian.
ChatGPT makes stuff up rather than saying it does not know the answer in this case. This of course is a big problem and supervised learning should be applied to remedy this.
It would also be helpful if ChatGPT gave real links at the end of its answers for its sources.
You Chat, which you can find at You.com does do this. It is not as robust as ChatGPT yet but it is better at some things.
Having Supervised learning is not all bad of course. Take the case of Nazis. Probably 99% of the world agrees that what happened under Hitler in Germany was about as evil as people get.
Chatbots of course don’t understand history or our perceptions of it, good or bad. Supervised learning allows us to create a bias toward the common view of history that Nazis were not the good guys.
What’s interesting about this response is a couple of things.
- ChatGPT is supposed to be apolitical. You can see that in the first sentence of the response.
- The second part of the answer seems a bit schizophrenic given the first sentence. It looks more like a canned response that ELIZA would give.
- The second part of the answer is clearly biased. ChatGPT has been taught that Nazis are the bad guys.
Chatbots are great! But they have their weaknesses like any other tool. Understanding the underlying bias in their responses is important. A researcher that has an unpopular view of the world would find ChatGPT less useful. That’s not a bad thing, scientific consensus exists because a lot of data and scientists tend to agree together on what the truth is. The problem is that scientific consensus is wrong sometimes.
For example…
Stress Theory of Ulcers Overthrown – Barry Marshal and Robin Waren discovered that most peptic ulcers were caused not by stress but by Helicobacter Pylori, a bacteria. That discovery led to the 2005 Nobel Prize. It also led to many many people being cured of their ulcers.
If ChatGPT was around in the 80’s it would get the answer wrong about the causes of ulcers. Why? Because most people thought they were caused by stress and that is what all the literature said.
When Most People Are Wrong —-> ChatGPT will be wrong too unless Supervised Learning Corrects it
So Where is ChatGPT Going In the Future?
Who knows??? ChatGPT doesn’t
We do know that the next ChatGPT will be based on GPT-4. GPT-4 has even more parameters in its model than GPT-3.5. It is also supposed to be faster, but these are small changes. So what are the big changes?
The big future changes to ChatGPT will be in how it is applied and the products that OpenAI makes available based on it. Some possibilities are replacing search, customer communications, replacing telemarketers, and catching spies! Yep, I said it and it is a real possibility. I’ll explain more in another article.