2.3K
TILet me explain what AI inference is, how it works, and why AI companies are racing to reduce the cost of it.
You already know from my other videos what AI training is. Training is the learning phase. Inference is everything that happens after that.
Every single time you type something into ChatGPT and it gives you a response, that’s inference. The model is not learning anything new from your message. It’s using what it already learned during training to generate a response in real time.
Think of it like school. Training is when you study for the exam. Inference is when you actually take it.
So here’s what actually happens when you send a message. First, your text gets broken up into tokens. These are small chunks of words or parts of words. Then each token gets converted into a vector, which is basically a long list of numbers that represents the meaning of that token.
Those vectors get fed through the transformer, which is the neural network architecture behind all of this. It processes them through dozens of layers, figuring out how every word relates to every other word. And at the end, it predicts the most likely next token. Then it does that again and again until you get a full response.
All of this happens in seconds.
Inference is much cheaper and less compute intensive than training. But when you’re doing that for millions of users at the same time, the cost adds up fast. That’s why AI companies are racing to make inference faster and cheaper, and building specialized chips to do it.
So when someone says a model was “trained,” that’s the learning phase. When you actually use it, that is the inference phase.
This video is part of a new series where I explain common tech terms. If you like content like this, definitely follow along for more.
@timothybramlett
