The power of LLama - Part 4: Does size really matter?

TL;DR. Parameter count is like engine displacement in a car: it sets a ceiling, but it does not decide who wins the race. Real model strength comes from how effectively those parameters are used: architecture, training data, optimization techniques, post-training, and reasoning capabilities.

I keep watching teams pick a model the way a teenager picks a first car: by the biggest number on the spec sheet. Bigger must be better.

That instinct is understandable, and it is also how you end up paying for capability you never use, or shipping something that feels sluggish in the one workflow that matters to you.

So let me walk through what really matters when evaluating a model.

Parameter count is engine displacement, not horsepower

Parameter count is the engine displacement of an AI model.

Just as a larger engine has the potential to burn more fuel and produce more power, a larger neural network has the potential to represent more complex relationships and more nuanced behavior.

Potential is the key word here.

Let's take a look at two models: Llama3.1-70b and Qwen3.6-27b. The former is a behemoth, having 70 billion parameters (about 70x larger than its small cousin, our old friend Llama3.2-1b). The latter is an Alibaba's offering, amassing a much more modest 27 billion parameters.

On paper, this shouldn't even be close. One model has over twice as many parameters as the other. If parameter count were all that mattered, Llama3.1-70b should dominate every benchmark.

It doesn't.

The thing is, there is a generational leap between Llama3.1-70b and Qwen3.6-27B: the former was released about 2 years before the latter.

Comparing parameter counts across different generations or different model families is like comparing the horsepower of a modern turbocharged engine with a 20 years old naturally aspirated engine. More modern techniques can dramatically increase what each parameter accomplishes.

That is why a modern 27-billion-parameter model can outperform a 70-billion-parameter model using an outdated architecture. The newer model isn't breaking the rules. Each of its parameter is simply being used more effectively thanks to improvements in architecture, training, optimization, and post-training.

Every token sends the model back to its weights

In the first article of this series, we followed the model as it completed the sentence:

The cat jumped onto the table because it was scared.

We learnt that the model referenced every word in the context when generating an answer. This algorithm - the attention algorithm - is by its nature quadratic and one of the main reasons as to why inference is so expensive.

That was only part of the problem.

In reality, on top of comparing to every other token in the context, the model must compare each token to every parameter in the model.

Every single one of them.

For instance, to predict jumped, the inference engine loads the model's parameters from memory and performs billions of mathematical operations.

To predict onto, it does it again.

Then again for the.

Again for table.

Again for because.

Again for "it".

Again for "was".

And finally, again for "scared".

The memory wall

You may be wondering that this would required an enormous amount of computing power. And you'd be right. But the thing that surprised me the most is that computing isn't the real bottleneck.

The real bottleneck is memory bandwidth.

Let's put some numbers on it. Qwen3.6-27b has 27 billion parameters. In full precision (2 bytes per parameter), these parameters alone occupy roughly 54 GB of memory.

To generate a single token, the inference engine has to read almost all of those weights once. If the model generates 30 tokens per second, that translates into approximately:

54 GB × 30 ≈ 1.6 TB/s

Let's take today flagship NVidia GPU: the almighty GeForce RTX 5090. It has, according to its specs, about 1.8 TB/s of memory bandwidth. By these numbers, this GPU can only output about 30-ish TPS.

Ok, you might argue that this is a consumer grade GPU, and a datacenter GPU would have much more memory bandwidth.

Let me humor you. At the time of writing, the B300 is the top of line datacenter GPU from NVidia. It comes in a rack of GPUS which has combined 8.0 TB/s of memory bandwidth. So this gargantuan GPU would at most generate about 120 TPS.

Not what you would expect of multi thousand appliance.

But it gets (way) worse.

Models such as GLM 5.2 boast hundreds of billions of parameters (753B, to be precise). Running something this size at 30 TPS would need approximately:

1.50 TB × 30 ≈ 45 TB/s

In this hellish scenario, even the beast of a GPU the B300 is would output about 6 TPS.

But how then can we have models this size running?

How we cheat physics

I have to confess that I wasn't completely honest in the previous section. It is not like the math is wrong or that is a theoretical problem. The thing is:

Nobody runs a model in full precision.

In the real world, we almost never deploy the model in full precision. Most of the time, we use 1 byte per parameter instead of 2. But often, we use even less: the models we used in ollama are quantized to 4 bits per parameter (aka Q4)

💡 We call the process of changing how much memory a parameter occupies quantization. While it is out of the scope of this series to delve into its details, you can read more about this here.

But quantization is only part of the equation. Let's take the last example. Even if we divide the memory bandwidth by four by going Q4, we would end up with 21.3 TPS. A far cry from the performance we see in the wild.

There must be something else at play.

MoE vs Dense

Up until now, we’ve talked about neural networks as single, monolithic blocks of brain. every single word you type forces the GPU to load every single parameter in the entire network. We call those models dense models.

Massive models like DeepSeek and GLM operate differently. Instead of one massive brain, they are organized like a small organization made up of small, highly specialized teams ("Experts"), coordinated by a manager ("The Router").

We call these kind of model a Mixture-of-Experts (or MoE). They are the real reason we can run a model with hundreds of billions of parameters without waiting five seconds per word is a massive architectural shift.

When you feed a token into an MoE model:

1. The router looks at the token;
2. It quickly determines which specific experts are best suited to handle it;
3. It activates only those specific experts for that token;

⚠️ It is natural to think of an expert as a piece of the model that is good at a particular task (e.g., coding, writing). In practice experts aren't divided based on the task. The router and the experts co-evolve during training, and the divisions that emerge are whatever the optimizer finds useful for minimizing loss, which turns out to have almost nothing to do with the tidy subject-matter boxes we'd unconsciously draw.

When we are dealing with MoEs, we have to track two different numbers on the spec sheet: Total Parameters and Active Parameters.

GLM 5.2 might have over 700 billion of total parametes sitting in the memory but only activate a fraction of that (40 billion) per token. Another good example is DeepSeek V4 Pro. While it boast 1.6 trillion parameters, only about 49 billion are active per token.

Let's revise the math we did for GLM, now taking into account it is a MoE. It uses only 40B per token during inference. So it would need to achieve 30 TPS, it would need:

80GB × 30 ≈ 2.40 TB/s

It still needs a massive amount of memory bandwith. But in this case, the B300 would yield about 100 TPS. Even the 5090 RTX would output a passable 15-ish TPS if it had enough memory fit the model.

💡 Some models like Gemma4 and Qwen3.6, state the number of parameters (total and active) in their name (e.g., Qwen3.6-35b-a3b means 35b total parameters with 3b active while Gemma4-26b-a4b means 26b total parameters with 4b active).

Talking about Qwen, let's see how the MoE and dense -- Qwen3.6-35b-a3b and Qwen3.6-27b, respectively -- compare to each other. The MoE has substantially more parameters than the dense version, so it should perform better, right ?

🔍 Check out this video by TokenChaser comparing both the dense and MoE versions of Qwen in assorted tasks.

WTF? Why doesn't a 35-billion-parameter MoE completely crush a 27-billion dense model? It has 8 billion more parameters under the hood afterall.

The fact is that, like many things in life. MoEs are a tradeoff. We are roughly trading 20% of reasoning power for 90% of savings in inference cost.

Take Qwen3.6-35b-a3b for instance, while it has 35 billion parameters to draw from, it only activates 3 billion of them per token (roughly, for each given token, it acts like a highly optimized, dynamic 3-billion-parameter model on steroids). Its dense counterpart, on the other hand, uses all the cognitive power of its 27B parameters for each token. This reflects in the latter's more cohesive output.

So far, we’ve looked at how engineers squeeze more performance out of parameters. But there's another way to get more out of your parameters that has nothing to do with the GPU, and everything to do with behavior

Thinking vs. Instruct: Nice guys finish last (and better)

If we look at Gemma4's output in the last article, it looks a little different from the output on Llama. In the latter's case, the answer seems direct. In the former's, however, there is thinking block. If we click on that block, we are presented the rationale the model used to arrive to its conclusion.

The models outputs differ due to their paradigms. Up until recently, all LLMs behaved like LLamma. They acted like contestants on Jeopardy: the second the host finishes asking a question, the contestant is forced to start speaking.

It mirrors a form of cognition that Daniel Kahneman calls System 1 thinking, where the person acts quickly, instinctively, and is highly reliant on immediate pattern recognition.

Models that use this form of reasoning follow the Instruct paradigm.

Instruct models are fantastic for quick language translation or writing basic email templates. But they fall short when trying to solve complex problem: since they generate the very first word of its answer immediately, they must commit to a logical path before it has actually planned how the sentence will end. If they make a logical error early on, they cannot backtrack and they are forced to awkwardly hallucinate their way forward.

Kahneman also introduced another form of thinking where the person approach to problem solving is slow, analytical, deliberate, and logical. He called this approach System 2 thinking.

Reasoning models mimics System 2 form of thinking: instead of shouting out the first token that pops into its weights, a reasoning model is allowed to pause, draft a plan, and verify its logic before saying a word.

But how does it reason?

Here's the thing: they don't 🤡.

A reasoning model isn't running some different algorithm under the hood. Strip away the branding, and it's still the same next-token prediction machine as we know. Same transformer architecture, same "guess the next word" mechanic.

What differs is how they are trained and how that shapes how they give their answers.

Same weights, different habits

An instruct model is trained to go straight from question to answer and is rewarded by each correct answer. A reasoning model goes through an extra stage: it is rewarded by both the answer being correct and for producing a chain of intermediate reasoning that leads to a correct answer (much like us when we were at school). Over thousands of these training examples, the model learns a habit: before committing to a final response, generate a bunch of exploratory text (restate the problem, try an approach, notice a mistake, try again, converge on something) and only then write the answer.

So the model isn't "thinking" in some mystical sense. It's just been trained to write a draft of its reasoning first, because empirically, models that do that before answering get it more right.

There is no free lunch

So if reasoning models think before they answer, and this makes them more accurate, why not make every model a reasoning model and call it a day?

Because that pause, as everything in life, isn't free.

It's paid for in tokens.

Remember the memory wall from earlier: every single token, whether it's part of the final answer or part of the model's internal reasoning scratchpad, forces the model to check its weights. A reasoning model multiplies this cost: before it writes the first word of what you actually wanted, it might generate hundreds or thousands of tokens working through the problem.

All of that "thinking" has to be generated one token at a time, same as everything else. So a question that an instruct model answers in 20 tokens might take a reasoning model 2,000 tokens to arrive at the same answer. You're not getting a smarter model for free. You're trading latency and compute for accuracy.

This is why asking a reasoning model "what's the capital of France?" feels like massive overkill, because it is (to answer that question, Gemma4:e2b used 114 tokens).

The Homer Simpson Crayon Trick

🤓 In the classic Simpsons episode HOMR, Homer has a crayon lodged in his brain since childhood, removing it makes him a genius, but being smart makes him miserable (he becomes painfully aware of how dumb everyone around him is), so he puts the crayon back in to go back to being happy (and dumb).

Reasoning models have their own version of Homer's crayon. You can dumb them down by selectively disable their reasoning.

In our stack (Ollama + Open WebUI), we can do this by:

1. Opening the Chat Controls (the sliders icon on the top right side of the chat);
2. Expanding the Advanced Parameters section;
3. Locating the think (Ollama) option and setting it to Off.

💡 Some models and inference engines allow a more fine grained control of how much thinking a reasoning model does. Sadly, ollama just allow us to turn it on or off.

If we ask our model what's the capital of france once again, we'll get the following response.

Notice the response doesn't have a throught process when reasoning is disabled: the model chose to answer the question directly. The same model, different reasoning types. More importantly, the answer required only 9 output tokens instead of 114. That's over 12× fewer generated tokens to produce exactly the same answer.

That's the real tradeoff behind reasoning models. You're not paying for a better answer every time: you are paying for the opportunity to produce a better answer when the problem actually requires deeper thinking.

Conclusion

We are conditioned to look at the single biggest number on a spec sheet and assume it tells the whole story. But as we've seen, parameter count is just a like the displacement of an engine. While it can hint how powerfull is an engine, how it actually performs on how it is built and driven.

Choosing a model isn't about finding the biggest brain. It's about matching the model's architectural tradeoffs to the specific constraints of your workload.

Where do we go from here

Notice something about that thinking block we've been toggling on and off: it's fenced. The model draws a clean line between "here's my scratchpad" and "here's my answer," and it does that on command, every single time.

If a model can reliably wall off its reasoning into its own tagged section, what stops it from wrapping its final answer in structure too — not prose, but JSON, a table, a schema you define? And once your model's output is something a program can parse instead of something a human has to read, what would you build with that?

FAQ

Does a higher parameter count always mean a smarter model?

No, it’s like engine displacement: A massive parameter count sets a high potential ceiling, but if the architecture is outdated or the training data is garbage, a much smaller, newer model will easily outperform it. A modern 27B model will routinely beat a 70B model from two years ago because newer techniques squeeze far more logic out of every single parameter.

What is the actual difference between a Dense model and a Mixture-of-Experts (MoE) model?

Think of a Dense model as one massive, monolithic brain. Every single time you type a word, the GPU is forced to load and use every single parameter in the entire network to predict the next word.

An MoE model is structured like a company with a manager (called the Router) and a bunch of specialized teams (the Experts). When you type a word, the Router quickly looks at it, decides which specific Experts are best equipped to handle that specific token, and only activates them. The rest of the model's parameters sit completely idle for that token. Because it only uses a small fraction of its total brainpower at any given moment, an MoE model can be massive in size but incredibly fast and cheap to run.

Are Reasoning models actually "thinking" or using a secret new algorithm?

Nope. Under the hood, they are the exact same next-token-prediction machines. The difference is purely behavioral. During training, they were rewarded not just for the right answer, but for generating a step-by-step chain of thought. They aren't thinking in a mystical sense; they've just been trained to write a rough draft of their logic first.

When should I turn off the "Thinking" mode on a Reasoning model?

Anytime the task is simple. Remember, every token in that thinking block costs memory bandwidth and latency. If you ask a reasoning model "What is the capital of France?" and it spends 114 tokens pondering the history of the French Republic before answering Paris, you are burning compute for absolutely no reason. Turn thinking off for basic summarization, translation, or simple Q&A, and save it for complex logic, coding, or math where it actually pays off.