The power of LLama - Part 3: The facts and the reason

This is the third weekly post in a series on running large language models locally. Do not forget other installments of the series:

• Introductory post;
• The Brain, the Engine, and Your First Llama on Ollama.
• From terminal to a ChatGPT style chat

TL;DR. LLM hallucinations happen because facts are a side effect of training, not a stored database. The model predicts the next token by probability, so when the right facts aren't represented in its learned parameters, it guesses fluently. Providing the correct context greatly reduces hallucinations because it shifts the probability toward the correct answer.

Start with a question the model gets wrong

Let's put our local Llama model to the test.

Question: Did the British government heavily ration the use of plastic during the First World War?

The model answers instantly and gives me a confident answer, citing the role of a so called Plastic Department within the Ministry of Munitions in introducing measures to control the usage of plastics.

And it is also utterly wrong.

The British did not ration plastics during WWI due to the fact that plastics didn't really exist. While early synthetic materials like Bakelite and Celluloid were used in limited military applications, they were not produced on a scale that required public or widespread industrial rationing.

That's the uncomfortable truth about these systems. A wrong answer and a right answer look identical in tone. The model has no built-in signal that says "I'm improvising now."

We call this phenomenon Hallucination.

What a hallucination actually is

A hallucination is when a language model produces text that is fluent, plausible, and false. It's not lying, because lying requires knowing the truth. It's not a bug in the usual sense either. It's the model doing exactly what it was trained to do: continue the text in the most probable way.

People expect a model to work like a search engine with a database behind it. It doesn't. There is no lookup table of facts inside the weights. When you ask a question, the model isn't retrieving an answer. It's generating one, token by token, based on statistical patterns.

How a model generates text (redux)

If you read the first article of this series, you already know that an LLM generates text one token at a time. At every step, it looks at everything that came before and asks a surprisingly simple question:

Given everything written so far, what should come next?

The important detail is that there is never just one possible next token.

Let's go back to our question:

Did the British government heavily ration the use of plastic during the First World War?

Before producing its first word, the model internally considers many possible continuations, each with its own probability.

These numbers are only illustrative, but they capture what happens inside every LLM. The model doesn't retrieve a stored fact from a database. Instead, it estimates which is the statistically most likely word given everything it has learned during training and its current context.

Suppose "Yes" wins. It immediately asks the same question again:

Given everything written so far, what should come next?

This cycle repeats hundreds of times every second. Each generated token changes the context, which changes the probabilities for the next token, which changes the context again.

Eventually, the model produces an answer like:

Yes, the British government established a Plastic Department within the Ministry of Munitions...

The model never reached a point where it decided to invent a Plastic Department. It simply kept choosing whichever next token appeared most probable at each step. Once the answer started down the wrong path, every subsequent prediction became conditioned on the fiction that had already been written.

This is a crucial insight.

The model doesn't know anything. It isn't reasoning, "I don't know the answer, so I'll make one up.". It's not a matter of epistemics. It is just predicting the most probable continuation of the text based on the patterns encoded in its parameters.

And saying that the British government established a Plastic Department during WWI is the most probably sequence.

Factual accuracy is a byproduct of training

Here's the part most people miss. The model was never trained to be correct. It was trained to predict the next token across a huge pile of text.

During training, the objective is narrow: given some text, guess what comes next, then adjust the weights when you're wrong. Repeat this trillions of times. Nobody labeled facts as true or false.

Factual knowledge emerges as a side effect of that process. The model reproduces a fact that appears often enough and consistently enough in the training. The phrase The capital of France is Paris shows up so many times that predicting that Paris comes after the capital of france becomes overwhelmingly probable.

But here's the crucial point: the goal of training was never to teach the model facts. The objective is simply to learn the statistical rules that make good next-token predictions. A robust AI system should never depend on the facts encoded inside the model.

But how can we fix this?

Context shifts the probabilities

If the root cause of a hallucination is that the model is playing a blind game of probability, the solution is neither to make the model magically smarter nor training the model with extra information.

We only need to change the math of the game.

And we do this by providing context.

Let’s look at what happens when we ask the exact same question, but this time, we paste accurate historical information about plastics during WWI directly into the prompt before the model can answer.

What just happened? We didn't retrain the model. We didn't alter its weights. We didn't change a single line of its neural code.

By injecting the correct information into the prompt, we shifted the statistical landscape. Because the phrases didn't exist in WWI and plastics are now sitting right there in its short-term memory, the tokens did not suddenly becomes the overwhelmingly dominant mathematical choice.

From there, the cascade effect works for us instead of against us. The model is guided down a path of factual accuracy, not because it suddenly "knows" history, but because the truth has become the path of least statistical resistance.

From one fact to a whole library

So injecting one accurate paragraph into the prompt fixed our WWI question.

Great — but that trick only works because we already knew the answer and pasted it in ourselves.

In the real world, you don't want to paste the right paragraph into every single question someone asks. This would defeat the need for a LLM. You want the system to find the right paragraph on its own, every time, out of potentially thousands of documents.

That's the problem RAG was built to solve.

What RAG is, in plain terms

RAG stands for Retrieval-augmented generation. Don't let the fancy name intimidate you. The idea is very simple:

Before the model answers, go find the relevant text and hand it a cheat sheet.

That's it, the whole. No retraining, no touching the model's weights, no magic. Just: look something up, then let the model read it before it answers.

Here's the flow in three steps:

1. You give it a library by point the system at a folder of documents — policies, manuals, articles, whatever contains the facts you care about.
2. When someone asks a question, the system searches that library and pulls out the pieces of text that seem related to the question. Something akin to a smart find that looks for meaning, not just matching words.
3. The relevant passages get pasted directly into the prompt, alongside the user's question (in the exact same way we manually pasted our WWI paragraph earlier).
4. The model then answers with that material on its context.

Notice that nobody touched the model's brain. RAG doesn't teach the model. It just makes sure that, for this one question, the relevant information is close enough that guessing wrong becomes statistically unlikely.

⚠️ Keep in mind that this is extremely simplified view of how RAG works. We'll deep dive into the moving parts (e.g., a vector database works) later on this series.

Build a knowledge base in Open WebUI

We don't need to wire up a vector database and setup a complex stack to implement RAG. Open WebUI has everything we need built in. With it, you can create a knowledge base in a few minutes and have an experience similar to NotebookLM. This time, however, completely local.

Lets walk through it.

Step 1: Open the Knowledge section

In Open WebUI, go to Workspace, then Knowledge. This is where your document collections live. Each collection is a separate knowledge base you can attach to chats or models.

Step 2: Create a new knowledge base

Click Create Knowledge. Give it a clear name and description. I called mine "Information on WWI" so I'd know exactly what's inside. The name matters later when you attach it.

💡 The knowledge base in the example above is private. This means only your user has access to it. It is also possible to make public knowledge bases (all users have access) or only allow selected users to use them.

Step 3: Add data to your knowledge base

After creating a new knowledge base, it is time to add actual knowledge to it. First, we need to navigate to the list of knowledge bases. Follow Workspace > Knowledge and you should a screen similar to screen below.

Then click on the first entry of the knowledge base entries -- Knowledge on WWI. You'll be shown which information belongs to that knowledge base. Let's now add a new piece of information  -- one that informs the model that plastics didn't exist during WWI -- to the knowledge base. Click in the + (plus) button in the right side of the screen and choose Add text context.

💡 Open WebUI knowledge bases supports a plethora of formats such as PDFs, plain text, web pages and etc.

Now let's add a title for this piece of information -- WWI Trivia -- and add a entry explicitly stating that plastics didn't exists during WWI.

Hit Save to save the. You'll go back to the page that shows the information sources this knowledge base holds. Notice that our new entry is now showing.

Let's ask the same question again

Now it is the moment of truth. Let's ask the very same question to the model, but now we'll make it use our knowledge base.

Open a new chat window, and click on the + (plus) button in the left bottom side of the message panel. Chose the option Attach Knowledge and them click on Knowledge on WWI. This will attach it to the given chat window. The screenshot above shows the chat windows with the knowledge base attached.

Let's now ask once more whether the British government heavily ration plastics during WWI.

Now the model did correctly identify that plastics did not exist during that time period. Moreover, it stated that there were no evidence suggesting that plastics were used or ration during this time.

There is another cool addition that is worth checking out. Now the answer contains references. This allows us to check from where the model took each piece of information during its reasoning.

A beefier example

To see if we really understood RAG, lets do another example. Based on a post-mortem of a production incident on Project Phoenix -- a high-throughput financial orchestration platform designed to process real-time ledger updates and downstream accounting notifications -- during a high load period. The post-mortem report is composed of three files.

• infrastructure-logs.md: Contains the observation on what has happened in production during the incident;
• architecture-review.md: A review from a 3rd party principal architect in regards of the architectural decisions of Project Phoenix;
• performance-lead-notes.md: The notes of the SRE lead on their vision on what has happended in the incident.

Building the knowledge base

Open WebUI is capable of holding multiple independent knowledge base. We'll create a new one, called Troubleshooting Database, to hold the post-mortem reports. To do it, just follow the 2nd step on the Build a knowledge base in Open WebUI section.

We could add the post-mortem reports to the knowledge base one-by-one. There is however a cool Open WebUI feature we can explore. It allows us to upload a whole directory at once, with each file in the directory being indexed and made available.

To do it, choose the Upload directory option instead of the Add text content. You'll be shown a dialog box to choose which directory you want to upload. After a while, Open WebUI will have all the content of the directory indexed and ready for use.

💾 You can download the files of the post-mortem report here

Let's ask a question

By looking at the documents we can easily infer that the issue was neither an architectural nor an issue with Kafka's rebalance mechanism. In fact, the issue was the lack of synchronization between the timeouts between the services and the kafka brokers.

Easy peasy for a LLM model with RAG, right ?

Let's try it with our good'ol llama 3.2:1b. Attach the Troubleshooting Database knowledge base to the chat and ask the question below.

Analyze the technical failure of Project Phoenix. Was the architectural choice of synchronous blocking calls the root cause of the throughput drop, or was it an infrastructure misconfiguration?

According to llama3.2:1b, the culprit of the production incident was the synchronous blocking calls. This contradicts information on the performance-lead-notes.md file: the SRE lead is adamant that real culprit is the failure to synchronize the timeouts of the service mesh and kafka.

You may wonder whether the model use all the data sources in its reasoning. However, it is explicitly stated that it used three sources -- the files on the Troubleshooting Database knowledge base.

What is happening? Wasn't RAG supposed to fix this?

The sour truth - RAG doesn't fix everything

Let me be candid, because RAG gets oversold.

RAG fixes hallucinations that come from missing knowledge. If the fact exists in a document and retrieval finds it, the model will almost always use it. That covers most enterprise use cases: policies, product specs, internal wikis, support docs.

RAG does not fix everything:

• If retrieval pulls the wrong chunk, the model confidently answers from bad context. Garbage in, garbage out.
• If the answer isn't in any document, the model can still hallucinate. Add a system instruction like "If the context doesn't contain the answer, say you don't know."

And more importantly. It cannot improve the cognitive power of a lesser model (I'm looking at you, llama3.2:1b), no matter how good it is.

A beefier model

So, you may be wondering if stronger model would do better. If you followed the second article of this series, you must already have a second model on your computer:

The *Gemma4:e2b" model we used for its multimodal capabilites.

Besides being larger than Llama3.2:1b, Gemma4:e2b also belongs to a noticeably stronger generation of language models. While there isn't a single benchmark that compares these two models head-to-head across reasoning tasks, Google's published results show Gemma4 performing strongly on challenging evaluations such as MMLU-Pro (60.0%), GPQA Diamond (43.4%), and BigBench Extra Hard (21.9%). Those benchmarks measure different aspects of a model's reasoning ability:

• MMLU-Pro: evaluates reasoning across a wide range of academic subjects using more challenging multiple-choice questions than the original MMLU (more info here);
• GPQA Diamond: contains graduate-level science questions specifically designed to be difficult even for experts, rewarding careful reasoning rather than memorization (more info here);
• BigBench Extra Hard (BBEH): is a collection of especially difficult reasoning tasks that test logic, multi-step inference, and the ability to solve problems beyond simple factual recall (more info here).

Together, they paint a consistent picture: Gemma4 is generally the more capable reasoning model. In practice, that usually translates into better reading comprehension, more nuanced answers, and a greater ability to connect pieces of information from your knowledge base.

Let's see if that extra capability makes a difference. We'll ask exactly the same question, with exactly the same knowledge base attached.

The takeaway is simple: RAG and the model itself solve different problems. RAG gives the model access to the information it needs, but it's still up to the model to interpret that information correctly. A stronger model can draw better conclusions from the same context, while a weaker one may still miss the answer, even when every relevant fact is sitting right in front of its face.

But what makes a model smarter?

A naive answer would say it is all about the parameter count: since Gemma4:e2b is almost seven times larger than Llama3:1b, so its only natural it is smater.

The reality, however, is a little more complex.

In the next article, we'll answer that question by looking inside modern language models. We'll explore why some models outperform others; what dense and Mixture-of-Experts architectures actually are; and how base, instruct, and reasoning models differ. Understanding those ideas will make choosing the right model far less mysterious and explain why not all LLMs think alike.

FAQ

Does RAG eliminate hallucinations?

No. RAG reduces hallucinations caused by missing information, but it doesn't eliminate them.

If the retrieval step finds the wrong documents, the model will answer using incorrect context. Likewise, if the answer isn't present in the knowledge base, the model may still generate a plausible but incorrect response.

If the model reads the documents, does it memorize them?

No.

The retrieved documents are only part of the model's context for the current conversation. The model doesn't permanently learn or remember that information. The next conversation starts out fresh and the same documents are retrieved again if needed.

Why are some models so much better than others if they all predict the next token?

Because predicting the next token is only the training objective, not the whole story. Modern language models differ in architecture, training data, scale, instruction tuning, and reasoning techniques. Those differences have a dramatic impact on how well they interpret context and solve problems.

That's exactly what we'll explore in the next article.

Can RAG answer questions that require combining multiple documents?

Yes—and that's one of its biggest strengths.

A good retrieval system can provide several relevant passages from different documents. The language model can then synthesize those pieces of information into a single answer, often producing insights that aren't explicitly written in the source documents.