BLOG
PT
← Back to blog

Daniel Garcia

llamafile on Windows: what I learnt from using the same GGUF models in Ollama

Versão portuguesa →

One executable, one model and the promise of making local AI simpler were enough to spark my curiosity. I spent a morning and part of the afternoon trying llamafile, comparing its responses with Ollama and discovering that some of the most instructive problems involved PowerShell and the generated code.

Illustration of a laptop displaying Ollama and a terminal displaying llamafile; between them, a file with a neural-network symbol points towards a command-line icon.

In this article

Part I — The experience

Why I wanted to try llamafile

I came across llamafile in Mozilla.ai’s Open Tools collection and was intrigued by its premise: making local language models easier to run, including the option to distribute a model and the software needed to run it in a single file.

Mozilla provides examples and recommends models to get started with llamafile. But we decided not to follow that route. After all, where’s the fun in simply running the suggested models? ;)

Instead, we chose two models ourselves, with different characteristics and purposes: Qwen3 1.7B for conversation and general tasks, and Qwen2.5 Coder 3B for programming. We wanted to find out whether we could run GGUF models of our own choosing in both llamafile and Ollama, rather than relying on the examples prepared by the tool’s developers.

I was already using Ollama and knew how it worked, so it seemed a useful point of reference. Rather than comparing performance, I wanted to see how easy llamafile was to set up and how practical it would be for everyday use.

My Linux homelab was out of action due to a hardware fault, so I couldn’t repeat the experiment on any operating system other than Windows.

If you’re new to local AI, the article “Local AI: when does running a model on your own computer make sense?” provides some background on hardware and model choices. Here, I’ll focus on what I did with these two tools.

Essential concepts for understanding the experiment

A model is a set of parameters trained to perform particular tasks. In this case, I used models from the Qwen3 and Qwen2.5 Coder families.

GGUF (GPT-Generated Unified Format) is a binary file format developed within the llama.cpp ecosystem to store AI models and make them easier to run locally. A .gguf file contains the model weights, information about its architecture and other metadata needed to interpret and run it, including tokenisation information.

One of its main advantages is support for different quantisation levels, which can reduce file sizes and memory requirements, potentially at some cost to response quality. In this experiment, GGUF files let us run the same models in Ollama and llamafile without converting them between the two tools.

The designation Q5_K_M identifies a quantisation method that reduces the precision used to store model weights, decreasing file size and memory requirements. This reduction may affect response quality, depending on the model and the quantisation applied.

llama.cpp is the inference project underpinning several of these solutions. llamafile combines that foundation with Cosmopolitan Libc and its own tools to make distribution and execution easier. The project’s technical overview describes this combination.

Ollama provides a way to manage and run models, using local names, commands and an API. It also lets you import an existing GGUF file, which is exactly what I did here.

There are also two ways to use llamafile: an executable with the model built in, or a standalone executable that loads a separate GGUF file. I used the second option. This experiment did not demonstrate distributing everything in one file or moving it between different computers.

What was kept the same — and what wasn’t controlled

Both tools ran on the CPU, with the context set to 4096 tokens. A token is a unit of text processed by the model; it does not necessarily correspond to a word. The context window limits how much text can be considered in an interaction, including relevant conversation history.

In llamafile, I used -ngl 0. In Ollama, ollama ps confirmed 100% CPU, a context of 4096 and the llamacpp runner.

These were the files recorded locally:

Purpose Model and GGUF file Recorded size
Conversation Qwen3 1.7B — Qwen3-1.7B-Q5_K_M.gguf 1,257,880,032 bytes, approximately 1.26 GB
Programming Qwen2.5 Coder 3B Instruct — qwen2.5-coder-3b-instruct-q4_k_m.gguf 2,104,932,800 bytes, approximately 2.10 GB

These are file sizes, not measurements of memory usage while running the models.

The versions identified were llamafile 0.10.6 (visible in a screenshot) and Ollama v0.40.6 (reported later). I did not record every generation parameter, and neither conversation history nor caching was kept strictly consistent. I also did not measure RAM usage, temperature, power consumption or time to first token. The figures below therefore describe exploratory tests on that particular computer, not a controlled benchmark.

Qwen3: three sentences can take longer than you might think

I started with a straightforward question:

Responde em português de Portugal: explica em três frases a diferença entre memória RAM e armazenamento SSD.

In English, this asks the model to explain the difference between RAM and SSD storage in three sentences, using European Portuguese.

With reasoning mode, or thinking, enabled, the model could generate quite a lot of text before producing its final answer. Two llamafile runs recorded 394 and 424 tokens, with displayed times of 31 and 36 seconds. In Ollama, one run generated 368 tokens and took approximately 22.5 seconds in total.

Asking for three sentences does not mean that only those sentences are involved in generation. How thinking is controlled also depends on the model and the tool, as explained in the Ollama documentation.

I then ran tests with thinking disabled. In the final series, with three repetitions per tool, I recorded:

Tool Run Tokens generated Displayed time, seconds Generation rate, tokens/s
llamafile 1 149 11 13.32
llamafile 2 139 10 12.96
llamafile 3 149 13 11.46
Ollama 1 136 8.736 17.92
Ollama 2 106 6.041 18.53
Ollama 3 120 6.636 18.27

The Ollama times in this table have been rounded to three decimal places. The arithmetic means of the reported generation rates are 12.58 tokens/s for llamafile and 18.24 tokens/s for Ollama: roughly 45% higher for Ollama than for llamafile in this set of runs.

That result does not mean Ollama will always be 45% faster. Responses differed in length; conversation history and caching varied; and the reported times may include different processing stages. Even using the same GGUF does not standardise the chat template or the parameters that affect generation.

Some responses also sounded closer to Brazilian Portuguese, and one Ollama output incorrectly claimed that SSDs were slower than HDDs. I noted the error in that particular response without treating it as a general verdict on the tool.

Qwen2.5 Coder: generated code still needs testing

For the programming tests, I first asked for a calcularDesconto(preco, percentagem) function that would return the final price rounded to two decimal places. It had to reject non-finite numbers, negative prices and percentages outside the range from 0 to 100.

The response generated through llamafile contained the condition preco <= 0. That rejected a price of zero, even though the prompt only ruled out negative prices. A simple test exposes the difference:

calcularDesconto(0, 50); // Expected result: 0

The response generated through Ollama allowed this case and matched the functional requirements more closely. Even so, the prompts in this first test were not exactly identical, and there were deviations from the instruction to output code only. It would be wrong to use this to make a general judgement about response quality.

In the second exercise, I asked the models to fix a function containing deliberate errors. The calculation had to apply a discount to the subtotal and then charge 23% VAT on the discounted amount. The tax rate was part of the exercise.

This was the starting code:

function calcularTotal(preco, quantidade, desconto) {
  if (preco < 0 && quantidade < 0) {
    throw new Error('Valores inválidos');
  }
  const subtotal = preco + quantidade;
  const valorDesconto = desconto / 100;
  const total = subtotal + valorDesconto * 1.23;
  return Math.round(total * 100) / 10;
}

The prompt also required finite numbers, a price and quantity greater than zero, a discount between 0 and 100, the original function name and three parameters, and rounding to two decimal places.

Both responses fixed parts of the problem, but both got the final calculation wrong. Using calcularTotal(100, 2, 10), we could check the results:

Implementation Result
Reference: 200 × 0.9 × 1.23 221.40
Response from llamafile 2046.00
Response from Ollama 184.60

In llamafile, the response retained division by 10 in the rounding calculation, failed to validate the discount and used the wrong amount as the basis for VAT. In Ollama, rounding was corrected, but VAT was calculated on the discount itself, and the validation allowed inputs that should have been rejected.

It is worth distinguishing isNaN() from Number.isFinite(): the former may convert values before checking them, whereas the latter requires a finite number. The MDN JavaScript reference explains the difference.

For this code-correction exercise, the reported generation rates were 10.20 tokens/s in llamafile and 10.72 in Ollama. They are close, but come from just one response per tool, with differing lengths. In the first programming exercise, I did not capture a reliable llamafile duration, so I have not compared their speeds for that test.

The most useful lesson was the importance of checking the result: a model can fix lines of code, confidently explain what it has done and still be wrong. Two known test inputs can reveal more than a convincing paragraph.

What I found useful in practice

I managed to load external GGUF files and use llamafile without relying on a pre-packaged model. Its web interface also made it easier to copy responses after some difficulty selecting text in the terminal.

Ollama continued to offer the workflow I was already used to: import a model, give it a name and run it. For me, the appeal of llamafile was mainly the way it let me work with an executable and separate model files.

I did not test portability across operating systems. And simplifying distribution does not remove limitations involving memory, CPU, drivers or inference backends. What I found was another way to run local AI, with practical advantages that do not depend on winning every speed comparison.

Part II — How to repeat the process on Windows

1. Prepare the files and record the versions

The following examples use PowerShell and a dedicated folder at D:\AI-Tests. Change the drive letter if necessary. These commands create example files; choose fresh directories if there is existing work you want to preserve.

New-Item -ItemType Directory -Force -Path 'D:\AI-Tests\llamafile', 'D:\AI-Tests\models', 'D:\AI-Tests\ollama\qwen3', 'D:\AI-Tests\ollama\qwen25', 'D:\AI-Tests\results'

Ollama was already installed during my experiment. To set up another computer, follow the Windows installation instructions.

Download the standalone executable from the official llamafile releases page. At the time of the review on 18 October 2026, the latest release was 0.10.6, with an asset named llamafile-0.10.6. Save it as D:\AI-Tests\llamafile\llamafile.exe, adding the file extension. Version 0.10.6 was also visible in a screenshot taken during the experiment.

The quick-start documentation explains how to use external model weights, including the Windows limitation affecting executables larger than 4 GB. This executable-size limitation should not be confused with a blanket 4 GB limit for external GGUF files.

There is an important point about the origins of the model files:

Store the GGUF files in models, using the filenames shown in the commands below. Use the same local file in both tools without changing the quantisation partway through. Filenames and sizes help identify a file; a hash provides a more reliable identity check.

Get-ChildItem 'D:\AI-Tests\models\*.gguf' | Select-Object Name, Length
Get-FileHash -Path 'D:\AI-Tests\models\*.gguf' -Algorithm SHA256
ollama --version
& 'D:\AI-Tests\llamafile\llamafile.exe' --version

Record this output alongside the results. The versions identified in this experiment were llamafile 0.10.6 and Ollama v0.40.6; in a new session, it is worth checking and recording the versions actually used.

2. Run llamafile and open the browser

To make it explicit that the server should only be accessible from the local computer:

Set-Location 'D:\AI-Tests\llamafile'
.\llamafile.exe --server --host 127.0.0.1 --port 8080 -m 'D:\AI-Tests\models\Qwen3-1.7B-Q5_K_M.gguf' -c 4096 -ngl 0 --gpu disable

-m selects the GGUF file, -c 4096 sets the context, and -ngl 0 prevents model layers from being offloaded to the GPU. I also added --gpu disable to the example to make the CPU-only choice explicit. --server selects web-server mode, --host 127.0.0.1 restricts access to the local machine, and --port selects the port.

These options are described in the llamafile 0.10.6 argument reference. During the experiment, I also used the combined mode, which provides terminal chat and a web server; the example above selects the server alone to keep the instructions clear.

Open http://localhost:8080 and send a simple question. Seeing the page load is not enough: check that the model actually produces a response.

To try the programming model, stop the previous process with Ctrl+C and run:

.\llamafile.exe --server --host 127.0.0.1 --port 8080 -m 'D:\AI-Tests\models\qwen2.5-coder-3b-instruct-q4_k_m.gguf' -c 4096 -ngl 0 --gpu disable

If the port is already in use, choose another one, such as --port 8081, and open the corresponding address. Stop the server when the session is over.

3. Disable thinking in the appropriate mode

In llamafile 0.10.6, --nothink is documented for CLI mode. Do not add it indiscriminately to the web-server start-up command. To send an independent Qwen3 prompt:

.\llamafile.exe --cli --nothink -m 'D:\AI-Tests\models\Qwen3-1.7B-Q5_K_M.gguf' -c 4096 -ngl 0 --gpu disable -p 'Responde em português de Portugal: explica em três frases a diferença entre memória RAM e armazenamento SSD.'

This example makes a single terminal call with a prompt. It does not automatically recreate the state of the earlier conversations. For other versions or modes, check the help first:

.\llamafile.exe --help
.\llamafile.exe --cli --help
.\llamafile.exe --server --help

4. Import the same GGUF files into Ollama

A Modelfile is a configuration recipe, not a command to enter directly into PowerShell. FROM identifies the model; PARAMETER sets options.

To keep the two recipes separate, we’ll use a folder for each model. In PowerShell, copy the entire block:

Set-Location 'D:\AI-Tests\ollama\qwen3'
@"
FROM D:\AI-Tests\models\Qwen3-1.7B-Q5_K_M.gguf
PARAMETER num_ctx 4096
PARAMETER num_gpu 0
"@ | Set-Content -LiteralPath .\Modelfile -Encoding ascii

Get-Content -LiteralPath .\Modelfile
ollama create teste-qwen3 -f .\Modelfile

The block between @" and "@ is a here-string: a string spanning several lines. Enter it as one complete construct. The closing line begins with "@, with no leading spaces. The PowerShell documentation explains this syntax.

The file must be named Modelfile, without a .txt extension. Set-Content replaces the contents of this example file. ASCII encoding is sufficient for these lines, which use paths without accented characters.

In the original experiment, the recipe set num_ctx. I have added num_gpu 0 to the guide to request CPU execution. The parameter exists in the Ollama v0.40.6 source code consulted; the practical check is still to run ollama ps while the model is active.

For the second model:

Set-Location 'D:\AI-Tests\ollama\qwen25'
@"
FROM D:\AI-Tests\models\qwen2.5-coder-3b-instruct-q4_k_m.gguf
PARAMETER num_ctx 4096
PARAMETER num_gpu 0
"@ | Set-Content -LiteralPath .\Modelfile -Encoding ascii

Get-Content -LiteralPath .\Modelfile
ollama create teste-qwen25 -f .\Modelfile

The Modelfile reference documents the FROM directive, paths and context setting. These three lines configure model loading; they do not standardise all the generation settings across the two tools.

5. Chat and confirm CPU usage

In PowerShell:

ollama run teste-qwen3 --verbose

To disable thinking, enter this inside the Ollama chat:

/set nothink

This is a chat command, not a PowerShell command. The syntax was confirmed in the Ollama v0.40.6 interactive-mode source code. It depends on model support; hiding the display of thinking should not be confused with disabling it.

Then send the same three-sentence prompt. For the programming model, run ollama run teste-qwen25 --verbose and use the exercises described in Part I.

While the model is loaded, open a second PowerShell window and run:

ollama list
ollama ps

Confirm 100% CPU and a context of 4096. If the output shows GPU usage, or a mixture of CPU and GPU, it does not match the setup described here. The Ollama FAQ explains the PROCESSOR column.

Leave the chat with /bye. Before testing the other tool, you can unload the models from Ollama:

ollama stop teste-qwen3
ollama stop teste-qwen25

Stopping unused models prevents them from remaining loaded during the next test. These commands are covered in the CLI reference.

6. Collect results without confusing the metrics

In Ollama, --verbose shows token counts and timings. Prompt evaluation processes the input; response generation produces the subsequent tokens. A high prompt-evaluation rate is not the same as the speed at which the response is generated.

In llamafile, I used the counters and statistics available in the interface, including /stats in the environment I tested. Check /help in your installed interactive mode to confirm the available commands.

For a new experiment, save the complete prompt, the response as text, the model, its hash, the software versions, the context and the thinking setting. Start fresh chats to avoid accumulated history and record whether the model was already loaded. This improves comparability but does not automatically make it a controlled study.

Do not calculate tokens/s by blindly dividing the token count by the total time: that time may include model loading and input processing. In the earlier table, I kept the generation rates reported by the tools.

7. The difficulties I actually encountered

One early source of confusion was treating the opening and closing markers of the here-string as separate commands. Another was finding >> characters saved inside the Modelfile itself.

When >> appears at the start of a terminal line, it may be PowerShell’s continuation prompt: the shell is waiting for you to finish a construct. You must not copy that prompt into the file. The same characters can also act as a redirection operator within a command; context matters.

The Modelfile must begin with the FROM directive, without >> in front of it. If the file contents are wrong, recreate the block and check it with Get-Content before trying ollama create again. To check the model path:

Test-Path 'D:\AI-Tests\models\qwen2.5-coder-3b-instruct-q4_k_m.gguf'

The expected output is True. If it returns False, correct the path or filename. If the terminal is stuck waiting for an incomplete instruction, Ctrl+C cancels the input so you can start again.

I also had trouble selecting text with the mouse in the terminal I used for llamafile. I found an export option and eventually used the browser interface on localhost, where copying responses was easier. This was a limitation of my setup, not necessarily of every terminal.

Converting an image to text also introduced what looked like a space inside the JavaScript operator ||, turning it into | |. I did not attribute that apparent syntax error to the model because the transcription was uncertain. When examining code, exporting the text is preferable to reconstructing it from an image.

A reference function to check the calculation

This implementation was prepared as a reference for checking the result; it is not presented as a response from either model:

function calcularTotal(preco, quantidade, desconto) {
  if (!Number.isFinite(preco) || preco <= 0 ||
      !Number.isFinite(quantidade) || quantidade <= 0) {
    throw new Error('Preço ou quantidade inválidos');
  }
  if (!Number.isFinite(desconto) || desconto < 0 || desconto > 100) {
    throw new Error('Desconto inválido');
  }
  const subtotal = preco * quantidade;
  const totalComDesconto = subtotal * (1 - desconto / 100);
  const totalComIVA = totalComDesconto * 1.23;
  return Math.round(totalComIVA * 100) / 100;
}

calcularTotal(100, 2, 10); // 221.4, or 221.40 when formatted

This example helps check the exercise’s usual requirements. Money calculations in production may require decimal arithmetic or integer minor currency units (such as cents), because of how JavaScript represents numbers in binary; this is not an invoicing library.

What I took away from the experience

llamafile did what I was most interested in testing: I could run it on Windows, load external GGUF models and chat with them. Ollama was a familiar reference point and produced higher generation rates in the small Qwen3 series I recorded, without that settling the broader comparison between the tools.

The more lasting benefit was understanding the process better: keeping the executable and model separate, confirming CPU use and context, importing the same file and checking responses against known cases.

A practical tool saves you steps. A useful experiment also teaches you to question results that sound a little too convincing. There was room for both during that morning and part of the afternoon.

Related articles