Running AI on your own machine
In one line: you can download a model and run it on your own PC — free, private, no internet — and it is genuinely good enough for a great deal of everyday work.
Why bother
Nothing leaves. Your text never goes anywhere. For client work, medical or financial material, or anything under an NDA, this is not a preference — it is the only version you are allowed to use.
It is free. No per-use cost, no account, no bill that grows.
It works offline. On a plane, on bad wifi, in a building with no signal.
It cannot be taken away. The model is a file on your disk. It will not be deprecated, repriced, or have its terms changed under you.
The cost: it is a step or two behind the best hosted models, and a big one will be slower than the cloud unless you have a good graphics card.
"Open source" — what it really means here
Mostly you will see open weights: you can download the model and run it, but you do not necessarily get the training data or the recipe. Llama and Mistral are like this. It is not open source in the strict sense, and it is what almost everyone means by "open model".
A few are properly open — the weights, the data and the method. Worthy and rarer.
What matters to you: can I download it, run it, and use it for what I need under its licence? For Llama, Mistral, Qwen, Gemma and DeepSeek, generally yes, including commercially — but read the licence if you are building a business on it.
How to actually do it
The easy way — Ollama. One install, then one command per model. It handles the download, the format and the serving. → ollama.com
ollama run llama3.2That is genuinely the whole thing. First run downloads the model; after that it is local and offline.
With a window rather than a terminal:
- LM Studio — a proper app, browse and download models, chat with them. The easiest start if you do not like a command line.
- Jan — open source, similar idea.
- GPT4All — simple, modest hardware.
Where the models come from:
- Hugging Face — effectively the library for all of this. Thousands of models, and where the ones above download from.
What your PC can handle
Roughly, and the number that matters is RAM — or VRAM if you have a decent graphics card:
| You have | You can comfortably run | Good for | | --- | --- | --- | | 8 GB | 3B–7B models | tidying, summarising, drafting | | 16 GB | 7B–14B | most everyday work | | 32 GB | 14B–32B | genuinely capable | | 64 GB+ or a strong GPU | 70B | close to hosted quality |
"B" is billions of parameters — roughly, how big the model is. Bigger is better and slower. Start with a small one. A 7B model that answers instantly is more useful in practice than a 70B one that takes a minute.
Which to try first
Names change; check the tool's own library. As a starting point, look for the current small Llama, Mistral, Qwen or Gemma — and for code specifically, whatever the current Qwen Coder or DeepSeek Coder is.
→ Ollama's model library is the fastest way to see what is current and how big each one is.
Where AIOS comes in
AIOS already runs entirely on your machine — your notes never leave unless you switch sync on. Its own understanding of what you type is not a model at all: it reads your sentences with rules, which is why it works with no internet, no account and no cost.
And the Tools tab can build you a Linux virtual machine in a few clicks, which is where a lot of people prefer to run local models.
Try this
Install Ollama, run one small model, and ask it something you would normally ask a hosted one. It will be a bit worse. Notice how much of your everyday work it would nonetheless handle perfectly well — that is the number that matters, and it surprises most people.
---
Next: Letting AI use tools · All lessons