Choose your first local language model

By Emmanuel Sauvage · Published · Updated

This page may contain Amazon affiliate links. A purchase after a click can earn the site a commission, at no extra cost to you. The advice stays independent. How affiliation works.

After you install Ollama or finish a first local AI in 15 minutes, the next question is almost always the same: which model file should you download? An LLM (a large language model) comes in several sizes (3B, 7B, 13B… where B means billions of parameters) and several quantizations (compression of the GGUF file). This guide matches the tag to your RAM and your VRAM.

What you need

  • Ollama installed and working (ollama --version answers).
  • A rough idea of your RAM (16, 32, or 64 GB) and, if you have it, your VRAM (graphics-card memory).
  • Disk space: plan on 3–8 GB per quantized model. See SSD storage for models.

Estimated time: 25 minutes, including the download at your connection speed.

An instruct variant: what you want for an assistant

On Ollama, prefer tags that contain instruct. Those models are trained to follow instructions (summarize, translate, “answer in three bullets”). A so-called base variant, with no instruct, often answers like free-form text, which is less useful day to day.

Valid tag examples (October 2026; confirm them on the model’s official page):

  • llama3.2:3b-instruct-q4_K_M — very light, a good fit for 16 GB RAM.
  • mistral:7b-instruct-q4_K_M — a solid balance of quality and size.
  • llama3.1:8b-instruct-q4_K_M — a bit hungrier for memory.

GGUF quantization table (the heart of the choice)

Quant. Disk size, ~7B Quality Rough RAM/VRAM
Q4_K_M ~4.5 GB A good household compromise 8–12 GB free
Q5_K_M ~5.5 GB A little more nuance 10–14 GB
Q8_0 ~7 GB Close to the original 12–16 GB

A worked example for 32 GB RAM: you can aim for an 8B instruct Q4_K_M (~5 GB) and still leave room for Windows, the browser, and Open WebUI. On 16 GB, prefer a 3B instruct Q4_K_M, or a 7B only when few other apps are open.

Numbered steps

  1. Write down the memory. Windows: Settings, System, About. macOS: Apple menu, About This Mac, Memory. Linux: in a terminal, free -h (the “total” and “available” lines). NVIDIA GPU: nvidia-smi (the “Memory-Usage” line). AMD GPU (with ROCm installed): rocm-smi. No dedicated graphics card (Intel or AMD integrated graphics only): skip nvidia-smi and rocm-smi and go by RAM alone. See GPU, iGPU, and eGPU if you are unsure which you have.

  2. Pick a size.

    • 16 GB RAM: 3B–7B instruct Q4.
    • 32 GB: 7B–8B is comfortable. 13B is possible on CPU if you close everything else.
    • A GPU with 8 GB VRAM: often a 7B–8B with most of it on the GPU. Detail is in which model for your VRAM.
  3. Download with the exact command.

    ollama pull mistral:7b-instruct-q4_K_M

    Replace the tag with the one you copied from ollama.com/library.

  4. Start a test session.

    ollama run mistral:7b-instruct-q4_K_M

    Ask: “Summarize in three bullets why local AI can help a household.”

    What you want to see: in under 60 seconds, the model answers with exactly three bullets in English, not one paragraph and not a numbered list standing in for the dashes. If it ignores the format, check the instruct tag before you change size.

    Optional note — context window Context (context length) is the text (prompt plus history) the model can “see” at once. A token is a unit of text for the model (a word or a piece of a word). On Ollama, household models often sit around 8k tokens (about 6,000–8,000 words), which is enough for an email or a page of notes. Long PDFs belong in n8n or in a manual split. Do not jump to a bigger model for that reason alone.

  5. Judge speed and quality. If a paragraph takes more than 30–60 seconds on CPU, try a smaller model. If the replies ignore the format, check the instruct tag.

Mistakes you can see (and what to do)

Mistake What happens Fix
Download a 70B on 16 GB RAM The disk grinds, replies crawl, or the app crashes ollama rm, then a 3B–7B model
Skip instruct The model ignores your instructions Pull again with an instruct tag
Mix up VRAM and RAM The wrong GPU model Reread VRAM
A tag with no visible quantization A heavier file than you expected Choose q4_K_M on purpose

A family interface, and what comes next

To share the chat in the household: Open WebUI for beginners (a web page on the same Ollama) or install LM Studio (a graphical app with no terminal, handy for comparing models before you lock an Ollama tag). If you split CPU and GPU by hand in LM Studio, GPU offload walks through it. For real jobs: a private family assistant, Home Assistant, and n8n.

Hardware if you buy a dedicated machine

Use the hardware selector together with choose hardware for home AI. A rough mini PC range of €500–800 (check the market price, October 2026): 32 GB mini PCs on Amazon (affiliate link — affiliation).

Cost and privacy

Compare 3B, 7B, 8B, and 13B (household use)

Size Why you would pick it The limit
3B A modest PC, quick tests Less nuance in the replies
7B–8B The best everyday compromise Can fill 16 GB of RAM
13B Richer writing Slow on CPU with no GPU; 32 GB or more
14B and up Comfortable machines Swap risk — see 32/64 GB of RAM

For a first download, stay on 7B instruct Q4 unless the hardware clearly says otherwise (16 GB RAM points to a 3B).

How to read a page on ollama.com/library

  1. Open ollama.com/library and search for mistral, llama3.2, qwen2.5, and so on.
  2. Open the Tags tab and find a row that contains instruct and q4_K_M (the exact capitalization shown on the page).
  3. Copy the exact tag (capitalization and colon included) into ollama pull.
  4. Note the size shown (~4.5 GB for a 7B Q4) and compare it with your SSD — model storage.

When several quantizations exist, Q4_K_M is the household default. Choose Q5 when you have RAM to spare and want a bit more quality.

English, French, and other languages

Most recent instruct models follow the language you request. Say “Reply in English” for everyday use at home. Voice is a separate job: transcription belongs to Whisper, not to the LLM. For hobby code, Continue (the editor extension of that name) often accepts an Ollama backend. See the local Continue coding assistant once the right model is pulled.

Managing disk space

List what you have:

ollama list

Remove a model you no longer use:

ollama rm nom_du_tag

Replace nom_du_tag with the tag from ollama list. Keep one general-purpose model plus a small backup. A dozen near-copies of the same model only fills the disk.

When to change models

Signal What to do
A short reply takes more than 1 minute Move to a 3B–7B, or turn the GPU on
Invented answers about recent facts Normal with no web search. That is separate from a broken model
Format is ignored every time Check the instruct tag
Quality is too thin on hard tasks Try an 8B Q5 or a 13B if RAM and GPU allow

LocAIhome scenarios (a model that fits)

  • Living room / Open WebUI: 8B instruct Q4 — family assistant.
  • Smart home: 3B–8B with low latency — Home Assistant.
  • Images: a different stack (ComfyUI), not the same chat model.
  • Mac: the same size logic, with unified memory — Mac Mini M4.

A minimum of safety

Leave the Ollama port (11434) off the public internet until the service has a login. For a Docker stack, see Docker, Ollama, and Open WebUI, and leave port 8080 unpublished until it is protected.

Common questions

What is a good first model for a household?

A 7B–8B instruct model (trained to follow instructions), quantized as q4_K_M, if you have 16–32 GB of RAM or 8 GB or more of GPU VRAM.

How do I copy the right ollama pull tag?

On ollama.com/library, open the model page and find a tag that contains instruct and q4_K_M, for example llama3.2:3b-instruct-q4_K_M or mistral:7b-instruct-q4_K_M.

Why does my model ignore my instructions?

You may have downloaded a base variant (no instruct), or a model so heavy that it swaps to disk and answers in a jumble.

Sources

Narrow it down with the hardware selector · French version of this section