VRAM: which local model fits your GPU

By Emmanuel Sauvage · Published · Updated

This page may contain Amazon affiliate links. A purchase after a click can earn the site a commission, at no extra cost to you. The advice stays independent. How affiliation works.

VRAM (the memory on the graphics card) mostly decides which model you can load entirely on the GPU, how fast the tokens come back, and whether you have to run part of the work on the CPU. That CPU path is slower. On an RTX 4060 it is sometimes the only way a 13B fits at all. This guide is for households installing Ollama or LM Studio on a mini PC, a tower, or a Mac: tables for 8, 12, 16, and 24 GB, how that lines up with system RAM, and a separate section for FLUX and ComfyUI when text chat is no longer the only job.

Quick read

You came for… Read next
Chat only (Ollama, Open WebUI, LM Studio text) Chat tables for 8, 12, and 16 GB. You can skip the whole FLUX section if you are not making images.
FLUX or ComfyUI images VRAM beyond chat. Image models use memory separately from a chat LLM (images without a cloud).

The 24 GB tier and dense 32B+ models: advanced-use appendix (little use if the only job is family chat).

Who this is for, and how long

Profile What you need first Rough time Path
Beginner (first local model) A PC that is on, ideally with Ollama already installed ~8 min Quick read for chat only, plus the 8 GB tier
Intermediate (13B, code, long context) You have already tried local chat, and you have read GGUF and Q4_K_M below ~12 min The 12 GB and 16 GB tables, plus running the rest on the CPU
Images and text on one machine You are considering ComfyUI or a FLUX project ~15 min Chat first, then FLUX images
Upgrading a card or a mini PC You already know VRAM is the bottleneck (slow replies, or an out-of-memory error) ~5 min Measure, then the hardware selector

VRAM, RAM, and GGUF: three pieces of the puzzle

A quantized GGUF file (Q4_K_M, Q5_K_M, and similar tags) takes a certain number of gigabytes on disk. GGUF is the compressed model format Ollama and LM Studio load. Once it is loaded:

  • part of it, ideally the whole model, lives in VRAM if you have a compatible NVIDIA GPU, AMD GPU, or integrated GPU;
  • the rest runs on the CPU and in system RAM. Ollama does this on its own when VRAM runs out;
  • the context (the chat history, and documents you paste in) uses more memory on top of the model weights.

A practical rule: leave 1–3 GB of VRAM free for context, the driver, and spikes. If you fill the card to 100%, you risk an out-of-memory error, or a sudden drop onto the CPU.

Quantization levels households actually use:

GGUF tag Disk size, about 7B Quality Memory for the weights alone
Q4_K_M ~4.5 GB A solid compromise ~5–6 GB
Q5_K_M ~5.5 GB A bit more nuance ~6–7 GB
Q8_0 ~7 GB Close to the original ~8–9 GB

To narrow the first download: choose a first model.

Pick a tier in 30 seconds

What you have Start with Typical use
No dedicated GPU, 16–32 GB of RAM 3B–7B Q4 on the CPU or iGPU Modest family chat
8 GB of VRAM 7B–8B Q4_K_M The best Ollama chat compromise
12 GB of VRAM 13B Q4 or 8B Q8 A code assistant, and room for context
16 GB or more 13B–32B Q4 Coding, and a first try at ComfyUI
Regular FLUX dev images 16 GB of VRAM (24 GB is comfortable) See the ComfyUI section below

VRAM tiers for Ollama chat (rough, October 2026)

Assumption: Llama 3.x, Mistral, or Qwen instruct models, family chat, a moderate context (4k–8k tokens). These are not guarantees. Some 8B models are heavier than the name suggests.

Before the tiers. An RTX 4060, a 4070, a 5070 Ti, a 5090, an RX 6600, and the rest are October 2026 examples, not a promise that any of them is in stock. The actual VRAM (8, 12, 16, or 24 GB) matters more than the name on the box. Check the maker’s spec sheet and what the store has in stock. The same series often ships with more than one memory size.

8 GB of VRAM (RTX 4060 laptop, RTX 3070, RX 6600, and similar)

Model you want Quantization Fully on the GPU? Comment
3B–4B Q4–Q8 Yes Very fast, modest quality
7B–8B Q4_K_M Yes Best compromise at 8 GB
7B–8B Q5 / Q8 Sometimes A long context pushes layers onto the CPU
13B Q4 Rarely Some layers on the CPU, and the wait grows
32B and up — No Unrealistic

Household advice: stay on an 8B Q4 for Open WebUI. If you step up to a 13B, accept the slower pace, or upgrade the GPU.

12 GB of VRAM (RTX 4070, GeForce RTX 3060 12 GB, RX 7700 XT, and similar)

Model Quantization Comment
8B Q5–Q8 Comfortable, with room for context
13B Q4_K_M Realistic fully on the GPU, or very close
14B–16B Q4 Depends on the exact architecture
32B Q3–Q4 Often partly on the CPU

A good price-to-speed balance for a mini PC with a mid-range dedicated card. See the mini PC comparison.

16 GB of VRAM (RTX 4080, 5070 Ti, RX 7800 XT, and similar)

Model Quantization Comment
13B–14B Q5–Q8 Demanding chat, and Continue (a VS Code extension for code assisted by a local model)
32B Q4_K_M Advanced use is possible
70B Q2–Q3 Experimental, and quality varies

Large MoE models (mixture of experts: several internal blocks, only one active for each token) are not in this table. Their VRAM use changes while they run. See only the 24 GB appendix.

Appendix — the 24 GB tier (advanced use)

Few households need 24 GB for family chat alone. It starts to matter with FLUX dev images, a long context, and 32B+ models. For MoE models, measure at load time (nvidia-smi or the equivalent) instead of guessing from a “13B Q4” row in the chat tables.

ModelQuantizationComment
32BQ5–Q8Close to a local premium setup
70BQ4Possible, depending on the exact model
Large MoEQ4Real VRAM at load time. Expert use.

Typical cards: RTX 4090, 5090, RTX 6000 (October 2026 examples; see the store note above).

When part of the model runs on the CPU

When VRAM is short, Ollama or LM Studio keeps some layers on the GPU and runs the rest on the CPU, using system RAM. That is the whole story: part of the model is running on the CPU. Do not mix it up with the phrase “GPU offload” in US docs, which sometimes means the opposite, moving layers onto the GPU and off the CPU.

What you will notice:

  • The first token takes longer, because the path is split.
  • Speed (tokens per second) drops if most layers are on the CPU.
  • Power draw: the CPU sits at 100%, and the mini PC fans get loud. Read noise and power of a mini PC.
  • System RAM: 32 GB becomes the comfortable minimum if you have 8–12 GB of VRAM and a 13B that is only partly on the GPU. See 32 vs 64 GB of RAM.

The num_gpu setting on the Ollama command line can force how many layers stay on the GPU. That is useful for diagnosis. It does not create extra VRAM.

LM Studio has “GPU layers” sliders. Start with about 80% of the layers on the GPU, run a test chat, then raise or lower the slider if VRAM fills up. Full walkthrough: Set GPU layers in LM Studio.

No dedicated GPU: iGPU, Apple Silicon, CPU only

Platform Unified or shared memory A realistic profile
Mac mini M4, 16 GB unified An 8B is comfortable, a 13B is tight
Mac mini M4, 32 GB unified 13B–14B, mixed use
Ryzen 7840HS or Core Ultra iGPU shared RAM 7B–8B, moderate
NAS or N100, CPU only RAM only 3B–7B, slow

The Apple page: Mac mini M4 for local AI. On a PC with no card, add RAM before you buy an RTX.

VRAM and image generation (beyond chat)

Chat only? Stay with the chat tables and choose a first model. The Images only block below is for FLUX and ComfyUI (images without a cloud).

Images only — FLUX and ComfyUI (open this if you are generating pixels)

Text chat and FLUX, SDXL, or Stable Diffusion do not need the same VRAM as a 7B Ollama model. ComfyUI chains nodes (the model, CLIP, the KSampler, an upscale, and so on). FLUX.1 dev is heavier than SD 1.5. Household overview, including privacy and the stack: images without a cloud.

Rough figures for 1024×1024, a simple workflow, October 2026:

SetupFLUX schnellFLUX devSDXL
8 GB of VRAMDifficult, or drop the resolutionImpracticalOK with optimizations
12 GBschnell is OK, dev is tightSlow, or partly on the CPUComfortable
16 GBdev is usableA good household compromiseEasy
24 GBComfortableComfortable, plus room to upscaleEasy

If VRAM fills up: fp8 or quantized checkpoints, CPU offload of the text encoder (here that means saving VRAM, which is not the same as spilling a language model onto the CPU), a 768 resolution and then an upscale, and quit Ollama before you open ComfyUI. Step by step: ComfyUI and FLUX locally. On an 8 GB GPU, a lighter Stable Diffusion model, or an OCuLink eGPU, is a better daily driver than FLUX dev. Ollama and ComfyUI on the same PC: 32 GB of RAM, and one heavy GPU job at a time.

Measure before you buy or upgrade

  1. On the machine you have now, ollama run the model you want. In Windows, open Performance → GPU and note Dedicated GPU memory once it levels off.
  2. On Linux, run nvidia-smi during a generation.
  3. If use stays above 90% of VRAM, lower the quantization, the model, or the context.
  4. If you plan to make images, run one ComfyUI FLUX session. A chat test alone is not enough.

What you should know at the end: whether your current tier matches the 8, 12, or 16 GB table, or whether you will have to let some layers run on the CPU, before you compare cards online.

Hardware selector: line up VRAM, RAM, and what you actually do

Why this comes after a measurement. A search for “RTX 4070” that ignores the VRAM you measured often mixes 8 GB and 12 GB cards on the same store page. The hardware selector lines up what you do (7B chat, 13B, occasional images), a rough budget, and, if you want, the household price per kilowatt-hour, so you can estimate noise and a 24-hour power bill.

Three clicks (free, no account required):

  1. Use the VRAM you measured, or the tier you picked in the tables above.
  2. Check the jobs you actually do (text only, code, images, and so on). The selector drops profiles that do not fit 32 vs 64 GB of RAM and SSD storage.
  3. Read the suggested profiles as a short list to check against choosing hardware. Nothing here places an order.

Affiliation: if you then follow a partner search in the section below, do that after this sort. The site may earn a commission at no extra cost to you. Details are on how affiliation works (October 2026).

Several people, and a long context

Open WebUI with four family accounts does not multiply VRAM by four. A consumer card runs one heavy generation at a time. A 32k-token context (pasting in school PDFs) can add several temporary gigabytes. If you keep long conversations, test with your real use, not a one-line hello from a demo. For sensitive documents, use a smaller model and summarize in smaller pieces. See privacy when AI stays home.

Common mistakes

  • Mixing up the gigabytes of the file on disk and the memory it needs while running (context plus overhead).
  • Buying 64 GB of RAM and keeping 8 GB of VRAM for a 32B model. The CPU becomes the bottleneck.
  • Running FLUX dev on an 8 GB RTX 4060 and concluding that local AI does not work.
  • Forgetting that Docker plus Open WebUI also uses a little RAM, and a little VRAM indirectly.

More hardware guides

Software after the hardware

Install Ollama, Open WebUI for beginners, or the Docker stack. To try it with no commitment: a first local AI in 15 minutes.

A sensible order: (1) test the model you want on the GPU you have, (2) the hardware selector, (3) only then compare store listings. The links below open an Amazon.com search as a starting point. The site may earn a commission at no extra cost to you. Price, actual VRAM, and availability still have to be checked on each listing (how affiliation works, October 2026).

VRAM you need (this guide) Example search Reminder
8 GB, 7B–8B Q4 chat RTX 4060 on Amazon.com Matches the 8 GB tier. Confirm 8 GB on the listing, not only the name “4060”.
12 GB, a comfortable 13B Q4 RTX 4070 on Amazon.com Matches the 12 GB tier. A 16 GB card (some recent variants) is not interchangeable with “12 GB” in these tables.

Case size, the power supply, and card length: check those against GPU, iGPU, and OCuLink before you order.

Conclusion

8 GB is the sweet spot for 7B–8B chat. 12–16 GB covers a 13B and a real start on images. 24 GB (the appendix) covers 32B text and a reasonable FLUX dev in ComfyUI. When VRAM runs out, running the rest on the CPU keeps the setup alive, at a cost in speed and noise. Matching the model to the card is the better plan. If you want images without sending prompts to a third party, continue with images without a cloud and ComfyUI and FLUX.

Rough figures for October 2026. Check the exact tags and file sizes on the model page before you buy a GPU.

Common questions

Is 8 GB of VRAM enough for local chat?

Often yes for a 7B–8B in Q4_K_M. A 13B gets tight, and Ollama will run some layers on the CPU, which slows the replies. Check the 8 GB tier in this guide with an `ollama run` test on your own machine, rather than relying on this answer alone.

How do I see how much VRAM I have free?

Windows Task Manager (GPU), macOS Activity Monitor, or nvidia-smi / rocm-smi on Linux.

What happens with no dedicated graphics card?

Ollama uses system RAM and, when it can, the integrated GPU. Aim for smaller models, and at least 32 GB of RAM for a comfortable 7B.

Sources

Narrow it down with the hardware selector · French version of this section