Chapter 2: What you’ll need

You can run a language model on a Raspberry Pi. It will be slow and the model will be small, but it will run. On the other end, a dual-RTX-5090 workstation with 64 GB of VRAM chews through 70-billion-parameter models at conversational speeds. Most of the useful territory sits between these extremes. This chapter maps it.

Learning objectives

After this chapter you will be able to:

• Estimate VRAM requirements for any model size and quantization level
• Choose between Apple Silicon and NVIDIA hardware for your use case and budget
• Identify which hardware tier fits the model sizes you plan to run
• Understand the CPU-only fallback and when it is viable

Why this matters

Hardware is the barrier and the enabler. Buy too little and your models crawl or do not load at all. Buy too much and you paid for VRAM you will never fill. The goal is to match your hardware to your workload without overshooting. The numbers are straightforward once you know the formula.

The VRAM equation

Model size dictates VRAM requirement. The relationship is linear: more parameters, more memory. Quantization shrinks the multiplier. Here is the rule of thumb for GGUF quantized models at Q4_K_M, the default Ollama quantization:

The “realistic VRAM need” column adds roughly 20 to 30 percent overhead for the KV cache and runtime memory. This is not optional padding. The KV cache stores the model’s attention state for every token in the context. It grows linearly with context length and can consume several gigabytes on its own for larger models. If your total VRAM minus the model file size leaves less than 15 percent headroom, expect the model to spill into system RAM and slow down dramatically.

Actual Ollama GGUF file sizes as of mid-2026 confirm these estimates:

A 70B model needs room to breathe. Your 24 GB RTX 4090 will not load a 70B Q4 model with any useful context. The math is simple: 43 GB model plus a modest 8 to 12 GB for KV cache at reasonable context lengths equals 51 to 55 GB total. That fits in a Mac Studio with 128 GB unified memory. It does not fit in a single consumer GPU from any manufacturer.

Apple Silicon: capacity over throughput

Apple’s M-series chips use unified memory architecture. The CPU and GPU share the same pool of RAM. There is no separate VRAM bank, no transfer between CPU and GPU memory, no hard limit on model size other than total system RAM. If your Mac has 48 GB of unified memory, the GPU can address all 48 GB. That one architectural decision makes Apple Silicon the most practical local inference platform for large models today.

The tradeoff is bandwidth. An M4 Max delivers roughly 546 GB/s of memory bandwidth. An RTX 4090 delivers roughly 1,008 GB/s. NVIDIA’s dedicated VRAM is faster per byte by nearly a factor of two. For a given model that fits in both, the NVIDIA card will produce more tokens per second. But the model you want to run might not fit in the NVIDIA card at all.

This is the central tension in local inference hardware: Apple wins on capacity, NVIDIA wins on speed. Which one you need depends on which models you run.

Practical Apple Silicon recommendations:

M1/M2 with 16 GB. You already own this machine. It runs models up to about 9B parameters at Q4, which covers Qwen 3.5 9B and smaller Gemma 4 variants. Tokens per second will be in the 15 to 30 range for 7B models and 8 to 15 for 9B models. This is usable for chat and code completion. It is not fast, but it works.

M4 Pro with 48 GB at $2,399. The mid-range sweet spot. 48 GB handles any 31 to 35B model at Q4 with comfortable KV cache headroom. Gemma 4 31B, Ornith 35B, and Qwen 3.6 35B all fit and run at 20 to 35 tokens per second. This tier handles nearly all practical coding and analysis tasks without compromise.

Mac Studio M4 Max with 128 GB at $3,999. The enthusiast tier. 128 GB runs 70B models like Llama 3.3 at Q4 with generous context windows. You can even run Qwen 3.5 122B if you accept a tighter KV cache budget. Token rates on 70B models are roughly 15 to 25 per second - not NVIDIA-fast, but conversational.

M4 Max MacBook Pro with 128 GB. Same silicon as the Studio in a portable package, at roughly $4,699. The thermal envelope is tighter under sustained load, but the memory advantage travels with you.

Apple Silicon’s efficiency is an underappreciated benefit. A MacBook Pro running inference draws roughly 40 watts from the wall. An RTX 4090 alone draws up to 450 watts under load, before you add the rest of the system. The power difference matters if you run models for hours at a time or if you pay European electricity rates.

NVIDIA: speed over capacity

NVIDIA GPUs use dedicated GDDR6X or GDDR7 VRAM connected to the GPU over a wide memory bus. The bandwidth advantage translates directly into tokens per second. For models that fit entirely in VRAM, NVIDIA cards produce output at 50 to 120 tokens per second for 7B models and 30 to 50 for 34B models. Those speeds make local inference feel indistinguishable from cloud API latency.

The hard limit is VRAM capacity:

RTX 5070 Ti, 16 GB, roughly $749. Fits models up to about 13B parameters at Q4. Qwen 3.5 9B, Gemma 4 12B, and smaller coding models run comfortably. 16 GB is tight for a 13B with any meaningful context, but workable with KV cache quantization enabled.

RTX 4090, 24 GB, roughly $1,599. The previous-generation flagship. 24 GB handles models up to about 22B parameters at Q4 with comfortable overhead. You can push a 34B model into 24 GB if you lower the context window aggressively and enable KV cache quantization, but it is a squeeze. The 4090 is the most practical high-end NVIDIA card for local inference today because the used market has driven prices below MSRP and the 24 GB buffer covers the most popular model sizes.

RTX 5090, 32 GB, roughly $1,999. The current-generation flagship. 32 GB is the new ceiling for single-consumer-GPU inference. It handles models up to about 28B at Q4 with comfortable context, and can squeeze in 34 to 35B models at the cost of context budget. The 32 GB frame buffer is the main reason to choose the 5090 over the 4090 for inference.

Multi-GPU setups multiply capacity but add complexity. Two RTX 4090s give you 48 GB of VRAM and the ability to run 70B models, but the software support for tensor parallelism across consumer GPUs is less mature than on the data center side. Ollama supports multi-GPU inference, as does llama.cpp with the appropriate build flags. It works. Expect to spend an afternoon tuning the configuration.

Three hardware tiers

Rather than list every configuration, here are three tiers matched to realistic budgets and workloads:

Budget tier: existing hardware or under $700. An M1/M2 Mac with 16 GB you already own, or a used RTX 4060 Ti 16 GB desktop. Runs models up to 9 to 12B parameters. Good for code completion, basic chat, and document Q&A on smaller models. If you already have the hardware, start here. If you need to buy, consider whether you could stretch to the mid-range instead - the jump from 16 GB to 48 GB gives you access to a much wider model catalog.

Mid-range tier: $2,000 to $2,500. An M4 Pro MacBook Pro with 48 GB, or a desktop with an RTX 4090 (24 GB) or RTX 5090 (32 GB). Runs models up to 31 to 35B at comfortable speeds with generous context. This tier covers everything a software engineer does daily: coding assistance, code review, documentation analysis, test generation. You will rarely hit a task that genuinely needs a larger model than this tier provides.

Enthusiast tier: $4,000 to $7,000. A Mac Studio M4 Max with 128 GB or a dual-RTX-5090 workstation with 64 GB combined VRAM. Runs 70B models and larger. 70B models are materially better at complex reasoning, multilingual tasks, and following nuanced instructions than 30B models. Whether that difference matters for your work is a judgment call, not a performance number. For most engineers, the mid-range tier is the rational ceiling.

CPU-only inference

If you have no GPU and no Apple Silicon, you can still run models on CPU and system RAM. A desktop with 64 GB of DDR5 RAM will run a 70B Q4 model entirely from system memory. It will generate 5 to 15 tokens per second instead of the 50+ you would get from a GPU. Reading speed is roughly 5 to 10 words per second for most people, so CPU inference at 5 tokens per second means you wait roughly as long as you would spend reading the output.

CPU-only inference is viable for batch processing, overnight summarization jobs, and use cases where latency does not matter. It is frustrating for interactive chat. If CPU-only is your reality, lean toward smaller models (7B to 13B) where token rates will be higher simply because there are fewer weights to process per token.

The CPU-only tier exists to make the point that you do not need specialized hardware to start. If you have a laptop with 16 GB of RAM and no dedicated GPU, download a 3B model and test the workflow. The model will run. It might run at 20 to 30 tokens per second on a modern CPU. That is fast enough to evaluate whether local inference fits your needs before you spend money on hardware.

A note on rented hardware

If you want to evaluate a model that does not fit on your machine, GPU cloud instances are a practical bridge. A single A100 80 GB instance rents for roughly $1 to $3 per hour on providers like Lambda Labs or RunPod. For $10, you can spend an afternoon testing a 70B model at full precision with generous context, measuring exactly how much memory it uses, and deciding whether the experience justifies a hardware purchase. Renting a high-VRAM instance for one afternoon is cheaper than buying the wrong GPU. Use it to validate before you commit.

Storage: the hidden requirement

Model files are large and multiplying. A single 70B Q4 model is 43 GB. A 122B model is 81 GB. A 671B model is 404 GB. Keep three or four models of different sizes on your machine and you can easily consume 100 to 200 GB of storage.

NVMe SSDs are cheap and necessary. A 1 TB drive offers comfortable storage for the operating system, applications, and a curated collection of models. Model downloads are sequential reads, so even a mid-range PCIe 3.0 NVMe drive is fast enough. What matters is capacity, not transfer speed.

Plan for growth. The model landscape shifts every month. The model you download today will have a successor in three months that is slightly larger and slightly better. Leave storage headroom so you are not deleting yesterday’s model to make room for today’s.

Key takeaways

• VRAM requirement equals model file size plus 20 to 30 percent overhead for KV cache and runtime memory. Measure accurately before you buy hardware.
• Apple Silicon wins on model capacity. NVIDIA wins on token speed. The right choice depends on which models you need to run.
• The budget tier runs models up to 9 to 12B. The mid-range tier runs 31 to 35B and covers nearly all daily engineering tasks. The enthusiast tier runs 70B+.
• CPU-only inference works for batch processing and small models. It is frustrating for interactive use with large models.
• A 1 TB NVMe drive is the practical minimum for a machine that stores multiple models.

Common pitfalls

Buying a GPU with too little VRAM. The 16 GB cards (RTX 5070 Ti, RTX 4060 Ti 16 GB) are the minimum viable for serious local inference. An 8 GB or 12 GB card limits you to models under 7B parameters or forces extreme quantization that degrades quality. Spend the extra money on VRAM. It is the one spec you cannot upgrade later.

Overestimating CPU-only performance. A 70B model on CPU at 5 tokens per second is usable for batch processing. It is not usable for a coding assistant where you expect sub-second responses. Match your expectations to the hardware. If you need interactive speeds, stay under 13B on CPU.

Ignoring power and cooling. An RTX 4090 under sustained inference load draws roughly 350 to 450 watts and dumps that heat into your room. A MacBook Pro M4 Max draws roughly 40 to 60 watts for the same workload. If you work in a small office or live in a warm climate, the power difference translates into real comfort and real electricity bills.

Forgetting about context length. A model that fits in VRAM at 2,048 tokens of context might not fit at 32,768 tokens. The KV cache grows linearly. When budgeting VRAM, budget for the context length you actually plan to use, not the minimum.

Further reading

• Apple M4 Max specs - official unified memory bandwidth and configuration options
• NVIDIA RTX 50 series - official VRAM and bandwidth specifications
• Ollama hardware guide - GPU compatibility and configuration for the most popular local inference runtime

Checkpoint exercise

List the machine you currently use for development. Write down its total RAM (or VRAM plus system RAM if it has a discrete GPU). Using the VRAM table in this chapter, identify the largest model size that fits comfortably on your hardware at Q4_K_M quantization with 2,048 tokens of context. Then identify the model you would actually want to run for your most common AI task. Does it fit? If not, what hardware change would close the gap?