Best local AI models for GeForce RTX 4070

A solid 7–9B machine. 14B fits at Q4_K_M with a short context and no room for much else.

Specification

VRAM12 GB

Usable for models11 GB

System RAM32 GB

Bandwidth504 GB/s

Local trainingSupported

Operating systemswindows, linux

Runs well

27 models fit this machine

Estimated

Model	Params	Quantization	Memory	~tok/s	Fit	Fine-tune
Qwen2.5 Coder 7B Instruct	7.6B	Q6_K	6.8 GB	63	Good fit	Yes
Gemma 2 9B Instruct	9.2B	Q4_K_M	8.1 GB	70	Good fit	Yes
Qwen2.5 7B Instruct	7.6B	Q6_K	6.8 GB	63	Good fit	Yes
Llama 3.1 8B Instruct	8B	Q6_K	7.7 GB	59	Good fit	Yes
Mistral 7B Instruct v0.3	7.25B	Q4_K_M	5.7 GB	89	Excellent fit	Yes
Qwen3 8B	8.2B	Q5_K_M	7.1 GB	67	Good fit	Yes
Code Llama 7B Instruct	6.7B	Q4_K_M	8.3 GB	96	Good fit	Yes
StarCoder2 15B	16B	Q4_K_M	10 GB	40	Tight fit	No
Mistral Nemo 12B Instruct	12.2B	Q4_K_M	9.1 GB	53	Tight fit	Yes
Qwen3 14B	14.8B	Q4_K_M	10 GB	44	Tight fit	No
Phi-3.5 Mini Instruct	3.8B	Q4_K_M	5.7 GB	170	Excellent fit	Yes
Gemma 3 4B Instruct	4.3B	Q5_K_M	4.7 GB	128	Excellent fit	Yes
Gemma 3 12B Instruct	12.2B	Q4_K_M	10 GB	53	Tight fit	Yes
Llama 3.2 3B Instruct	3.2B	Q8_0	4.6 GB	115	Excellent fit	Yes
Qwen2.5 Coder 32B Instruct	32.8B	Q4_K_M	21 GB	20	Runs with CPU offload	No
Qwen2.5 32B Instruct	32.8B	Q4_K_M	21 GB	20	Runs with CPU offload	No
Mixtral 8x7B Instruct	46.7B	Q4_K_M	29 GB	50	Runs with CPU offload	No
SmolLM2 1.7B Instruct	1.7B	F16	5.2 GB	115	Excellent fit	Yes

Fine-tunable here

QLORA · 7.0 GB peak

QLORA · 8.2 GB peak

QLORA · 7.0 GB peak

QLORA · 7.2 GB peak

QLORA · 6.8 GB peak

QLORA · 7.4 GB peak

QLORA · 6.4 GB peak

QLORA · 10 GB peak

Out of reach