How to Run LLMs Locally on a Laptop: Memory, Disk, Model Choice

Running a language model on your own laptop keeps your prompts on your machine and avoids per-token fees. Whether it works well depends on a few numbers you can work out before downloading anything. This guide shows how to do that arithmetic, how to measure speed yourself, and how to start a local server, using two recent open projects as worked examples.

Quick answer: a quantized model file needs roughly (parameters x bits per weight / 8) bytes, plus room for the context cache and your other programs. A 4-bit "Q4_K_M" file is about 4.9 bits per weight, so a 24B model is roughly 14.7 GB. Mixture-of-experts (MoE) models need memory for all their weights but compute only a few, which makes them faster than their size suggests. Measure your own tokens per second rather than trusting anyone else's chart.

Checked against the source on November 5, 2026. The Janas-LLM README gives the licence as GPL v3 or later, calls the project "an experiment, not a product" and dates its latest measurements to October 2026. The LFM2.5-24B-A2B model card lists 24.0B total and 2.45B active parameters, and the llama.cpp README shows an MIT licence. This was a read of those pages, not a hands-on test.

• Janas-LLM: GPL v3 or later; Linux x86-64 and Windows 10 (1903+) or 11; the GPU path is optional and uses Vulkan. No version number is stated in the README.
• LFM2.5-24B-A2B: 131,072-token context; licence listed as "lfm1.0"; formats include GGUF, ONNX, MLX 4-bit and FP8.
• llama.cpp: MIT licence; backends listed include CUDA, HIP, Metal, Vulkan and SYCL.
• Ollama: offered for macOS, Linux and Windows on its download page.

The sizing formula

Model files are mostly weights, and each weight is stored in a fixed number of bits. So:

file size (bytes) = number of parameters x bits per weight / 8

Quantization lowers the bits per weight. In the llama.cpp project's own table for an 8B model (Llama-3.1-8B), the measured figures were:

Source: the llama.cpp quantize README, one machine, 128-token generation. Lower precision generates faster because less data moves per token. Accuracy loss grows as bits fall, but that page publishes no accuracy numbers, so test quality on your own tasks.

Worked example (illustration, not a measurement)

Take a 24-billion-parameter model such as Liquid AI's LFM2.5-24B-A2B at Q4_K_M:

• 24,000,000,000 x 4.8944 / 8 = 14.68 GB (13.67 GiB)
• At Q8_0: 24e9 x 8.5008 / 8 = 25.5 GB. At Q3_K_M: about 12.0 GB.
• Unquantized 16-bit: 24e9 x 16 / 8 = 48 GB.

Real files differ a little because some tensors keep higher precision. As a check, Janas-LLM lists its Qwen3-Next-80B-A3B file as 48.4 GB. The formula at Q4_K_M gives 48.9 GB for 80B parameters, about 1 percent off. The same file implies 48.4e9 x 8 / 80e9 = 4.84 bits per weight.

The file is only the start. The context (the "KV cache") also takes memory and grows with prompt length, and your operating system and browser need room. Janas-LLM's long-context tests used an eight-bit KV cache for that reason. Treat file size as a floor, not a budget.

Mixture-of-experts: memory versus speed

A dense model uses every weight for every token. A mixture-of-experts model has many specialist sub-networks ("experts") and a router picks a few per token. For LFM2.5-24B-A2B the model card lists 64 experts per MoE block with top-4 routing, 24.0B total parameters and 2.45B active.

Two consequences follow:

1. Memory is set by total parameters. All 24B must be available, so the file is about 14.7 GB at Q4_K_M, not the 1.5 GB an "active only" size would suggest.
2. Speed is set by active parameters. Each token reads roughly 2.45e9 x 4.8944 / 8 = 1.5 GB of weights, similar to a small dense model. That is why Liquid AI reports 184 tokens per second on an Apple M5 Max and 114 on an AMD Ryzen AI Max+ 395 (the company's figures, measured with llama.cpp at Q4_K_M and a 4K-token input).

The trade-off: you need the memory of a big model but get near the speed of a small one. If the model cannot fit, speed collapses because experts must be fetched from somewhere slower. That is the situation in the next example.

Example 1: Janas-LLM streams experts from disk

Janas-LLM is a free C program (GPL v3 or later) that keeps weights on disk, streams the experts a token needs from an NVMe SSD and caches the busy ones in free memory. Its README calls it "an experiment, not a product", says the API and command line change without notice, and says stability is not verified.

Setup (Linux x86-64, from the README):

curl -LO https://github.com/prabanta-dev/janas/releases/latest/download/janas-linux-x86_64.tar.gz
tar xf janas-linux-x86_64.tar.gz && cd janas
./janas-chat

On Windows 10 (1903 or later) or 11, download janas-windows-x86_64.zip, unpack it and run .\janas-chat in the janas folder. Models come already converted from Hugging Face. --cache <GiB> sets a hard memory limit, and JANAS_GPU=0 turns the GPU path off.

Developer's results (not independently reproduced). Test machine: Debian 13, Intel Core Ultra 9 185H, integrated Arc GPU, 32 GB RAM, NVMe SSD reading about 5.7 GB/s. On 21 October 2026 the README reported, for Qwen3-Next-80B-A3B, 68.4 tokens per second reading a prompt and 26.4 writing, against 17.7 and 3.0 for llama.cpp (Vulkan backend, build 2b18470). The earlier figures in our coverage (37 tokens per second with draft-model help, 24 without) came from 11 October runs.

Why disk speed matters, as arithmetic (illustration). Suppose all 3B active parameters had to come from disk for every token: 3e9 x 4.8944 / 8 = 1.84 GB per token. At 5.7 GB/s that caps you near 3.1 tokens per second. Janas reports 26.4, so most reads must hit the memory cache; at that speed the disk can supply at most about 12 percent of each token's bytes. Free memory is the lever. In one test with a browser and an editor open, the model kept about three quarters of its speed but waited five times longer on the disk.

Limits: one test machine, Intel only. AMD CPUs, discrete GPUs, slower disks and other memory sizes are untested.

Example 2: LFM2.5-24B-A2B for local agents

Liquid AI's model is open-weight and always reasons before answering. From the Hugging Face model card: 40 layers, a 131,072-token context, a 128K vocabulary, and variants in GGUF, ONNX, MLX 4-bit and FP8 plus a base model. Recommended sampling is temperature 0.1, top_k 50 and repetition penalty 1.05. The card does not publish GGUF file sizes, so use the formula above and check the actual file listing.

Liquid AI's own benchmarks (not independently verified) show it ahead on tool-use and instruction-following tests, for example 27.69 on AutomationBench against 8.72 for Qwen3.6-35B-A3B, and 96.75 on IFEval against 93.06. The same table has Qwen3.6-35B-A3B ahead on SWE-Bench Verified (66.20 against 51.00), GPQA Diamond (83.69 against 74.42) and SciCode (37.35 against 32.71). Liquid AI suggests retrieval or web search for knowledge-heavy work. A draft model for speculative decoding was listed as "coming soon".

Setting up a local runner

A correction to older write-ups: Liquid AI's post lists 8080 for llama.cpp, but the current llama.cpp server README states a default of 9931 (8080 appears only in its Docker examples), and you can change it with --port. Ollama's OpenAI-compatible API offers /v1/chat/completions, /v1/models and /v1/responses, and requires an API key value that it ignores. For the other servers, Liquid AI lists port 8000 for vLLM (vllm serve "LiquidAI/LFM2.5-24B-A2B") and 30000 for SGLang. Janas-LLM's server also speaks Ollama-style (start it with --port 11434) and Anthropic-style APIs.

A minimal test against any OpenAI-compatible server:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma4:e2b","messages":[{"role":"user","content":"Say hello"}]}'

Change the port and model name for other runners. For an app built around local chat, see our look at a private budgeting app with local AI chat.

Measure tokens per second yourself

Two numbers matter: prompt processing (reading) and generation (writing). llama.cpp's llama-bench reports them as pp and tg:

./llama-bench -m model.gguf -p 1024 -n 128 -r 3

-p is prompt tokens (default 512), -n is generated tokens (default 128) and -r is how many repetitions are averaged (default 5). Janas ships janas-bench, for example janas-bench --prompt 1024 --gen 16 --rounds 2. One caveat from the Janas README: llama-bench writes without first processing a prompt, while janas-bench writes after 1,024 tokens, so compare like with like. Close heavy apps first and note which backend you used.

Which approach suits whom

Checklist

• Compute file size with the formula, then add room for context and the OS.
• Check free RAM and whether your drive is NVMe.
• Benchmark reading and writing separately, on an idle machine.
• Add retrieval or search instead of expecting a small model to know everything.

Who should skip it

• Anyone who needs stability: the Janas-LLM README says the API and command line change without notice and that stability is not verified.
• Anyone on AMD CPUs, a discrete GPU, a slower disk or a different memory size than 32 GB: the README says these are untested. It also warns that using the GPU once caused a driver reset that took the desktop with it, and that the engine can push a machine with little free memory into swap.
• Anyone who wants LFM2.5-24B-A2B as a general knowledge assistant: the model card says it is not recommended for knowledge-heavy tasks on its own and suggests pairing it with retrieval or web search. Its licence is listed as "lfm1.0" and the terms were not reviewed here, so read them before commercial use.

Alternatives worth a look

• llama.cpp: the MIT-licensed engine this guide benchmarks against; its README lists many backends and install routes (script, Docker, pre-built binaries, source).
• Ollama: a desktop app and command line for macOS, Windows and Linux that downloads and serves models for you, trading some low-level control for an easier start.
• LM Studio: its docs describe starting a local server from the Developer tab or with lms server start. Pricing and licence terms were not stated on the pages fetched, so check them yourself.

Frequently asked questions

Can a laptop run a model larger than its RAM? Janas-LLM reports running a 48.4 GB file on 32 GB by streaming experts, but speed then depends on the SSD and free memory, and the project is experimental.

Do I need a GPU? Janas-LLM is built to run with or without one. In llama.cpp the -ngl option sets how many layers go to the GPU.

Is a local model as good as a cloud model? Not for every task. Liquid AI's own table shows a larger rival ahead on coding and knowledge tests.

Benchmark figures are the developers' own and have not been independently reproduced. Computed examples are illustrations using the stated assumptions.

Sources

• https://raw.githubusercontent.com/prabanta-dev/janas/main/README.md
• https://raw.githubusercontent.com/prabanta-dev/janas/main/docs/serving.md
• https://huggingface.co/LiquidAI/LFM2.5-24B-A2B
• https://raw.githubusercontent.com/ggml-org/llama.cpp/master/README.md
• https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/server/README.md
• https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/llama-bench/README.md
• https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/quantize/README.md
• https://docs.ollama.com/quickstart.md
• https://docs.ollama.com/api/openai-compatibility.md
• https://ollama.com/download
• https://lmstudio.ai/docs/developer/openai-compat
• https://lmstudio.ai/docs/developer/core/server

Related: Apogee: A Private AI Summarizer That Runs in Your Browser · Claude Code and Codex Tools: Cut Costs, Mix Models, Monitor