GPU memory is the main LLM hardware requirement: the weights plus working memory for every conversation must fit. Up to about 30 billion parameters, one 48 to 96 GB GPU is enough; around 120 billion needs one 80 GB GPU or more. The largest open models need a full eight-GPU server.
The short answer
- Weights take about 2 GB of GPU memory per billion parameters at 16-bit. 8-bit halves that; 4-bit quarters it.
- gpt-oss-120b runs on one 80 GB GPU. Llama 4 Maverick and Mistral Large 3 need a full eight-GPU server.
- European clouds such as Verda rent every GPU class by the hour, from about €1.37 for an L40S to about €7.53 for a B300. Sizing can be tested before buying.
The self-hosted LLM guide covers the full decision. The guide to what drives inference cost turns hardware into a cost per token.
Why is GPU memory the main constraint?
A model reads all its active weights from GPU memory for every word it generates. So the weights must fit in one GPU, or in several working together. Compute sets the speed; memory decides whether the model runs at all.
Hugging Face puts weight memory at roughly 2 GB per billion parameters in 16-bit precision (bfloat16 or float16).1 8-bit halves that; 4-bit quarters it. Gemma 4 31B needs about 62 GB at 16-bit. Google states its unquantised weights fit on a single 80 GB H100.2
Mixture-of-experts (MoE) models change speed, not memory. Mistral Small 4 has 119 billion parameters, with only 6 billion active per token.3 All 119 billion must still be loaded. Memory follows total parameters; speed follows active ones.
Working memory for context grows with users
Each conversation also needs working memory on the GPU: the key-value cache (KV cache). NVIDIA says it grows with batch size, sequence length, layer count and hidden size. For Llama 2 7B, one 4,096-token conversation takes about 2 GB.4 Long documents and many users multiply that. The vLLM team found that serving systems before PagedAttention wasted 60 to 80 percent of this memory. PagedAttention cut the waste to under 4 percent.5 Sizing starts from context length and concurrency, not the model alone.
What does quantisation change?
Quantisation stores weights at lower precision, such as 8 or 4 bits instead of 16. It cuts memory; the effect on speed depends on hardware and inference software. Hugging Face notes it trades accuracy for memory, and 4-bit models can give different results than 8-bit or 16-bit ones.1 Some developers ship reduced precision themselves: gpt-oss-120b stores its expert weights in the 4-bit MXFP4 format.6 Any quantised model needs testing on the organisation's own tasks first.
Precision decides the memory bill
At 4-bit, weights need a quarter of the memory they take at 16-bit.
A 119-billion-parameter model needs about 238 GB for its weights at 16-bit, and about 60 GB at 4-bit.
GPU memory needed for model weights, by precision, GB
1 Weights only. The KV cache for context and concurrent users comes on top. Real 4-bit files are slightly larger, as some layers keep higher precision.
Source: Parameter counts from the model cards of IBM, utter-project, Google and Mistral AI; 2 bytes per parameter at 16-bit per Hugging Face; 8-bit and 4-bit scaled by bit width; accessed 28 September 2026
View data table
| Category | Series 1 | Series 2 | Series 3 |
|---|---|---|---|
| Granite 4.2 8B | 16 | 8 | 4 |
| EuroLLM 22B | 45 | 23 | 11 |
| Gemma 4 31B | 61 | 31 | 15 |
| Mistral Small 4 119B | 238 | 119 | 60 |
Series 1: 16-bit. Series 2: 8-bit. Series 3: 4-bit. Unit: GB
How much GPU memory do popular open models need?
The table groups the Model Index by size. Hardware in the first four rows is as each developer states it; the last row is indicative.
| Size class | Example models | Weights at 8-bit | Minimum hardware |
|---|---|---|---|
| Up to 31B, dense | Gemma 4, Granite 4.2, EuroLLM-22B | up to 31 GB | 1× H100 80 GB for Gemma 4 31B |
| About 110B to 120B, MoE | gpt-oss-120b, Llama 4 Scout | 109 to 117 GB | 1× 80 GB GPU at 4-bit |
| About 400B, MoE | Llama 4 Maverick | about 400 GB | One 8× H100 server, FP8 |
| About 675B, MoE | Mistral Large 3 | about 675 GB | One 8× H200 or B200 server, FP8 |
| 1.6T to 2.8T, MoE | DeepSeek V4 Pro, Kimi K3, Qwen3.8 | 1.6 to 2.8 TB | Full 8× B200 or B300 server, or more |
Runs on one GPU Needs a multi-GPU server
Sources: model cards and announcements of Google, IBM, utter-project, OpenAI, Mistral AI, Meta, DeepSeek, Moonshot AI and Alibaba; weights at 1 byte per parameter; last row from the Lindstead Model Index (indicative); accessed 28 September 2026.
gpt-oss-120b fits one 80 GB GPU because it ships in 4-bit.6 For Mistral Small 4, Mistral states a minimum of "4x NVIDIA HGX H100, 2x NVIDIA HGX H200, or 1x NVIDIA DGX B200", well above its 238 GB of 16-bit weights.3 Meta says Llama 4 Maverick fits one H100 host in FP8, and Llama 4 Scout one H100 at 4-bit.7 Mistral Large 3 fits one node of B200 or H200 GPUs in FP8, or of H100 GPUs in the 4-bit NVFP4 format.8
Which GPU suits a self-hosted LLM?
Six NVIDIA data-centre GPUs cover almost every enterprise deployment. They differ in memory and in how they link, which matters once a model spans several cards.
| GPU | Memory | Link between GPUs | Rental at Verda (FI) | Typical use |
|---|---|---|---|---|
| L40S | 48 GB | PCIe Gen4 | €1.37 per hour | Up to about 30B at 8-bit |
| RTX PRO 6000 | 96 GB | PCIe Gen5 | €1.75 per hour | Up to about 120B at 4-bit |
| H100 SXM | 80 GB | NVLink 900 GB/s | €3.17 per hour | gpt-oss-120b; 8× for Maverick |
| H200 SXM | 141 GB | NVLink 900 GB/s | €4.13 per hour | 8× for Mistral Large 3 |
| B200 | 180 GB | NVLink 1.8 TB/s | €6.10 per hour | 8× for the largest models |
| B300 | 288 GB | NVLink 1.8 TB/s | €7.53 per hour | 8× for the largest models |
PCIe card SXM module on an NVLink board
Sources: NVIDIA product pages and DGX user guides (memory, interconnect); verda.com/pricing, on-demand price per GPU, converted at the ECB reference rate of 2 October 2026; accessed 28 September 2026.
The European cloud GPU price index compares these rates across European and US providers, including H100s at Scaleway and OVHcloud and the Nebius price increase of 1 October 2026.
What does a self-hosted LLM server look like?
Besides GPUs, an inference server needs CPUs and system memory, fast storage for model files of hundreds of gigabytes, and network links to the applications. The GPUs dominate price and power draw.
What is the difference between PCIe and SXM GPUs?
PCIe cards such as the L40S and RTX PRO 6000 fit a standard server slot and talk over the PCIe bus. PCIe Gen5 offers 128 GB/s in both directions combined.9 SXM modules such as the H100, H200 and B200 sit on an NVIDIA HGX board with NVLink: 900 GB/s per GPU for H100 and H200, 1.8 TB/s for B200.910 A model split across eight GPUs exchanges data for every token; the faster link keeps the GPUs busy. For models that fit one card, PCIe is enough and simpler to host.
What does a server rack need?
Power and cooling set the limit in most existing data centres. NVIDIA specifies a maximum of about 10.2 kW for a DGX H100. A DGX B200, with eight B200 GPUs and 1,440 GB of GPU memory, reaches about 14.3 kW.1011 An RTX PRO 6000 Server Edition draws up to 600 W, an L40S up to 350 W.9 Many server rooms built for ordinary IT cannot power or cool 14 kW in one machine. That is one reason to rent eight-GPU capacity or place it in colocation. The comparison of on-premise and private cloud AI weighs these options.
Should an organisation start with one GPU or a full server?
Most organisations start small. One rented GPU in a European cloud can prove a use case with a model of up to about 120 billion parameters. It also yields the number no published sizing table can: measured throughput on the organisation's own prompts and documents. Owned hardware is then sized from measured load, not estimates.
Whether buying pays off depends on volume and on the API it replaces. The break-even calculator shows where an owned server matches an API bill. Lindstead offers infrastructure sizing per workload, comparing rented GPUs, owned servers and APIs.
What server software does a self-hosted LLM need?
The stack has four layers: operating system and GPU drivers; an inference server such as vLLM, which loads the model and batches requests;5 a gateway for authentication, access rights and logging; and monitoring of load, latency and errors. The self-hosted LLM guide covers the full architecture.
Hardware matters most where data may not leave the organisation: healthcare, the public sector and industrial companies protecting designs and process knowledge. Lindstead sizes GPU capacity for an organisation's models and workloads and deploys the first use case on the chosen infrastructure. Schedule a meeting to discuss hardware plans.
Frequently asked questions
-
One with enough memory for the model at its precision, plus room for context. Up to about 30 billion parameters at 8-bit, a 48 GB L40S or 96 GB RTX PRO 6000 holds the weights. Around 120 billion parameters need about 60 GB at 4-bit: an 80 or 96 GB GPU. The largest open models need a full eight-GPU H200, B200 or B300 server.
-
Yes, if quantised. At 16-bit its weights take about 140 GB, leaving no room for context even on a 141 GB H200. At 8-bit, about 70 GB, they fit a 96 GB RTX PRO 6000 with some headroom. At 4-bit, about 35 GB, they fit a 48 GB L40S.
-
For the weights, about 2 GB per billion parameters at 16-bit, 1 GB at 8-bit and 0.5 GB at 4-bit. Context memory comes on top and grows with conversation length and user count.
-
No. Models up to about 30 billion parameters run on PCIe cards such as the L40S or RTX PRO 6000 in a standard server. NVLink boards with H100, H200 or B200 GPUs are needed once a model spans several GPUs, as with open models of 400 billion parameters and up.
Sources
- Hugging Face, Optimizing LLMs for speed and memory (Transformers documentation), accessed 28 September 2026. huggingface.co/docs/transformers/main/en/llm_tutorial_optimization
- Google, Gemma 4 model card and announcement, accessed 28 September 2026. ai.google.dev/gemma/docs/core/model_card_4
- Mistral AI, Mistral Small 4, 16 March 2026, accessed 28 September 2026. mistral.ai/news/mistral-small-4
- NVIDIA Technical Blog, Mastering LLM Techniques: Inference Optimization, 17 November 2023, accessed 28 September 2026. developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization
- vLLM, Easy, fast and cheap LLM serving with PagedAttention, 20 June 2023, accessed 28 September 2026. vllm.ai/blog/2023-06-20-vllm
- OpenAI, gpt-oss-120b model card, accessed 28 September 2026. huggingface.co/openai/gpt-oss-120b
- Meta, Llama 4 Maverick and Scout model cards, accessed 28 September 2026. huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct
- Mistral AI, Mistral Large 3 model card, accessed 28 September 2026. huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512
- NVIDIA, L40S, RTX PRO 6000 Blackwell Server Edition, H100 and H200 product pages, accessed 28 September 2026. nvidia.com/en-us/data-center/h200
- NVIDIA, DGX B200 and HGX product pages, and DGX B300 user guide, accessed 28 September 2026. nvidia.com/en-us/data-center/dgx-b200
- NVIDIA, DGX H100 datasheet, accessed 28 September 2026. nvidia.com, DGX H100 datasheet
- Other model cards: IBM Granite 4.2, utter-project EuroLLM-22B, DeepSeek V4 Pro, Moonshot AI Kimi K3 and Qwen3.8-2.4T-A95B on Hugging Face; Verda GPU pricing, verda.com/pricing; all accessed 28 September 2026.