Self-hosted LLMs for European enterprises: the 2026 guide

A self-hosted LLM is a large language model on infrastructure the organisation controls: its own servers, rented racks or dedicated GPUs at a European cloud provider. Prompts and outputs stay in-house. The case rests on control and regulation; cost favours it only at high, steady volumes.

What is a self-hosted LLM?

It is an open-weight model that the organisation downloads and runs on hardware it owns or rents exclusively. Requests never reach the model maker. The organisation decides where it runs, who uses it, what is logged and when it is replaced.

Three look-alikes are not self-hosted:

• A closed-model API. OpenAI or Anthropic process every prompt, even when contracts limit data use.
• A private endpoint at a hyperscaler. An EU region of AWS, Azure or Google Cloud isolates the capacity, but the operator is a US company.
• A local LLM on a laptop. Fine for experiments, not a production service with access control, logging and support.

Is a private LLM the same as a self-hosted LLM?

Not necessarily. Vendors use "private LLM" for anything from a hyperscaler endpoint to on-premise servers. This guide calls a deployment self-hosted only if the organisation itself, or a provider under EU jurisdiction only, operates the hardware and controls who reads prompts, outputs and logs.

Why do European enterprises host their own LLM?

Boards in regulated sectors usually weigh five reasons, in this order.

1. Control over sensitive data. Client files, patient records, privileged legal documents and trade secrets stay in-house. For many use cases this alone decides.
2. Jurisdiction. Under the US CLOUD Act, US providers must disclose data in their possession, custody or control "regardless of whether" it is stored inside or outside the United States.10 Microsoft France told the French Senate under oath that it could not guarantee such a request would never be granted.11
3. Sector rules. DORA requires financial entities to manage ICT third-party risk and plan exits. In November 2025 the European supervisors designated 19 critical ICT providers, including AWS, Google Cloud and Microsoft.12 Health data is a special category under GDPR Article 9.17 NIS2 adds supply-chain duties for essential and important entities.
4. Independence from provider changes. A provider can change, reprice or retire a hosted model. A self-hosted model stays as tested.
5. Cost at volume. At high, steady volumes, GPUs can beat per-token pricing. It is rarely the main argument.

Demand is broad. In an Accenture vendor survey of 1,928 organisations (November 2025), 62 percent of European organisations said they were actively seeking sovereign AI solutions.9 In December 2025 HSBC said it would use "self-hosted AI models that operate on HSBC's internal technology systems" under its partnership with Mistral AI.8 Sector pages cover financial services, healthcare, legal and professional services and the public sector.

Where to host an LLM: on-premise, private cloud or European GPU cloud?

Four options differ in hardware ownership, upfront capital, operating effort and jurisdiction.

Four hosting options, from most to least control over the hardware

EU jurisdiction only US jurisdiction can also apply

Sources: 18 U.S.C. 2713; provider websites; Lindstead classification. Accessed 28 September 2026.

On-premise means the most control and the heaviest burden: power, cooling, spare parts and GPU repairs. Colocation removes the building, not the hardware. An EU-owned GPU cloud rents dedicated GPUs, and only EU law applies to the operator. A US hyperscaler region in the EU has the widest catalogue, but the CLOUD Act question remains: jurisdiction follows ownership, not data centre location.

On-premise or private cloud AI for regulated industries weighs owning against renting. The European cloud GPU price index tracks prices monthly.

Which models can be self-hosted?

Any model with published weights. Capability, size and licences vary. Below is a shortlist from the Lindstead Model Index by maker origin; hardware is indicative.

Leading open-weight models by origin, September 2026

Permissive licence Custom licence, review terms Restrictions for large or EU companies

Sources: model cards and licence texts on Hugging Face and the makers' websites, as listed in the Lindstead Model Index; accessed 28 September 2026. MoE: mixture of experts; "active" is the share of parameters used per token.

What do open-weight licences allow a large company to do?

"Open" does not mean unrestricted. The Mistral Medium 3.5 licence grants no rights to a company whose global consolidated monthly revenue "exceeds $20 million" in the preceding month. Such companies need a commercial licence.18 Meta's Llama 4 policy withholds multimodal rights from EU-based companies.19 Kimi K3, the largest Qwen3.8 and other Chinese flagships now use custom licences with separate terms for internal use and resale. Apache 2.0 or MIT still covers Mistral Small 4 and Large 3, gpt-oss-120b, Gemma 4, Granite 4.2, EuroLLM-22B and DeepSeek V4 Pro, among others. Counsel should read each licence before production; Lindstead's model selection work includes this review.

Can a European company use Chinese open models?

Legally, usually yes, if the licence allows it and the weights run in-house: no data flows to the maker. The open questions are behaviour and security. The US standards body NIST found that agents built on DeepSeek's most secure model were on average 12 times more likely than US frontier models to follow malicious instructions. DeepSeek models also echoed four times as many inaccurate and misleading Chinese Communist Party narratives as US reference models.20 Local weights remove the data transfer, not what was trained into the model. Many boards set a model-origin policy and use a European or US model one tier lower for sensitive work.

What hardware does a self-hosted LLM need?

GPU memory comes first. The weights must fit, plus the KV cache, which grows with context length and concurrent users. Each parameter needs 2 bytes at 16-bit precision, 1 byte at 8-bit and half a byte at 4-bit. Quantisation to 8 or 4 bits shrinks the model and usually costs some quality; measure it on real tasks.

Memory per GPU: NVIDIA L40S 48 GB, H100 80 GB, RTX PRO 6000 Blackwell Server Edition 96 GB, H200 141 GB.2426 An eight-GPU DGX B200 has 1,440 GB in total, or 180 GB per GPU.25 Three tiers cover most needs:

• Up to about 30 billion parameters (Gemma 4, Granite 4.2, EuroLLM-22B): one L40S or RTX PRO 6000. Hetzner rents a dedicated RTX PRO 6000 server for €1,199 a month plus a €599 setup fee.7
• Around 120 billion parameters (gpt-oss-120b, Mistral Small 4): one 80 to 96 GB GPU for gpt-oss, two H200 for Mistral Small 4 at full precision.
• Frontier-scale open models (DeepSeek V4 Pro, Kimi K3, Mistral Large 3): a full eight-GPU H200 or B200 node.

Throughput, the tokens per second served to all users together, depends on GPU, inference server, batch size and prompt and answer length. Measure it with a load test before buying. Full method: hardware for Llama, Mistral or Qwen.

What does a self-hosted LLM architecture look like?

Production needs seven layers, not just a model on a GPU. Requests pass from the application through the gateway to the inference server. Retrieval adds company documents to the prompt. Every step is logged and monitored.

Reference architecture for a self-hosted LLM, top to bottom

Sources: vLLM, SGLang, llama.cpp and Safetensors documentation; OWASP Top 10 for LLM Applications 2025. Accessed 28 September 2026. Examples are illustrative, not a ranking.

Two choices matter most. The gateway is the control point. It handles single sign-on, document permissions, quotas and the audit log. Applications never call the inference server directly. Retrieval must respect access rights. Otherwise the index shows confidential documents to anyone who asks the right question.

Which inference server to use

The inference server loads the model, batches requests and streams answers. Common open-source options:

• vLLM offers an HTTP server compatible with the OpenAI Completions, Chat Completions and Responses APIs, so existing applications switch easily.27
• SGLang is an open-source serving framework for language and multimodal models.29
• llama.cpp runs smaller models on CPU or mixed hardware, at the edge or on workstations.30

Hugging Face's Text Generation Inference is in maintenance mode; Hugging Face now recommends vLLM, SGLang and local engines such as llama.cpp.28 Test two candidates on the real workload, not on published benchmarks.

What does it cost to self-host an LLM, and when does it break even?

Self-hosting swaps a cost per token for a fixed cost per server. The payoff depends on volume, utilisation and, above all, the API it replaces. Lindstead's break-even analysis uses a vendor estimate of €20.22 per server-hour for an owned eight-GPU B200 system at 70 percent utilisation over three years: about €14,790 a month before staff.1

Renting lowers the entry point. At list prices, Verda rents an H100 for €3.14 an hour and Nebius for €4.01 from 1 October 2026, against €6.13 at AWS and €9.86 at Google Cloud.456 Dollar prices are converted at the ECB reference rate of 2 October 2026.

Four factors usually push break-even further out:

• Staff. Patching, monitoring, upgrades and on-call duty are not in the server price.
• Utilisation. An idle server costs money at night and at weekends; an API does not.
• Price trends. API prices per level of capability have fallen steeply every year,3 while GPU rental prices rose in 2026.
• Quality. A cheaper model that needs more human review is not cheaper.

Academic research agrees. Owned hardware pays back within about three months for small models, within 6 to 24 months for medium models and often only after more than two years for the largest. It is mainly viable at high, steady volumes or under strict data-residency rules.2 LLM inference cost explained has the full logic; Lindstead's infrastructure and cost work models it per organisation.

How to secure and evaluate a self-hosted LLM

Self-hosting removes third-party access, not model risk. The OWASP Top 10 for LLM Applications 2025, the most used checklist, ranks prompt injection first, then sensitive information disclosure, supply chain, and data and model poisoning.32

• Prompt injection. The UK National Cyber Security Centre warns it may never be fully mitigated.33 Limit what the model can do: least-privilege access, human approval for actions, output checks before anything runs.
• Supply chain. Download weights only from the maker's official repository and verify checksums. Prefer Safetensors, which stores tensors without the code-execution risk of Python pickle files.31
• Poisoning. Anthropic, the UK AI Security Institute and the Alan Turing Institute showed around 250 poisoned documents can plant a backdoor in models of very different sizes.34 Control what enters fine-tuning data and the retrieval index.
• Sensitive information. Prompts and logs hold the same confidential data as the sources. Encrypt them, set retention and restrict access.

How to evaluate quality in production

Before go-live, build an evaluation set: real questions with reference answers, reviewed by domain experts. Run it on every candidate model, before every model or prompt change, and on a sample of live traffic. Log model version, prompt template, retrieved documents and latency per request. Track refusal rate, accuracy and user feedback. New open models appear monthly; a fixed evaluation set makes upgrades defensible.

What does regulation require of a self-hosted LLM?

Self-hosting changes who is responsible, not whether rules apply.

• AI Act roles. An organisation that uses an AI system under its own authority is a deployer. It becomes a provider of a high-risk system if it puts its name on the system or substantially modifies it (Article 25).15 Fine-tuning makes it a provider of a general-purpose model only in significant cases. The Commission's indicative criterion is training compute of more than a third of the original model's.16
• AI Act dates. The Digital Omnibus on AI entered into force on 27 July 2026. It moved high-risk obligations to 2 December 2027 for stand-alone systems and 2 August 2028 for AI in regulated products. Article 50 transparency duties apply from 2 August 2026. Article 4 now requires measures that support AI literacy, not a guaranteed level.14
• GDPR. Self-hosting avoids a transfer to a model provider, but a lawful basis, purpose limitation and security of processing still apply. Large-scale processing of sensitive data usually needs a data protection impact assessment under Article 35. A GPU cloud provider that hosts the data is a processor under Article 28.17
• DORA. Financial entities must list the GPU cloud or hardware supplier in their ICT third-party register and exit plans.12
• NIS2. The Dutch Cyberbeveiligingswet has applied since 15 August 2026. It brings duties of care, including supply-chain security, to more than 8,000 organisations.13

The briefing Where should your AI run? shows how these rules interact.

How to deploy a self-hosted LLM, step by step

Eight steps take one use case to production. Each ends in an output that security, compliance and the business can sign off.

1. Pick one use case with sensitive data and a measurable outcome. Document review or internal search are typical. Define "good" in numbers.
2. Classify the data. The rules involved (GDPR Article 9, professional secrecy, DORA, NIS2) decide which hosting options are acceptable.
3. Build the evaluation set. Domain experts supply real questions and reference answers before any model is chosen.
4. Shortlist models and check licences. Test two or three models from the Model Index on the evaluation set; review each licence and origin.
5. Choose hosting and size the hardware. Rent from an EU-owned provider unless data rules require on-premise. Size memory from the model, throughput from a load test.
6. Build the platform. Inference server, gateway with single sign-on and audit log, retrieval that respects access rights, monitoring and encrypted log storage.
7. Secure and test. Run the OWASP checklist, red-team for prompt injection and data leakage, and rerun the evaluation set.
8. Go live with a limited group, then hand over. Monitor quality and cost, document the runbook, train the operators and schedule model reviews.

Frequently asked questions

• What is a self-hosted LLM?

  A large language model on infrastructure the organisation controls: its own servers, colocation racks or dedicated cloud GPUs. The organisation runs the model and decides who can send prompts and read outputs. No third-party model provider sees the data.

• Is a private LLM the same as a self-hosted LLM?

  Not always. Vendors use the term for anything from a US hyperscaler endpoint to a server in the basement. Self-hosted means the organisation runs the model itself, under a jurisdiction it chooses.

• Is self-hosting an LLM cheaper than using an API?

  Only at high, steady volumes, and only against some APIs. On Lindstead's analysis, an owned eight-GPU B200 server at about €14,790 a month matches a premium closed-model API below one billion tokens a month, but a cheap hosted open-model API only above twenty billion. Staff, power and idle capacity push break-even further out.

• What is the best self-hosted LLM for an enterprise?

  No single model is best; it depends on task, languages, hardware and licence. Mistral Small 4, gpt-oss-120b and Gemma 4 have permissive licences and run on one or a few GPUs. The strongest open models are Chinese, and several leading licences restrict large or EU-based companies.

• Can a self-hosted LLM be GDPR compliant?

  Yes, but not automatically. A lawful basis, purpose limitation, access control and retention rules for prompts and logs still apply. Large-scale or sensitive processing usually needs a data protection impact assessment. Self-hosting only removes the transfer to an external model provider.

• What hardware do we need to host our own LLM?

  Models up to about 30 billion parameters run on one 48 to 96 GB GPU. At around 120 billion parameters, gpt-oss-120b runs on one 80 to 96 GB GPU. Mistral Small 4 at full precision needs two H200 GPUs. Frontier-scale open models such as DeepSeek V4 Pro need a full eight-GPU H200 or B200 node.

• How long does it take to deploy a self-hosted LLM?

  Serving a model on rented GPUs is quick. Most time goes into data classification, access control, retrieval, evaluation and sign-off by security and compliance. Buying hardware adds procurement and delivery time, so many organisations start in a European GPU cloud.

Sources

1. Mercatus, B200 server price and cost per hour, 16 July 2026 (vendor analysis). www.mercatus-ai.com/blog/b200-server-price
2. Pan et al. (Carnegie Mellon University and independent researchers), cost-benefit analysis of on-premise LLM deployment, arXiv 2509.18101. arxiv.org/abs/2509.18101
3. Epoch AI, LLM inference price trends, 12 March 2025. epoch.ai/data-insights/llm-inference-price-trends
4. Verda GPU pricing. verda.com/pricing
5. Nebius pricing, including changes from 1 October 2026. nebius.com/prices
6. getdeploying.com, NVIDIA H100 rental price comparison (AWS, Azure, Google Cloud). getdeploying.com/gpus/nvidia-h100
7. Hetzner, GEX131 dedicated GPU server announcement. www.hetzner.com/pressroom/new-gex131/
8. HSBC, HSBC and Mistral AI join forces to accelerate AI adoption across global bank, 1 December 2025. www.hsbc.com/news-and-views/news/media-releases/2025/hsbc-and-mistral-ai-join-forces-to-accelerate-ai-adoption-across-global-bank
9. Accenture, Europe seeking greater AI sovereignty, November 2025 (vendor survey). newsroom.accenture.com/news/2025/europe-seeking-greater-ai-sovereignty-accenture-report-finds
10. 18 U.S.C. 2713, Required preservation and disclosure of communications and records (CLOUD Act). www.law.cornell.edu/uscode/text/18/2713
11. The Register, Microsoft exec admits it cannot guarantee data sovereignty, 25 July 2025. www.theregister.com/off-prem/2025/07/25/microsoft-exec-admits-it-cannot-guarantee-data-sovereignty/458553
12. EIOPA, ESAs designate critical ICT third-party providers under DORA, 18 November 2025. www.eiopa.europa.eu/european-supervisory-authorities-designate-critical-ict-third-party-providers-under-digital-2025-11-18_en
13. Rijksoverheid, Cyberbeveiligingswet and Wet weerbaarheid kritieke entiteiten in force from 15 August 2026, 7 July 2026. www.rijksoverheid.nl/actueel/nieuws/2026/07/07/cyberbeveiligingswet-en-wet-weerbaarheid-kritieke-entiteiten-vanaf-15-augustus-2026-van-kracht
14. Lewis Silkin, The Digital Omnibus on AI enters into force today, 27 July 2026. www.lewissilkin.com/insights/2026/07/27/the-digital-omnibus-on-ai-enters-into-force-today-102nedo
15. Regulation (EU) 2024/1689 (AI Act), EUR-Lex. eur-lex.europa.eu/eli/reg/2024/1689/oj
16. European Commission, Guidelines on the scope of the obligations for general-purpose AI models, C(2025) 5045, 18 July 2025. ai-act-service-desk.ec.europa.eu/sites/default/files/2025-07/guidelines_on_the_scope_of_the_obligations_for_generalpurpose_ai_models_established_by_regulation_1cx2atxgq79us4n3x8jfgyy1qlm_118340-3.pdf
17. Regulation (EU) 2016/679 (GDPR), EUR-Lex. eur-lex.europa.eu/eli/reg/2016/679/oj
18. Mistral AI, Mistral-Medium-3.5-128B model card and licence, Hugging Face. huggingface.co/mistralai/Mistral-Medium-3.5-128B
19. Meta, Llama 4 Acceptable Use Policy. www.llama.com/llama4/use-policy
20. NIST CAISI, Evaluation of DeepSeek AI models finds shortcomings and risks, 30 September 2025. www.nist.gov/news-events/news/2025/09/caisi-evaluation-deepseek-ai-models-finds-shortcomings-and-risks
21. OpenAI, gpt-oss-120b model card, Hugging Face. huggingface.co/openai/gpt-oss-120b
22. EuroLLM-22B-Instruct-2512 model card, Hugging Face. huggingface.co/utter-project/EuroLLM-22B-Instruct-2512
23. Google, Gemma 4 31B model card, Hugging Face. huggingface.co/google/gemma-4-31B-it
24. NVIDIA, H100 and H200 specifications (80 GB and 141 GB). www.nvidia.com/en-us/data-center/h200/
25. NVIDIA, DGX B200 specifications (1,440 GB across eight GPUs). www.nvidia.com/en-us/data-center/dgx-b200/
26. NVIDIA, RTX PRO 6000 Blackwell Server Edition (96 GB) and L40S (48 GB) specifications. www.nvidia.com/en-us/data-center/rtx-pro-6000-blackwell-server-edition/
27. vLLM documentation, online serving and OpenAI-compatible server. docs.vllm.ai/en/latest/serving/online_serving/
28. Hugging Face, Text Generation Inference documentation (maintenance mode notice). huggingface.co/docs/text-generation-inference/index
29. SGLang project repository. github.com/sgl-project/sglang
30. llama.cpp project repository. github.com/ggml-org/llama.cpp
31. Hugging Face, Safetensors documentation. huggingface.co/docs/safetensors/index
32. OWASP, Top 10 for LLM Applications 2025. genai.owasp.org/llm-top-10/
33. UK NCSC, on prompt injection, December 2025. www.ncsc.gov.uk/news/mistaking-ai-vulnerability-could-lead-to-large-scale-breaches
34. Anthropic, UK AI Security Institute and Alan Turing Institute, A small number of samples can poison LLMs of any size, October 2025. www.anthropic.com/research/small-samples-poison

Method: desk research of public primary sources, all accessed 28 September 2026. Figures are published list prices and specifications, not Lindstead measurements. Vendor-sponsored data is labelled. Corrections: contact@lindstead.com.