Running a large language model on premises
What a self hosted LLM needs in hardware, software and people, and when your own cloud tenancy is the better home.
Last updated 15 October 2026
The short answer
Yes, you can run a capable large language model on your own servers. Open weight models can be downloaded and run on GPU hardware in your data centre, so documents, prompts and answers need never leave it. The work is in choosing the right model and hardware, building the platform around it, and running it well over time.
Open weight models
An open weight model is one whose trained weights are published, so you can run it yourself rather than call someone else's service. There is now a wide choice, from small models that run on a single GPU to large ones that need several. Licences vary: some allow broad commercial use, and others restrict it. Check the licence before you build on a model.
The hardware
Language models run on GPUs, and GPU memory is usually the limit. As a rough rule, a model's weights need about two bytes per parameter at 16 bit precision. A 70 billion parameter model therefore needs around 140GB of GPU memory for its weights alone, before any memory for the conversations in progress. Quantisation, which stores weights at lower precision, can cut that substantially for a small loss in quality.
The other driver is concurrency: how many people will ask questions at the same time, and how long their documents and answers are. Sizing for the busiest hour, not the average, avoids a platform that slows to a crawl at nine in the morning.
The software around the model
The model is one component. A usable platform also needs:
- an inference server that loads the model and serves requests efficiently
- an orchestration layer, often Kubernetes, to run and scale the components
- the retrieval pipeline: document ingestion, embeddings and a vector database
- identity and access control tied to your existing directory
- logging, monitoring and alerting
- infrastructure as code, so the whole thing can be rebuilt
Choosing a model
Choose on evidence, not leaderboards. The questions that matter are how well the model handles your tasks and your documents, whether its licence fits your use, whether it fits the hardware you have, and how often it is updated. The only reliable test is a set of real questions from your own work, answered and checked. That is what a proof of concept is for.
Running it over time
A self hosted LLM is a service, not a project. Someone has to patch the servers and the software, watch capacity, test and roll out new model versions, check that answer quality holds as documents change, and support the people using it. Plan for that from the start, whether your own team runs it or a partner runs it alongside them.
On premises, in your cloud, or both
On premises gives the most control, and suits organisations with a data centre, existing GPU capacity or data that must not leave the building. Your own cloud tenancy gives GPU capacity without buying hardware, while the data stays in a subscription you control. A hybrid keeps documents on premises and runs inference in your cloud, or the other way round. Our private AI platform build covers all three.
Start with an assessment.
An assessment sets out the hosting choices for your workload and your estate before you buy anything.
Thank you. Your request is on its way. We will reply from info@tlgsolutions.co.uk to arrange a first discussion.
That did not send. Please email info@tlgsolutions.co.uk instead, and we will reply from there.