Training, fine-tuning or inference
- Inference (running an existing model, for example a chatbot over your documents or image recognition on CCTV) needs enough GPU memory to hold the model, and usually one or two GPUs.
- Fine-tuning (adapting a model to your data) needs more GPU memory and benefits from several GPUs.
- Training from scratch needs many GPUs with fast links between them. Most businesses do not need this.
GPU memory comes first
The model has to fit in GPU memory (VRAM). As a rough guide, a model needs about 2 GB of VRAM per billion parameters at full 16-bit precision, and about a quarter to half of that when compressed (quantized) to 4 or 8 bits, plus room for the conversation context.
| Model size | Compressed (4-bit), roughly | Typical setup |
|---|---|---|
| 7 to 8 billion parameters | 6 to 8 GB | One 16 to 24 GB GPU |
| 13 to 14 billion | 10 to 12 GB | One 24 GB GPU |
| 30 to 34 billion | 20 to 24 GB | One 48 GB GPU or two 24 GB GPUs |
| 70 billion | 40 to 48 GB | Two 48 GB GPUs, or four 24 GB GPUs |
These are estimates for planning; the exact figure depends on the model, the software and the context length.
Server, CPU and RAM
- Slots and space: data-centre GPUs are full-height, double-width cards. Check how many the server supports and whether it needs a GPU riser or GPU enablement kit.
- CPU: the CPU mostly feeds the GPUs; a mid-range dual Xeon is usually enough. More PCIe lanes help with several GPUs.
- System RAM: plan at least as much system memory as total GPU memory, often double.
- Storage: fast NVMe or SSD storage for models and datasets.
Power, cooling and space
- Each data-centre GPU can draw 250 to 350 watts or more. A 4-GPU server can need 2 kW or more, so check the power supplies, your UPS and the rack's circuit.
- GPU servers are loud and hot. They belong in a server room or rack with good airflow, not under a desk.
- Passively cooled data-centre GPUs rely on the server's own fans; they will overheat in a normal desktop case.
Own server vs cloud GPUs
Cloud GPUs are good for short experiments. If you run AI every working day, or your data must stay on your premises, owning a refurbished GPU server often pays for itself within months. Renting a server is a middle path when you want to test before you buy: see our server rentals.
Tell us the models you want to run and we will suggest a configuration. Read the refurbished server buying guide for what to check before you buy.
Common questions
Can I use a gaming graphics card in a server?
Sometimes, but most gaming cards are too big, take power from the wrong side or need airflow a rack server does not provide. Data-centre cards are designed for servers.
How many GPUs do I need for a company chatbot?
For a small team running a 7 to 14 billion parameter model over company documents, one GPU with 24 GB of memory is a common starting point.
Is a refurbished GPU server reliable?
A well-tested refurbished server with a warranty is reliable for AI work. Ask for a long burn-in test with the GPUs under load before delivery.