GPUs run LLMs because memory movement is the real workload

Large language models perform enormous amounts of parallel arithmetic, but the practical hardware limit is often simpler: whether the model, its context and its active users fit in accelerator memory and can be moved through that memory fast enough.

A GPU is often described as a faster processor.

That description is convenient and incomplete.

A modern CPU is designed to execute varied instruction streams, respond quickly to branches, manage operating systems and perform a great deal of irregular work. A GPU is designed to apply similar operations across large collections of data in parallel.

Large language models contain exactly that kind of work.

During inference, the system repeatedly applies large matrices of learned parameters to the current representation of the input. Every generated token requires another pass through much of the model.

The arithmetic is substantial.

The movement of parameters and intermediate state is often the harder physical problem.

The central test

An LLM accelerator should be selected from model memory, memory bandwidth, context, concurrency and software support—not from a headline count of cores or operations per second.

A language model is mostly a large numerical structure

A model described as “7B” contains roughly seven billion learned parameters.

Those parameters must be represented in memory.

At 16 bits per parameter, the weights alone require approximately:

7 billion × 2 bytes ≈ 14 GB
13 billion × 2 bytes ≈ 26 GB
70 billion × 2 bytes ≈ 140 GB

Reducing precision changes the capacity requirement:

Model size	16-bit weights	8-bit weights	4-bit weights
7B	~14 GB	~7 GB	~3.5 GB
13B	~26 GB	~13 GB	~6.5 GB
70B	~140 GB	~70 GB	~35 GB

These are minimum weight estimates.

A real runtime also needs memory for:

temporary activations
execution buffers
the attention cache
model metadata
kernels and runtime state
batching
memory fragmentation
parallelism between devices

A 70B model quantised to four bits may have approximately 35GB of raw weights.

That does not mean a 40GB card will provide a practical 70B service.

The model may load and still leave too little space for useful context or concurrency.

Why GPUs suit the arithmetic

Transformer models rely heavily on matrix multiplication.

The same operation is applied across many elements, heads, tokens and batches. This parallel structure maps well to GPU execution units and specialised matrix hardware.

A CPU can run the same mathematical operations.

It has fewer, more general-purpose cores and usually much lower dedicated memory bandwidth. It remains valuable for:

tokenisation
request handling
orchestration
retrieval
data preparation
small models
low-throughput execution
tasks with irregular control flow

The GPU is not replacing the CPU.

It is accelerating the part of the workload whose structure rewards parallel execution.

Request handling and orchestration
                CPU
                 |
                 v
        Model tensors and kernels
                GPU
                 |
                 v
          Generated tokens
                 |
                 v
      Validation and application logic
                CPU

A good inference server is a balanced CPU–GPU system.

An expensive GPU waiting for the CPU, storage or network is still waiting.

Memory capacity decides whether the model fits

The first accelerator question is capacity.

Can the required model representation, context and runtime fit in one device?

A single-device deployment is usually easier to operate. It avoids distributing each layer or tensor across several cards and reduces inter-device communication.

When the model does not fit, the choices include:

lower precision
a smaller model
CPU offload
splitting layers across GPUs
tensor parallelism
pipeline parallelism
a larger-memory accelerator

Each choice changes latency, throughput, software complexity and failure behaviour.

Quantisation is useful because it reduces memory and memory traffic. It can also change model behaviour and may require hardware or kernels optimised for the selected format.

The smallest representation is not automatically the best.

The selected precision should be evaluated against the tasks the organisation actually needs.

Context and concurrency consume the space left over

The attention mechanism needs information about previous tokens.

Inference engines commonly preserve key and value tensors so the model does not recompute the entire conversation for every new token. This is the KV cache.

Its size grows with factors including:

model architecture
context length
number of active sequences
cache precision
batch organisation

This is why a model can work for one user and fail as a service.

A single short conversation leaves most memory available.

Several users with long contexts can consume the remaining capacity quickly.

Common mistake

Sizing the GPU from the model file alone.

Better framing

Size the complete serving workload: weights, runtime overhead, context length, active sequences and the latency target.

Long context is not a free model feature.

It is an infrastructure commitment.

Memory bandwidth determines how quickly weights can be reused

During token-by-token generation at low batch sizes, the accelerator may repeatedly move large quantities of model data to perform the arithmetic for each token.

If computation units are waiting for data, adding more theoretical compute does not improve throughput proportionally.

This is why high-end AI accelerators are built around unusually fast memory.

Current examples show the emphasis clearly. NVIDIA specifies 80GB of HBM and 3.35TB/s memory bandwidth for the H100 SXM, and 141GB of HBM3e with 4.8TB/s for the H200. AMD specifies 192GB of HBM3 and up to 5.3TB/s for the MI300X. ([NVIDIA][8])

Those numbers are not direct measures of real application performance.

They show what the hardware is designed to optimise: keeping large numerical models close to many parallel execution units.

A GPU with greater arithmetic throughput but insufficient memory may be unusable for the selected model.

A GPU with enough capacity but lower bandwidth may run it with unacceptable token latency.

Both dimensions matter.

GPU classes make different compromises

Consumer GPUs

Consumer cards can provide substantial compute and memory bandwidth at comparatively accessible prices.

NVIDIA’s GeForce RTX 5090, for example, has 32GB of GDDR7 memory and a stated 1.792TB/s memory bandwidth. ([NVIDIA][9])

These cards can be valuable for:

local experimentation
development
model evaluation
small-team inference
fine-tuning smaller models
batch work that tolerates interruption

Their limitations may include:

lower memory capacity
workstation-style active cooling
fewer enterprise management features
limited multi-GPU interconnect
different support expectations
physical size and power requirements
no practical partitioning between independent tenants

A consumer GPU can be excellent engineering hardware.

It should not be treated as a data-centre accelerator merely because the model loads.

Professional and workstation GPUs

Professional cards often provide more memory, ECC support, validated drivers and form factors intended for workstations or general-purpose servers.

AMD’s Radeon PRO W7900 provides 48GB of ECC-capable GDDR6 memory and 864GB/s peak memory bandwidth. NVIDIA’s RTX PRO 6000 Blackwell Server Edition provides 96GB of ECC GDDR7 and a stated 1.597TB/s. ([AMD][10])

This class can be attractive for:

engineering workstations
visual computing
moderate LLM inference
mixed graphics and AI
servers that cannot support SXM or OAM accelerator modules

The software stack remains decisive. A card with suitable memory may still be a weak choice if the required inference engine, kernels or framework are poorly supported.

Data-centre PCIe accelerators

Data-centre PCIe cards are designed for server airflow, sustained operation, ECC memory, management and enterprise software environments.

Many use passive cooling. They expect the server chassis to force sufficient air through the card.

Putting a passive server GPU into a workstation case without the intended airflow is not a small compromise. It can make the card thermally unsafe.

PCIe cards are flexible because they fit conventional server expansion slots.

Their connection to other GPUs may be much slower than their local memory.

SXM, OAM and integrated accelerator platforms

High-end accelerator modules place GPUs on specialised baseboards with high-bandwidth memory and direct GPU-to-GPU fabrics.

They are built for:

large-model training
high-throughput inference
tensor parallelism
distributed scientific computing

They are not ordinary expansion cards.

The baseboard, power delivery, cooling, firmware and fabric are part of one platform. Replacing the accelerator independently may not be as simple as replacing a PCIe card.

The organisation is purchasing a multi-accelerator system, not a server with several GPUs added.

Multi-GPU memory is not one large pool without cost

Four 24GB cards do not behave exactly like one 96GB card.

Software can divide a model across them, but the division creates communication.

If different layers reside on different GPUs, data must move between cards as inference progresses. If a tensor is split across several GPUs, partial results must be exchanged during each operation.

The speed of that exchange depends on:

PCIe topology
direct GPU interconnect
switch architecture
NUMA placement
collective communication libraries
model-parallel strategy

A topology diagram matters more than the total printed memory.

CPU 0 ── PCIe ── GPU 0
  |                 |
  |              GPU link
  |                 |
CPU 1 ── PCIe ── GPU 1

If the process runs on CPU 0 while feeding a GPU attached to CPU 1, data may cross the inter-socket link before it even reaches the accelerator.

If four GPUs share restricted PCIe paths, the server may contain enough total memory while communication limits useful throughput.

Multi-GPU design begins with the physical topology.

Training is not larger inference

Inference holds model weights and the state needed to generate outputs.

Training also needs to preserve or calculate:

gradients
optimiser states
activations
master copies of parameters
distributed-training buffers

The memory requirement can be several times larger than the weights alone.

Full training of a large model is therefore a different infrastructure class from running that model.

Fine-tuning sits between them.

Parameter-efficient methods can train a smaller set of additional weights while keeping most of the base model fixed. This can reduce memory and compute demand substantially, but it does not remove the need for data preparation, evaluation, checkpointing and reproducibility.

An organisation should state which activity it needs:

inference
batch inference
embedding generation
parameter-efficient fine-tuning
full fine-tuning
pretraining

“AI server” is too broad to size.

Throughput and latency pull the design in different directions

Batching several requests allows the GPU to reuse work and keep more execution units busy.

That can increase total token throughput.

It may also make an individual request wait longer before processing begins.

An interactive assistant values time to first token and steady per-user generation.

An overnight document-processing job values total throughput.

A shared inference service may need both.

The hardware should therefore be evaluated with workload-level measures:

time to first token
tokens per second per sequence
total tokens per second
maximum concurrent sequences
response under long context
energy per completed workload
behaviour after one GPU fails

A benchmark with an unknown batch size, prompt length or output length cannot answer these questions.

Peak operations per second is not a service objective.

Software support can outweigh the specification sheet

Accelerator performance depends on software capable of using the hardware.

The stack includes:

firmware
kernel driver
user-space runtime
communication libraries
model framework
inference engine
quantisation kernels
scheduler
monitoring
model format

A theoretically attractive accelerator may perform poorly if its kernels are immature for the selected model architecture.

A slightly less capable card with a stable and well-understood software path may provide a more ownable platform.

This is especially important when choosing between accelerator vendors or adopting a new GPU generation.

The migration test should include:

model loading
supported precision
attention implementation
multi-GPU operation
observability
failure recovery
upgrade and rollback
reproducible container or environment
availability of source and documentation

The GPU is one component.

The runtime determines whether its capability becomes infrastructure.

A practical LLM hardware review

Question	Weak signal	Stronger evidence
Will the model fit?	The model file is smaller than VRAM	Weights, buffers, KV cache and concurrency have been measured
Is the GPU fast enough?	It has high peak operations	Target latency and throughput pass on the real model
Is memory sufficient?	One prompt works	Required contexts and simultaneous users remain stable
Are several GPUs useful?	Their memory adds up	Topology and model-parallel communication have been tested
Is the server compatible?	The card fits the slot	Power, airflow, firmware, PCIe and NUMA placement are validated
Is quantisation acceptable?	The model becomes smaller	Task-specific evaluation confirms behaviour
Is the software mature?	A framework lists the GPU	Required kernels, runtime, monitoring and recovery work
Can it be operated?	The endpoint returns text	Capacity, queueing, evidence and failure ownership are defined
Can it be replaced?	Another GPU has similar specs	Model artifacts and evaluation run on a second supported stack
Is it sustainable?	The GPU is highly efficient	Useful work is measured against power and utilisation

The decision begins with the service, not the card.

The best GPU may be the one you do not need

Some workloads described as AI do not require a large language model.

A search index, classifier, rules engine, small embedding model or conventional database query may provide a better answer with less complexity.

Other workloads need an LLM but not the largest available one.

A smaller model can be easier to evaluate, cheaper to operate and simpler to recover. It may fit on one card, avoid distributed inference and provide lower latency.

The correct accelerator is not the fastest card the organisation can obtain.

It is the smallest complete platform that meets the required behaviour, capacity and operating standard.

One question for the next AI hardware review

Do not begin by asking which GPU has the most compute.

Ask:

For the model, context and number of simultaneous users we actually intend to support, where will the first bottleneck appear: memory capacity, memory bandwidth, interconnect, software or power?