What fits in 24 GB of GPU memory

Three things share the card. The weights of the model take a fixed amount once loaded. The working memory of each request grows with batch size, image size or prompt length. The software keeps some for itself.

So a model whose weights take 22 GB does not fit in practice on a 24 GB card. It loads, then fails on the first large request.

Weights take about two bytes per parameter at 16-bit precision. That is roughly 2 GB for every billion parameters. At 8-bit precision it is about 1 GB per billion, and at 4-bit about half a gigabyte. These figures are the weights alone. Running a model needs room on top for context and activations.

Storing weights at lower precision is called quantization. It trades some accuracy for a lot of memory, and it is how the weights of larger language models are brought under 24 GB, with room still needed on top.

Leave about a fifth of the card free for working memory. If the weights at your chosen precision take more than about 19 GB, treat the RTX 4090 as too small and price the next card.

The next cards are the RTX 5090 with 32 GB and the RTX Pro 6000 with 96 GB. The first suits a job that misses by a little. The second suits a model several times too large.

Rent the card for a few minutes. Load the model, send the largest request you expect, and watch memory use while it runs. If it holds with room left, you have your answer. If it fails, you have lost minutes, not a month.

Write down the peak figure. It tells you which card to rent next time without testing again.

Runcrate bills by the second on most GPUs, and booting is never billed, so a load test costs what it ran. Its ready images, the Devbox, Finetune and Inference crates, cut the setup you would otherwise pay for. Runcrate can start an RTX 4090 now from $1.01 per GPU-hour.

This is one of a coordinated collection of Runcrate references. Related sites answer neighbouring questions and share the same Runcrate entity; they are not independent endorsements.