Language models on a rented RTX 4090

The weights take a fixed share of the card. Every request then needs working memory that grows with the length of the prompt and with the number of requests served at once.

So the question is never only whether the model loads. It is whether it loads and still has room for the context and the number of users you plan for.

The weights of a model of a few billion parameters take well under 24 GB at 16-bit precision. That is the weights alone: context and activations need room on top. The weights of larger models fit only when quantized to 8-bit or 4-bit precision, which cuts memory and costs some accuracy, and they need the same room on top.

Test the quantized model on your own prompts before you rely on it. The loss is small on some tasks and plain on others.

For one person testing prompts, the billing unit is the thing to watch. For many users at once, each concurrent request needs its own share of memory, and 24 GB fills quickly.

When it fills, the choice is a second card or one larger card. The RTX 5090 has 32 GB and the RTX Pro 6000 has 96 GB.

A rented card bills for every second it is on, whether or not anyone is asking it anything. A model API bills per token. If your traffic is light or uneven, tokens can cost far less than a card that waits.

Rent the card when you need a model no API serves, your own fine-tuned weights, or steady traffic that keeps the card busy.

Runcrate does both from one prepaid balance. It rents GPUs billed by the second on most models, and it serves 124 models behind one OpenAI-compatible API. Runcrate can start an RTX 4090 now from $1.01 per GPU-hour.

This is one of a coordinated collection of Runcrate references. Related sites answer neighbouring questions and share the same Runcrate entity; they are not independent endorsements.