Over the past six months we have run a steady stream of open-source model workloads — fine-tuning, batch inference, embedding generation, eval sweeps — on on-demand GPU providers, with RunPod carrying most of the load. This is a field report: what worked, what bit us, and the decision rules we now use with clients.
The context matters. Several of our engagements involve open-weight models (Llama, Qwen, Mistral families) either for data-residency reasons, cost reasons, or because a fine-tuned 8B model beats a frontier API model on a narrow task at a fraction of the unit cost. All of those workloads need GPUs, and none of our clients wanted to sign a hyperscaler committed-use deal to find out if the approach worked.
The economics are hard to argue with
The headline numbers, as of when we ran them: on-demand H100 SXM instances in the $2.50–$3.50/hr range, A100 80GB around $1.50–$2.00/hr, and consumer-grade cards (RTX 4090-class) under $0.70/hr. Comparable hyperscaler on-demand pricing ran 2–4x higher when capacity was available at all — and "available at all" was doing real work in that sentence for H100s in certain regions.
Concrete example: a QLoRA fine-tune of an 8B model on roughly 40k instruction pairs cost us about $14 on a single rented A100 — around five hours end to end including environment setup. An eval sweep across six model variants and three prompt formats, embarrassingly parallel across cheap consumer cards, ran overnight for under $50. Numbers like these change behavior: engineers run experiments they would have skipped, because the cost of curiosity dropped below the cost of a team lunch.
Spot/interruptible pricing cuts costs further — often 40–60% — but only makes sense for checkpointed training jobs. We got burned once by running a non-checkpointed job on an interruptible instance. Once.
Where it bit us
An honest field report includes the scar tissue.
Cold starts and image pulls. Our first serving experiments had container images north of 20GB (CUDA base, PyTorch, vLLM, model weights baked in). Pull times made scale-from-zero latency measured in minutes, not seconds. Fixes that worked: bake weights into a network volume instead of the image, trim the image to the serving essentials, and keep one warm worker during business hours. Serverless GPU offerings have improved here, but test your own cold-path latency before promising anyone "scales to zero."
Hardware variance. Renting from a marketplace of data centers means the same instance type is not always the same experience. We saw meaningful variance in disk I/O and inter-GPU bandwidth between hosts. For multi-GPU training, we now benchmark NCCL all-reduce for a few minutes before committing a long job to a host, and we kill and re-provision if it is off. That five-minute ritual has saved us multiple times its cost.
Ephemerality discipline. Instances disappear, get preempted, or occasionally just degrade. Everything important lives on network volumes or object storage; anything on instance-local disk is presumed lost. Checkpoint every N steps, make jobs resumable, and treat the GPU as a stateless compute brick. This is standard cloud hygiene, but the failure frequency is higher than hyperscaler baseline, so the discipline is non-optional rather than theoretical.
Idle burn. The failure mode nobody talks about: the $2/hr instance someone forgot over a long weekend. We now run automated idle detection (GPU utilization below threshold for 30 minutes triggers an alert, then a stop) and tag every pod with an owner and a project. Our untracked spend went to effectively zero after that; before it, roughly 15% of our first month's bill was idle time.
For regulated workloads, read the provider's data-processing terms before uploading anything. Community-cloud tiers on GPU marketplaces can mean your container runs in a third-party data center. Secure-cloud tiers with vetted facilities exist and are worth the premium for client data — or keep sensitive data out of the rented environment entirely and ship only de-identified training sets.
The stack that emerged
After six months, our default setup is boring and reliable:
- vLLM for serving open-weight models; Axolotl or plain HF + PEFT for fine-tunes.
- One slim base image per workload class, weights on network volumes.
- Everything launched via API/CLI from CI — no hand-created pods, so every job is reproducible and attributable.
- Metrics shipped to our own observability stack (GPU utilization, tokens/sec, job success rate), because the provider's dashboard is for humans, not alerting.
When we recommend it — and when we don't
Our current decision rules with clients:
On-demand rented GPUs win for experimentation, fine-tuning, batch inference, eval infrastructure, and bursty workloads — anything where utilization is under ~40% or the future is uncertain. Which, in month one of an AI initiative, is everything.
Hyperscalers win when the workload must sit next to data already in that cloud, when procurement or compliance mandates it, or when you need the surrounding managed services more than you need cheap FLOPs.
Owning hardware wins almost never at the scale of a pilot, and only starts to pencil out with sustained 60%+ utilization on a stable workload — a bar most enterprises never actually clear, whatever the capacity plan says.
The meta-lesson matches our production-or-nothing bias: cheap, elastic GPU access removes the last excuse for deciding architecture questions on slides. For a few hundred dollars, you can measure whether the fine-tuned open model actually beats the API baseline on your evals, on your data. So measure it.