Your GPUs may wait on storage rather than compute: drive throughput alone ignores random small-file reads, concurrent dataloader workers, augmentation, and checkpoint writes. For NVMe for AI model training vs HDD for datasets, keep inexpensive HDDs or object storage as the durable source, but stage active shards, caches, and checkpoints on NVMe.
Size and workers set the real storage need
Storage demand comes from the pipeline: sample size, global samples per second, worker concurrency, access pattern, and cache-hit rate determine the required read performance.
Calculate the minimum read rate
Use this starting formula: average stored sample size × global samples per second = minimum storage read rate. A four-GPU job consuming 800 samples per second with 1.5 MB samples needs about 1.2 GB/s before overhead. Add 30% to 100% headroom for random access, compression, uneven workers, metadata operations, and temporary bursts. Measure actual samples per second from logs because gradient accumulation, stalls, and variable step times change the demand.
File count changes the answer
Ten TB in 4 GB WebDataset shards can be easier for HDDs than 1 TB containing millions of JPEG and JSON files. Large shards favor sequential reads; tiny files trigger metadata lookups, seeks, and random-read latency. One HDD can stream a large shard acceptably, but it cannot cheaply imitate NVMe random-read behavior across many dataloader workers. Compression lowers bytes read but can shift the bottleneck to CPU decoding, tokenization, or decompression.
A useful sizing pass starts with the global batch: multiply per-GPU batch size by the number of GPUs, then divide by measured step time to estimate samples per second. Multiply that result by the compressed bytes fetched per sample, not the decoded in-memory size, and adjust for the training data cache hit rate. For example, eight GPUs running batches of 32 every 0.8 seconds consume 320 samples per second. If each fetched sample averages 3 MB and the cache serves 40% of reads, storage must sustain about 576 MB/s before headroom.
Dataloader workers affect how much concurrency is needed to reach that rate: more workers can hide latency, but they also intensify random I/O on HDDs. Size NVMe capacity for the active shard window, preprocessing output, and checkpoint retention needed between durable uploads.
NVMe and HDD costs for active training data
NVMe is the speed tier and HDD is the capacity tier; for most small US teams, the economical design assigns each medium a separate job.
| Storage option | Typical usable performance | US cost reference | Best job in training |
|---|
| Single 7200 RPM HDD | 150 to 250 MB/s sequential, roughly 100 to 250 random IOPS | About $15 to $22 per TB for 16 to 20 TB enterprise-class drives | Master dataset, backups, large sequential shards |
| PCIe 4.0 NVMe SSD | 5 to 7 GB/s sequential reads, far lower read latency | About $55 to $90 per TB for 2 to 4 TB consumer drives | Active shards, cache, scratch space |
| AWS EBS gp3 in US East | Baseline volume performance with configurable IOPS and throughput | Published list pricing has commonly been near $0.08 per GB-month, before extras | Persistent cloud block storage |
| Amazon S3 Standard | High aggregate throughput, but request and network latency apply | Published US list pricing has commonly been near $0.023 per GB-month | Durable source copy and checkpoint archive |
Cloud figures are list-price references, not quotes. Provisioned IOPS, snapshots, requests, and egress can change the real bill, so verify the current rate card from Amazon Web Services pricing before committing.
Cost per TB is not cost per run
A $300 4 TB NVMe drive can look expensive beside a $320 20 TB HDD, yet idle GPUs can waste that difference during a few poorly fed runs. Compare storage cost against GPU hours lost, not only dollars per TB. If local NVMe raises sustained utilization from 55% to 85% on a $20-per-hour node, avoided idle time can repay the flash premium quickly.
Keep capacity separate from speed
HDDs and object storage are economical homes for retained datasets, prior versions, and backups. NVMe should cover the active training window rather than inactive artifacts. In cloud plans, confirm whether “SSD” means physically attached NVMe or network block storage; that distinction matters more than the label because their latency, persistence, and bandwidth behavior differ substantially.
Choose HDD for large sequential dataset shards
HDD is sensible when reads are mostly sequential, active data fits in RAM cache, and measured GPU demand stays below aggregate disk throughput.
Pros
- HDD capacity costs roughly three to five times less per TB than consumer NVMe in the US retail market.
- Large RAID arrays can deliver several hundred MB/s to multiple GB/s of sequential aggregate throughput.
- HDDs work well for immutable Parquet, Arrow, TFRecord, tar, and WebDataset shards.
Cons
- Seek latency makes small-file reads slow, even when the drive reports unused sequential bandwidth.
- Random I/O can collapse under many PyTorch DataLoader workers.
- Checkpoint writes on the same HDD can cause long queue depths and unstable batch timing.
Choose HDD when data is sharded into files of hundreds of MB or several GB, reads move mostly forward, and an A/B run shows stable GPU utilization above about 80%. It also fits cold data, backups, and source datasets too large for affordable flash. Although RAID improves throughput, RAID 0 raises failure exposure, and RAID is never a substitute for backups.
Avoid HDD as the direct worker source for millions of individual image, audio, or text files across four or more GPUs. Avoid it when p95 batch time rises while disk queue depth remains high. HDD is best when access is large and sequential, not uncontrolled and random.
Choose NVMe for random reads and hot data
NVMe earns its price when data loading, rather than model compute, leaves GPUs waiting for batches.
Pros
- PCIe Gen4 NVMe often delivers 5 to 7 GB/s sequential reads for local active data.
- Low storage latency supports many workers reading shuffled samples at once.
- Fast writes suit token caches, decoded samples, shuffled indexes, and local scratch data.
Cons
- NVMe costs more per TB and cannot economically hold every long-term dataset.
- Consumer drives can throttle during sustained writes if cooling is poor.
- Frequent large checkpoints consume TBW endurance and can disturb simultaneous reads.
Choose local NVMe for random-access image datasets, heavy sample shuffling, token caches, embedding generation, and multi-GPU jobs whose active shards fit on flash. NVMe improves training most when GPUs compete for small, scattered samples rather than one long stream. It also helps cloud instances when shared storage or network latency introduces batch jitter, provided data can be staged again from durable storage.
Avoid buying NVMe for an entire repository when only a small portion is active during each run. Avoid keeping the only copy of a valuable dataset or final checkpoint on instance-local flash. PCIe Gen5 is not automatically useful if CPU decoding, networking, or worker design is the actual limit.
Split cache, source data, and checkpoints
A three-tier layout prevents different storage jobs from competing: source data stays durable, active data stays fast, and recovery copies remain separate.
A simple hybrid layout
Use an HDD array, NAS, or object store as the canonical repository. Copy the next shard set to local NVMe before training, then remove it after verification or retain it as a reusable cache. Put shuffled manifests, transformed images, tokenized text, embedding files, and preprocessing output on NVMe because they are read or written frequently enough for latency to affect job speed.
A practical split for a 24 TB dataset: retain all 24 TB on HDD or object storage, reserve 2 to 4 TB of NVMe for the current shard window and cache, and send milestone checkpoints to a separate durable destination. This avoids paying flash prices for dormant data.
Keep checkpoint writes off the read path
Checkpointing writes model weights, optimizer state, and sometimes data-state metadata. A checkpoint can range from a few GB for fine-tuning to tens or hundreds of GB for distributed jobs. Write frequent recovery checkpoints to a separate NVMe namespace or volume when possible, then copy milestones asynchronously to object storage, replicated block storage, or HDD backup. Saving checkpoints to the dataloader volume can create write spikes, read latency, and p95 step-time jumps.
Measure before changing hardware
Track GPU utilization, dataloader wait time, samples per second, disk queue depth, CPU saturation, and network throughput. Test source HDD, staged NVMe, and synthetic in-memory data while keeping model, seed, batch size, workers, augmentations, and steps unchanged. Storage is likely responsible only when slow batches correlate with I/O waits while CPU and network capacity remain available. Peak vendor figures cannot represent every real pipeline.
Do not treat NVMe as the answer when the active dataset already fits in RAM, the CPU is saturated decoding samples, the network is capped, or the job loads all data before epochs begin. In those cases, faster local storage may not change GPU utilization at all.
Compare p50 and p95 step times across those tests before renting high-IOPS storage. The resulting measurement is a clearer buying case than a specification sheet.
Treat the AI data pipeline as separate stages with different storage requirements. The master corpus can remain in Amazon S3 Standard or on HDDs, while active dataset shards move to a local NVMe SSD before each run. A training data cache should retain recently used shards, tokenized records, and deterministic preprocessing outputs so repeated epochs do not repeatedly fetch or transform the same data. Measure storage throughput at the worker level as well as for the host: a high aggregate benchmark is not useful if individual dataloader workers receive uneven bandwidth or wait on metadata operations.
This layout keeps durable capacity inexpensive while giving the hot path predictable latency.
What people ask
Is NVMe necessary for AI training?
NVMe is necessary when random reads or concurrent workers leave GPUs waiting for data. It is unnecessary when sharded, sequential, RAM-cached data already sustains the required sample rate.
Can I train AI models from an HDD?
You can train from an HDD when files are large and mostly sequential. Measure whether its actual sustained rate covers the pipeline with sufficient headroom.
How do I know storage is starving my GPU?
Low GPU utilization, high dataloader wait time, and elevated disk queue depth are indicators. Confirm using an in-memory dataset test, because CPU decoding and network delays create similar symptoms.
Should checkpoints go on NVMe?
Frequent checkpoints can use NVMe for fast local recovery, but final recovery copies should live elsewhere. Separate checkpoint writes from active dataset reads when checkpoints are large.
Is RAID HDD fast enough for AI datasets?
RAID HDD can serve sharded sequential datasets and large reads. It still has much higher random-read latency than NVMe, making it unsuitable for millions of shuffled tiny files.
Is cloud object storage too slow for training?
Object storage works when data is sharded and cached locally before training. Direct small-object reads add request latency, network variance, and possible egress or request charges.
Does PCIe Gen5 matter for GPU training?
PCIe Gen5 matters only after measurements show a Gen4 NVMe limit. Most single-node jobs benefit more from sharding, CPU workers, and an active NVMe cache.
What is the cheapest storage design for AI?
The cheapest reliable design usually keeps the master copy on HDD or object storage and uses NVMe only for active data. This limits flash capacity to the current training window.
Choose the tiered design unless tests prove otherwise
The strongest default is cheap durable capacity underneath and fast local flash where batches and temporary writes need it. Buy all-NVMe only when active-data size, random-read behavior, and measured GPU idle cost justify it. Direct HDD remains valid for sequential shards and modest worker counts; direct NVMe still requires durable backups.
- The essentials: Dataset size alone does not decide the drive type; samples per second and access pattern do.
- The essentials: HDD is cost-effective for source data and large sequential shards, not uncontrolled small-file concurrency.
- The essentials: NVMe earns its cost when random I/O and worker bursts reduce GPU utilization.
- The essentials: Keep checkpoints separate from the active read path and preserve a durable recovery copy.
Workload type changes the best tier. Computer-vision training from shuffled JPEGs usually benefits most from NVMe because random small-file reads, image decoding, and many workers create bursty demand; converting files into WebDataset shards can make an HDD array more viable. LLM pretraining commonly reads large tokenized shards sequentially, so HDD, network storage, or object storage plus a local cache can work if aggregate throughput is sufficient. Fine-tuning often has smaller active datasets and checkpoints, making a modest NVMe volume practical even on a local workstation.
In a multi-GPU cluster, each node should ideally stage a local shard subset or use a shared system proven to sustain aggregate demand; otherwise, adding GPUs can increase storage contention faster than training throughput.
Related sources
These articles can help you explore the topic in more depth:
