Are build times spiking, caches failing, or artifacts disappearing mid-pipeline? For teams running self-hosted or cloud CI runners, storage choice often explains inconsistent build speed, failed caching, and hidden costs.
This guide lays out the practical tradeoffs of persistent vs ephemeral storage for CI runners, provides a reproducible step-by-step persistent disk setup, shows how to secure reproducible builds with ephemeral runners, and includes cost and performance models for 2026 environments.
Key takeaways: what to know in 1 minute
- Ephemeral storage equals strong isolation and predictable cleanup. Ephemeral runners reduce cross-build contamination but need caching to avoid re-fetch costs.
- Persistent storage saves expensive rebuild time at the cost of operational complexity and potential stale state. Use it for large caches, container layers, and artifact retention.
- High-performance NVMe/SSD local disks drastically cut CI latency for I/O-heavy workloads. Network block storage can match throughput but often adds latency and cost.
- Hybrid architectures (ephemeral runner + persistent cache) provide the best cost-to-performance ratio for most teams. Implement strict garbage collection policies.
- Choose persistent disks when build variance, cache miss penalties, or Docker layer rebuilds exceed the storage cost. Monitor IO wait, cache hit ratio, and build time variance.
Ephemeral vs persistent storage for CI: technical tradeoffs
Ephemeral storage
- Typical behavior: runner provisions a temporary workspace, used only for the duration of a job, then destroyed.
- Pros: full isolation, zero accumulation of stale state, lower cross-job data leak risk, simpler autoscaling.
- Cons: repeated downloads and rebuilds unless an external cache exists; possible increased external network egress and longer cold-starts for heavy dependencies.
Persistent storage
- Typical behavior: runner mounts a block or file volume that survives across jobs and reboots.
- Pros: fast cache hits, retained artifacts, reduced network fetches, and quicker incremental Docker builds.
- Cons: potential stale state, more complex lifecycle and GC, possible multi-tenant security surface, cost for provisioned capacity and IO.
IO patterns and practical impact
- Small-file workloads (npm, pip): dominated by metadata ops and many small reads—latency matters more than raw throughput.
- Large-file or artifact-heavy workloads (maven, gradle, large test datasets): throughput matters alongside latency.
- Docker layer reuse: benefits heavily from persistent layer caches; ephemeral runners without a shared cache rebuild layers every time.
Benchmarks to watch (recommended KPIs)
- Median build time (with and without cache)
- Build time variance (P90 - P50)
- Cache hit ratio
- IO wait (iowait) and per-job IOPS
- Artifact retention costs (GB-month)
Sources for disk performance: GCP persistent disk performance, AWS EBS volume types.
Step-by-step guide to attach persistent disks to CI runners
Overview and prerequisites
- Requirement: CI orchestration (self-hosted runner, Kubernetes, or VM fleet) with permission to attach volumes.
- Typical choices: block storage (AWS EBS, GCP PD), network file systems (NFS, EFS), or host-local NVMe volumes.
- Security note: ensure runner identity/instance role has minimal attach/detach perms.
Step 1: choose the disk type
- For lowest latency, pick host-local NVMe/SSD (bare-metal or NVMe-backed VMs).
- For flexible resizing and snapshotting, choose cloud block storage (EBS gp3/io2, GCP PD SSD).
- For many small concurrent clients, choose a managed file service (EFS/Filestore) or a CSI-backed distributed filesystem.
- Create volume with IO provisioning suited for peak CI load (IOPS and throughput). Example: gp3 with baseline IOPS and throughput tuning on AWS.
- Format with ext4 or xfs with noatime and appropriate inode tuning for many small files.
Step 3: mount policies and permissions
- Mount under /var/lib/ci-cache or /home/runner/cache.
- Use dedicated Unix users/groups and restrictive permissions for isolation.
- For multi-tenant runners, use filesystem ACLs or separate volumes per tenant.
Step 4: integrate with runner config
- GitHub Actions self-hosted: configure runner service to use the mounted path as RUNNER_WORKDIR or cache location. Docs: GitHub Actions runners.
- GitLab Runner: configure volumes in config.toml and map caches/artifacts. Docs: GitLab Runner.
- Kubernetes: use PersistentVolume (PV) + PersistentVolumeClaim (PVC) with a CSI driver and node affinity for local volumes.
Step 5: snapshot and backup
- Automate snapshots pre-upgrade and weekly for critical caches.
- Validate restores periodically to ensure snapshot consistency.
Step 6: implement garbage collection and retention policies
- Set TTL for caches (e.g., 30 days) and implement automated GC jobs to delete old artifacts.
- Monitor disk fullness and trigger scaling or eviction at thresholds (70% warn, 85% scale).
Example: minimal GitHub Actions self-hosted runner config snippet
- Mount persistent disk at /var/lib/gha-cache
- Set RUNNER_WORKDIR to /home/runner/_work and configure build scripts to use /var/lib/gha-cache
Secure reproducible builds with ephemeral storage
Ephemeral runners excel for security-sensitive pipelines where reproducibility and secret leakage must be minimized.
Best practices
- Use ephemeral runners for steps that handle secrets or produce sensitive artifacts.
- Combine ephemeral workspace with a remote content-addressable cache or artifact repository (e.g., Nexus, Artifactory, or S3-backed caches) to keep builds reproducible without persisting runner-local state.
- Sign or hash artifacts before publishing to prevent tampering.
Concrete measures
- Use ephemeral container images that run as unprivileged users and drop capabilities.
- Mount secrets through short-lived credentials or secret managers (HashiCorp Vault, AWS Secrets Manager) rather than writing them to disk.
- Use reproducible-build tooling (Nix, Bazel, Reproducible Builds initiative) and store build inputs in an immutable store.
References: Reproducible Builds and CNCF guidance on secure supply chains at CNCF.
Cost comparison: persistent vs ephemeral storage for CI
Cost drivers
- Provisioned capacity (GB-month) and IOPS for persistent disks.
- Snapshot and backup costs.
- Bandwidth and network egress from repeated downloads on ephemeral setups.
- Operational costs for GC, security, and monitoring.
Simple cost model (example numbers, 2026):
- Ephemeral-only: storage cost ~ minimal; network egress and external cache cost may grow with build frequency.
- Persistent-block (gp3/io2 style): pay for storage (e.g., $0.08–$0.12/GB-month) + IOPS throughput reservation.
- High-performance local NVMe: higher VM price but included IO performance with no per-IO cost.
HTML table: comparative summary
| Metric |
Ephemeral |
Persistent block (EBS/GCP PD) |
Host-local NVMe |
| Isolation |
High |
Medium (volume sharing possible) |
High (local to node) |
| Cold-start cost |
High |
Low (if attached) |
Very low |
| IO latency |
Variable (depends on provisioned cache) |
Moderate |
Lowest |
| Operational complexity |
Low |
Medium |
High |
Analysis: for teams with hundreds of daily builds and heavy Docker layer reuse, persistent or host-local NVMe typically pays back in reduced build minutes and lower external egress.
When to use persistent disks in CI: decision checklist
Use persistent disks when one or more of these are true:
- Cache miss cost per build is > 30–60 seconds on average and cache hit ratio is low.
- Docker layer rebuilds add minutes per pipeline and container layer sharing is required.
- Artifacts or large test datasets (>10s of GB) need retention between jobs.
- Build variance and flaky timings impact SLAs and developer productivity.
Avoid persistent disks when:
- Builds must be cryptographically isolated every run.
- Team prefers simple autoscaling with short-lived runners.
- Storage operational overhead and GC risk exceed the value for small teams.
Measured effects
- NVMe local drives reduce small-file latency by ~3–10x compared with network block storage in many tests; throughput for large file transfers also improves by 2–4x.
- Docker image pulls and layer extraction are noticeably faster with SSD-backed caches, cutting cold-start pipeline times.
Practical tuning
- Use noatime and proper scheduler settings (deadline or mq-deadline) for workloads with many small reads.
- Tune inode ratios when using ext4/xfs for dependency-heavy projects.
Reference measurements: vendor docs and community benchmarks at AWS EBS performance and GCP local SSD guidance.
Signs you need persistent storage in CI
Monitor these signals:
- Cache hit ratio consistently below 30% and network egress rising.
- P90 build times much higher than P50 with large variance.
- Repeated Docker layer rebuilds or long image extraction times.
- Disk thrashing on controller-level metrics and high IO wait.
- Developer complaints about slow re-runs for iterative work.
Action thresholds (example):
- If average incremental build > 5 minutes and > 50 daily runs, evaluate persistent caching.
- If cache size grows > 200 GB with frequent reuse, consider provisioned persistent volumes.
Alternatives to persistent storage for reproducibility
- Remote artifact repositories (S3, Nexus, Artifactory) with content-addressable storage.
- Build caches backed by object storage and a cache index service.
- Immutable container registries and prebuilt images for common stages.
- Distributed cache layers (Redis/HTTP cache) for dependency metadata.
- Reproducible build systems (Bazel, Nix) that reduce reliance on runner-local state.
Each alternative trades off latency, cost, and complexity—object stores are cheap for capacity but add latency; local caches reduce latency but increase operational overhead.
CI runner storage flow: ephemeral + hybrid cache
🧩
Step 1 → provision ephemeral runner container
⚡
Step 2 → check local cache mount (/var/lib/cache), fall back to remote object store
🔁
Step 3 → run build using layers and remote cache for reproducibility
🗑️
Step 4 → ephemeral workspace destroyed; persistent cache retained with GC
✅ Hybrid: isolation for security + persistent cache for speed
Advantages, risks and common mistakes
✅ Benefits / when to apply
- Significant build-time savings for dependency-heavy pipelines.
- Reduced network egress and lower per-build data transfer costs.
- Faster iteration for developers when caches and Docker layers are persistent.
⚠️ Risks and mistakes to avoid
- Allowing unbounded cache growth without GC leading to full disks and failed pipelines.
- Sharing writable volumes between tenants without proper isolation.
- Over-provisioning IOPS or storage without monitoring usage.
Mitigations
- Implement size and age-based eviction policies.
- Use immutable cache keys and content-addressable storage.
- Separate volumes per runner or use filesystem-level quotas.
Frequently asked questions
What is ephemeral storage in CI?
Ephemeral storage is runner-local storage that exists only for the job lifetime. It ensures clean state per run and reduces long-term management of caches.
When should persistent storage be avoided?
Avoid persistent storage when strict isolation is required, when operations prefer simple autoscaling, or when team size is too small to justify the operational overhead.
How much does persistent storage cost compared to ephemeral?
Persistent cost includes GB-month and IOPS; ephemeral shifts costs to network egress and external cache. Exact numbers depend on provider pricing and build frequency.
Can ephemeral runners be fast enough for large monorepos?
Yes, if combined with an object-backed cache or prebuilt artifacts. Hybrid approaches are common for monorepos.
How to secure persistent caches against secrets leakage?
Use strict file permissions, mount options, secrets managers for credentials, and isolate caches by project or tenant.
SSD/NVMe improves most I/O-bound tasks, especially many small files and Docker layer extraction. CPU or network-bound tasks may not see as much benefit.
Are there managed services for CI caches?
Yes: third-party caching services, cloud object storage + CDN, and some CI platforms offer integrated smart caching.
Conclusion
Next steps
- Measure current KPIs: collect median build time, P90, cache hit ratio, IO wait, and network egress for a 7-day window.
- Prototype a hybrid setup: ephemeral runners + shared persistent cache (object store + host-local SSD for hot data) and measure delta over 2 weeks.
- Implement retention and GC policies before enabling persistent volumes in production to avoid disk saturation.
