WD unveils tiered storage architectures for AI workloads

WD has introduced validated storage reference architectures for artificial intelligence and high-performance computing workloads, focused on tiered and disaggregated storage for data-intensive environments.

The new architectures are intended to address a common constraint in large AI and HPC clusters: moving data to graphics processors can limit performance when many users access the same datasets at once.

At the centre of the approach is policy-driven auto-tiering, which places data based on access patterns and workload stage. Faster NVMe storage handles active ingestion and training, while higher-capacity hard disk systems store warmer and colder data for retraining, analytics and governance.

The setup is designed to sustain throughput across mixed workloads while reducing the need to keep all data on flash systems. The aim is to help organisations balance performance and storage costs as data volumes grow.

WD outlined three validated architectures for different deployment models. One was developed with IBM Storage Scale, another uses BeeGFS for distributed file systems, and a third is based on the open-source Ceph platform.

The IBM-based design uses a single file system across two storage pools, combining an NVMe tier with a hard disk tier. Policy controls such as migration, compression and clean-up determine where data is placed and how it moves over time.

The BeeGFS configuration combines WD OpenFlex Data24 NVMe-oF EBOF systems with Ultrastar Data Series JBOD hardware. The design extends tiering to distributed file systems and supports environments where performance-driven and capacity-driven workloads share a namespace.

It is also positioned as a high-availability option for IT and HPC workloads involving heavy concurrent access, with the goal of simplifying data management and supporting flexible scaling.

The Ceph-based architecture uses Ultrastar Data60 and Ultrastar Data102 systems. In that setup, object storage tiering is managed through storage classes and lifecycle policies, with NVMe handling metadata and active workloads while JBOD platforms provide bulk capacity.

Data can move automatically between those tiers, while the distributed design supports scaling and resilience through replication and erasure coding. WD also said a JBOD-based Ceph arrangement can reduce costs and make storage easier to expand over time.

Data bottleneck

WD argues that infrastructure decisions in AI are increasingly shaped by data movement rather than processing power alone. In large shared environments, sustained data delivery can matter more than headline peak performance, particularly when thousands of clients draw on common datasets.

Jason Strawderman, Director of Platform Solutions at WD, described that issue as central to the company's latest work.

"In large-scale AI and HPC environments, the challenge is sustained data delivery to the GPU under concurrency, not peak performance," said Strawderman. "When thousands of clients access shared datasets, performance depends on how efficiently data is placed and moved across tiers. These validated architectures show how policy-driven tiering can maintain sustained throughput and utilisation as AI and HPC workloads scale."

Broader shift

The architectures also reflect a broader move towards disaggregated storage, in which compute and storage resources are separated and managed independently. The model has gained attention in AI and research computing as organisations look to scale clusters without matching every increase in compute with equivalent amounts of expensive flash storage.

WD's latest announcement places it alongside other storage suppliers positioning tiered systems as a way to manage the economics of AI infrastructure. The premise is that not all data in a training or inference pipeline needs the same speed or media type at the same time.

By validating designs across IBM Storage Scale, BeeGFS and Ceph, WD is targeting organisations with different storage software preferences and operating models. The common thread is a mix of NVMe for active data and hard drives for larger, less frequently accessed datasets.

The new reference architectures are intended to help organisations maintain throughput and manage costs across the full lifecycle of AI and HPC data.