PEAK Open pNFS launched to address AI & HPC storage scaling

PEAK:AIO has introduced PEAK Open pNFS, an open-source, vendor-neutral technology designed to address the growing data storage challenges faced by organisations deploying artificial intelligence (AI) and high-performance computing (HPC) workloads. The new solution is positioned as the first fully open metadata server for the parallel network file system (pNFS), offering a path for scalable storage systems without dependence on a single vendor.

Scalability challenges

The expansion of AI and HPC workloads places significant pressure on data storage infrastructure, particularly in managing vast quantities of metadata. Legacy file system architectures commonly produce performance bottlenecks during large-scale metadata operations, complicating both the scaling process and ongoing management. PEAK Open pNFS aims to address these obstacles by separating metadata and storage performance, allowing each to scale independently and in line with organisational demands.

Open architecture

The core of PEAK Open pNFS is a fully open source metadata server. Developed in collaboration with Los Alamos National Labouratory and Carnegie Mellon University, the server is designed to support the latest Flex Files standard. The open design enables organisations to integrate the PEAK Open pNFS metadata server with their preferred storage hardware, thereby removing proprietary constraints that have previously limited scalability and flexibility.

"Emerging AI workloads present scaling demands similar to HPC needs but require different access patterns. To achieve a multi-decadal solution to address the combined and growing requirements benefits greatly from a standards and community-based solution, Open-source solutions like PEAK:AIO's PEAK Open pNFS promises to address the growing requirements and performance needs and is a very important step towards achieving a flexible base on which community innovation can proceed," said Gary Grider, High Performance Computing Division Leader, Los Alamos National Labouratory.

Integration and expansion

PEAK Open pNFS is engineered for modular deployment, enabling system administrators to add additional capacity and performance node by node. According to the company, this allows organisations to start with a single high availability node and scale seamlessly to exabytes, with no changes required to the underlying architecture. The metadata server is tuned for both AI and HPC workloads and integrates with the company's own AI Data Server, targeting deployments from edge environments to hybrid cloud models.

Industry partnerships

Vendors across the technology sector have voiced support for open and modular data storage systems in response to the escalating requirements for AI performance. Solidigm is among the partners to align with the introduction of PEAK Open pNFS. The partnership aims to deliver consolidated solutions for customers, focusing on demanding, data-intensive applications.

"Solidigm recognizes the urgent need for an open source parallel filesystem tailored to the demanding performance and scalability requirements of AI workloads. As data-intensive AI applications continue to evolve, the industry requires solutions that not only deliver high throughput but also offer operational simplicity and cost efficiency," said Greg Matson, Head of Marketing, Product and Business Strategy, Solidigm. "PEAK:AIO's PEAK Open pNFS appears to address these critical needs with its innovative, modular architecture that scales linearly with demand. As a PEAK:AIO partner, Solidigm sees this technology as a valuable complement to our industry leading AI storage portfolio, enabling customers to maximize the performance potential of next-generation AI infrastructures."

Performance metrics

Pairing Open pNFS with PEAK:AIO AI Data Server is claimed to maximise performance and storage density. The AI Data Server reportedly reaches throughput up to 320GB/sec in a compact 2U configuration, which the company claims results in up to six times the density and energy efficiency compared to established alternatives.

Operational impact

With the introduction of PEAK Open pNFS, PEAK:AIO aims to provide a system that enables customers to scale according to their needs without requiring fundamental changes to existing infrastructure or processes. As workloads in AI and HPC domains continue to evolve, the emphasis is on maintaining operational simplicity and cost control, themes echoed by PEAK:AIO's leadership.

"AI and HPC are pushing storage systems to their limits. PEAK Open pNFS removes metadata bottlenecks, while our AI Data Server ensures storage performance scales alongside it. The result is a simple to manage modular system that grows linearly with workload demands - without compromise," said Roger Cummings, President and CEO, PEAK:AIO.