Today, we’re proud to announce that Google Cloud is a founding member of the DAOS Foundation, launched today by the Linux Foundation. Created in collaboration with partners, the DAOS Foundation will accelerate investment and development of the world-record-setting open-source DAOS storage stack, which supports next-generation HPC and AI/ML workloads.
Intel Corporation initiated investment in DAOS in 2012 to drive a new paradigm in storage for high performance computing (HPC). The combination of fully-distributed metadata management, a key-value store architecture, and the lack of dependency on kernel modifications has led to notable breakthroughs, such as:
- impressive gains in throughput, I/O, and metadata operations performance
- easier container/Kubernetes integration
- increased architectural flexibility and decreased infrastructure overhead (eliminating the need for metadata servers)
The upcoming deployment at Argonne National Laboratories, which is targeted to deliver >25TB/s of throughput for HPC, demonstrates the significant impact DAOS has had on the world of scientific computing.
Further, we recently launched Parallelstore, a scratch storage service based on DAOS that delivers up to 6.3x read-throughput performance compared to competitive Lustre scratch offerings. Both large language models (LLMs) and HPC have demanding IOPS, throughput, MDOPS, and ultra-low latency requirements. Customer demand for AI/ML training based on large data sets with small files that require low-latency operations aligns extremely well with the performance characteristics and architecture of DAOS.
“Google is delighted to be a founding member of the DAOS Foundation,” said Sameet Agarwal, Google Storage VP/GM. “This industry-leading architecture underpins our new storage service, Parallelstore, and we are excited to accelerate the growth of HPC and AI/ML workloads in the cloud.”
The DAOS Foundation will be established as a Linux Foundation project, which provides a structure for shared governance and continued investment in the open source community. The collaboration across companies has been strong and Google Cloud is excited to continue this investment.