HPC News Bytes – 20261012

SPEAKER_1 (0:05)

Welcome to HPC News Bytes, a weekly show about important news in the world of supercomputing, AI, quantum computing and other advanced technologies.

Doug Black (0:14)

Hi, everyone. Welcome to HPC News Bytes. I'm Doug Black, and with me is Shahin Khan. Tech analyst firm Gartner has already put out its predictions for 2027 and beyond. And they have a recurring theme, the operating consequences of AI.

By 2030, Gartner expects 80% of organizations that offer public-facing AI to experience cost exhaustion attacks where abusive queries by online attackers create large computing bills.

By 2028, 60% of global 500 companies will put AI FinOps controls directly into inference to govern these costs in real time. The firm also predicts that by 2029, 80% of new applications will be deliberately disposable, used for less than a year. It also expects a quarter of global 500 firms to keep refreshing componentized AI products so rapidly that traditional fast follower strategies become obsolete. Now, these are forecasts, of course, rather than measured trends.

Shahin Khan (1:28)

Yeah, they go on to say that by 2030, more than 10 billion autonomous agents would flood public services with requests, and physical AI would assist 80% of frontline employees at international companies. They also predict that $10 trillion worth of enterprise-owned energy assets will turn some large corporations into electricity suppliers. Also interesting, Gartner expects insurance companies to shape AI governance as businesses want coverage against AI-related liabilities, and CIOs or chief AI officers will become responsible for maintaining auditable records of consequential decisions made by AI and actions taken as a result.

Another forecast says most companies deploying AI will need a dedicated function linking its total cost to business value.

Doug Black (2:26)

Argonne National Lab has expanded the Aurora Exascale Supercomputers DAOs storage system to 800 servers and reports transfer rates as high as 30 terabytes per second across thousands of compute nodes. This supports simulations, AI workloads and checkpointing. Storage is a very complex area with many approaches and products. DAOs is a distributed object store designed for HPC, also offering file-oriented interfaces. It distributes data and metadata and can use user space, low overhead IO paths.

In storage circles, POSIX compliance is mentioned frequently as an important requirement that can also constrain performance. POSIX defines how applications interact with files, but maintaining its semantics across thousands of clients can be expensive. DAOs offers familiar file interfaces with some limitations, while its native interfaces can bypass much of that overhead.

Shahin Khan (3:31)

It solves a lot of problems in storage. It combines distributed metadata with transactional objects, kernel bypass, and high-performance networks. That's attractive when you have thousands of clients issuing small operations, stream data, or write checkpoints. But separating metadata from data has been around for decades. Sun's QFS and SAMFS did that, and today, Versity's Scout FS goes much further with distributed metadata management and sophisticated indexing. Lustre has also evolved beyond a single metadata server.

But while setup, integration, failure recovery, and long-term support for DAOs are all improving, there are mature products out there that have gone through many years of operational engineering to get there. For example, Google introduced DAOs-based Parallel Store, but is retiring it in favor of DDN-based managed Lustre. Google has similar commercial arrangements with NetApp for storage, VMware for virtualization, and Oracle for databases. So that by itself is not so unusual. One possible factor in Google's decision is a weakness in automated multi-level storage tiering in DAOs that makes it hard to tier down to lower cost storage and requires explicit data movement, which in a public cloud environment can run into bandwidth, operational overhead and associated costs.

We should really mention that the idea of storage tiering or hierarchical storage management goes back to the 1960s. It became a part of the mainframe stack in the 1970s.

Cray's data migration facility, DMF, brought it to supercomputing in the early 90s. It was baked into the OS. It was part of the native directory structure, and it did amazing things, like the OS was able to stop an application, go mount an actual tape cartridge through a robot arm, find the data, and resume execution where it left off.

SGI then integrated it into IREX and XFS, but the functionality moved over to the storage side, right around when the industry moved to Linux. It all ended up at HPE, which carries it on, though it just announced that DMF has reached end of support and will be phased out in 2028 End of an era, really. So yeah, like you said, storage is complex with a long history. DAOS could yet be a long-term winner, but the market may adopt its ideas without adopting DAOS everywhere.

Doug Black (6:01)