The report "Council on Competitiveness and USC-ISI Broad Study of Desktop Technical Computing End Users and HPC" [pdf] by the Council on Competitiveness and sponsored by DARPA in 2008 found that a large number of companies deployed virtual prototyping and data modeling software (e.g., Computational Fluid Dynamics [CFD]) on entry level High Performance Computing (HPC) or desktop computers as opposed to realizing the performance and fidelity gains available with true HPC systems. In a subsequent follow up report [pdf] the authors coined the phrase 'the missing middle' to describe the gap in HPC usage.

Background

The report noted that HPC adoption did not strongly correlate with large companies (though in another report it did), so the missing middle here refers to HPC users not company size. The report concluded that "...the studies also revealed some major gaps in the rapidly growing HPC market - and each gap represents an important opportunity for bolstering U.S. economic and national security strength." I contend that this conclusion extends to many other countries too.

Problems

The report suggests that the lack of HPC adoption is primarily due to:

• Lack of easy-to-use application software
• Lack of sufficient human expertise
• Costs (hardware, software, and personnel)

The report was published in 2008 and was based on studies performed during 2005-2008, I would argue that there has been major progress made in most areas.

Progress

I can speak for the CFD field which has seen a marked shift in recent times toward unified simulation environments, as with our own Caedium system, where the entire CFD workflow is better supported. No longer is there a need to switch between multiple disparate applications to perform certain tasks. This has made CFD much easier to learn (a single GUI) and use. I think the 'lack of easy-to-use application software' and the 'lack of sufficient human expertise' at the application software level is being addressed.

An added bonus is that the performance/cost ratio for HPC hardware continues to improve, especially if you factor in the emergence of GPUs - thank you Moore's Law.

Another game changer is the availability of cloud-based HPC services as an alternative to the hassle of hosting and maintaining your own HPC cluster, which reduces both upfront hardware costs and human expertise requirements.

Application Software Costs

The Council on Competitiveness report cited cost as a major constraint preventing further adoption of HPC. It's clear that application costs are still a stubborn barrier preventing further HPC adoption. As an example, consider that many CFD vendors are still using legacy (and lucrative) per-core pricing - in an era when even a phone can have a dual-core processor. In somewhat of a compromise these same vendors have begun to offer unlimited parallel licenses, but at eye-watering premiums. It is worth noting that the software cost to develop CFD to run on 2-processes compared to a 100-processes is the same.

It is time to abandon legacy pricing models to promote HPC applications for the missing middle. It is time to offer reasonable parallel pricing to let HPC fulfill its promise.