ACD | CGD | HAO | MMM | TIIMES

Preparing WRF for Petascale Computing

The National Science Foundation and other U.S. government agencies are planning to build and operate petascale computer facilities by the end of the decade. The challenge to scientific computing groups will be to have codes and applications that efficiently exploit sustained computational rates of 1015 floating point operations per second on these machines. MMM software developers John Michalakes and Tom Henderson are working with computer vendors and petascale system planners to ensure that the Weather Research and Forecast Model (WRF) will be ready.

WRF was designed from the beginning as a high-performance computing application [1]. It is supported on the range of mid- to high-end computing systems deployed by WRF users; among these, operational NWP and research computing systems listed in the Top500 Supercomputing Sites. On current systems, WRF runs on up to one or two thousand processors. Petascale systems, however, will require efficient scaling to tens of thousands to hundreds of thousands of processors.

Recently, in preparation for responding to the NSF $200 million dollar call for proposals to build and install a petascale computing system by 2010, Cray Inc. engineers ran a standard WRF benchmark case on very large Cray XT3 system. The figure above shows performance up to 6.7 Trillion floating point operations per second on 10,400 processors, a WRF model record but only about 1/150th of the way towards a sustained petaflop per second. WRF is also undergoing scaling and performance optimization on the IBM Blue Gene system at the University of Colorado and NCAR, San Diego Supercomputing Center and at IBM's T.J. Watson Research Center. Significant work lies ahead to ensure that WRF will scale to hundreds of thousands of processors and that datasets on the order of 100s to 1000s of trillions of bytes of output can be efficiently written, stored, and analyzed.

At the same time, NCAR scientists are moving forward with scientifically meaningful frontier simulations and accompanying dynamic analyses that will be enabled by WRF running on petascale computing systems. Petascale computing power will be brought to bear on problems such as regional climate modeling, hurricane forecasting, multi-scale organization of tropical convection, predictability, and fundamental dynamics of the atmosphere:

  • Cloud-system resolving modeling (CRM) where a primary goal is to quantify the global role of organized precipitating convection. A related effort is superparameterization, an approach in which CRMs are applied in place of contemporary convective parameterization. The multi-scale organization of tropical convection is the centerpiece. This choice recognizes the fundamental nature of tropical convection as well as the fact that vexing uncertainties in global models (weather and climate) stem from the incomplete representation of tropical convection and how it interacts with higher latitudes. (Contact: M. Moncrieff, moncrief@ucar.edu)
  • Mesoscale nature runs to facilitate studies in predictability, stochastic parameterization, and fundamental dynamics. A nature run is intended to be a representation of the atmosphere that is realistic enough to contain dynamics relevant to a particular problem. In this case, mesoscale and planetary-scale dynamics are simulated with grid spacing less than 10km on a hemisphere, providing a data set that includes a wide range of scales. Statistics gathered from these runs can potentially guide the construction of stochastic closure schemes for climate simulation at much coarser resolution, and help describe wave-wave and wave-turbulence interactions. Perturbation experiments using individual realizations may also lead to an understanding of how predictability interacts across these scales. (Contact: J. Hacker, hacker@ucar.edu)

Additional information on the Weather Research and Forecast Model can be found at http://www.wrf-model.org.

[1] Michalakes, J., J. Dudhia, D. Gill, T. Henderson, J. Klemp, W. Skamarock, and W. Wang: The Weather Research and Forecast Model: Software Architecture and Performance. Proceedings of the Eleventh ECMWF Workshop on the Use of High Performance Computing in Meteorology. Eds. Walter Zwieflhofer and George Mozdzynski. World Scientific, 2005, pp 156-168

[2] Courtesy Peter Johnsen and John Levesque, Cray Inc.