The HECToR Service is now closed and has been superceded by ARCHER.

Implementing OpenIFS on HECToR

The OpenIFS project is a new initiative from the European Centre for Medium-Range Weather Forecasts (ECMWF) that will deliver a portable version of its Integrated Forecasting System (IFS) to the academic community. OpenIFS will allow UK researchers use of one of the world's leading weather forecasting models for the first time and provide new and significant scientific opportunities for research into: weather and weather related processes, short to seasonal range forecasting, forecast verification, as well as a range of other applications.

OpenIFS is the forecast only part of the full Integrated Forecast System (IFS) at ECMWF. The IFS model uses an efficient parallel I/O system operationally at ECMWF. Without this system the academic version, OpenIFS, will pass all I/O through the master task - limiting the types of experiments that can be carried out on HECToR. This project will address this deficiency by implementing the necessary parts of the ECMWF I/O software on HECToR.

The overall aims of this project were:

The individual achievements of the project are summarised below:

  • The support libraries (GRIB_API and FDB) were installed on HECToR for the Cray compiler environment (CCE) and the GNU complier suite (GCC).
  • The OpenIFS software was built and coupled to the FDB library. Results were validated against those of the "traditional" GRIB serial writer files.
  • The metric for four cases was established as the forecast days per day and this was used to record the performance of the FDB-enabled software. However, timing and tuning results were indicative of a busy multiuser machine.
  • The differences achieved by changing the LUSTRE parameters were not very noticeable compared to the variation seen in multiple runs. The lack of significant speed up from LUSTRE parameters is attributable to the fact that although FDB is identified as a parallel IO method, the parallel nature is between the data streams rather than an equivalent MPI-IO call to parallel file writing. If the number of IO streams is similar to the number of OSTs then it is unlikely that there would be any performance gain. In that case the striping is more likely increase the load on the OSTs and cause contention.
  • The recommendation is to use one writer per node although it is expected that there is an increase in intra node communications for the GRIB encoding.
  • Hugepages were investigated and a subtle improvement was found when using 2MB pages. The advice for configuring OpenIFS is to use the hugepages2M module and then set a directory stripe pattern with one strip of size 4MB.
  • ECMWF have found this work beneficial as it was the first port of these libraries to non-IBM systems. It has also extended their knowledge to use heterogeneous CMake (cross-compilation). Their development machines use Global File System so the results of the behavior of Lustre has been very helpful.

Please see PDF for a report which summarises this project.