The HECToR Service is now closed and has been superceded by ARCHER.

Seismic wave calculation at the Hz level for a general Earth

The SPECFEM3D_GLOBE package is used for simulating three dimensional global and regional seismic wave propagation. SPECFEM3D_GLOBE may be used to simulate the propagation of seismic waves inside the Earth. It employs the (continuous Galerkin) spectral finite-element method to solve the acoustic-viscoelastic wave equation in the time domain. It also allows the incorporation of the effects of gravity, attenuation, surface topography, and fully-general elastic structure in the mantle. As the spectral finite-element method (SEM) is used, only one communication round between elements is required per time step, which makes the application especially suitable for large-scale computing.

The use of SPECFEM3D_GLOBE on HECToR has been limited, due to certain numerical instabilities which occur when performing high-frequency simulations. These are caused by pervasive compiler optimisations, and the aim of this project is to resolve these and further improve performance by improving the initial mesh input operation. This will then enable simulations on long-wavelength seismic phenomena, with maximum frequency f~0.5 Hz. The work will involve:

  • Performing a full stability analysis of SPECFEM3D_GLOBE on HECToR Phase 3. Rigorous testing of the code's stability will be necessary to ensure reliable performance for a range of frequencies (f~0.01-1.0 Hz). This part of the work will also involve resolving over/underflow problems caused by particular compiler optimisations. Although instabilities may be straightforward to identify, to eradicate them will be difficult. Throughout this part of the work good code optimisation will also need to be maintained (e.g. loop unrolling and in-line replacement of subroutines/functions).
  • Improving the initial mesh input operation by reducing the initial I/O requirements. New code will be developed to combine the mesh generation routine xmeshfem3d with the main solver xspecfem3D. This will enable the mesh generation and decomposition to be performed on the fly, thus reducing the I/O at the beginning of every run.

The overall outcome of this work may be summarised as follows:

  • Throughout the work, compilation was performed with the Cray compiler (v. 8.1.4). The CoMITAC version of SPECFEM3D_GLOBE was also used throughout.
  • Compilation of the xcreate_header_file program was improved. This computes the size of the statically-allocated arrays required for the two executables (the mesher, xmeshfem3D, and the solver, xspecfem3D). The improvement was achieved by updating the Makefile to compile for the Istanbul cpus on the head nodes. This was required since on HECToR Phase 3, the compute and head nodes have a different architecture: the compute nodes have the newer AMD Interlagos processors, whilst the head nodes have the Istanbul type.
  • For the main compilation, several compiler flags for the Cray compiler needed to be set in order to overide the default ones to: create .mod files for modules, ignore OpenMP directives, and use system malloc rather than Google's tcmalloc library.
  • Initial attempts at profiling SPECFEM3D_GLOBE using CrayPAT were unsucessful due to what appears to be a bug in CrayPAT. However, a temporary workaround enabled a .APA file to be generated. Scalasca was also used to provide the required information on the most expensive subroutines.
  • To improve portability, selected_real_kind was used to determine the kind type parameters for set precisions throughout the code.
  • During the course of this project a number of bugs which affected the code's stability were discovered and then fixed by the main SPECFEM_GLOBE authors.
  • The I/O bottleneck was also removed by merging the mesher and solver into a single application. The need for I/O access to externally mounted filestore in order to pass data between the mesher and solver parts of SPECFEM3D_GLOBE was removed, along with the need to use any disk space for intermediate files.
  • Following this work, a benchmark simulation with maximum frequency of f~0.2 Hz was performed. Although this is less than the original goal of f~0.5 Hz, in reality, most observations have f_max~0.1 Hz, therefore f~0.2 Hz is a big improvement over the original limitation with the range of f~0.01-1.0 Hz.
  • This work mean that SPECFEM_GLOBE may now be used on HECToR and other HPC systems to study much larger problem sizes that were previously impossible.
  • The developments from this work have been made available to all HECToR users of SPECFEM3D_GLOBE and will be merged into the main development branch at the first opportunity.

Please see PDF or HTML for a report which summarises this project.