Combined-Multicore Parallelism for the UK electron-atom scattering Inner Region R-matrix codes on HECToR
Electron collisions with atoms were among the earliest problems studied using quantum mechanics. However, the accurate computation of much of the data required in astrophysics, plasma physics and optical physics requires large computational resources. Even on the latest generation of high-performance computer architectures, such as the Cray XE series these calculations still present huge computational challenges and future calculations require substantial increases in both the numbers of channels and scattering energies involved in the R-matrix propagations.
This Distributed Computational Science and Engineering (dCSE) project describes how many of the computational challenges in electron atom collisions have been addressed using parallel programs based on the 'R-matrix' ab initio approach. Following an earlier successful project on the 'outer region' code PFARM the current project concentrates on the 'inner region' codes RAD, ANG and HAM which build up (and diagonalize) the full Hamiltonian matrices for the collision problem, along with associated multipole matrices, in a finite volume throughout which electron exchange effects must be included.
The overall work may be summarized as: i) by substantial improvement of the parallel performance of the radial integrals RAD code by augmenting existing fine-grain OpenMP parallelism with coarse-grain MPI parallelism, and by improving serial efficiency of the existing code; ii) development of new parallel versions of the ANG (angular couplings) and HAM (matrix elements from RAD and ANG data and other work) codes, using intra-node shared-memory segments to utilize HECToR's multicore structure and memory, and MPI-2 including a new asynchronous parallel I/O package particularly suited to the R-matrix codes. The latter code developments are in 'future-proof' Fortran 2003, making full use of object-oriented features to combine efficient code with direct 'easy-to-understand' higher-level modules in which the many-electron anti-symmetric configuration-interaction mathematics and associated spherical tensor algebra may be directly related to the coding.
The individual achievements of the project are summarised below:
- The RAD code efficiency within a node has been greatly improved, and the code now scales to more than one node. The coding also demonstrates the appropriate targeting of fine- and coarse- grained parallelization using mixed-mode OpenMP plus MPI.
- The ANG (and HAM) codes, previously serial, now scale to hundreds of cores for collision problems currently being tackled by users and previously impossible to run on batch system HPC machines. The MPI-2 parallelism with passive remote memory access takes care of load-balancing problems that occur in coarse-grain parallelism over loops with unpredictable amounts of work to perform. Fortran 2003 features allow improved clarity of coding.
-
Two new utility packages have been developed which are of general interest and
availability, as well as being particularly suited to the PRMAT codes:
- 'shm', a shared-memory segment package for shared-memory utilization in pure MPI coding, with segments and semaphores etc presented as Fortran 2003 objects with simple user interfaces. Interactions with the system are in low-level C routines, accessed using iso_c_binding.
- 'pfh', a new double-buffered, asynchronous parallel I/O package using MPI-IO, developed from a serial direct-access file package present in the original codes. Communicators for parallel writing and reading may be completely independent of each other. Fortran 2003 objects are used, with simple access routines for users.
- The codes have been and are in use as test routines for debugging the latest versions of various compilers.
Please see PDF or HTML for a report which summarises this project.
