The HECToR Service is now closed and has been superceded by ARCHER.

Multigrid solver module for ONETEP, CASTEP and other codes

ONETEP (Order-N Electronic Total Energy Package) is a linear-scaling code for quantum-mechanical calculations based on density-functional theory (DFT). CASTEP is a state-of-the-art DFT code based on the plane-wave pseudopotential approach. This Distributed Computational Science and Engineering (dCSE) project is concerned with the development of a stand alone Multigrid solver module which will enable the use of solvent models for simulations of biomolecules, nanostructures, molecular systems, materials and surfaces with ONETEP, CASTEP and other electronic structure codes.

A great deal of physics and chemistry of matter happens in the presence and due to the influence of solvent. Including explicitly all the molecules of solvent in a simulation cell ("waterbox") is extremely costly as typically it requires a huge number of atoms. Implicit solvation models provide a computationally affordable way to model solvent, by representing it as a dielectric continuum which surrounds and is polarised by the solute and "reacts" by back-polarising the solute. In first principles electronic structure calculations this is taken into account in the selfconsistent process so that the electronic density is deformed accordingly by the presence of the solvent. Furthermore, a number of explicit solvent molecules can be included when necessary (e.g. water molecules forming hydrogen bonds with an exposed protein active site).

Therefore the overall aim of this project will be to develop an external module Multigrid solver module which will be capable of:

  • Solving the Nonhomogeneous Poisson Equation with continuous charge distributions represented on a grid. There are already a number of solvers available for point charges, but few that work with continuous charge and none with good parallel performance.
  • Using flexible boundary conditions, e.g. Dirichlet, more general (Neumann - i.e. derivative values) and mixed Dirichlet and Neumann.

In particular, the following work will be performed:

  • Develop a simple two-level multigrid code parallelised in a mixed mode fashion (using OpenMP for intra-node data-sharing and MPI for data distribution and communication between nodes).
  • Expand the two-level multigrid code to a full multigrid V-cycle (with 4 levels of multigrid). The multigrid solver will be developed in a general way to be able to cope with an arbitrary number of levels. The algorithms implemented will provide the best performance on multicore architectures.
  • Extend the capabilities of the multigrid solver to non-linear differential equations using a full approximation scheme with the implementation of argument-capping functions to treat the exponential non-linearity. Particularly relevant for the implicit solvent model is the inclusion of ionic strength in the simulations via Boltzmann terms in the Poisson-Boltzmann extension of the Nonhomogeneous Poisson Equation.
  • Develop data interfaces for the multigrid solver for use with ONETEP and CASTEP.

The individual achievements of the project are summarised below:

  • A parallel hybrid (MPI+OpenMP) multigrid solver for Poisson and Poisson-Boltzmann Equation was written from scratch in Fortran 95.
  • Use of a 3D domain decomposition for MPI parallelism.
  • OpenMP threads operate on adjustable grid block.
  • Almost 16 times faster than the previous solver used in ONETEP.
  • Good scaling up to 2048 cores (128MPIx16threads) for a representative ONETEP problem which uses a global grid with 449x545x609 points, 1D MPI topology.
  • Good scaling for 3D MPI decomposition up to 256 MPI ranks for a global grid with 129x129x129 points which is representative for CASTEP computations.
  • Thoroughly tested and integrated in ONETEP main branch.
  • Solver was presented to users attending "ONETEP Masterclass 2013" training.
  • Written as a library to allow easy usage in other applications.

This work will help ONETEP and CASTEP user communities to develop their research activity in the following simulations:

  • materials for solar cell applications,
  • the interaction of proteins with carbon-based nano-particles,
  • drug optimisation,
  • enzyme catalysis,
  • adsorbtion and chemical reaction on surface of metallic nanoparticles.

These applications will require around 15 millions of AUs per year.

Please see PDF or HTML for a report which summarises this project.