The HECToR Service is now closed and has been superceded by ARCHER.

Opening up High Performance Computing to the Discrete Element Method User Community

The Discrete Element Modelling (DEM) method is similar to molecular dynamics (MD) in many ways. One application of DEM is to simulate granular response in materials for civil and geotechnical engineering applications. However, current simulations are restricted due to lack of required functionality in the HPC variants of these applications. The Large-scale Atomic/Molecular Massively Parallel Simulator LAMMPS is a widely used MD simulator. Due to physical and algorithmic analogies, LAMMPS is a very good platform for DEM simulations. The granular LAMMPS application is one such code and although it is highly scalable, it lacks necessary functionality for DEM.

The main objective of this project is to develop granular LAMMPS for added functionality by implementing new C++ code for two sets of boundary conditions and a new contact model for bonding between two grains. This will enable highly-scalable DEM simulations to be performed on HECToR. The work will consist of:

  • Developing new boundary conditions will be achieved by firstly implementing the capability to deal with rigid stress controlled boundaries, and secondly the inclusion of membrane boundaries. The former is necessary because granular LAMMPS currently only allows rigid boundaries to be modeled in a periodic space with no stress control, which is very limiting for DEM. The latter will allow more accurate simulation of laboratory tests of granular materials, by removing model artifacts created by the use of flat boundaries.
  • Further developing the contact models already available in granular LAMMPS. This part of the work is concerned with implementing an existing model which is commonly used for bonding between two grains. New C++ classes will be developed which will work on top of the existing communication class in LAMMPS, so that the new contact models are expected to have very good scalability.

The overall outcome of this work may be summarised as follows:

  • To develop granular LAMMPS for moving and rigid stress controlled boundaries, modifications were made to the FixWallGran class. For the addition of the membrane boundary condition, an algorithm based on the weighted Voronoi graph method was implemented within the FixMembraneGran class. This method uses the Voro++ package to provide the Voronoi calculation of the membrane forces. The LAMMPS input script was also updated to enable the user to specify the new boundary conditions.
  • To develop the existing contact models in granular LAMMPS to allow the bonding between two grains, firstly a new non-linear elastic contact spring model (Hertz-Mindlin) was included. This was acheived by implementing a new class, PairGranShmHistory, as a child of an existing class (PairGranHookeHistory) and using the necessary member functions. Secondly, for the addition of the new bonding model, the new class PairGranHookeHistoryPbond was developed. The LAMMPS input script was also updated to enable the user to specify the new inter-grain interaction(contact) model. The new code was verified against results from the commercial code PFC3D.
  • To demonstrate the potential of these developments, a large-scale simulation was performed where grains were initially rained under gravity and then equilibrated inside a cubic container. After raining of the particles, the top of the container was flattened by removing approximately 500,000 grains so that the sample used for the subsequent simulations contained 1,428,532 grains. The sample, which was stressed at 100kPa was then used for a simulation of wave propagation from a point source, which is analogous to a bender element laboratory test. The stressing was performed on HECToR using 8 fully populated nodes (256 cores). Approximately 48 CPU hours were required to reach a stress of 1MPa, representing a total real simulation time of 5 seconds, i.e. more than 20,000,000 timesteps.
  • The stress-controlled demonstration enabled by this work, therefore represents a significant development as this is the first known DEM simulation for geomechancis where more than 1,000,000 particles have been compressed in a controlled manner.
  • It is now possible for users of the DEM with granular LAMMPS to run stress-controlled simulations on HPC systems such as HECToR, or use an analogue to a membrane as part of simulations of laboratory tests of granular materials.
  • Efforts are also underway to add the new LAMMPS classes developed in this work to the main LAMMPS code base through discussions with the main developers. The new classes already form part of the Imperial College local version of LAMMPS.

Please see PDF or HTML for a report which summarises this project.