The HECToR Service is now closed and has been superceded by ARCHER.

Mesh reordering in Fluidity using Hilbert space-filling curves

Fluidity is an open-source, multi-scale, general purpose, finite-element CFD model developed by the Applied Modelling and Computation Group (AMCG) at Imperial College. Fluidity is capable of resolving oceanic flows simultaneously on global, basin, regional, and process scales, enabling oceanographic research that is not possible with other models. It is an adaptive model that can optimise the number and placement of computational degrees of freedom dynamically through the course of a simulation. Previous Distributed Computational Science and Engineering (dCSE) projects for developing Fluidity are described here, here and here.

The key objective of this dCSE project was to improve the ordering of topological and computational meshes in Fluidity and thus speed up the matrix assembly and matrix solve steps. The reordering has been implemented using a Hilbert space-filling curve (HSFC) to order the nodes and elements of the mesh in each MPI process. The Zoltan library implementation of HSFC ordering was used. Zoltan is already used in Fluidity for mesh decomposition and load balancing across MPI processes.

Reordering the mesh gives a 5% speed-up overall, 10% for the pressure calculation (assemble and solve) and 5% for the velocity calculation (assemble and solve), based on OpenMP runs of a 3D backward-facing step simulation with 1,000,000 nodes (see Figure 1). The performance varies with thread placement over the UMA regions in the HECToR node. Hilbert space-filling curve reordering could be applied to other finite element codes running on HECToR to give a similar speed-up.

Figure 1: 3D backward-facing step flow at simulation time = 0.04 (= 20 time-steps followed by the first adapt step). Speed-up from reordering the mesh. Simulation size is ~1,000,000 nodes. Single HECToR node. OpenMP thread placement is chosen to accentuate NUMA effects, e.g. for 2 threads, thread 0 is on die 1, thread 1 is on die 2.
\includegraphics[width=0.88\textwidth]{allscaling2.eps}

Please see PDF or HTML for a report which summarises this project.