The Zoltan based mesh re-partitioning solution was originally incorporated as a compile time option accessed using the flag -with-zoltan when configuring Fluidity. At this stage all but the short report on the Zoltan implementation and unit tests from M2 were complete. It was decided that detailing the implementation in a short report would not be possible. Unit tests for Zoltan are complicated as the functionality needs to run on full simulations to be tested. As such it was decided to not add unit tests but to add tests to the Fluidity test suite which compared the output from an adaptive run using Zoltan with the output from both a run with the previous implementation and a serial run. This is test 2d_circle_adapt_zoltan_sam in the Fluidity test suite. To test the functionality of Zoltan on element types which could not be done by Sam a number of additional test were also added, such as parallel_p0_consistent_interpolation and diffusion_2d_p0_adaptive_parallel.
From Fluidity revision 3530, Zoltan has been the default solution for the Fluidity trunk, development branches and releases. All tests run as part of the Fluidity buildbot system are using the Zoltan build and run successfully with Zoltan. The previous solution, Sam, is now a compile time option accessed by configuring with the flag -with-sam. All options for Zoltan are fully documented and available through the Fluidity options package, diamond, with full details also given in the Fluidity manual. This meets all of the work in M3.
Section 4 shows the performance of the Zoltan implementation on one of the HPC benchmark cases: the backward facing step. Zoltan resulted in a drop in performance from 16 processors onwards, though at this number of elements per core and with the frequency of adapts it is not surprising that the performance difference between Zoltan and Sam becomes more pronounced. The expected performance increase in both the assembly and solve was not evident in this test case. However, other tests may show better performance, however at this stage it is not clear which category of simulation would show better performance and there is a limited number of test cases that can be used as a comparison between Zoltan and Sam.
M1 was to become familiar with the Fluidity source. This was achieved and the completion of the subsequent work packages shows this. The short report was not written as it was felt it was more important to continue with the other work. Initial profiling work was done but only for a single benchmark, the small backward facing step. This was because profiling work was completed as preparation for this dCSE and presented in the original dCSE proposal.
This work as well as aiming to improve the scaling performance of Fluidity was to improve the maintainability, extensibility and functionality of the repartitioning solution. The new Zoltan based implementation has already been extended to allow parallel periodic problems to be solved by Fluidity. Detectors have also been implemented allowing them to be used within all parallel adaptive simulations as set out as the last element of WP3. The greatest benefit of the Zoltan implementation though is that it is general purpose. It allows any element type to be used in parallel adaptive simulations. This will enable Fluidity to be used for new science not previously possible. For example, modelling froths and foams using Fluidity benefits from a P0P1 discretisation and an adaptive mesh Brito-Parada2011. Limiting this to serial simulations only is prohibitive and therefore the inclusion of Zoltan is enabling new science to be carried out.
Attendance and participation in the HPC workshops run by the Fluidity group was not possible as none were held during the course of this dCSE project. However details of the work were presented at the HECToR dCSE Technical Workshop on 4th and 5th October, 2011 and the slides from this presentation are available http://www.hector.ac.uk/cse/distributedcse/technical2/.
Zoltan has been installed as a module on HECToR available to all HECToR users. All partitioners (ParMETIS, PT-Scotch and Zoltan graph/hypergraph) are available when using this Zoltan module. A centrally installed copy of Fluidity using Zoltan is also available and details have been added to the Fluidity webpages detailing how to compile Fluidity on HECToR.
During the course of this project several times software or hardware upgrades to HECToR caused Fluidity to either fail to compile or fail to run. This often caused significant delays and it was decided to setup a build test. This is a serial job run once a day which attempts to checkout, configure and compile Fluidity with the results transferred to AMCG systems. These data are then automatically processed and used to update the status of the HECToR build on the Fluidity buildbot system. This means that compilation failures of Fluidity on HECToR can be noted earlier and fixed quicker. Despite being outside the original work plan this work was essential and very beneficial.
Fluidity is an open source project so this project as well as benefiting all Fluidity users is of wider benefit to all those interested in dynamic load balancing and adaptive mesh methods. The inclusion of Zoltan, whilst not increasing performance as hoped, has vastly improved the maintainability of Fluidity and enabled new areas of science to exploit the adaptive remeshing capabilities of Fluidity in parallel using all element types.
Jon Hill 2012-03-02