The major points in need of improvement are the routines for charges, band energy energies, and force calculations and the structure constant generation. We will start with the structure constants since it is outside the self-consistency loop and is largely independent of the rest, as can be seen from the diagram in Fig 1.
In the early stages of the development, routines for conversions between the global actual and virtual (represented through local pieces) arrays were written avoiding p{d,z}elset{,2} and thus saving on the immense latency of millions of connections. Initially for a scatter style routine a single block buffer was formed from the global value and set to the appropriate position in the local array, for a gather, the buffer was sent to a root process which then broadcasts the global array after the complete assembly. A more elegant approach was implemented later by employing the MPI darray type and mpi_alltoallw. There is now a trifold speedup compared to the buffer implementation which is likely to be due to the blocking blacks routines. Since alltoallw is very general, an attempt was made to configure it so that all processes would receive the result instead of just the root but this showed significantly worse performance than the broadcast approach. This is now integrated in the basic library (slatsm), which is used by all programs in the LMTO suite together with a wrapper around the unified diagonalising routine which scatters the input matrices, calls the diagonaliser then gathers and broadcasts them. The overhead is small for the gather operation and negligble for the scatter on a QDR Infiniband network. The gather approach is likely to be faster than the reduce one which is used in the PLATO code, due to a difference in the basic operations but it was not tested.