Implementation of k-point parallelism

In the original code the tag NPAR is used to create intra-band and inter-band communicators. MPI processes are bound to an intra-band communicator to work on specific bands and when information is required about the other bands, MPI processes communicate through the inter-band communicators. To exploit k-point parallelism, I constructed an extra layer of communication using the value of KPAR. KPAR intra-k-point communicators (named COMM) and MPI processes/KPAR inter-k-point communicators (named COMM_CHAIN_K) are created. Each COMM communicator has all the information needed for the k-points it is assigned. Subsequently, as before the MPI processes in the COMM communicator are divided into MPI processes bounded to inter- and intra-band communicators. When a loop over k-points is encountered, the processes in each COMM communicator iterate through all the k-points they are responsible for and as soon the loop exits the COMM_CHAIN_K communicators are used to gather together the results. This is the general design followed, although the main difficulties at the development process arose at places where information from all k-points was required e.g., for implementing the linear tetrahedron method in the evaluation of the Fermi level etc.

Also extra care was needed when a full, non-symmetrized k-grid was to be employed, as some particular quantities have to be stored in all communicators, even when they are referring to k-points the communicator is not responsible for. The cases where the Hartee-Fock exchange is used were further complicated as they involve nested k-loops. Extra communication is needed there (compared to the non-Hartee Fock) cases, but as seen in the test cases ([*]) and ([*]) this does not hinder the scaling of the code, at least for the cases studied.

In regards with I/O, depending on the case I have chosen either to communicate all relevant information on the master node, or have a `leader' in each COMM communicator to do the writing (using a loop so that information was written in orderly form).

Asimina Maniopoulou 2011-07-09