Distributing $\beta$-Projectors and $\ensuremath{\left\langle \left. \beta \right. \right\vert \left. \phi \right\rangle}$

As noted in section 4.2.2, the $\beta$-projectors are not currently distributed over the band-group. This is a problem for large systems as the number of projectors is proportional to the number of atoms. For the al3x3 benchmark there are 88,184 plane-waves and 2,160 $\beta$-projectors, so their total storage comes to almost 3GB per k-point-there is no way a purely band- or mixed k-point/band-parallel calculation can run on HECToR under these circumstances simply because of insufficient RAM. Distributing the associated workload over the band-group should also improve the run-time, provided the extra communications can be kept to a minimum.

In addition to the $\beta$-projectors themselves, the inner products of the projectors with the wavefunction bands are also non-distributed. These so-called $\ensuremath{\left\langle \left. \beta \right. \right\vert \left. \phi \right\rangle}$ data are computed and stored in the wavefunction, wavefunction_slice and band data types for efficiency. For ease of use the array is stored in unpacked form, but this is undesirable for large systems since the addition of a single new species of ion can increase the size of the array dramatically.

Because the $\ensuremath{\left\langle \left. \beta \right. \right\vert \left. \phi \right\rangle}$ array has data for each band, it is naturally distributed over the band-group, as implemented in this dCSE project. However the projector dimension is not distributed, and it is not distributed over the gvector-group; thus the array could be distributed by projector index over the gvector-group to further reduce the memory overhead and improve scaling.

Both of these proposed distributions would save memory and improve run-time. However care must be taken in the implementation, as projector data would be distributed differently depending on whether it was the projectors themselves (which are currently distributed over the gvector-group and would be then distributed over the band-group too), or the $\ensuremath{\left\langle \left. \beta \right. \right\vert \left. \phi \right\rangle}$ arrays (which are already distributed over the bands, and would now be distributed over gvector-group as well).