Parallelisation of the Hamiltonian Construction.

Since the (N+1)-electron Hamiltonian matrix (hereafter referred to simply as the Hamiltonian matrix) is real and symmetric the construction phase sees the construction of only the lower-triangular part of the matrix (along with the main diagonal) which is then written to disk in the serial version. It sweeps through several deeply nested loops associated with each of these blocks and writes the column and row indices along with the matrix values to a fortran binary sequential access file (lower triangular part only). The matrix is stored on disk in what amounts to a coordinate (COO) sparse storage format. The storage format is not a genuinely COO format in that the matrix is unordered and only the lower part is stored.

The Hamiltonian is made up of three separate blocks that are hereafer named the Continuum-Continuum (CC) block, the Bound-Continuum (BC) block and the Bound-Bound (BB) block.

Figure 2: The structure of the Hamiltonian associated with a small test-case H2 O calculation. Note the internal "class" structure of the CC block which is always dense. The BC block is also always dense. Note also that the BB block is the only sparse part of the matrix and that it is unstructured and changes from molecule to molecule.

Figure [*] shows the structure of the Hamiltonian associated with a water molecule and it can be seen that the only genuinely sparse part of the matrix is found within the BB block. The sparsity of the BB block is unstructured and changes from molecule to molecule. The CC and BB blocks are always dense blocks. It should be noted straight away that the H$ _2$O test case represents a very small Hamiltonian and for the case of larger molecular systems, such as phosphate and guanine, the dimension of the BB block is several orders of magnitude larger than the CC block (e.g in phosphate CC order $ \sim$ 700 and BB order $ \sim$ 120,000), so that for molecules of future interest, the BB block dominates the Hamiltonian matrix. A schematic of such a Hamiltonian (a phosphate test-case that has made the BC block smaller than preferred) can be seen in figure [*]

Figure 3: Schematic of the Hamiltonian associated with a test- case phosphate calculation. Typically, the CC (red) and BC blocks (blue) would be larger, but still much smaller than the BB block.

Rather than write the sparse matrix elements (i.e. the rows, columns and values) to disk we insert said elements into the PETSc Mat object (PETSc Matrix) as the matrix is being constructed. Unfortunately, this is not entirely straight forward due to the fact that PETSc only allows for a ordered row-wise insertion of the upper-triangular part of a sparse symmetric matrix. The fact that in SCATCI the Hamiltonian is generated as a lower-triangular matrix means that, for both the CC block and the BC blocks, elements first need to be inserted into arrays so that the blocks can be sorted before they are inserted into the PETSc Mat object.

A description of how each of the blocks is constructed and inserted into the PETSc matrix now follows.

Paul Roberts 2012-06-01