The PETSc assembly stage

Figure 7: A schematic of the assembled PETSc matrix. Note that, for this example, the CC block has now been partitioned among processes 0, 1 and 2 automatically during the assembly process. Processes are numbered according to IDs in PETSC COMM WORLD.

Once all the elements have been inserted into the PETSc mat object as described above, PETSc assembly routines are called (MatAssemblyBegin() and MatAssemblyEnd()). Little communication between MPI processes should occur during this phase as most of the elements have been inserted by the processes that subsequently own them during the diagonalization phase. It can be seen from figure 6 that, for the example given above, the CC block elements have been passed between some processes, as only the master process has inserted the CC elements and processes 1 and 2 have been assigned elements within the CC block for the diagonalization phase.