Detectors

Detectors are either passive particles injected into the flow (Lagrangian detectors) or static probes within the flow. Lagrangian detectors can be used for a number of applications including calculating mixing within the flow or as agents in individual-based biological modelling. Both require high numbers of detectors to be injected into the flow. In a parallel-adaptive run the detectors must be moved along with their owning element when Zoltan deems that element should be moved to a new processor.

Within the Zoltan implementation detectors are dealt with in the callback functions for migrating the fields data. This is because each detector is associated with a particular element rather than a node and hence it is not known which process should own each detector until the transfer fields stage. There is also some work involved before and after the use of Zoltan to migrate the fields data to update certain data structures and to deal with corner cases.

Detectors are implemented as a Fortran derived type with all the information; position, name, element, type, id, etc. contained within the derived type. For the Zoltan implementation routines for packing/unpacking the detector information between the derived type and a real buffer were produced. Originally these were only used within Zoltan but they have since been adopted for use throughout the detector code in Fluidity whenever detectors are transferred between processes.

Before the fields are transferred the detector lists must be pre-processed. This is done because the detectors are stored in one or more unordered linked lists and in pre-processing them we avoid searching these lists during the packing callback. The pre-processing is done as a subroutine call from the callback function for determining the field data sizes. The pre-processing is done at this point as this is when the required list of elements being transferred from the process is available. The pre-processing consists of:

1. Determining the element associated with each detector.
2. If the detector's element is to be transferred, moving the detector from its current detector list to a list of detectors to be sent.
3. Keeping a count of the number of detectors to be sent with each element being transferred.

The pseudo code below details how the three pre-processing steps are implemented:

The implementation aims to avoid traversing the detector lists more than once. For each detector it is checked if its owning element is on the list of elements being transferred. If a match is found then the process which will own the element in the new partitioning is determined. If the new owner is the current owner then the element is only having its data transferred as it will be in another processes halo in the new partitioning. In Fluidity detectors are only dealt with by the process which owns the element containing the detector, so for halo elements detectors do not need to be transferred. If the new owner is not the current owner then the ownership of the element is changed in the new partitioning and the detector must be transferred. The detector is moved from its current detector list to a list of detectors being transferred and the count of the number of detectors in the element is incremented.

The pre-processing of the detector lists gives an array containing the number of detectors in each of the elements being transferred and an ordered list of the detectors to be transferred. The order in which detectors were added to the send detector list is the same as the order they will be removed from the list when the detectors are packed into the communications buffer with the other fields data. The pre-processing therefore allows the send detector list to be traversed serially during the packing process and avoids needing to search the send detector list for detectors which must be sent with each element.

The following modifications are made to the callback functions to add detector functionality to parallel adaptive simulations in Fluidity:

1. During field data sizes callback the pre-processing of the detector lists is carried out. The callback also ensures enough memory is available to pack all the detector information along with the element's field data using the count of detectors in each element calculated during the pre-processing.
2. During the field data packing callback, for each element loop over the number of detectors in the element (as calculated in the pre-processing) and remove each from the send detector list and pack them into the communications buffer.
3. During the field data unpacking callback for each element read the number of detectors being sent with it. Loop over the number of detectors transferred with the element checking to see if they are owned by this process in the new partitioning. If the process now owns the detector allocate a new detector in the unpacked detector list and unpack the detector data into it.

Unpacking the detectors within the unpacking of fields data callback also makes use of a temporary detector list, the unpacked detector list. This is so after the migrate the detectors which were not transferred can be updated in the normal lists before merging in the detectors that were transferred. Elements may be transferred to more than one process in the migrate phase. This is because it is not just owned elements data which is being transferred but halo elements data as well. When unpacking detectors each process checks the owner of the element containing the detector and only unpacks to the unpacked detectors list if it is the element owner in the new partitioning.

As was briefly mentioned above once the call to Zoltan_Migrate has been made and the field data has been transferred the detector lists containing detectors which were not transferred must be updated. Local element numbers change between the old and new partitioning and each detector is associated with a particular local element number so this value must be updated. This is done using two mappings:

• Between old local element number and the universal element number created from the old mesh before any data migration.
• Between universal element number and new local element number created from the new mesh after the node migrations and construction of the new mesh.

Each detector has the element number of the element that owns it first mapped the the universal element number and then from the universal element number to a new local element number. The new local element number is stored in the detector.

An additional complication when migrating detectors can be found at this point. A corner case exists where a detector does not get flagged as being in an element that is being transferred during the pre-processing of the detector lists, but it is also in an element which will not be owned (or even known about) on the current owning process in the new partitioning. In this case the detector is broadcast to all other processors until one accepts it as being within an element that processor owns.

Finally once the detector lists have been updated those detectors received when migrating the fields data are merged in from the unpacked detector list. Each detector in the unpacked detector list was transferred with the element owner set to the elements universal element number. The universal element number is mapped to the new local element number, the detectors element owner is updated and the detector is moved from the unpacked detectors list to the appropriate detector list.