The dCSE Project

The main goal of this dCSE project is to port AVS/Express DDR to the Cray XT4 hardware provided by HECToR (Phase2a). This will allow the visualization of large datasets that currently exceed the capabilities of our GPU-based visualization systems. Working with Materials Science we have access to datasets that are typically 50-500GB in size. The datasets provide material density data acquired by CT scanning equipment, some of which is provided by the Henry Moseley X-Ray Imaging Facility at Manchester [6]. We also expect to acquire datasets from the I12 JEEP beamline at the Diamond Light Source, RAL. It should be noted that AVS/Express DDR is not limited to this type of data and so will be of use to researchers working with other forms of data. In particular AVS has NetCDF [7] and HDF5 [8] readers in addition to the generalised AVS Field reader which allows many forms of data to be described and read in to the application.

10 months of development time were allocated to the project, beginning May 2009, with the possibility of 2 months of NAG CSE support to assist with optimisation during the 10 month period. This option was not taken up as it was deemed unnecessary to conduct very low level optimisation due to the interactive nature of the application.

The main development task was to modify the AVS/Express code such that it could operate within the HECToR runtime environment. This was broken down in to the following subtasks:

• Provide a mechanism to allow the AVS/Express main application (network editor, module user interfaces, visualization window) to operate on the login nodes, where X11 functionality is available, yet communicate with parallel module and rendering processes executing on the backend nodes. Modifications to the AVS source tree should be kept to a minimum, avoiding significant architectural changes that would impact other platforms. This will make AVS/Express appear more like the open source ParaView [9,10,11] application in structure.

• Modify the image compositing architecture within AVS/Express so that it can use MPI on the backend nodes. At the time of development an open source image compositing library is used by AVS/Express. This provides no MPI functionality and only supports dynamically linked applications.

• Optimise the existing MPI communication within the AVS/Express. It is known that the parallel renderer uses point-to-point communication which is expected to reduce scalability to large processor counts.

• Provide AVS networks that demonstrate the common visualization tasks that users will perform.

AVS/Express must be run from a login node with X11 forwarded over the ssh connection to the user's X server. The X server must support the GLX protocol. This is not a particularly efficient method of running an X11 application and results in the overall rendering frame rate having an upper bound that cannot be improved by parallel rendering because the limiting factor becomes the time needed to transfer the image from the login node to the user's X server. In the case of the AVS renderer the upper bound is approximately 5.0 frames per second (fps) for a 512x512 window and 1.1 fps for a 1024x1024 window when rendering remotely to a test desktop system running linux. While this appears to be low the visualization does in fact remain sufficiently responsive that the user can interact with the visualization. We are more concerned with the ability to scale out the memory usage to increase the size of dataset that can be visualized.

The AVS/Express source tree has been compiled with the default GNU compiler (currently 4.4.2). This is one of the compilers supported by the AVS build process for 64bit linux platforms (the other being the Intel compiler). The group in Manchester have a source code agreement with AVS allowing access to the AVS/Express source tree although certain components are only available as libraries. The build process is to compile the AVS base executable which then reads the V module description files (AVS/Express uses its own module description language named V). Processing of the V files generates C/C++/Fortran code for those modules which is then compiled as part of the build process. This produces the final express executable.