3. Programming Environment

3.1 Modules environment

The Cray XE system uses the modules environment to control the programming and runtime environment. For more information on using modules see the output of "man module" on the system or:

Some quick start commands for the modules environment usage:

To check which modules you have loaded, type

module list

To search all the available modules, type

module avail

To search for a specific available module(s), type

module avail [module_name]

To load a specific module, type

module load [module_name]

To unload a specific module, type

module unload [module_name]

To swap a loaded module with another one:

module swap [first_module_name] [second_module_name]

3.2 Compiler wrapper commands

No matter which programming environment you have loaded you access the compilers via the following commands:

  • ftn - Fortran compiler
  • cc - C compiler
  • CC - C++ compiler

Using these high-level compiler wrapper commands ensures that you are compiling for the correct processor architecture on the compute nodes and also makes sure that all the correct library versions are accessed.

You should not use the native compiler commands (e.g. pgf90, gfortran) on Cray XE systems unless you are compiling analysis tools for the service (login) nodes.

Examples of compilation procedures for a number of software packages are available on the individual software package pages. These may provide useful hints on how to compile your software.

3.3 Available compilers

A number of different compiler suites are available on HECToR. These are:

  • Cray compilers - the "PrgEnv-cray" module
  • Portland group compilers - the "PrgEnv-pgi" module
  • GNU compilers - the "PrgEnv-gnu" modules

The default programming environment is the Cray environment. You can switch compiler suites with the "module swap" command. For example, to switch from the Cray compiler suite to the GNU compiler suite you would use:

module swap PrgEnv-cray PrgEnv-gnu

You can also switch between different versions of compilers within a compiler suite by using the "module swap" command. For example, to change the version of the GNU compiler you are using:

module swap gcc gcc/4.6.1

The compiler version modules for the different compiler suites are:

  • cce - For the Cray compiler suite
  • pgi - For the PGI compiler suite
  • gcc - For the GNU compiler suite

3.3.1 Partitioned Global Address Space (PGAS) compiler support

The Cray compiler suite supports both the Coarray Fortran (CAF) and Unified Parallel C (UPC) PGAS language extensions. For more information on enabling these options see the appropriate section on parallel programming below. You can also find out more at:

Cray XE systems also support the Chapel PGAS language through the "chapel" compiler. You can use this compiler by adding the "chapel" module:

module add chapel

You can find more information on the Chapel language on the Chapel web site.

3.4 Available (vendor optimised) numerical libraries

The Cray CLE distribution comes with a range of optimised numerical libraries compiled for all the supported compiler suites listed above. The libraries are listed in the table below along with their current module names and a brief description. Generally, if you wish to use a library in your code you should only need to load the module before compilation.

LibraryModuleDescription
LibScicray-libsciCray Scientific Library includes BLAS, LAPACK, BLACS and ScaLAPACK
PETSccray-petscPortable, Extensible Toolkit for Scientific Computation
FFTWfftwFastest Fourier Transform in the West versions 2 and 3
Trilinoscray-trilinos
Global Arrayscray-ga

Many of these libraries use the Cray autotuning framework to improve the on-node performance. This framework automatically selects the best version of the library routines based on the size and nature of your problem at runtime.

More information on the library contents can be found on the following web:

3.5 Available MPI implementations

Cray XE systems use a version of the MPICH 2 library that has been optimised for the Gemini interconnect. The version of the MPI library is controlled by the "cray-mpich2" module. All users will have the default "cray-mpich2" module loaded when they connect to the system - for best performance we recommend using the default or later versions.

You can get a list of available versions of the "cray-mpich2" module by using the "module avail" command. For example:

module avail cray-mpich2

Once the cray-mpich2 module is loaded then compiling using the standard wrapper compiler commands will automatically include and link to the MPI headers and libraries - you do not need to specify any more options on the commnad line.

3.6 OpenMP Support

All of the compiler suites available on the Cray XE system support the OpenMP 2 standard. The Cray and GNU compiler suites also support OpenMP 3 statements.

Note: in the Cray compiler suite OpenMP functionality is turned on by default.

3.6.1 Compiler flags

The compiler flags to include OpenMP for the various compiler suites are:

CompilerEnable OpenMPDisable OpenMP
Cray-h omp-h noomp
PGI-mp-Mnoopenmp
GNU-fopenmpby omission of -fopenmp

You may find these links useful:

3.7 SHMEM

To compile code that uses SHMEM you should ensure that the cray-shmem module lis loaded. Loading this module will ensure that all the correct environment variables are set for linking to the libsma static and dynamic libraries. You can load the module with the command:

module load cray-shmem

For more information on using SHMEM, see the Cray man pages at: