3. Programming Environment
- 3.1 Modules environment
- 3.2 Compiler wrapper commands
- 3.3 Available compilers
- 3.4 Available (vendor optimised) numerical libraries
- 3.5 Available MPI implementations
- 3.6 OpenMP
- 3.7 SHMEM
3.1 Modules environment
The Cray XE system uses the modules environment to control the programming and runtime environment. For more information on using modules see the output of "man module" on the system or:
Some quick start commands for the modules environment usage:
To check which modules you have loaded, type
module list
To search all the available modules, type
module avail
To search for a specific available module(s), type
module avail [module_name]
To load a specific module, type
module load [module_name]
To unload a specific module, type
module unload [module_name]
To swap a loaded module with another one:
module swap [first_module_name] [second_module_name]
3.2 Compiler wrapper commands
No matter which programming environment you have loaded you access the compilers via the following commands:
- ftn - Fortran compiler
- cc - C compiler
- CC - C++ compiler
Using these high-level compiler wrapper commands ensures that you are compiling for the correct processor architecture on the compute nodes and also makes sure that all the correct library versions are accessed.
You should not use the native compiler commands (e.g. pgf90, gfortran) on Cray XE systems unless you are compiling analysis tools for the service (login) nodes.
Examples of compilation procedures for a number of software packages are available on the individual software package pages. These may provide useful hints on how to compile your software.
3.3 Available compilers
A number of different compiler suites are available on HECToR. These are:
- Cray compilers - the "PrgEnv-cray" module
- Portland group compilers - the "PrgEnv-pgi" module
- GNU compilers - the "PrgEnv-gnu" modules
The default programming environment is the Cray environment. You can switch compiler suites with the "module swap" command. For example, to switch from the Cray compiler suite to the GNU compiler suite you would use:
module swap PrgEnv-cray PrgEnv-gnu
You can also switch between different versions of compilers within a compiler suite by using the "module swap" command. For example, to change the version of the GNU compiler you are using:
module swap gcc gcc/4.6.1
The compiler version modules for the different compiler suites are:
- cce - For the Cray compiler suite
- pgi - For the PGI compiler suite
- gcc - For the GNU compiler suite
3.3.1 Partitioned Global Address Space (PGAS) compiler support
The Cray compiler suite supports both the Coarray Fortran (CAF) and Unified Parallel C (UPC) PGAS language extensions. For more information on enabling these options see the appropriate section on parallel programming below. You can also find out more at:
Cray XE systems also support the Chapel PGAS language through the "chapel" compiler. You can use this compiler by adding the "chapel" module:
module add chapel
You can find more information on the Chapel language on the Chapel web site.
3.4 Available (vendor optimised) numerical libraries
The Cray CLE distribution comes with a range of optimised numerical libraries compiled for all the supported compiler suites listed above. The libraries are listed in the table below along with their current module names and a brief description. Generally, if you wish to use a library in your code you should only need to load the module before compilation.
Library | Module | Description |
---|---|---|
LibSci | cray-libsci | Cray Scientific Library includes BLAS, LAPACK, BLACS and ScaLAPACK |
PETSc | cray-petsc | Portable, Extensible Toolkit for Scientific Computation |
FFTW | fftw | Fastest Fourier Transform in the West versions 2 and 3 |
Trilinos | cray-trilinos | |
Global Arrays | cray-ga |
Many of these libraries use the Cray autotuning framework to improve the on-node performance. This framework automatically selects the best version of the library routines based on the size and nature of your problem at runtime.
More information on the library contents can be found on the following web:
- Cray Application Developer's Environment User Guide
- PETSc Website
- FFTW Website
- Trilinos Website
- Global Arrays Website
3.5 Available MPI implementations
Cray XE systems use a version of the MPICH 2 library that has been optimised for the Gemini interconnect. The version of the MPI library is controlled by the "cray-mpich2" module. All users will have the default "cray-mpich2" module loaded when they connect to the system - for best performance we recommend using the default or later versions.
You can get a list of available versions of the "cray-mpich2" module by using the "module avail" command. For example:
module avail cray-mpich2
Once the cray-mpich2 module is loaded then compiling using the standard wrapper compiler commands will automatically include and link to the MPI headers and libraries - you do not need to specify any more options on the commnad line.
3.6 OpenMP Support
All of the compiler suites available on the Cray XE system support the OpenMP 2 standard. The Cray and GNU compiler suites also support OpenMP 3 statements.
Note: in the Cray compiler suite OpenMP functionality is turned on by default.
3.6.1 Compiler flags
The compiler flags to include OpenMP for the various compiler suites are:
Compiler | Enable OpenMP | Disable OpenMP |
---|---|---|
Cray | -h omp | -h noomp |
PGI | -mp | -Mnoopenmp |
GNU | -fopenmp | by omission of -fopenmp |
You may find these links useful:
- OpenMP Website
- GCC 4.6.1 OpenMP Manual
- OpenMP tutorial and reference
- Intel Getting Started with OpenMP
3.7 SHMEM
To compile code that uses SHMEM you should ensure that the cray-shmem module lis loaded. Loading this module will ensure that all the correct environment variables are set for linking to the libsma static and dynamic libraries. You can load the module with the command:
module load cray-shmem
For more information on using SHMEM, see the Cray man pages at: