7. Compilation

When compiling for the HECToR compute nodes you will always be using the (dual-core) frontend nodes which have a different architecture. Hence you will be "cross-compiling". This compilation is facilitated by the Cray compiler wrapper scripts "ftn", "cc" and "CC".

For examples of compilation procedures for software packages on HECToR please see the detailed individual software package pages which contain detailed compilation instructions. These examples contain many useful hints for compiling on HECToR.

You will also find that the HECToR Optimisation Guide contains a lot of useful information on compiling and getting the best performance out of the HECToR service.

Please note that HECToR is a heterogeneous system, with dual-core login and serial batch nodes. For more information on compiling for the login and serial batch nodes, see the Compiling for the frontend nodes section.

7.1 Cross-compiling for Compute Nodes

The HECToR facilities have the Cray, PGI, and GNU compiler suites available. Irrespective of the compiler suite you chose, you should use the Cray provided wrapper scripts ftn, cc or CC to access the compilers. These wrapper scripts access the compiler suite as specified by the PrgEnv module you have loaded. Currently, the Cray compiler suite is the default for all users.

For best performance, the xtpe-interlagos should always be loaded (it is loaded by default for all users).

The wrapper scripts can be used to compile both sequential and parallel codes. Further information on the wrapper scripts can be obtained from the man pages, for example:

man ftn

To find out which compiler is being accessed by the wrapper use the -V option:

user@nid00004:~> ftn -V
/opt/cray/xt-asyncpe/5.03/bin/ftn: INFO: Compiling with XTPE_COMPILE_TARGET=native.
Cray Fortran : Version 7.4.4  Wed Nov 02, 2011  14:52:44

which tells us that the crayftn compiler is being used. Full details of the compiler options can then be obtained from:

man crayftn

The manual pages for the different compiler suites are:

  • Cray - Fortran man crayftn, C man craycc, C++ man crayCC
  • GNU - Fortran man crayftn, C/C++ man gcc
  • PGI - Fortran man pgf90, C man pgcc, C++ man pgCC

7.1.1 Compiling Fortran code

Use the ftn script to compile all Fortran code. See man ftn for further details.

For example:

ftn [compiler options] my_code.f90 -o my_code.x

7.1.2 Compiling C code

Use the cc script to compile C code. See man cc for further details.

For example:

cc [compiler options] my_code.c -o my_code.x

7.1.3 Compiling C++ code

Use the CC script to compile C++ code. See man CC for further details.

For example:

CC [compiler options] my_code.cpp -o my_code.x

The compiler options, [compiler options], will be passed from the wrapper script to the appropriate compiler, for example:

CC -O3 my_code.cpp -o my_code.x

will pass -O3 to the appropriate compiler.

7.1.4 Selecting compiler suites and compiler versions

You can choose which compiler suite the wrappers access by loading appropriate module. The names of the modules referring to the different compiler suites start with PrgEnv- followed by the name of the compiler. The compiler suites available are:

  • PrgEnv-cray - Cray Compiler Suite;
  • PrgEnv-pgi - PGI Compiler Suite;
  • PrgEnv-gnu - GNU Compiler Suite;

To check which compiler suite and programming environment is presently loaded, type

module list

at the prompt and search the output for the relevant name. Issuing:

module avail PrgEnv

will list all the alternatives available to you. To change the compiler suite you have to unload the present programming environment and load the new one. For example, to change from the Cray compiler suite to the GNU compiler suite, use

module unload PrgEnv-cray
module load PrgEnv-gnu

This can also be accomplished with a single statement using the module swap command:

module swap PrgEnv-cray PrgEnv-gnu

You can also swap between different versions of the same compiler. Switching between different versions of the same compiler does not require the programming environment to be changed as the same library paths, header files, etc. will be used. To find out which versions of a particular compiler are available to you, type:

module avail cce

which will list the available versions of the Cray compiler. If you want to change to version 7.4.3 use:

module swap cce cce/7.4.3

which will unload the current version of the Cray compiler and load version 7.4.3.

Please note: changing between compiler suites (e.g. PGI to GNU, Cray to PGI, etc.) must be carried out via the PrgEnv modules. Along with selecting the compiler, the programming environment also determines which numerical and MPI libraries are picked up. Attempts to change compiler suite without changing the programming environment may result in the wrong numerical or MPI libraries being used and your code may not compile or may fail to execute as a result.

Compiler version module names are:

  • cce - For the Cray compiler
  • pgi - For the PGI compiler
  • gcc - For the GNU compiler

Special Instructions for NAG Fortran Compiler

Note: The NAG compiler uses a C compiler at the last stage. It is most convenient to use the GNU C compiler. For this to work, load the GNU compiler (PrgEnv-gnu) and then the Nag-f95 module:

module swap PrgEnv-cray PrgEnv-gnu
module load Nag-f95
f95 [options] mycode.f90

In this configuration, f95 is the command that invokes the NAG compiler and ftn the command that invokes the GNU gfortran compiler. The f95 compiler uses the dressed cc script, not the gcc from the gnu module.

Access to the MPI library is either through "USE MPI" or "INCLUDE 'mpif.h'". The latter requires the option"-I $MPICH_DIR/include" to be added to the command line. The MPI library is always linked in by the back-end C compiler.

There is extensive documentation available through f95(1), nag_modules(3), fpp(1). Even more can be found on the NAG website and a list of useful options can be found in the Debugging Good Practice Guide

7.1.5 Compiling MPI code

The Cray wrappers ftn, cc or CC, which are explained here, will compile MPI code. You do not need to specify anything else to find the relevant headers or to link against the MPI library.

The Tuning section of the HECToR Optimisation Guide provides more information on optimising the MPI performance of your code:

7.1.6 Compiling OpenMP code

OpenMP codes can currently be run within a node. The PGI, Cray and GNU Fortran, C and C++ compilers support OpenMP. To compile an OpenMP code you need to specify the appropriate compiler option. For example, to compile an OpenMP code with the PGI compiler you would use:

ftn -mp=nonuma my_openmp_code.f90 my_openmp_code.x

for the Cray compiler OpenMP support is enabled by default (use -h noomp to disable it)

and for the GNU compiler use:

ftn -fopenmp my_openmp_code.f90 -o my_openmp_code.x

The Programming Environment section of the HECToR Optimisation Guide provides more information on compiling OpenMP code:

The HECToR Optimisation Guide provides more information on optimising the OpenMP performance of your code:

7.1.7 Compiling mixed (OpendMP/MPI) code

Mixed code is a mix of message-passing (MPI) and multithreading(OpenMP) to maximize performance. To compile hybrid code follow the instructions on Compiling OpenMP code. If thread support is required the MPI environment should be initialized with MPI_INIT_THREAD() instead of MPI_INIT().

7.1.8 Compiling Coarray Fortran code

Currently only the Cray Compiler Suite supports Coarray Fortran. To compile a Coarray Fortran code you need to specify the appropriate compiler option:

ftn -h caf my_caf_code.f90 -o my_caf_code.x

To fix the number of images at compile time you can use the compiler option -X:

ftn -h caf -X 128 my_caf_code.f90 -o my_caf_code.x

Useful compiler options

As a starting point we suggest the following compiler options which should provide a reasonable level of optimisation. You should also study the HECToR Optimisation Guide.

HECToR Optimisation Guide - Tuning

  • For Cray: -O3
  • For PGI: -fastsse
  • For GNU: -O3

To get more information on the optimisations (particularly the generation of SSE instructions) you can use the following flags:

  • For PGI: -Minfo -Mneginfo

7.2 Compiling for the frontend nodes

If you want to compile code for the frontend nodes (for use with the serial queues or login nodes, for example) you must change the architecture that the compilers are compiling for by issuing the command:

module swap xtpe-interlagos xtpe-istanbul

before you compile any code. This is required as, by default, the compilers are set up to compile for the compute nodes which (on phase 3) have a different processor architecture to the frontend nodes.

Note: if you try to run code compiled for the compute nodes on the frontend nodes then you will see a "Illegal instruction" error.

6. Running jobs | Contents | 8. Software libraries