The HECToR Service is now closed and has been superceded by ARCHER.

7. Compilation

When compiling for the HECToR compute nodes you will always be using the (dual-core) frontend nodes which have a different architecture. Hence you will be "cross-compiling". This compilation is facilitated by the Cray compiler wrapper scripts "ftn", "cc" and "CC".

For examples of compilation procedures for software packages on HECToR please see the detailed individual software package pages which contain detailed compilation instructions. These examples contain many useful hints for compiling on HECToR.

You will also find that the HECToR Optimisation Guide contains a lot of useful information on compiling and getting the best performance out of the HECToR service.

Please note that HECToR is a heterogeneous system, with dual-core login and serial batch nodes. For more information on compiling for the login and serial batch nodes, see the Compiling for the frontend nodes section.

7.1 Cross-compiling for Compute Nodes

The HECToR facilities have the Cray, PGI, and GNU compiler suites available. Irrespective of the compiler suite you chose, you should use the Cray provided wrapper scripts ftn, cc or CC to access the compilers. These wrapper scripts access the compiler suite as specified by the PrgEnv module you have loaded. Currently, the Cray compiler suite is the default for all users.

For best performance, the craype-interlagos should always be loaded (it is loaded by default for all users).

The wrapper scripts can be used to compile both sequential and parallel codes. Further information on the wrapper scripts can be obtained from the man pages, for example:

man ftn

To find out which compiler is being accessed by the wrapper use the -V option:

user@nid00004:~> ftn -V
/opt/cray/xt-asyncpe/5.03/bin/ftn: INFO: Compiling with XTPE_COMPILE_TARGET=native.
Cray Fortran : Version 7.4.4  Wed Nov 02, 2011  14:52:44

which tells us that the crayftn compiler is being used. Full details of the compiler options can then be obtained from:

man crayftn

The manual pages for the different compiler suites are:

  • Cray - Fortran man crayftn, C man craycc, C++ man crayCC
  • GNU - Fortran man crayftn, C/C++ man gcc
  • PGI - Fortran man pgf90, C man pgcc, C++ man pgCC

7.1.1 Compiling Fortran code

Use the ftn script to compile all Fortran code. See man ftn for further details.

For example:

ftn [compiler options] my_code.f90 -o my_code.x

7.1.2 Compiling C code

Use the cc script to compile C code. See man cc for further details.

For example:

cc [compiler options] my_code.c -o my_code.x

7.1.3 Compiling C++ code

Use the CC script to compile C++ code. See man CC for further details.

For example:

CC [compiler options] my_code.cpp -o my_code.x

The compiler options, [compiler options], will be passed from the wrapper script to the appropriate compiler, for example:

CC -O3 my_code.cpp -o my_code.x

will pass -O3 to the appropriate compiler.

7.1.4 Selecting compiler suites and compiler versions

You can choose which compiler suite the wrappers access by loading appropriate module. The names of the modules referring to the different compiler suites start with PrgEnv- followed by the name of the compiler. The compiler suites available are:

  • PrgEnv-cray - Cray Compiler Suite;
  • PrgEnv-pgi - PGI Compiler Suite;
  • PrgEnv-gnu - GNU Compiler Suite;

To check which compiler suite and programming environment is presently loaded, type

module list

at the prompt and search the output for the relevant name. Issuing:

module avail PrgEnv

will list all the alternatives available to you. To change the compiler suite you have to unload the present programming environment and load the new one. For example, to change from the Cray compiler suite to the GNU compiler suite, use

module unload PrgEnv-cray
module load PrgEnv-gnu

This can also be accomplished with a single statement using the module swap command:

module swap PrgEnv-cray PrgEnv-gnu

You can also swap between different versions of the same compiler. Switching between different versions of the same compiler does not require the programming environment to be changed as the same library paths, header files, etc. will be used. To find out which versions of a particular compiler are available to you, type:

module avail cce

which will list the available versions of the Cray compiler. If you want to change to version 7.4.3 use:

module swap cce cce/7.4.3

which will unload the current version of the Cray compiler and load version 7.4.3.

Please note: changing between compiler suites (e.g. PGI to GNU, Cray to PGI, etc.) must be carried out via the PrgEnv modules. Along with selecting the compiler, the programming environment also determines which numerical and MPI libraries are picked up. Attempts to change compiler suite without changing the programming environment may result in the wrong numerical or MPI libraries being used and your code may not compile or may fail to execute as a result.

Compiler version module names are:

  • cce - For the Cray compiler
  • pgi - For the PGI compiler
  • gcc - For the GNU compiler

Special Instructions for NAG Fortran Compiler

Note: The NAG compiler uses a C compiler at the last stage. It is most convenient to use the GNU C compiler. For this to work, load the GNU compiler (PrgEnv-gnu) and then the Nag-f95 module:

module swap PrgEnv-cray PrgEnv-gnu
module load Nag-f95
nagfor [options] mycode.f90

In this configuration, nagfor is the command that invokes the NAG compiler and ftn the command that invokes the GNU gfortran compiler. The nagfor compiler uses the dressed cc script, not the gcc from the gnu module. Note that in ealier versions of the NAG compiler, the compiler wrapper is called f95

Access to the MPI library is either through "USE MPI" or "INCLUDE 'mpif.h'". The latter requires the option"-I $MPICH_DIR/include" to be added to the command line. The MPI library is always linked in by the back-end C compiler.

There is extensive documentation available through nagfor(1), nag_modules(3), fpp(1). Even more can be found on the NAG website and a list of useful options can be found in the Debugging Good Practice Guide.

7.1.5 Compiling MPI code

The Cray wrappers ftn, cc or CC, which are explained here, will compile MPI code. You do not need to specify anything else to find the relevant headers or to link against the MPI library.

The Tuning section of the HECToR Optimisation Guide provides more information on optimising the MPI performance of your code:

7.1.6 Compiling OpenMP code

OpenMP codes can currently be run within a node. The PGI, Cray and GNU Fortran, C and C++ compilers support OpenMP. To compile an OpenMP code you need to specify the appropriate compiler option. For example, to compile an OpenMP code with the PGI compiler you would use:

ftn -mp=nonuma my_openmp_code.f90 my_openmp_code.x

for the Cray compiler OpenMP support is enabled by default (use -h noomp to disable it)

and for the GNU compiler use:

ftn -fopenmp my_openmp_code.f90 -o my_openmp_code.x

The Programming Environment section of the HECToR Optimisation Guide provides more information on compiling OpenMP code:

The HECToR Optimisation Guide provides more information on optimising the OpenMP performance of your code:

7.1.7 Compiling mixed (OpendMP/MPI) code

Mixed code is a mix of message-passing (MPI) and multithreading(OpenMP) to maximize performance. To compile hybrid code follow the instructions on Compiling OpenMP code. If thread support is required the MPI environment should be initialized with MPI_INIT_THREAD() instead of MPI_INIT().

7.1.8 Compiling Coarray Fortran code

Currently only the Cray Compiler Suite supports Coarray Fortran. To compile a Coarray Fortran code you need to specify the appropriate compiler option:

ftn my_caf_code.f90 -o my_caf_code.x

To fix the number of images at compile time you can use the compiler option -X:

ftn -X 128 my_caf_code.f90 -o my_caf_code.x

7.1.9 Useful compiler options

As a starting point we suggest the following compiler options which should provide a reasonable level of optimisation. You should also study the HECToR Optimisation Guide.

HECToR Optimisation Guide - Tuning

  • For Cray: -O3
  • For PGI: -fastsse
  • For GNU: -O3

To get more information on the optimisations (particularly the generation of SSE instructions) you can use the following flags:

  • For PGI: -Minfo -Mneginfo

7.1.10 Using dynamic linking/libraries

By default, executables on HECToR are built using static libraries (that is, all of the object code of referenced libraries are contained in the executable file) when using the wrapper scripts. This has the advantage that once an executable is created, whenever it is run in the future, it will always use the same object code and thus give the same results from the same input. However, executables compiled with static libraries have the potential disadvantage that when multiple instances are running simultaneously multiple copies are held in memory, even though much of the object code is identical. This can lead to large amounts of memory being used to hold the executable and not application data.

Alternatively, applications can be compiled to use shared/dynamic libraries (i.e. libraries which are loaded at run-time as and when needed by the application). This may be because static versions of certain libraries are unavailable, or to reduce the amount of memory executables take by sharing common sections of object codes between applications which use the same library.

To create an application that uses shared/dynamic libraries you must pass extra flags during compilation, or set an environment variable. You can either:

  1. Use the -dynamic flag when invoking the compiler for linking.
  2. Set the environment variable XTPE_LINK_TYPE=dynamic without any extra compilation/linking options.

By default an application compiled this way to use shared libraries will use the default version of the library installed on the system (just like any other Linux executable), even if the system modules were set differently at compile time. This means that the application may potentially be using slightly different object code each time the application runs as the defaults may change. This is usually the desired behaviou for many applications as any fixes or improvements to the default linked libraries are used without having to recompile the application, however some users may feel this is not the desired behaviour for their applications.

Instead, users may choose to hard code the location of the libraries used during compilation and linking into the resulting executable by setting the CRAY_ADD_RPATH=yes environment variable. The same object code, from the original shared libraries, will be used each time the executable is run, as long as the library files exist. This is very similar to how statically linked executables behave.

Finally, users may wish to choose which version of a library is used through the standard module environment. This is done prepending the value of CRAY_ LD_LIBRARY_PATH to the standard LD_LIBRAY_PATH environment variable once all the modules are loaded. This will place all the paths to libraries specified in the modules files ahead of the defaults and allow the Operating System to select the specified versions from the modules in place of the defaults.

Use the UNIX command ldd exe_file to check whether you are using an executable that depends on shared libraries. This utility will also report the shared libraries this executable will use with the current value of LD_LIBRARY_PATH (not CRAY_LD_LIBRARY_PATH).

(Thanks to Dr Tom Edwards of Cray for this text.)

7.2 Compiling for the frontend nodes

If you want to compile code for the frontend nodes (for use with the serial queues or login nodes, for example) you must change the architecture that the compilers are compiling for by issuing the command:

module swap craype-interlagos craype-istanbul

before you compile any code. This is required as, by default, the compilers are set up to compile for the compute nodes which (on phase 3) have a different processor architecture to the frontend nodes.

Note: if you try to run code compiled for the compute nodes on the frontend nodes then you will see a "Illegal instruction" error.

6. Running jobs | Contents | 8. Software libraries