HECToR

HECToR Optimisation Guide

Version 2.5 (27 January 2012)

Andrew Turner (EPCC) a.turner@epcc.ed.ac.uk
Xu Guo (EPCC) xguo@epcc.ed.ac.uk
Lilit Axner (KTH) lilit@kth.se

This guide provides more advanced information on using the HECToR system than is available in the HECToR User Guide. It includes sections detailing the hardware, optimising your code (in serial and parallel), profiling your code and debugging your code. This guide is also being continually updated with more content.

The Optimisation Guide is produced by staff at EPCC and KTH (Sweden) as part of the PRACE (Partnership for Research and Advanced Computing in Europe) initiative.

Contents

1. Introduction

Description of the guide and useful links.

• 1.1 Useful Links

2. System Architecture and Configuration

Detailed description of the HECToR hardware and system software. This includes a in-depth look at the AMD Bulldozer architecture and memory layout.

• 2.1 Processor architecture
  • 2.1.1 Compute node hardware
  • 2.1.2 Service node hardware
• 2.2 Building block architecture
• 2.3 Memory architecture
• 2.4 Interconnect
• 2.5 I/O subsystem architecture
• 2.6 Available file systems
  • 2.6.1 "home"
  • 2.6.2 "work"
  • 2.6.3 Archive
• 2.7 Operating system (CLE)
  • 2.7.1 Cluster Compatibility Mode (CCM)

3. Programming Environment

Details on how to compile codes; use numerical libraries; MPI libraries and other parallel programming options.

• 3.1 Modules environment
• 3.2 Compiler wrapper commands
• 3.3 Available compilers
  • 3.3.1 Partitioned Global Address Space (PGAS) compiler support
• 3.4 Available (vendor optimised) numerical libraries
• 3.5 Available MPI implementations
• 3.6 OpenMP
  • 3.6.1 Compiler flags
• 3.7 SHMEM

4 Batch system / job command language

Advanced use of the HECToR batch system.

• 4.1 Batch system commands
• 4.2 Multiple aprun commands in a single job
• 4.3 Job arrays
• 4.4 Interactive jobs
• 4.5 Writing job submission scripts in Perl and Python

5. Performance analysis

How to use the performance analysis tools installed on the system.

• 5.1 Available Performance Analysis Tools
• 5.2 Cray Performance Analysis Tool (CrayPAT)
  • 5.2.1 Instrumenting a code with pat_build
  • 5.2.2 Analysing profile results using pat_report
  • 5.2.3 Using hardware counters
• 5.3 Allinea MAP
  • 5.3.1 Create MAP profiling MPI library
  • 5.3.2 Re-compile code linking to MAP MPI libraries
  • 5.3.3 Run the profiling experiment
  • 5.3.4 Visualise the profile data
• 5.2 General hints for interpreting profiling results
  • 5.2.1 Spotting load-imbalance

6. Tuning

Tips on how to optimise the performance of your code in both serial and parallel.

• 6.1 Optimisation summary
• 6.2 Serial (single-core) optimisation
  • 6.2.1 Compiler optimisation flags
  • 6.2.2 Using Libraries
  • 6.2.3 Writing Optimal Serial Code
  • 6.2.4 Cache Optimisation
• 6.3 Parallel optimisation
  • 6.3.1 Load-imbalance
  • 6.3.2 MPI Optimisation
  • 6.3.3 Mapping tasks/threads onto cores
• 6.4 Advanced OpenMP usage
  • 6.4.1 Environment variables
• 6.5 Memory optimisation
  • 6.5.1 Memory affinity
  • 6.5.2 Memory allocation (malloc) tuning
  • 6.5.3 Using huge pages
• 6.6 I/O optimisation

7. Debugging

How to use the debugging tools installed on the system.

• 7.1 Available Debuggers
• 7.2 Totalview
  • 7.2.1 Example: Debugging an MPI application
  • 7.2.2 Example: Using Totalview to debug a core file
  • 7.2.3 Limitations of Totalview implementation on HECToR
• 7.3 Cray ATP
  • 7.3.1 ATP Example
• 7.4 GDB (GNU Debugger)
  • 7.4.1 Launching your program using 'lgdb'
  • 7.4.2 Debugging the remote gdbserver using 'gdb'
  • 7.4.3 Useful GDB commands
  • 7.4.4 Example: debugging an MPI program using GDB
• 7.5 DDT Debugger
  • 7.5.1 Example: Debugging a parallel application