1. technology and computing
2. hardware
3. computer components
4. chips and processors

How to write code that will survive the
many-core revolution
Write once, deploy many(-cores)
F. Bodin, CTO Foreword
•  “How to write code that will survive the
many-core revolution?” is being setup as a collective
initiative of HMPP Competence Centers
o  Gather competences from universities across the world (Asia,
Europe, USA) and R&D projects: H4H, Autotune, TeraFlux
o  Visit http://competencecenter.hmpp.org for more information
November 2011
www.caps-entreprise.com
2
Introduction
•  Computing power comes from parallelism
o  Hardware (frequency increase) to software (parallel codes) shift
o  Driven by energy consumption
•  Keeping a unique version of the codes, preferably mono-language, is a
necessity
o  Directive-based approaches suitable
o  Write codes that will last many architecture generations
•  A context of fast evolving hardware targets
•  Addressing many-core programming challenge implies
o  Massively parallel algorithms
o  New development methodologies / code architectures
o  New programming environments
November 2011
www.caps-entreprise.com
3
Many-Cores
•  Massively parallel devices
o  Thousands of threads needed
Data/stream/vector
parallelism to be
exploited by GPUs
Data transfer between CPU and GPUs, mulBple address spaces November 2011
E.g. CUDA / OpenCL www.caps-entreprise.com
4
Where Are We Going?
forecast November 2011
www.caps-entreprise.com
5
HMPP Heterogeneous Multicore Parallel Programming
•  A directive based approach for many-core
o  Cuda device, OpenCL device
#pragma hmpp f1 codelet!
myfunc(...){
...
for()
for()
for()
...
main(){
...
...
}
#pragma hmpp f1 callsite
myfunc(V1[k],V2[k]);
...
}
November 2011
Available Q1 2www.caps-entreprise.com
012 CPU
version
GPU
version
6
Software Main Driving Forces
•  ALF - Amdahl’s Law is Forever
o  A high percentage of the execution time has to be parallel
o  Many algorithms/methods/techniques will have to be reviewed to scale
•  Data locality is expected to be the main issue
o  Limit on the speed of light
o  Moving data will always suffer latency
November 2011
www.caps-entreprise.com
7
What to Expect From the Architecture?
•  Strong Non Uniform Memory Access (NUMA) effects
o  Not all memories may be coherent
o  Multiple address spaces
•  Node is not an homogeneous piece of hardware
o  Not only one device
o  Balance between small and fat cores changes over time
•  Many parallelism forms
o  Increasing number of processing units
o  Some form of vector computing
November 2011
www.caps-entreprise.com
8
One MPI Process per Core Approach
•  Benefit
o  No need to change existing code
•  Issues
o  Extra latency compared to shared memory use
•  MPI implies some copying required by its semantics (even if efficient MPI implementations
tend to reduce them)
o  Excessive memory utilization
•  Partitioning for separate address spaces requires replication of parts of the data.
•  When using domain decomposition, the sub-grid size may be so small that most points are
replicated (i.e. ghost zone)
•  Memory replication implies more stress on the memory bandwidth which finally produces a
weak scaling
o  Cache trashing between MPI processes
o  Heterogeneity management
•  How are MPI processes linked to accelerator resources?
•  How to deal with different core speed?
•  The balance may be hard to find between the optimal MPI process that makes good use of the
CPU core and the use of the accelerators that may be more efficiently used with a different
ratio
November 2011
www.caps-entreprise.com
9
Thread Based Parallelism Approach
•  Benefits
o  Some codes are already ported to OpenMP/threads APIs
o  Known APIs
•  Issues
o  Data locality and affinity management
•  Data locality and load balancing (main target of thread APIs) are in general two
antagonistic objectives
•  Usual thread APIs make it difficult / not direct to express data locality affinity
o  Reaching a tradeoff between vector parallelism (e.g. using the AVX instruction
set), thread parallelism and MPI parallelism
•  Vector parallelism is expected to impact more and more on the performance
•  Current threads APIs have not been designed to simplify the implementation of such
tradeoff
o  Threads granularity has to be tuned depending on core characteristics (e.g. SMT,
heterogeneity)
•  Thread code writing style does not make it easy to tune
November 2011
www.caps-entreprise.com
10
An Approach for Portable Many-Core Codes
1.  Try to expose node massive data parallelism in a target independent way
2.  Do not hide parallelism by awkward coding
3.  Keep code debug-able
4.  Track performance issues
5.  Exploit libraries
6.  When allowed, do not target 100% of possible performance, the last 20%
are usually very intrusive in the code
November 2011
www.caps-entreprise.com
11
Express Parallelism, not Implementation
•  Rely on code generation for implementation details
o 
o 
o 
o 
Usually not easy to go from a low level API to another low level one
Tuning has to be possible from the high level
But avoid relying on compiler advanced techniques for parallelism discovery, …
You may have to change the algorithm!
•  An example with HMPP
Code generaBon process OpenMP/Threads Cuda/OpenCL Vector ISA November 2011
No automaBc translaBon HMPP www.caps-entreprise.com
#pragma hmppcg gridify(j,i) !
#pragma hmppcg unroll(4), jam(2)!
for( j = 0 ; j < p ; j++ ) {!
for( i = 0 ; i < m ; i++ ) {!
for (k = ...) { ...}!
vout[j][i] = alpha * ...;!
}!
}!
12
HMPP 3.0 Map Operation on Data Collection
#pragma hmpp <mgrp> parallel
for(k=0;k<n;k++) {
#pragma hmpp <mgrp> f1 callsite
myparallelfunc(d[k],n);
}
CPU 0 CPU 1 Main memory d0 d1 d2 d3 November 2011
GPU 0 GPU 1 GPU 2 Device memory Device memory Device memory d1 www.caps-entreprise.com
d2 d3 13
Exposing Massive Parallelism
•  Do not hide parallelism by awkward coding
o  Data structure aliasing, …
o  Deep routine calling sequences
o  Separate concerns (functionality coding versus performance coding)
•  Data parallelism when possible
o  Simple form of parallelism, easy to manage
o  Favor data locality
o  But sometimes too static
•  Kernels level
o  Expose massive parallelism
o  Ensure that data affinity can be controlled
o  Make sure it is easy to tune the ratio vector / thread parallelism
November 2011
www.caps-entreprise.com
14
Data Structure Management
•  Data locality
o  Makes it easy to move from one address space to another one
o  Makes it easy to keep coherent and track affinity
•  Do not waste memory
o  Memory per core ratio not improving
•  Choose simple data structures
o  Enable vector/SIMD computing
o  Use library friendly data structures
o  May come in multiple forms, e.g. sparse matrix representation
•  For instance consider “data collections” to deal with multiple
address spaces or multiple devices or parts of a device
o  Gives a level a adaptation for dealing with heterogeneity
o  Load distribution over the different devices is simple to express
November 2011
www.caps-entreprise.com
15
Collection of Data in HMPP 3.0
d[0] d[0].data d[1] CPU Memory d[1].data struct { float * data ; int size ; } Vec; mirroring GPU 1 Memory November 2011
GPU 2 Memory www.caps-entreprise.com
GPU 3 Memory 16
Performance Measurements
•  Parallelism is about performance!
•  Track Amdahl’s Law Issues
o  Serial execution is a killer
o  Check scalability
o  Use performance tools
•  Add performance measurement in the code
o  Detect bottleneck a.s.a.p.
o  Make it part of the validation process
November 2011
www.caps-entreprise.com
17
HMPP Wizard
GPU library usage detecBon Access to MyDevDeck web online resources 11/16/11
www.caps-entreprise.com
18
Debugging Issues
•  Avoid assuming you won’t have to debug!
•  Keep serial semantic
o  For instance, implies keeping serial libraries in the application code
o  Directives based programming makes this easy
•  Ensure validation is possible even with rounding errors
o  Reductions, …
o  Aggressive compiler optimizations
•  Use defensive coding practices
o  Events logging, parameterize parallelism, add synchronization points, …
o  Use debuggers (e.g. Allinea DDT)
November 2011
www.caps-entreprise.com
19
Allinea DDT – Debugging Tool for CPU/GPU
•  Debug your kernels:
o  Debug CPU and GPU concurrently
•  Examine thread data
o  Display variables
•  Integrated with HMPP
o  Allows HMPP directives
breakpoints
o  Step into HMPP codelets
11/16/11
www.caps-entreprise.com
20
Dealing with Libraries
• 
Library calls can usually only be partially replaced
o 
o 
o 
o 
• 
No one to one mapping between libraries (e.g.BLAS, FFTW, CuFFT, CULA,LibJacket )
No access to all code (i.e. avoid side effects)
Don’t create dependencies on a specific target library as much as possible
Still want a unique source code
Deal with multiple address spaces / multi-GPUs
o  Data location may not be unique (copies)
o  Usual library calls assume shared memory
o  Library efficiency depends on updated data location (long term effect)
• 
Libraries can be written in many different languages
o  CUDA, OpenCL, HMPP, etc.
• 
There is not one binding choice depending on applications/users
o  Binding needs to adapt to uses depending on how it interacts with the remainder of the code
o  Choices depend on development methodology
November 2011
www.caps-entreprise.com
21
Example of Library Uses with HMPP 3.0
C
CALL INIT(A,N)
Replaces the call to a proxy that handles GPUs CALL ZFFT1D(A,N,0,B) ! This call is needed to initialize FFTE
and allows to mix user GPU code with library ones CALL DUMP(A,N)
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
C TELL HMPP TO USE THE PROXY FUNCTION FROM THE FFTE PROXY GROUP
C CHECK FOR EXECUTION ERROR ON THE GPU
!$hmppalt ffte call , name="zfft1d", error="proxy_err"
CALL ZFFT1D(A,N,-1,B)
CALL DUMP(A,N)
C
C SAME HERE
!$hmppalt ffte call , name="zfft1d" , error="proxy_err"
CALL ZFFT1D(A,N,1,B)
CALL DUMP(A,N)
November 2011
www.caps-entreprise.com
22
Library Interoperability in HMPP 3.0
HMPP RunBme API ...!
call libRoutine1(…)!
...!
...!
#pragma hmppalt!
call libRoutine2(…)!
...!
...!
...!
...!
call libRoutine3(…)!
...!
November 2011
NaBve GPU RunBme API ...!
...!
...!
...!
...!
...!
proxy2(…)!
...!
...!
www.caps-entreprise.com
C/CUDA/… ! GPU Lib ...!
...!
...!
...!
...!
...! ! CPU Lib ...!
gpuLib(…)!
...! cpuLib1(…)!
...! ...!
...!
...!
...!
cpuLib3(…)!
...!
...!
23
Conclusion
•  Software has to expose massive parallelism
o  The key to success is the algorithm!
o  The implementation has “just” to keep parallelism flexible and easy to
exploit
•  Directive-based approaches are currently one of the most promising
track
o  Preserve code assets
o  May separate parallelism exposure from the implementation (at node level)
•  Remember that even if you are very careful in your choices you may
have to rewrite parts of the code
o  Code architecture is the main asset here
November 2011
www.caps-entreprise.com
See us on Booth #650 24
Accelerator Programming Model
Directive-based programming
GPGPU
Manycore programming
Hybrid Manycore Programming
Petaflops
Code speedup
HPC community
HPC open standard
Parallel computing
Multicore programming
Parallelization
Exaflops
NVIDIA Cuda
Hardware accelerators programming
High Performance Computing
Parallel programming interface
Massively parallel
Open CL
hap://www.caps-­‐entreprise.com hap://twiaer.com/CAPSentreprise hap://www.openhmpp.org