1. No category

CLE and How to S
Start Your Application
Y
A li i
Jan Thorbecke
Scalable S ft
Software Architecture
A hit t
2
Scalable Software Architecture: Cray Linux Environment (CLE)
Speciali
Specialized ed
Linux nodes
Microkernel on Compute nodes, full featured Linux on Service nodes.
Service PEs specialize by function
Software Architecture eliminates OS “Jitter”
Software Architecture enables reproducible run times
Compute Partition
Service Partition
Service Partition
3
Service nodes
• Overview
 Run full Linux (SuSe SLES), 2 nodes per service blade
• Boot node
 first XE6 node to be booted: boots all other components
• System Data Base (SDB) node
 hosts MySQL database
 processors, allocation, accounting, PBS information
processors allocation accounting PBS information
• Login nodes
 User login and code preparation activities: compile, launch
User login and code preparation activities: compile, launch
 Partition allocation: ALPS (Application Level Placement Scheduler)
4
Trimming OS – Standard Linux Server
nscd
cron
mingetty(s)
klogd
Portmap
cupsd
qmgr
master
…
sshd
powersaved
pickup
init
slpd
resmgrd
kdm
ndbd
Linux Kernel
5
FTQ Plot of Stock SuSE (most daemons removed)
28350
Count
28150
27950
27750
27550
0
1
2
3
Time - Seconds
6
noise amplification in a parallel program
pe 0
pe 1
pe 2
pe 3
pe 4
synchronization
time
•In each synchronization interval, one rank experiences a noise delay
•Because the ranks synchronize, all ranks experience all of the delays
•In this worst-case situation, the performance degradation due to the
noise is multiplied by the number of ranks
•Noise events that occur infrequently on any one rank (core) occur
frequently in the parallel job
S nchroni ation o
Synchronization
overhead
erhead amplifies the noise
7
Linux on a Diet – CNL
ALPS
client
syslogd
l d
Lustre
Client
klogd
init
Linux Kernel
8
FTQ plot of CNL
28350
Count
28150
27950
27750
27550
0
1
2
3
Time - Seconds
9
Cray XE I/O architecture
• All I/O is offloaded to service nodes
• Lustre
 High performance parallel I/O file system
 Direct data transfer between compute nodes and files
• DVS S
 Virtualization service
 Allows compute nodes to access NFS mounted on service Allows compute nodes to access NFS mounted on service
node
 Applications must execute on file systems mounted on compute nodes
• No local disks
• /tmp
/
i MEMORY fil
is a MEMORY file system, on each login node
hl i
d
10
Scaling Shared Libraries with DVS
Diskless
Compute Node 0
Node 0
/dvs
Diskless
Compute Node 1
Node 1
/dvs
Diskless
Compute Node 2
Node 2
/dvs
Diskless
Compute Node 3
Node 3
/dvs
Diskless
Compute Node N
Node N
/dvs
Cray
C
Interconnect
 Requests for shared libraries (.so files) are routed through DVS
are
routed through DVS Servers
 Provides similar functionality as NFS, but scales to 1000s of compute nodes
 C
Central point of administration for t l i t f d i i t ti f
shared libraries
 DVS Servers can be “re‐purposed” t
d
compute nodes
November 7, 2011
Cray Proprietary
DVS Server
Server Node 0
NFS
Shared
Libraries
11
Slide 11
Running an A li i
Application on the h
Cray XE6
Cray XE6
Running an application on the Cray XE ALPS + aprun
• ALPS : Application Level Placement Scheduler
• aprun is the ALPS application launcher
 It must
It
t be used to run application on the XE compute nodes
b
dt
li ti
th XE
t
d
 If aprun is not used, the application is launched on the login node (and will most likely fail)
(
y )
 aprun man page contains several useful examples
 at least 3 important parameters to control:
 The total number of PEs :
‐n
 The number of PEs per node:
‐N
 The number of OpenMP threads:
‐d
More precise : The ‘stride’ between 2 PEs in a node
13
Job Launch
Login Node
SDB Node
XE6 User
Compute Nodes
14
Job Launch
Login Node
SDB Node
Login &
Start App
qsub
b
B t h
Batch
XE6 User
Compute Nodes
15
Job Launch
Login Node
SDB Node
Login &
Start App
qsub
b
B t h
Batch
apsched
h d
Login Shell
XE6 User
Aprun (c)
apbasil(c)
Compute Nodes
16
Job Launch
Login Node
SDB Node
Login &
Start App
qsub
b
B t h
Batch
Apsched(dm)
Login Shell
XE6 User
aprun
apbasil
Compute Nodes
Apinit (dm)
apshepherd
Application
17
Job Launch
Login Node
SDB Node
Login &
Start App
qsub
b
apsched
h d
B t h
Batch
Login Shell
XE6 User
aprun
apbasil
Compute Nodes
apinit
apshepherd
Application
…
…
18
Job Launch
Login Node
SDB Node
Login &
Start App
qsub
b
apsched
h d
B t h
Batch
Login Shell
XE6 User
aprun
apbasil
Application
Runs on
apinit
compute
t nodes
d
apshepherd
Application
…
…
IO Nodes
s Impleme
ent I/O
Re
equest
Compute Nodes
IO Node
IO
daemons
IO Node
IO
daemons
IO Node
IO
daemons
19
Job Launch
Login Node
SDB Node
Login &
Start App
qsub
b
B t h
Batch
apsched
h d
Login Shell
XE6 User
aprun
apbasil
Compute Nodes
Job is
cleaned
l
d up
Apinit (dm)
20
Job Launch
Login Node
Login &
Start App
qsub
b
B t h
Batch
SDB Node
Nodes returned
apsched
h d
Login Shell
XE6 User
aprun
apbasil
Compute Nodes
Job is
cleaned
l
d up
apinit
21
Job Launch : Done
Login Node
SDB Node
XE6 User
Compute Nodes
22
Some Definitions
• ALPS is always used for scheduling a job on the compute nodes. It does not care about the programming model you used. So we need a few general ‘definitions’ :
 PE : Processing Elements
Basically an Unix ‘Process’, can be a MPI Task, CAF image, UPC tread, …
 Numa_node
Numa node
The cores and memory on a node with ‘flat’ memory access, basically one of the 4 Dies on the Interlagos processor and the direct attach memory.
y
 Thread A thread is contained inside a process. Multiple threads can exist within the same process and share resources such as
p
memory, while y,
different PEs do not share these resources.
Most likely you will use OpenMP threads.
23
Running an application on the Cray XE6
some basic examples
• Assuming a XE6 IL16 system (32 cores per node)
• Pure MPI application, using all the available cores in a node
$ aprun –n 32 ./a.out
• Pure MPI application, using only 1 core per node
 32 MPI tasks, 32 nodes with 32*32 cores allocated
 Can be done to increase the available memory for the MPI tasks
$ aprun –N
N 1 –n 32 –d
d 32 ./a.out
/
t
(we’ll talk about the need for the –d32 later)
• Hybrid MPI/OpenMP application, 4 MPI ranks per node
 32 MPI tasks, 8 OpenMP threads each
k
h d
h
 need to set OMP_NUM_THREADS
$ export
p
OMP_NUM_THREADS=8
$ aprun –n 32 –N 4 –d $OMP_NUM_THREADS
24
aprun CPU Affinity control
• CNL can dynamically distribute work by allowing PEs and threads to migrate from one CPU to another within a node
• In some cases, moving PEs or threads from CPU to CPU increases cache and In some cases moving PEs or threads from CPU to CPU increases cache and
Translation Lookaside Buffer (TLB) misses and therefore reduces performance
• CPU affinity options enable to bind a PE or thread to a particular CPU or a subset of CPUs on a node
subset of CPUs on a node
• aprun CPU affinity option (see man aprun)
 Default settings : ‐cc cpu
PEs are bound a to specific core depended on the d setting
PEs are bound a to specific core, depended on the –d setting
 Binding PEs to a specific numa node : ‐cc numa_node
PEs are not bound to a specific core but cannot ‘leave’ their numa_node
 No binding : ‐cc none
 Own binding : ‐cc 0,4,3,2,1,16,18,31,9,…
25
Memory affinity control
• Cray XE6 systems use dual‐socket compute nodes with 4 dies
 Each die (8 cores) is considered a NUMA‐node
• Remote‐NUMA‐node memory references, can adversely affect performance. Even if you PE and threads are bound to a specific numa node
Even if you PE and threads are bound to a specific numa_node, the memory used does not have to be ‘local’ • aprun memory affinity options (see man aprun)
 Suggested setting is –ss
a PE can only allocate the memory local to its assigned NUMA node If this is not possible your application will crash
node. If this is not possible, your application will crash. 26
Running an application on the Cray XE ‐ MPMD • aprun supports MPMD – Multiple Program Multiple Data
• Launching several executables on the same MPI_COMM_WORLD
$ aprun –n 128 exe1 : -n 64 exe2 : -n 64 exe3
• Notice : Each exacutable needs a dedicated node, exe1 and exe2 cannot share a node.
Example : The following commands needs 3 nodes $ aprun –n 1 exe1 : -n 1 exe2 : -n 1 exe3
• Use a script to start several serial jobs on a node :
$ aprun –a xt –n 3 script.sh
>cat script.sh
./exe1&
./exe2&
./exe3&
wait
>
27
How to use the interlagos 1/3
1 MPI Rank on Each Integer Core Mode
MPI Task 0
November 7, 2011
Shared
Components
MPI Task 1
Fetch
Decode
FP
Scheduler
L1 DCache
Cray Proprietary
Pipeline
Pipeline
Pipeline
128--bit FMA
128
AC
Int Core 1
128--bit FMA
128
AC
Pipeline
Pipeline
Int Core 0
Int
Scheduler
Pipeline
Int
Scheduler
Pipeline
each integer core
● Implications
● Each core has exclusive access to an Each core has exclusive access to an
integer scheduler, integer pipelines and L1 Dcache
● The 256‐bit FP unit and the L2 Cache is shared between the two cores
● 256‐bit AVX instructions are dynamically executed as two 128‐bit instructions if the 2nd FP unit is busy
instructions if the 2
FP unit is busy
● When to use
● Code is highly scalable to a large number of MPI ranks
● Code can run with 1 GB per core memory footprint (or 2 GB on 64 GB node)
● Code is not well vectorized
Pipeline
● In this mode, an MPI task is pinned to L1 DCache
Shared L2 Cache
28
Slide 28
How to use the interlagos 2/3 Wide AVX mode
November 7, 2011
Active
Components
Fetch
Decode
FP
Scheduler
L1 DCache
Cray Proprietary
Pipeline
Pipeline
Pipeline
128--bit FMA
128
AC
Int Core 1
128--bit FMA
128
AC
Pipeline
Pipeline
Int Core 0
Int
Scheduler
Pipeline
Int
Scheduler
Pipeline
used per core pair
● Implications
● This core has exclusive
This core has exclusive access to access to
the 256‐bit FP unit and is capable of 8 FP results per clock cycle
● The core has twice the memory The core has twice the memory
capacity and memory bandwidth in this mode
● The L2 cache is effectively twice as large
● The peak of the chip is not reduced
● When to use
● Code is highly vectorized and makes use of AVX instructions
● Code needs more memory per MPI rankk
Idle
Components
Pipeline
thi
d
l
i t
i
● IIn this mode, only one integer core is L1 DCache
Shared L2 Cache
29
Slide 29
How to use the interlagos 3/3 2‐way OpenMP Mode
November 7, 2011
Cray Proprietary
Shared
Components
OpenMP
Thread 1
Fetch
Decode
FP
Scheduler
L1 DCache
Pipeline
Pipeline
Pipeline
128--bit FMA
128
AC
Int Core 1
128--bit FMA
128
AC
Pipeline
Pipeline
Int Core 0
Int
Scheduler
S
h d l
Pipeline
Int
Scheduler
S
h d l
Pipeline
core pair
● OpenMP is used to run a thread on each i t
integer core
● Implications
● Each OpenMP thread has exclusive access to an integer scheduler integer
access to an integer scheduler, integer pipelines and L1 Dcache
● The 256‐bit FP unit and the L2 Cache is shared between the two threads
● 256‐bit AVX instructions are dynamically executed as two 128‐bit instructions if the 2nd FP unit is busy
● When to use
When to use
● Code needs a large amount of memory per MPI rank
p
p
p
● Code has OpenMP
parallelism exposed in each MPI rank
OpenMP
Thread 0
Pipeline
● In this mode, an MPI task is pinned to a L1 DCache
Shared L2 Cache
30
Slide 30
Aprun: cpu_lists for each PE • CLE was updated to allow threads and processing elements to have more flexibility in placement. This is ideal for processor architectures whose cores share resources with which they hit t
h
h
ith hi h th
may have to wait to utilize. Separating cpu_lists by colons (:) allows the user to specify the cores used by processing elements and their child processes or threads.
Essentially, this provides the user more granularity to specify
Essentially, this provides the user more granularity to specify cpu_lists for each processing element.
Here an example with 3 threads :
aprun n 4 N 4 cc 1 3 5 7 9 11 13 15 17 19 21 23
aprun ‐n 4 ‐N 4 ‐cc 1,3,5:7,9,11:13,15,17:19,21,23
• Note: This feature will be modified in CLE 4.0.UP03, however p
this option will still be valid.
31
‐cc detailed example
export OMP_NUM_THREADS=2
aprun -n 4 -N 2 -d 2 -cc 0,2:4,6:8,10:12,14 ./acheck_mpi
nid00028[ 0] on cpu 00 affinity for thread 0 is: cpu
nid00028[ 0] on cpu 02 affinity for thread 1 is: cpu
nid00028[ 1] on cpu 04 affinity for thread 0 is: cpu
nid00028[ 1] on cpu 06 affinity for thread 1 is: cpu
nid00572[ 2] on cpu 00 affinity for thread 0 is: cpu
nid00572[ 2] on cpu 02 affinity for thread 1 is: cpu
nid00572[ 3] on cpu 04 affinity for thread 0 is: cpu
nid00572[ 3] on cpu 06 affinity for thread 1 is: cpu
Application 3024546 resources: utime ~2s, stime ~0s
aprun -n 4 -N 4 -d 2 -cc 0,2:4,6:8,10:12,14 ./acheck_mpi
nid00028[ 0] on cpu 00 affinity for
nid00028[ 2] on cpu 08 affinity for
nid00028[ 0] on cpu 02 affinity for
nid00028[
id00028[ 1] on cpu 06 affinity
ffi it for
f
nid00028[ 3] on cpu 12 affinity for
nid00028[ 2] on cpu 10 affinity for
nid00028[ 1] on cpu 04 affinity for
p 14 affinity
y for
nid00028[ 3] on cpu
Application 3024549 resources: utime
thread 0 is: cpu
thread 0 is: cpu
thread 1 is: cpu
thread
th
d 1 is:
i
cpu
thread 0 is: cpu
thread 1 is: cpu
thread 0 is: cpu
thread 1 is: cpu
p
~0s, stime ~0s
0,
2,
4,
6,
0,
2
2,
4,
6,
mask
mask
mask
mask
mask
mask
mask
mask
1
001
00001
0000001
1
001
00001
0000001
0,
8,
2,
6
6,
12,
10,
4,
14,
mask
mask
mask
mask
k
mask
mask
mask
mask
1
000000001
001
0000001
0000000000001
00000000001
00001
000000000000001
32
Running a batch application with Torque
• The number of required nodes and cores is determined by the parameters specified in the job header
#PBS -l
l mppwidth=256
idth 256 (MPP width:
idth number
b
of
f PE’
PE’s )
#PBS -l mppnppn=4
(MPP number of PE’s per node)
•
•
•
•
This example uses 256/4=64 nodes
p
/
The job is submitted by the qsub command
At the end of the execution output and error files are returned to submission directory
PBS environment variable: $PBS_O_WORKDIR
Set to the directory from which the job has been submitted
Set to the directory from which the job has been submitted
Default is $HOME
man qsub for env. variables
33
Other Torque options
• #PBS ‐N job_name
the job name is used to determine the name of job output and error files
• #PBS ‐l walltime=hh:mm:ss
Maximum job elapsed time
Maximum job elapsed time
should be indicated whenever possible: this allows Torque to determine best scheduling startegy
g
gy
• #PBS ‐j oe
job error and output files are merged in a single file
• #PBS ‐q queue
request execution on a specific queue
34
Torque and aprun
Torque
aprun
‐lmppwidth=$PE
‐n $PE
Number of PE to start
‐lmppdepth=$threads
lmppdepth=$threads
‐d $threads
d $threads
#threads/PE
‐lmppnppn=$N
‐N $N
#(PEs per node)
<none>
‐S $S
#(PEs per numa_node)
‐lmem=$size
‐m $size[h|hs]
per‐PE required memory
• -B will provide aprun with the Torque settings for –n,-N,-d and –m
aprun –B ./a.out
/a out
• Using –S can produce problems if you are not asking for a full node.
If possible, ALPS will only give you access to a parts of a node if the Torque
settings allows this. The following will fail :
• PBS -lmppwidth=4 ! Not asking for a full node
• aprun –n4 –S1 …
! Trying to run on every die
doesn t use it
• Solution is to ask for a full node, even if aprun doesn‘t
35
Core specialization
• System ‘noise’ on compute nodes may significantly degrade scalability for some applications
• Core Specialization can mitigate this problem
 1 core per node will be dedicated for system work (service core)
 As many system interrupts as possible will be forced to execute
As man s stem interr pts as possible ill be forced to e ec te
on the service core
 The application will not run on the service core
pp
• Use aprun ‐r to get core specialization
$ aprun –r –n 100 a.out
• apcount provided to compute total number of cores required
$ qsub -l mppwidth=$(apcount -r 1 1024 16)job
aprun -n 1024 -r 1 a.out
36
Running a batch application with Torque
• The number of required nodes can be specified in the job header
• The job is submitted by the qsub command
• At the end of the exection output and error files are returned to submission directory
• Environment variables are En ironment ariables are
inherited by #PBS ‐V
• The job starts in the home j
directory. $PBS_O_WORKDIR
contains the directory from which the job has been submitted
the job has been submitted
Hybrid MPI + OpenMP
#!/bin/bash
#PBS –N hybrid
#PBS –lwalltime=00:10:00
lwalltime=00:10:00
#PBS –lmppwidth=128
#PBS –lmppnppn=8
#PBS –lmppdepth=4
lmppdepth=4
cd $PBS_O_WORKDIR
export
p
OMP_NUM_THREADS=4
aprun –n128 –d4 –N8 a.out
37
Starting an interactive session with Torque
• An interactive job can be started by the –I argument
 That is <capital‐i>
• Example: allocate 64 cores and export the environment variables to the job ( V)
the job (‐V)
$ qsub –I –V –lmppwith=64
• This will give you a new prompt in your shell from which you can Thi ill i
i
h ll f
hi h
use aprun directly. Note that you are running on a MOM node (shared resource) if not using aprun
38
Watching a launched job on the Cray XE
• xtnodestat
 Shows XE nodes allocation and aprun
Shows XE nodes allocation and aprun processes
 Both interactive and PBS
• apstat
 Shows aprun processes status
 apstat
overview
 apstat –a[ apid ]info about all the applications or a specific one
 apstat –n
info about the status of the nodes
• Batch qstat
B t h t t command
d
 shows batch jobs
39
Starting 512 MPI tasks (PEs)
#PBS
#PBS
#PBS
#PBS
#PBS
-N
N
-l
-l
-l
l
-j
MPIjob
MPIj
b
mppwidth=512
mppnppn=32
walltime=01:00:00
llti
01 00 00
oe
cd $PBS_O_WORKDIR
_ _
export MPICH_ENV_DISPLAY=1
export MALLOC_MMAP_MAX_=0
export MALLOC_TRIM_THRESHOLD_=536870912
aprun
-n 512 –cc cpu –ss ./a.out
40
Starting an OpenMP program, using a single node
#PBS
#PBS
#PBS
#PBS
#PBS
-N
-l
-l
l
-l
-j
OpenMP
mppwidth=1
mppdepth=32
walltime=01:00:00
oe
cd $PBS_O_WORKDIR
export MPICH_ENV_DISPLAY=1
export MALLOC_MMAP_MAX_=0
export MALLOC_TRIM_THRESHOLD_=536870912
export OMP
OMP_NUM_THREADS=32
NUM THREADS=32
aprun –n1 –d $OMP_NUM_THREADS –cc cpu –ss ./a.out
41
Starting a hybrid job single node, 4 MPI tasks, each with 8 threads
single node, 4 MPI tasks, each with 8 threads
#PBS
#PBS
#PBS
#PBS
#PBS
#PBS
-N
-l
l
-l
-l
-l
l
-j
hybrid
mppwidth=4
mppwidth
4
mppnppn=4
mppdepth=8
walltime=01:00:00
walltime
01:00:00
oe
cd $PBS_O_WORKDIR
export MPICH_ENV_DISPLAY=1
export MALLOC_MMAP_MAX_=0
export MALLOC_TRIM_THRESHOLD_=536870912
export OMP_NUM_THREADS=8
aprun –n4
4 –N4
N4 –d
d $OMP
$OMP_NUM_THREADS
NUM THREADS –cc cpu –ss ./a.out
/
t
42
Starting a hybrid job single node, 8 MPI tasks, each with 4 threads
single node, 8 MPI tasks, each with 4 threads
#PBS
#PBS
#PBS
#PBS
#PBS
#PBS
-N
-l
l
-l
-l
-l
l
-j
hybrid
mppwidth=8
mppwidth
8
mppnppn=8
mppdepth=4
walltime=01:00:00
walltime
01:00:00
oe
cd $PBS_O_WORKDIR
export MPICH_ENV_DISPLAY=1
export MALLOC_MMAP_MAX_=0
export MALLOC_TRIM_THRESHOLD_=536870912
export OMP_NUM_THREADS=4
aprun –n8
8 –N8
N8 –d
d $OMP
$OMP_NUM_THREADS
NUM THREADS –cc cpu –ss ./a.out
/
t
43
Starting a MPMD job on a non‐default projectid
using 1 master, 16 slaves, each with 8 threads
#PBS
#PBS
#PBS
#PBS
#PBS
#PBS
-N
-l
-l
-l
-j
-W
hybrid
mppwidth=160 ! Note : 5 nodes * 32 cores = 160 cores
mppnppn=32
walltime=01:00:00
oe
group_list=My_Project
_
_
cd $PBS_O_WORKDIR
export MPICH_ENV_DISPLAY=1
export MALLOC_MMAP_MAX_=0
export MALLOC_TRIM_THRESHOLD_=536870912
export OMP_NUM_THREADS=8
id # Unix command ‚id‘, to check group id
aprun –n1
n1 –d32
d32 –N1
N1 ./master.exe
/master exe :
-n 16 –N4 –d $OMP_NUM_THREADS –cc cpu –ss ./slave.exe
44
Starting an MPI job on two nodes
using only every second integer core
using only every second integer core
#PBS
#PBS
#PBS
#PBS
#PBS
#PBS
-N
-l
l
-l
-l
-l
l
-j
hybrid
mppwidth=32
mppwidth
32
mppnppn=16
mppdepth=2
walltime=01:00:00
walltime
01:00:00
oe
cd $PBS_O_WORKDIR
export MPICH_ENV_DISPLAY=1
aprun –n32 –N16 –d 2 –cc cpu –ss ./a.out
45
Starting a hybrid job on two nodes
using only every second integer core
using only every second integer core
#PBS
#PBS
#PBS
#PBS
#PBS
#PBS
-N
-l
l
-l
-l
-l
l
-j
hybrid
mppwidth=32
mppwidth
32
mppnppn=16
mppdepth=2
walltime=01:00:00
walltime
01:00:00
oe
cd $PBS_O_WORKDIR
export MPICH_ENV_DISPLAY=1
export OMP_NUM_THREADS=2
aprun –n32 –N16 –d $OMP_NUM_THREADS
–cc 0,2:4,6:8,10:12,14:16,18:20,22:24,26:28,30 –ss ./a.out
46
Running a batch application with SLURM
• The number of required nodes can be specified in the job header
• The job is submitted by the qsub command
• At the end of the exection output and error files are returned to submission directory
• Environment variables are En ironment ariables are
inherited
• The job starts in the directory j
y
from which the job has been submitted
Hybrid MPI + OpenMP
#!/bin/bash
#SBATCH –-job-name=“hybrid”
#SBATCH –-time=00:10:00
#SBATCH –-nodes=8
export OMP_NUM_THREADS
OMP NUM THREADS=6
6
aprun –n32 –d6 a.out
47
Starting an interactive session with SLURM
• An interactive job can be started by the SLURM salloc command
• Example: allocate 8 nodes
$ salloc –N 8
Further SLURM info available from CSCS web page: www.cscs.ch
User Entry Point / How to Run a Batch Job / Palu ‐ Cray XE6 48
Documentation
• Cray docs site
http://docs.cray.com
• Starting point for Cray XE info
http://docs.cray.com/cgi‐bin/craydoc.cgi?mode=SiteMap;f=xe_sitemap
//
/
/
• Twitter ?!?
Twitter ?!?
http://twitter.com/craydocs
49
End
50