NAME
dplace - a tool for controlling placement of processes
onto cpus
SYNOPSIS
dplace [-c cpu_numbers] [-s skip_count] [-n process_name] [-x skip_mask] [-p placement_file] command [command-args]
dplace -q
DESCRIPTION
The given program is executed after scheduling and memory
placement policies are set up according to command line
arguments,
By default, memory is allocated to a process on the node
that the process is executing on. If a process moves from
node to node while it running, a higher percentage of mem
ory references will be to remote nodes. Remote accesses
typically have higher access times. Process performance
may suffer.
Dplace is used to bind a related set of processes to spe
cific cpus or nodes to prevent process migrations. In some
cases, this will improve performance since a higher per
centage of memory accesses will to the local node.
Processes alway execute within a cpumemset. The cpumemset
specifies the cpus that are available for a process to
execute on. By default, processes usually execute in a
cpumemset that contains all the cpus in the system.
Dplace invokes a kernel hook (PAGG) to create a placement
container consisting of all (or a subset of) the cpus of
the cpumemset. The dplace process is placed in this con
tainer & (by default) is bound to the first cpu of the
cpumemset associated with the container. Then dplace
"execs" the <command>.
The command is executing within this placement container &
continues to be bound to the first cpu of the container.
As the command forks child processes, they inherit the
container & are bound to the next available cpu of the
container.
dplace maintains a global count of the number of active
processes that have been placed (by dplace) on each cpu.
If no placement file is specified, dplace will bind a pro
cess to the cpu that has the lowest number of active pro
cesses & is also in the users cpu list. For example, if
the current cpumemset consists of physical cpus 2, 5, 8,
and 9, and the user types:
mpirun -np 2 dplace -s1 app2
app1 will run on cpus 2 & 5, and app2 will run on cpus 8 &
9.
OPTIONS
-c Cpu_number(s). Specified as a list of cpu ranges:
Example: "-c1", "-c2-4", "-c1,4-8,3". Cpu numbers
are NOT physical cpu numbers. They are logical cpu
number that are relative to the cpus that are in
the set of allowed cpus as specified by the current
cpumemset or "runon" command. Cpu numbers start at
0. If this option is not specified, all cpus of the
current cpumemset are available. Note that a previ
ous "runon" command may be used to restrict the
available cpus.
-s Skip the first <skip_count> processes before start
ing to place processes onto cpus. This option is
useful if the first <skip_count> processes are
"shepherd" processes that are used only for launch
ing the application. If <skip_count> is not speci
fied, a default value of 0 is used.
-n Only processes named <process_name> are placed.
Other processes are ignored and are not explicitly
bound to cpus. Note: process_name is the basename
of the executible.
-x Provides the ability to skip placement of pro
cesses. <skip_mask> is a bitmask. If bit N of
<skip_mask> is set, then the N+1th process that is
forked is not placed. For example, setting the mask
to 6 will cause the 2nd & 3rd processes from being
placed. The first process (the process named by the
<command>) will be assigned to the first cpu. The
second & third processes are not placed. The fourth
process is assigned to the second cpu, etc.. ohis
option is useful for certain classes of threaded
apps that spawn a few helper processes that are not
typically do not use much cpu time. (Hint: Intel
OpenMP applications currently should be placed
using -x6. This could change in future versions of
OpenMP).
directives that are used to control process place
ment. [TBD]
-q List the global count of the number of active pro
cesses that have been placed (by dplace) on each
cpu in the current cpuset. Note that cpu numbers
are logical cpu numbers within the cpuset, NOT
physical cpu numbers.
EXAMPLES
The following examples assume the command is executed from
a shell running in a cpumemset consisting of physical cpus
8-15.
To execute a process on a specific logical cpu(s):
dplace -c2 date # date runs on physical cpu 10.
dplace make linux # gcc & related processes run on
# physical cpus 8-15.
dplace -c0-4,6 make linux # gcc & related processes run on
# physical cpus 8-12 or 14.
runon 4-7 dplace app # runon restricts execution to
# physical cpus 12-15. dplace
# binds processes to cpus 12-15.
MPI EXAMPLE
Most MPI applications are launched by mpirun and use N+1
threads. The first thread is mainly inactive and usually
does not need to be placed.
To launch an MPI application use the following syntax:
mpirun -np <process_count> dplace [dplace_args] app [args]
Example:
mpirun -np 8 dplace -s1 lu.8
BUGS
The placement file is not implemented yet. The
<skip_mask> is a kludge. A better solution is needed.
Dplace depends on a loadable kernel module named "numa
tools". If this modules is not loaded, dplace will fail &
print a message remind the user to load the numatools mod
ule.
SEE ALSO
cpumemsets(1), dlook(1), runon(1)
Man(1) output converted with
man2html