CpuMemSets, cmsQueryCMM, cmsSetCMM, cmsQueryCMS, cms
SetCMS, cmsGetCpu, cpu2node, runon, cpubind, nodebind,
numnodes, numcpus, lubcpunum, cmsFreeCMM, cmsFreeCMS -
Manage system CPU scheduling and memory allocation.
SYNOPSIS
#include <cpumemsets.h>
cpumemmap_t *cmsQueryCMM (cms_choice_t c, pid_t pid, void *start);
int cmsSetCMM (cms_choice_t c, pid_t pid, void *start, size_t len, cpumemmap_t *cmm);
cpumemset_t *cmsQueryCMS (cms_choice_t c, pid_t pid, void *start);
int cmsSetCMS (cms_choice_t c, pid_t pid, void *start, size_t len, cpumemset_t *cms);
cms_acpu_t cmsGetCpu(void);
cms_smem_t cpu2node(cms_scpu_t cpu);
int runon (cms_acpu_t cpu);
int cpubind (cms_scpu_t cpu);
int nodebind (cms_smem_t node);
int numnodes();
int numcpus();
cms_scpu_t lubcpunum();
void cmsFreeCMM (cpumemmap_t *cmm);
void cmsFreeCMS (cpumemset_t *cms);
DESCRIPTION
The CpuMemSets interface provides general purpose proces
sor and memory placement facilities to applications, sys
tem services and emulations of other CPU and memory place
ment interfaces.
CpuMemSets are implemented using two separate layers.
1) cpumemmap (cmm):
The bottom layer provides a simple pair of maps,
mapping system CPU and memory block numbers to
application CPU and memory block numbers. System
numbers are those used by the kernel task schedul
ing and memory allocation code, and typically
include all CPUs and memory in the system. Appli
cation numbers are those used by an application in
its cpumemset to specify its CPU and memory affin
ity for those CPU and memory blocks available in
its map.
Each process, each virtual memory area, and the
kernel has such a map. These maps are inherited
across fork(2), exec and the various ways to create
vm areas. Only a process with root privileges can
extend cpumemmaps to include additional system CPUs
or memory blocks. Changing a map will cause kernel
scheduling code to immediately start using the new
system CPUs, and cause kernel allocation code to
allocate additional memory pages using the new sys
tem memory blocks, but memory already allocated on
other means.
The bulk of the kernel is still written using what
ever system CPU and memory block numbers are appro
priate for a system. Changes to cpumemmaps are
converted at the time of the cmsSet*() calls into
changes to the system masks (such as cpus_allowed)
and lists (such as zone lists) used by the existing
scheduler and allocator.
2) cpumemset (cms):
The upper layer specifies on which of the applica
tion CPUs known to that process a task can be
scheduled, and in which application memory blocks
known to the kernel or that vm area, memory can be
allocated. The kernel allocators search the memory
block lists in the given order for available mem
ory, and a different list is specified for each CPU
that may execute the request. An application may
change the cpumemset of its tasks and vm areas, and
root may change the cpumemset used for kernel mem
ory allocation. Also root may change the cpumem
sets of any process, and any process may change the
cpumemsets of other processes with the same uid
(kill (2) permissions).
Each task has two cpumemsets, one defining its current CPU
allocation and created vm areas, and one that is inherited
by any child process it forks. Both the current and child
cpumemsets of a newly forked process are set to copies of
the child cpumemset of the parent process. Allocations of
memory to existing vm areas visible to a process depend on
the cpumemset of that vm area (as acquired from its creat
ing process at creation, and possibly modified since), not
on the cpumemset of the currently accessing task.
During system boot, the kernel creates and attaches a
default cpumemmap and cpumemset that is used everywhere.
By default this initial map and set contain all CPUs and
all memory blocks. The memory blocks are not necessarily
sorted in any particular order. An optional kernel boot
command line parameter causes this initial cpumemmap and
cpumemset to contain only the first CPU and one memory
block, rather than all of them, for the convenience of
system management services that wish to take greater con
trol of the system.
The kernel will only schedule a task on the CPUs in the
tasks cpumemset, and only allocate memory to a user vir
tual memory area from the list of memories in that areas
memory list. The kernel allocates kernel memory only from
the list of memories in the cpumemset attached to the CPU
executing the allocation request, except for specific
ory block.
Both the current and child cpumemmaps and cpumemsets of a
newly forked process are taken from the child settings of
its parent, and memory allocated during the creation of
the new process is allocated according to the parents
child cpumemset and associated cpumemmap, because that
cpumemset is acquired by the new process and then by any
vm area created by that process.
The cpumemset (and associated cpumemmap) of a newly cre
ated virtual memory area is taken from the current cpumem
set of the task creating it. In the case of attaching to
an existing vm area, things get more complicated. Both
mmap'd memory objects and System V shared memory regions
can be attached to by multiple processes, or even attached
to multiple times by the same process at different
addresses. If such an existing memory region is attached
to, then by default the new vm area describing that
attachment inherits the current cpumemset of the attaching
process. If however the policy flag CMS_SHARE is set in
the cpumemset currently linked to from each vm area for
that region, then the new vm area will also be linked to
this same cpumemset.
When allocating another page to an area, the kernel will
choose the memory list for the CPU on which the current
task is being executed, if that CPU is in the cpumemset of
that memory area, else it will choose the memory list for
the default CPU (see CMS_DEFAULT_CPU) in that memory areas
cpumemset. The kernel then searches the chosen memory
list in order, from the beginning of that memory list,
looking for available memory. Typical kernel allocators
search the same list multiple times, with increasingly
aggressive search criteria and memory freeing actions.
The cpumemmap and cpumemset calls with the CMS_VMAREA
apply to all future allocation of memory by any existing
vm area, for any pages in the range [start,start+len) ,
similar to the behavior of madvise(2), mincore(2) and
msync(2).
The cpu2node call returns which node the specified cpu is
on, or else NON_NODE_NUM.
The runon call causes the current process and any children
it subsequently forks to run on the application CPU speci
fied in the call. This call does not change the current
cpumemmap, and does not allow access to any CPUs not
already in that map, nor remove permission to runon
another of these mapped CPUs at some later time. This
call is a wrapper for several of the other calls.
dren it subsequently forks to be bound to the system CPU
specified in the call. If the specified system CPU is not
in the current cpumemmap, then it is added (requires root
privileges). No CPUs already in the current cpumemmap are
removed. This call is a wrapper for several of the other
calls.
The nodebind() call causes the current process and any
children it subsequently forks to be bound to the cpus and
memory on the system node (memory block) specified in the
call. If any of these cpus or memory nodes are not in the
current cpumemmap, then it is added (requires root privi
leges). No CPUs already in the current cpumemmap are
removed. This call is a wrapper for several of the other
calls.
The numnodes() call returns the number of nodes (memory
blocks) on the current system, or -1 on error. This call
depends on the presence of a valid /var/cpuset/cpu-node-
map file.
The numcpus() call returns the number of CPUs on the cur
rent system, or -1 on error. This call depends on the
presence of a valid /var/cpuset/cpu-node-map file.
The lubcpunum() call returns the least upper bound of all
CPU numbers on the current system, or 0 on error. This
call is useful for sizing arrays that are to be indexed by
CPU number. This call depends on the presence of a valid
/var/cpuset/cpu-node-map file.
CONSTANTS AND TYPEDEFS
#define CMS_DEFAULT 0x01 /* Memory list order */
#define CMS_SHARE 0x04 /* Share virtual memory area CMS */
typedef int cms_setpol_t; /* Type of policy argument for sets */
/* 16 bits gets us 64K CPUs ... that should be enough! */
typedef uint16_t cms_acpu_t; /* Type app CPU number */
typedef uint16_t cms_amem_t; /* Type app memory block number */
typedef uint16_t cms_scpu_t; /* Type sys CPU number */
typedef uint16_t cms_smem_t; /* Type sys memory block number */
#define CMS_DEFAULT_CPU ((cms_acpu_t)-1) /* Default Memory List */
#define NON_NODE_NUM ((cms_smem_t)-1) /* Invalid Node Number */
/* Calls to query and set cmm and cms specify which one ... */
#define CMS_CURRENT 0 /* task's own cmm/cms */
#define CMS_CHILD 1 /* task's children's cmm/cms */
#define CMS_VMAREA 2 /* vm area's cmm/cms */
#define CMS_KERNEL 3 /* kernel's cmm/cms */
typedef int cms_choice_t; /* Type cmm/cms choice argument */
/* cpumemmap: Type for the pair of maps ... */
typedef struct cpumemmap {
int nr_cpus; /* number of CPUs in map */
cms_scpu_t *cpus; /* map app to sys CPU num */
int nr_mems; /* number of mems in map */
cms_smem_t *mems; /* map app to sys mem num */
} cpumemmap_t;
/*
* How memory looks to (typically) a set of equivalent
* CPUs, including which memory blocks to search for
* memory, in what order, and the list of CPUs to which
* this list of memory blocks applies. The cpumemset
* is sufficiently complex that this portion of the data
* structure type is specified separately, then an array
* of cms_memory_list_t structures is included in the
* main cpumemset type.
*/
typedef struct cms_memory_list {
int nr_cpus; /* Num CPUs sharing memory list */
cms_acpu_t *cpus; /* CPUs sharing memory list */
int nr_mems; /* Num memory blocks in list */
cms_amem_t *mems; /* Memory blocks in list */
} cms_memory_list_t;
/*
* Specify a single cpumemset, describing on which CPUs to
* schedule tasks, from which memory blocks to allocate memory,
* and in what order to search these memory blocks.
*/
typedef struct cpumemset {
cms_setpol_t policy; /* CMS_* set policy flags */
int nr_cpus; /* Num CPUs in this cpumemset */
cms_acpu_t *cpus; /* CPUs in this cpumemset */
int nr_mems; /* Num memory lists in cpumemset */
cms_memory_list_t *mems;/* Memory lists in cpumemset */
} cpumemset_t;
/* cpumemmap: Type for the pair of maps ... */
typedef struct cpumemmap {
int nr_cpus; /* number of CPUs in map */
cms_scpu_t *cpus; /* array maps application to system CPU num */
int nr_mems; /* number of mems in map */
cms_smem_t *mems; /* array maps application to system mem num */
} cpumemmap_t;
ERRORS
EINVAL Every cpumemset must specify a memory list for
the CMS_DEFAULT_CPU, to ensure that regardless of
which CPU a memory request is executed on, a
ory. Attempts to set a cpumemset without a mem
ory list specified for the CMS_DEFAULT_CPU will
fail, with errno set to EINVAL.
ENOENT If a request is made to set a cpumemmap that has
fewer CPUs or memory blocks listed than needed by
any cpumemsets that will be using that cpumemmap
after the change, then that cmsSetCMM() will
fail, with errno set to ENOENT. That is, you
cannot remove elements of a cpumemmap that are in
use.
ENOENT If a request is made to set a cpumemset that ref
erences CPU or memory blocks not available in its
current cpumemmap, then that cmsSetCMS() will
fail, with errno set to ENOENT. That is, you
cannot reference unmapped application CPUs or
memory blocks in a cpumemset.
EPERM If a request is made to set a cpumemmap by a pro
cess without root privileges, and that request
attempts to add any system CPU or memory block
number not currently in the map being changed,
then that request will fail, with errno set to
EPERM.
EPERM If a cmsSetCMS() request is made on another pro
cess, then the requesting process must either
have root privileges, or the real or effective
user ID of the sending process must equal
the real or saved set-user-ID of the other pro
cess, or else the request will fail, with errno
set to EPERM. These permissions are similar to
those required by the kill(2) system call.
EINVAL If a request is made to set a cpumemset that has
the same CPU (application number) listed in more
than one array "cpus" of CPUs sharing any
cms_memory_list_t, then the request will fail,
with errno set to EINVAL. Otherwise, duplicate
CPU or memory block numbers are harmless, except
for minor inefficiencies.
ESRCH The operations to query and set cpumemmaps and
cpumemsets can be applied to any process (any
pid). If the pid is zero, then the operation is
applied to the current process. If the specified
pid does not exist, then the operation will fail
with errno set to ESRCH.
EPERM If the cpubind(cpu) routine is called for a sys
tem cpu not in the current cpumemmap, and the
current process lacks the root privileges
will fail (return -1) with errno set to EPERM.
ENOSYS Kernel does not support CpuMemSets.
ENOMEM CpuMemSets library unable to allocate memory.
EPERM Request to cpubind() to a cpu that was not in the
process's current cpumemmap, and process lacked
permission to add the cpu, probably because pro
cess lacked root privileges.
EINVAL An application cpu number in a requested cpumem
set is neither CMS_DEFAULT_CPU nor defined by the
associated cpumemmap (not in the range of zero to
nr_cpus-1)
EINVAL An application memory number in a requested
cpumemset is not defined by the associated
cpumemmap (not in the range of zero to
nr_mems-1).
EINVAL The cms_choice_t parameter of a cmsQueryCMM(),
cmsSetCMM(), cmsQueryCMS() or cmsSetCMS() call is
not one of the allowed values.
EINVAL A call to one of cmsQueryCMM(), cmsSetCMM(), cms
QueryCMS() or cmsSetCMS() is made with
cms_choice_t of CMS_VMAREA, and a "start" address
that is larger than any valid address in the vir
tual address space of the affected process.
EFAULT A directly or indirectly referenced parameter is
outside the invoking process's accessible address
space.
EBADF One of the routines nodebind(), cpu2node(),
numnodes(), numcpus(), or lubcpunum() was invoked
without a valid configuration file
/var/cpuset/cpu-node-map. The system initializa
tion script /etc/init xxx creates cpu-node-map
each system boot.
EINVAL The 'cpu' requested of cpubind() or cpu2node()
was not listed in the configuration file
/var/cpuset/cpu-node-map.
EINVAL The configuration file /var/cpuset/cpu-node-map
has no valid entries, or has two inconsistent
entries mapping one cpu to two distinct nodes.
ENOENT The 'node' requested of nodebind() was not listed
in the configuration file /var/cpuset/cpu-node-
map.
/var/cpuset/cpu-node-map. Perhaps cpumemsets is
not properly installed, or 'cpunodemap' is not
chkconfig'd on.
ERANGE Internal range check error in nodebind().
EXAMPLES
This example shows how to query the currently allowed
cpus.
/*
* sample1 - display current cpumemset cpus
*
* Compile with:
* cc sample1.c -o sample1 -lcpumemsets
* Displays on stdout the number and a list of the cpus
* on which the current process is allowed to execute.
*/
#include "cpumemsets.h"
main()
{
int i;
cpumemset_t *pset;
pset = cmsQueryCMS(CMS_CURRENT, (pid_t)0, (void *)0);
if (pset == (cpumemset_t *)0) {
perror("cmsQueryCMS");
exit (1);
}
printf("Current CpuMemSet has %d cpu(s):\n\t", pset->nr_cpus);
for (i = 0; i < pset->nr_cpus; i++)
printf("%s%d", (i > 0 ? ", " : ""), pset->cpus[i]);
printf("0);
exit(0);
}
This example shows how to run a child process (/bin/sh,
here) only on cpu 0.
/*
* sample2 - change child cpumemset cpus to just cpu 0
*
* Compile with:
* cc sample2.c -o sample2 -lcpumemsets
* Change the cpus which the child task is allowed
* execute on to just cpu 0. Start a subshell,
* instead of just exiting, so that the user has
* the opportunity to verify that the change occurred.
*/
#include <malloc.h>
main()
{
int i;
cpumemset_t *pset;
pset = cmsQueryCMS(CMS_CHILD, (pid_t)0, (void *)0);
if (pset == (cpumemset_t *)0) {
perror("cmsQueryCMS");
exit (1);
}
pset->nr_cpus = 1;
free(pset->cpus);
pset->cpus = (cms_acpu_t *) malloc(sizeof(cms_acpu_t));
pset->cpus[0] = (cms_acpu_t)0;
if (cmsSetCMS(
CMS_CHILD, (pid_t)0, (void *)0, (size_t)0, pset) < 0) {
perror ("cmsSetCMS");
exit(1);
}
cmsFreeCMS (pset);
printf ("Invoking subshell running on cpu 0.0);
execl("/bin/sh", "sh", 0);
exit (2);
}
Try running sample1 from the subshell started by sample2.
You should see that sample1 reports that it is only
allowed to run on cpu 0:
# PS1='Sub> ' ./sample2
Invoking subshell running on cpu 0.
Sub> ./sample1
Current CpuMemSet has 1 cpu(s):
0
Sub> exit
SEE ALSO
runon(8), madvise(2), mincore(2), msync(2).
Man(1) output converted with
man2html