CpuMemSets,   cmsQueryCMM,  cmsSetCMM,  cmsQueryCMS,  cms­
       SetCMS, cmsGetCpu,  cpu2node,  runon,  cpubind,  nodebind,
       numnodes,  numcpus,  lubcpunum,  cmsFreeCMM,  cmsFreeCMS -
       Manage system CPU scheduling and memory allocation.


SYNOPSIS

       #include <cpumemsets.h>

       cpumemmap_t *cmsQueryCMM (cms_choice_t c, pid_t pid, void *start);
       int cmsSetCMM (cms_choice_t c, pid_t pid, void *start, size_t len, cpumemmap_t *cmm);
       cpumemset_t *cmsQueryCMS (cms_choice_t c, pid_t pid, void *start);
       int cmsSetCMS (cms_choice_t c, pid_t  pid, void *start, size_t len, cpumemset_t *cms);
       cms_acpu_t cmsGetCpu(void);
       cms_smem_t cpu2node(cms_scpu_t cpu);
       int runon (cms_acpu_t cpu);
       int cpubind (cms_scpu_t cpu);
       int nodebind (cms_smem_t node);
       int numnodes();
       int numcpus();
       cms_scpu_t lubcpunum();
       void cmsFreeCMM (cpumemmap_t *cmm);
       void cmsFreeCMS (cpumemset_t *cms);


DESCRIPTION

       The CpuMemSets interface provides general purpose  proces­
       sor  and memory placement facilities to applications, sys­
       tem services and emulations of other CPU and memory place­
       ment interfaces.

       CpuMemSets are implemented using two separate layers.

       1) cpumemmap (cmm):
              The  bottom  layer  provides a simple pair of maps,
              mapping system CPU  and  memory  block  numbers  to
              application  CPU  and memory block numbers.  System
              numbers are those used by the kernel task  schedul­
              ing  and  memory  allocation  code,  and  typically
              include all CPUs and memory in the system.   Appli­
              cation  numbers are those used by an application in
              its cpumemset to specify its CPU and memory  affin­
              ity  for  those  CPU and memory blocks available in
              its map.

              Each process, each virtual  memory  area,  and  the
              kernel  has  such  a map.  These maps are inherited
              across fork(2), exec and the various ways to create
              vm  areas.  Only a process with root privileges can
              extend cpumemmaps to include additional system CPUs
              or memory blocks.  Changing a map will cause kernel
              scheduling code to immediately start using the  new
              system  CPUs,  and  cause kernel allocation code to
              allocate additional memory pages using the new sys­
              tem  memory blocks, but memory already allocated on

              other means.

              The bulk of the kernel is still written using what­
              ever system CPU and memory block numbers are appro­
              priate  for  a  system.   Changes to cpumemmaps are
              converted at the time of the cmsSet*()  calls  into
              changes  to the system masks (such as cpus_allowed)
              and lists (such as zone lists) used by the existing
              scheduler and allocator.

       2) cpumemset (cms):
              The  upper layer specifies on which of the applica­
              tion CPUs known to  that  process  a  task  can  be
              scheduled,  and  in which application memory blocks
              known to the kernel or that vm area, memory can  be
              allocated.  The kernel allocators search the memory
              block lists in the given order for  available  mem­
              ory, and a different list is specified for each CPU
              that may execute the request.  An  application  may
              change the cpumemset of its tasks and vm areas, and
              root may change the cpumemset used for kernel  mem­
              ory  allocation.   Also root may change the cpumem­
              sets of any process, and any process may change the
              cpumemsets  of  other  processes  with the same uid
              (kill (2) permissions).

       Each task has two cpumemsets, one defining its current CPU
       allocation and created vm areas, and one that is inherited
       by any child process it forks.  Both the current and child
       cpumemsets  of a newly forked process are set to copies of
       the child cpumemset of the parent process. Allocations  of
       memory to existing vm areas visible to a process depend on
       the cpumemset of that vm area (as acquired from its creat­
       ing process at creation, and possibly modified since), not
       on the cpumemset of the currently accessing task.

       During system boot, the  kernel  creates  and  attaches  a
       default  cpumemmap  and cpumemset that is used everywhere.
       By default this initial map and set contain all  CPUs  and
       all  memory blocks.  The memory blocks are not necessarily
       sorted in any particular order.  An optional  kernel  boot
       command  line  parameter causes this initial cpumemmap and
       cpumemset to contain only the first  CPU  and  one  memory
       block,  rather  than  all  of them, for the convenience of
       system management services that wish to take greater  con­
       trol of the system.

       The  kernel  will  only schedule a task on the CPUs in the
       tasks cpumemset, and only allocate memory to a  user  vir­
       tual  memory  area from the list of memories in that areas
       memory list.  The kernel allocates kernel memory only from
       the  list of memories in the cpumemset attached to the CPU
       executing the  allocation  request,  except  for  specific

       ory block.

       Both the current and child cpumemmaps and cpumemsets of  a
       newly  forked process are taken from the child settings of
       its parent, and memory allocated during  the  creation  of
       the  new  process  is  allocated  according to the parents
       child cpumemset and  associated  cpumemmap,  because  that
       cpumemset  is  acquired by the new process and then by any
       vm area created by that process.

       The cpumemset (and associated cpumemmap) of a  newly  cre­
       ated virtual memory area is taken from the current cpumem­
       set of the task creating it.  In the case of attaching  to
       an  existing  vm  area, things get more complicated.  Both
       mmap'd memory objects and System V shared  memory  regions
       can be attached to by multiple processes, or even attached
       to  multiple  times  by  the  same  process  at  different
       addresses.   If such an existing memory region is attached
       to, then by  default  the  new  vm  area  describing  that
       attachment inherits the current cpumemset of the attaching
       process.  If however the policy flag CMS_SHARE is  set  in
       the  cpumemset  currently  linked to from each vm area for
       that region, then the new vm area will also be  linked  to
       this same cpumemset.

       When  allocating  another page to an area, the kernel will
       choose the memory list for the CPU on  which  the  current
       task is being executed, if that CPU is in the cpumemset of
       that memory area, else it will choose the memory list  for
       the default CPU (see CMS_DEFAULT_CPU) in that memory areas
       cpumemset.  The kernel then  searches  the  chosen  memory
       list  in  order,  from  the beginning of that memory list,
       looking for available memory.  Typical  kernel  allocators
       search  the  same  list  multiple times, with increasingly
       aggressive search criteria and memory freeing actions.

       The cpumemmap and  cpumemset  calls  with  the  CMS_VMAREA
       apply  to  all future allocation of memory by any existing
       vm area, for any pages in the  range  [start,start+len)  ,
       similar  to  the  behavior  of  madvise(2), mincore(2) and
       msync(2).

       The cpu2node call returns which node the specified cpu  is
       on, or else NON_NODE_NUM.

       The runon call causes the current process and any children
       it subsequently forks to run on the application CPU speci­
       fied  in  the call.  This call does not change the current
       cpumemmap, and does not  allow  access  to  any  CPUs  not
       already  in  that  map,  nor  remove  permission  to runon
       another of these mapped CPUs at  some  later  time.   This
       call is a wrapper for several of the other calls.

       dren it subsequently forks to be bound to the  system  CPU
       specified in the call.  If the specified system CPU is not
       in the current cpumemmap, then it is added (requires  root
       privileges).  No CPUs already in the current cpumemmap are
       removed.  This call is a wrapper for several of the  other
       calls.

       The  nodebind()  call  causes  the current process and any
       children it subsequently forks to be bound to the cpus and
       memory  on the system node (memory block) specified in the
       call.  If any of these cpus or memory nodes are not in the
       current  cpumemmap, then it is added (requires root privi­
       leges).  No CPUs already  in  the  current  cpumemmap  are
       removed.   This call is a wrapper for several of the other
       calls.

       The numnodes() call returns the number  of  nodes  (memory
       blocks)  on the current system, or -1 on error.  This call
       depends on the presence of a  valid  /var/cpuset/cpu-node-
       map file.

       The  numcpus() call returns the number of CPUs on the cur­
       rent system, or -1 on error.  This  call  depends  on  the
       presence of a valid /var/cpuset/cpu-node-map file.

       The  lubcpunum() call returns the least upper bound of all
       CPU numbers on the current system, or 0  on  error.   This
       call is useful for sizing arrays that are to be indexed by
       CPU number.  This call depends on the presence of a  valid
       /var/cpuset/cpu-node-map file.


CONSTANTS AND TYPEDEFS

       #define CMS_DEFAULT 0x01 /* Memory list order */
       #define CMS_SHARE   0x04 /* Share virtual memory area CMS */
       typedef int cms_setpol_t;     /* Type of policy argument for sets */

       /* 16 bits gets us 64K CPUs ... that should be enough! */
       typedef uint16_t cms_acpu_t;  /* Type app CPU number */
       typedef uint16_t cms_amem_t;  /* Type app memory block number */
       typedef uint16_t cms_scpu_t;  /* Type sys CPU number */
       typedef uint16_t cms_smem_t;  /* Type sys memory block number */

       #define CMS_DEFAULT_CPU ((cms_acpu_t)-1)  /* Default Memory List */

       #define NON_NODE_NUM ((cms_smem_t)-1)     /* Invalid Node Number */

       /* Calls to query and set cmm and cms specify which one ... */
       #define CMS_CURRENT 0    /* task's own cmm/cms */
       #define CMS_CHILD   1    /* task's children's cmm/cms */
       #define CMS_VMAREA  2    /* vm area's cmm/cms */
       #define CMS_KERNEL  3    /* kernel's cmm/cms */
       typedef int cms_choice_t;     /* Type cmm/cms choice argument */

       /* cpumemmap: Type for the pair of maps ... */
       typedef struct cpumemmap {
               int nr_cpus;            /* number of CPUs in map */
               cms_scpu_t *cpus;       /* map app to sys CPU num */
               int nr_mems;            /* number of mems in map */
               cms_smem_t *mems;       /* map app to sys mem num */
       } cpumemmap_t;

       /*
        * How memory looks to (typically) a set of equivalent
        * CPUs, including which memory blocks to search for
        * memory, in what order, and the list of CPUs to which
        * this list of memory blocks applies.  The cpumemset
        * is sufficiently complex that this portion of the data
        * structure type is specified separately, then an array
        * of cms_memory_list_t structures is included in the
        * main cpumemset type.
        */

       typedef struct cms_memory_list {
               int nr_cpus;            /* Num CPUs sharing memory list */
               cms_acpu_t *cpus;       /* CPUs sharing memory list */
               int nr_mems;            /* Num memory blocks in list */
               cms_amem_t *mems;       /* Memory blocks in list */
       } cms_memory_list_t;

       /*
        * Specify a single cpumemset, describing on which CPUs to
        * schedule tasks, from which memory blocks to allocate memory,
        * and in what order to search these memory blocks.
        */

       typedef struct cpumemset {
               cms_setpol_t policy;    /* CMS_* set policy flags */
               int nr_cpus;            /* Num CPUs in this cpumemset */
               cms_acpu_t *cpus;       /* CPUs in this cpumemset */
               int nr_mems;            /* Num memory lists in cpumemset */
               cms_memory_list_t *mems;/* Memory lists in cpumemset */
       } cpumemset_t;

       /* cpumemmap: Type for the pair of maps ... */
       typedef struct cpumemmap {
               int nr_cpus;          /* number of CPUs in map */
               cms_scpu_t *cpus;     /* array maps application to system CPU num */
               int nr_mems;          /* number of mems in map */
               cms_smem_t *mems;     /* array maps application to system mem num */
       } cpumemmap_t;


ERRORS

       EINVAL   Every  cpumemset  must  specify a memory list for
                the CMS_DEFAULT_CPU, to ensure that regardless of
                which  CPU  a  memory  request  is executed on, a

                ory.   Attempts to set a cpumemset without a mem­
                ory list specified for the  CMS_DEFAULT_CPU  will
                fail, with errno set to EINVAL.

       ENOENT   If  a request is made to set a cpumemmap that has
                fewer CPUs or memory blocks listed than needed by
                any  cpumemsets that will be using that cpumemmap
                after the  change,  then  that  cmsSetCMM()  will
                fail,  with  errno  set  to ENOENT.  That is, you
                cannot remove elements of a cpumemmap that are in
                use.

       ENOENT   If a request is made to set a cpumemset that ref­
                erences CPU or memory blocks not available in its
                current  cpumemmap,  then  that  cmsSetCMS() will
                fail, with errno set to  ENOENT.   That  is,  you
                cannot  reference  unmapped  application  CPUs or
                memory blocks in a cpumemset.

       EPERM    If a request is made to set a cpumemmap by a pro­
                cess  without  root  privileges, and that request
                attempts to add any system CPU  or  memory  block
                number  not  currently  in the map being changed,
                then that request will fail, with  errno  set  to
                EPERM.

       EPERM    If  a cmsSetCMS() request is made on another pro­
                cess, then the  requesting  process  must  either
                have  root  privileges,  or the real or effective
                user ID of the   sending   process   must   equal
                the  real  or saved set-user-ID of the other pro­
                cess, or else the request will fail,  with  errno
                set  to  EPERM.  These permissions are similar to
                those required by the kill(2) system call.

       EINVAL   If a request is made to set a cpumemset that  has
                the  same CPU (application number) listed in more
                than  one  array  "cpus"  of  CPUs  sharing   any
                cms_memory_list_t,  then  the  request will fail,
                with errno set to EINVAL.   Otherwise,  duplicate
                CPU  or memory block numbers are harmless, except
                for minor inefficiencies.

       ESRCH    The operations to query and  set  cpumemmaps  and
                cpumemsets  can  be  applied  to any process (any
                pid).  If the pid is zero, then the operation  is
                applied to the current process.  If the specified
                pid does not exist, then the operation will  fail
                with errno set to ESRCH.

       EPERM    If  the cpubind(cpu) routine is called for a sys­
                tem cpu not in the  current  cpumemmap,  and  the
                current   process   lacks   the  root  privileges

                will fail (return -1) with errno set to EPERM.

       ENOSYS   Kernel does not support CpuMemSets.

       ENOMEM   CpuMemSets library unable to allocate memory.

       EPERM    Request to cpubind() to a cpu that was not in the
                process's current cpumemmap, and  process  lacked
                permission  to add the cpu, probably because pro­
                cess lacked root privileges.

       EINVAL   An application cpu number in a requested  cpumem­
                set is neither CMS_DEFAULT_CPU nor defined by the
                associated cpumemmap (not in the range of zero to
                nr_cpus-1)

       EINVAL   An  application  memory  number  in  a  requested
                cpumemset  is  not  defined  by  the   associated
                cpumemmap   (not   in   the   range  of  zero  to
                nr_mems-1).

       EINVAL   The cms_choice_t parameter  of  a  cmsQueryCMM(),
                cmsSetCMM(), cmsQueryCMS() or cmsSetCMS() call is
                not one of the allowed values.

       EINVAL   A call to one of cmsQueryCMM(), cmsSetCMM(), cms­
                QueryCMS()    or   cmsSetCMS()   is   made   with
                cms_choice_t of CMS_VMAREA, and a "start" address
                that is larger than any valid address in the vir­
                tual address space of the affected process.

       EFAULT   A directly or indirectly referenced parameter  is
                outside the invoking process's accessible address
                space.

       EBADF    One  of  the  routines  nodebind(),   cpu2node(),
                numnodes(), numcpus(), or lubcpunum() was invoked
                without    a     valid     configuration     file
                /var/cpuset/cpu-node-map.  The system initializa­
                tion script /etc/init  xxx  creates  cpu-node-map
                each system boot.

       EINVAL   The  'cpu'  requested  of cpubind() or cpu2node()
                was  not  listed  in   the   configuration   file
                /var/cpuset/cpu-node-map.

       EINVAL   The  configuration  file /var/cpuset/cpu-node-map
                has no valid entries,  or  has  two  inconsistent
                entries mapping one cpu to two distinct nodes.

       ENOENT   The 'node' requested of nodebind() was not listed
                in the configuration  file  /var/cpuset/cpu-node-
                map.

                /var/cpuset/cpu-node-map.  Perhaps cpumemsets  is
                not  properly  installed,  or 'cpunodemap' is not
                chkconfig'd on.

       ERANGE   Internal range check error in nodebind().


EXAMPLES

       This example shows how  to  query  the  currently  allowed
       cpus.

           /*
            * sample1 - display current cpumemset cpus
            *
            * Compile with:
            *  cc sample1.c -o sample1 -lcpumemsets
            * Displays on stdout the number and a list of the cpus
            * on which the current process is allowed to execute.
            */

           #include "cpumemsets.h"

           main()
           {
               int i;
               cpumemset_t *pset;

               pset = cmsQueryCMS(CMS_CURRENT, (pid_t)0, (void *)0);
               if (pset == (cpumemset_t *)0) {
                   perror("cmsQueryCMS");
                   exit (1);
               }
               printf("Current CpuMemSet has %d cpu(s):\n\t", pset->nr_cpus);
               for (i = 0; i < pset->nr_cpus; i++)
                   printf("%s%d", (i > 0 ? ", " : ""), pset->cpus[i]);
               printf("0);
               exit(0);
           }

       This  example  shows  how to run a child process (/bin/sh,
       here) only on cpu 0.

           /*
            *  sample2 - change child cpumemset cpus to just cpu 0
            *
            *  Compile with:
            *  cc sample2.c -o sample2 -lcpumemsets
            *  Change the cpus which the child task is allowed
            *  execute on to just cpu 0.  Start a subshell,
            *  instead of just exiting, so that the user has
            *  the opportunity to verify that the change occurred.
            */

           #include <malloc.h>


           main()
           {
               int i;
               cpumemset_t *pset;

               pset = cmsQueryCMS(CMS_CHILD, (pid_t)0, (void *)0);
               if (pset == (cpumemset_t *)0) {
                   perror("cmsQueryCMS");
                   exit (1);
               }
               pset->nr_cpus = 1;
               free(pset->cpus);
               pset->cpus = (cms_acpu_t *) malloc(sizeof(cms_acpu_t));
               pset->cpus[0] = (cms_acpu_t)0;
               if (cmsSetCMS(
                   CMS_CHILD, (pid_t)0, (void *)0, (size_t)0, pset) < 0) {
                   perror ("cmsSetCMS");
                   exit(1);
               }
               cmsFreeCMS (pset);
               printf ("Invoking subshell running on cpu 0.0);
               execl("/bin/sh", "sh", 0);
               exit (2);
           }

       Try running sample1 from the subshell started by  sample2.
       You  should  see  that  sample1  reports  that  it is only
       allowed to run on cpu 0:

           # PS1='Sub> ' ./sample2
           Invoking subshell running on cpu 0.
           Sub> ./sample1
           Current CpuMemSet has 1 cpu(s):
                   0
           Sub> exit


SEE ALSO

       runon(8), madvise(2), mincore(2), msync(2).


Man(1) output converted with man2html