lib: Better available cpu count determination on Linux

At startup, the unit router process creates a number of threads, it
tries to create the same number of threads (not incl the main thread) as
there are 'cpus' in the system.

On Linux the number of available cpus is determined via a call to

  sysconf(_SC_NPROCESSORS_ONLN);

in a lot of cases this produces the right result, i.e. on a four cpu
system this will return 4.

However this can break down if unit has been restricted in the cpus it's
allowed to run on via something like cpuset()'s and/or
sched_setaffinity(2).

For example, on a four 'cpu' system, starting unit will create an extra
4 router threads

  $ /opt/unit/sbin/unitd
  $ ps -efL | grep router
  andrew   234102 234099 234102  0    5 17:00 pts/10   00:00:00 unit: router
  andrew   234102 234099 234103  0    5 17:00 pts/10   00:00:00 unit: router
  andrew   234102 234099 234104  0    5 17:00 pts/10   00:00:00 unit: router
  andrew   234102 234099 234105  0    5 17:00 pts/10   00:00:00 unit: router
  andrew   234102 234099 234106  0    5 17:00 pts/10   00:00:00 unit: router

Say we want to limit unit to two cpus, i.e.

  $ taskset -a -c 2-3 /opt/unit/sbin/unitd
  $ ps -efL | grep router
  andrew   235772 235769 235772  0    5 17:08 pts/10   00:00:00 unit: router
  andrew   235772 235769 235773  0    5 17:08 pts/10   00:00:00 unit: router
  andrew   235772 235769 235774  0    5 17:08 pts/10   00:00:00 unit: router
  andrew   235772 235769 235775  0    5 17:08 pts/10   00:00:00 unit: router
  andrew   235772 235769 235776  0    5 17:08 pts/10   00:00:00 unit: router

So despite limiting unit to two cpus

  $ grep Cpus_allowed_list /proc/235772/status
  Cpus_allowed_list:	2-3

It still created 4 threads, probably not such an issue in this case, but
if we had a 64 'cpu' system and wanted to limit unit two cpus, then we'd
have 64 threads vying to run on two cpus and with our spinlock
implementation this can cause a lot of thread scheduling and congestion
overhead.

Besides, our intention is currently to create nr router threads == nr
cpus.

To resolve this, on Linux at least, this patch makes use of
sched_getaffinity(2) to determine what cpus unit is actually allowed to
run on.

We still use the result of

  sysconf(_SC_NPROCESSORS_ONLN);

as a fallback, we also use its result to allocate the required cpuset
size (where sched_getaffinity() will store its result) as the standard
cpu_set_t only has space to store 1023 cpus.

So with this patch if we try to limit unit to two cpus we now get

  $ taskset -a -c 2-3 /opt/unit/sbin/unitd
  $ ps -efL | grep router
  andrew   236887 236884 236887  0    3 17:20 pts/10   00:00:00 unit: router
  andrew   236887 236884 236888  0    3 17:20 pts/10   00:00:00 unit: router
  andrew   236887 236884 236889  0    3 17:20 pts/10   00:00:00 unit: router

This also applies to the likes of docker, if you run docker with the
--cpuset-cpus="" option, unit will now create a number of router threads
that matches the cpu count specified.

Perhaps useful if you are running a number of unit docker instances on a
high cpu count machine.

Link: <https://github.com/nginx/unit/issues/1042>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>

1 files changed, 23 insertions, 4 deletions

diff --git a/src/nxt_lib.c b/src/nxt_lib.c
index aba07dda..de23ce0a 100644
--- a/src/nxt_lib.c
+++ b/src/nxt_lib.c
@@ -32,7 +32,7 @@ const char *malloc_conf = "junk:true";
 nxt_int_t
 nxt_lib_start(const char *app, char **argv, char ***envp)
 {
-    int           n;
+    int           n = 0;
     nxt_int_t     flags;
     nxt_bool_t    update;
     nxt_thread_t  *thread;
@@ -87,13 +87,32 @@ nxt_lib_start(const char *app, char **argv, char ***envp)
 #ifdef _SC_NPROCESSORS_ONLN
     /* Linux, FreeBSD, Solaris, MacOSX. */
     n = sysconf(_SC_NPROCESSORS_ONLN);
+#endif
+
+#if (NXT_HAVE_LINUX_SCHED_GETAFFINITY)
+    if (n > 0) {
+        int        err;
+        size_t     size;
+        cpu_set_t  *set;
+
+        set = CPU_ALLOC(n);
+        if (set == NULL) {
+            return NXT_ERROR;
+        }
+
+        size = CPU_ALLOC_SIZE(n);
+
+        err = sched_getaffinity(0, size, set);
+        if (err == 0) {
+            n = CPU_COUNT_S(size, set);
+        }
+
+        CPU_FREE(set);
+    }
 
 #elif (NXT_HPUX)
     n = mpctl(MPC_GETNUMSPUS, NULL, NULL);
 
-#else
-    n = 0;
-
 #endif
 
     nxt_debug(&nxt_main_task, "ncpu: %d", n);