MAJOR: config: remove parsing of the global "nbproc" directive

This one was deprecated in 2.3 and marked for removal in 2.5. It suffers
too many limitations compared to threads, and prevents some improvements
from being engaged. Instead of a bypassable startup error, there is now
a hard error.

The parsing code was removed, and very few obvious cases were as well.
The code is deeply rooted at certain places (e.g. "for" loops iterating
from 0 to nbproc) so it will not be that trivial to remove everywhere.
The "bind" and "bind-process" parsers will have to be adjusted, though
maybe not completely changed if we later want to support thread groups
for large NUMA machines. Some stats socket restrictions were removed,
and the doc was updated according to what was done. A few places in the
doc still refer to nbproc and will have to be revisited. The master-worker
code also refers to the process number to distinguish between master and
workers and will have to be carefully adjusted. The MAX_PROCS macro was
reset to 1, this will at least reduce the size of some remaining arrays.

Two regtests were dependieng on this directive, one with an explicit
"nbproc 1" and another one testing the master's CLI using nbproc 4.
Both were adapted.
This commit is contained in:
Willy Tarreau 2021-06-11 16:50:29 +02:00
parent eb778248d9
commit b63dbb7b2e
10 changed files with 30 additions and 132 deletions

View File

@ -46,7 +46,7 @@ syn keyword hapParam enabled
syn keyword hapParam fullconn syn keyword hapParam fullconn
syn keyword hapParam gid group syn keyword hapParam gid group
syn keyword hapParam maxconn monitor-uri syn keyword hapParam maxconn monitor-uri
syn keyword hapParam nbproc noepoll nopoll syn keyword hapParam noepoll nopoll
syn keyword hapParam pidfile syn keyword hapParam pidfile
syn keyword hapParam quiet syn keyword hapParam quiet
syn keyword hapParam redispatch retries syn keyword hapParam redispatch retries

View File

@ -990,7 +990,6 @@ The following keywords are supported in the "global" section :
- lua-load-per-thread - lua-load-per-thread
- lua-prepend-path - lua-prepend-path
- mworker-max-reloads - mworker-max-reloads
- nbproc
- nbthread - nbthread
- node - node
- numa-cpu-mapping - numa-cpu-mapping
@ -1645,30 +1644,15 @@ mworker-max-reloads <number>
SIGTERM. This option helps to keep under control the number of workers. SIGTERM. This option helps to keep under control the number of workers.
See also "show proc" in the Management Guide. See also "show proc" in the Management Guide.
nbproc <number> (deprecated)
Creates <number> processes when going daemon. This requires the "daemon"
mode. By default, only one process is created, which is the recommended mode
of operation. For systems limited to small sets of file descriptors per
process, it may be needed to fork multiple daemons. When set to a value
larger than 1, threads are automatically disabled. USING MULTIPLE PROCESSES
IS HARDER TO DEBUG AND IS REALLY DISCOURAGED. This directive is deprecated
and scheduled for removal in 2.5. Please use "nbthread" instead. See also
"daemon" and "nbthread".
nbthread <number> nbthread <number>
This setting is only available when support for threads was built in. It This setting is only available when support for threads was built in. It
makes HAProxy run on <number> threads. This is exclusive with "nbproc". While makes HAProxy run on <number> threads. "nbthread" also works when HAProxy is
"nbproc" historically used to be the only way to use multiple processors, it started in foreground. On some platforms supporting CPU affinity, the default
also involved a number of shortcomings related to the lack of synchronization "nbthread" value is automatically set to the number of CPUs the process is
between processes (health-checks, peers, stick-tables, stats, ...) which do bound to upon startup. This means that the thread count can easily be
not affect threads. As such, any modern configuration is strongly encouraged adjusted from the calling process using commands like "taskset" or "cpuset".
to migrate away from "nbproc" to "nbthread". "nbthread" also works when Otherwise, this value defaults to 1. The default value is reported in the
HAProxy is started in foreground. On some platforms supporting CPU affinity, output of "haproxy -vv".
when nbproc is not used, the default "nbthread" value is automatically set to
the number of CPUs the process is bound to upon startup. This means that the
thread count can easily be adjusted from the calling process using commands
like "taskset" or "cpuset". Otherwise, this value defaults to 1. The default
value is reported in the output of "haproxy -vv". See also "nbproc".
numa-cpu-mapping numa-cpu-mapping
By default, if running on Linux, HAProxy inspects on startup the CPU topology By default, if running on Linux, HAProxy inspects on startup the CPU topology
@ -2226,11 +2210,10 @@ maxcompcpuusage <number>
Sets the maximum CPU usage HAProxy can reach before stopping the compression Sets the maximum CPU usage HAProxy can reach before stopping the compression
for new requests or decreasing the compression level of current requests. for new requests or decreasing the compression level of current requests.
It works like 'maxcomprate' but measures CPU usage instead of incoming data It works like 'maxcomprate' but measures CPU usage instead of incoming data
bandwidth. The value is expressed in percent of the CPU used by HAProxy. In bandwidth. The value is expressed in percent of the CPU used by HAProxy. A
case of multiple processes (nbproc > 1), each process manages its individual value of 100 disable the limit. The default value is 100. Setting a lower
usage. A value of 100 disable the limit. The default value is 100. Setting value will prevent the compression work from slowing the whole process down
a lower value will prevent the compression work from slowing the whole and from introducing high latencies.
process down and from introducing high latencies.
maxpipes <number> maxpipes <number>
Sets the maximum per-process number of pipes to <number>. Currently, pipes Sets the maximum per-process number of pipes to <number>. Currently, pipes
@ -2718,8 +2701,8 @@ tune.ssl.cachesize <number>
idle entries are purged and reassigned. Higher values reduce the occurrence idle entries are purged and reassigned. Higher values reduce the occurrence
of such a purge, hence the number of CPU-intensive SSL handshakes by ensuring of such a purge, hence the number of CPU-intensive SSL handshakes by ensuring
that all users keep their session as long as possible. All entries are that all users keep their session as long as possible. All entries are
pre-allocated upon startup and are shared between all processes if "nbproc" pre-allocated upon startup. Setting this value to 0 disables the SSL session
is greater than 1. Setting this value to 0 disables the SSL session cache. cache.
tune.ssl.force-private-cache tune.ssl.force-private-cache
This option disables SSL session cache sharing between all processes. It This option disables SSL session cache sharing between all processes. It
@ -4039,12 +4022,6 @@ bind /<path> [, ...] [param*]
only in log-forward sections. only in log-forward sections.
- 'unix@' -> address is a path to a local unix socket - 'unix@' -> address is a path to a local unix socket
- 'abns@' -> address is in abstract namespace (Linux only). - 'abns@' -> address is in abstract namespace (Linux only).
Note: since abstract sockets are not "rebindable", they
do not cope well with multi-process mode during
soft-restart, so it is better to avoid them if
nbproc is greater than 1. The effect is that if the
new process fails to start, only one of the old ones
will be able to rebind to the socket.
- 'fd@<n>' -> use file descriptor <n> inherited from the - 'fd@<n>' -> use file descriptor <n> inherited from the
parent. The fd must be bound and may or may not already parent. The fd must be bound and may or may not already
be listening. be listening.
@ -4187,7 +4164,7 @@ bind-process [ all | odd | even | <process_num>[-[<process_num>]] ] ...
bind 10.0.0.4:80 bind 10.0.0.4:80
bind-process 1-4 bind-process 1-4
See also : "nbproc" in global section, and "process" in section 5.1. See also : "process" in section 5.1.
capture cookie <name> len <length> capture cookie <name> len <length>
@ -10497,10 +10474,6 @@ stats admin { if | unless } <cond>
The admin level allows to enable/disable servers from the web interface. By The admin level allows to enable/disable servers from the web interface. By
default, statistics page is read-only for security reasons. default, statistics page is read-only for security reasons.
Note : Consider not using this feature in multi-process mode (nbproc > 1)
unless you know what you do : memory is not shared between the
processes, which can result in random behaviors.
Currently, the POST request is limited to the buffer size minus the reserved Currently, the POST request is limited to the buffer size minus the reserved
buffer space, which means that if the list of servers is too long, the buffer space, which means that if the list of servers is too long, the
request won't be processed. It is recommended to alter few servers at a request won't be processed. It is recommended to alter few servers at a
@ -10534,7 +10507,7 @@ stats admin { if | unless } <cond>
stats http-request auth unless AUTH stats http-request auth unless AUTH
stats admin if AUTH_ADMIN stats admin if AUTH_ADMIN
See also : "stats enable", "stats auth", "stats http-request", "nbproc", See also : "stats enable", "stats auth", "stats http-request",
"bind-process", section 3.4 about userlists and section 7 about "bind-process", section 3.4 about userlists and section 7 about
ACL usage. ACL usage.
@ -10998,10 +10971,6 @@ stick match <pattern> [table <table>] [{if | unless} <cond>]
way, it becomes very easy to insert cookies and match on IP addresses in way, it becomes very easy to insert cookies and match on IP addresses in
order to maintain stickiness between HTTP and HTTPS. order to maintain stickiness between HTTP and HTTPS.
Note : Consider not using this feature in multi-process mode (nbproc > 1)
unless you know what you do : memory is not shared between the
processes, which can result in random behaviors.
Example : Example :
# forward SMTP users to the same server they just used for POP in the # forward SMTP users to the same server they just used for POP in the
# last 30 minutes # last 30 minutes
@ -11020,7 +10989,7 @@ stick match <pattern> [table <table>] [{if | unless} <cond>]
server s1 192.168.1.1:25 server s1 192.168.1.1:25
server s2 192.168.1.1:25 server s2 192.168.1.1:25
See also : "stick-table", "stick on", "nbproc", "bind-process" and section 7 See also : "stick-table", "stick on", "bind-process" and section 7
about ACLs and samples fetching. about ACLs and samples fetching.
@ -11034,10 +11003,6 @@ stick on <pattern> [table <table>] [{if | unless} <condition>]
to both keywords for details. It is only provided as a convenience to both keywords for details. It is only provided as a convenience
for writing more maintainable configurations. for writing more maintainable configurations.
Note : Consider not using this feature in multi-process mode (nbproc > 1)
unless you know what you do : memory is not shared between the
processes, which can result in random behaviors.
Examples : Examples :
# The following form ... # The following form ...
stick on src table pop if !localhost stick on src table pop if !localhost
@ -11065,7 +11030,7 @@ stick on <pattern> [table <table>] [{if | unless} <condition>]
server s1 192.168.1.1:443 server s1 192.168.1.1:443
server s2 192.168.1.1:443 server s2 192.168.1.1:443
See also : "stick match", "stick store-request", "nbproc" and "bind-process". See also : "stick match", "stick store-request" and "bind-process".
stick store-request <pattern> [table <table>] [{if | unless} <condition>] stick store-request <pattern> [table <table>] [{if | unless} <condition>]
@ -11131,10 +11096,6 @@ stick store-request <pattern> [table <table>] [{if | unless} <condition>]
established, so that the table will contain the real server that processed established, so that the table will contain the real server that processed
the request. the request.
Note : Consider not using this feature in multi-process mode (nbproc > 1)
unless you know what you do : memory is not shared between the
processes, which can result in random behaviors.
Example : Example :
# forward SMTP users to the same server they just used for POP in the # forward SMTP users to the same server they just used for POP in the
# last 30 minutes # last 30 minutes
@ -11153,7 +11114,7 @@ stick store-request <pattern> [table <table>] [{if | unless} <condition>]
server s1 192.168.1.1:25 server s1 192.168.1.1:25
server s2 192.168.1.1:25 server s2 192.168.1.1:25
See also : "stick-table", "stick on", "nbproc", "bind-process" and section 7 See also : "stick-table", "stick on", "bind-process" and section 7
about ACLs and sample fetching. about ACLs and sample fetching.
@ -13770,7 +13731,7 @@ process <process-set>[/<thread-set>]
the same IP:port but not the same process in a listener, so that the system the same IP:port but not the same process in a listener, so that the system
can distribute the incoming connections into multiple queues and allow a can distribute the incoming connections into multiple queues and allow a
smoother inter-process load balancing. Currently Linux 3.9 and above is known smoother inter-process load balancing. Currently Linux 3.9 and above is known
for supporting this. See also "bind-process" and "nbproc". for supporting this. See also "bind-process".
proto <name> proto <name>
Forces the multiplexer's protocol to use for the incoming connections. It Forces the multiplexer's protocol to use for the incoming connections. It
@ -17366,11 +17327,6 @@ lat_ns_tot : integer
meth(<method>) : method meth(<method>) : method
Returns a method. Returns a method.
nbproc : integer
Returns an integer value corresponding to the number of processes that were
started (it equals the global "nbproc" setting). This is useful for logging
and debugging purposes.
nbsrv([<backend>]) : integer nbsrv([<backend>]) : integer
Returns an integer value corresponding to the number of usable servers of Returns an integer value corresponding to the number of usable servers of
either the current backend or the named backend. This is mostly used with either the current backend or the named backend. This is mostly used with
@ -17391,9 +17347,8 @@ prio_offset : integer
set-priority-offset". set-priority-offset".
proc : integer proc : integer
Returns an integer value corresponding to the position of the process calling Always returns value 1 (historically it would return the calling process
the function, between 1 and global.nbproc. This is useful for logging and number).
debugging purposes.
queue([<backend>]) : integer queue([<backend>]) : integer
Returns the total number of queued connections of the designated backend, Returns the total number of queued connections of the designated backend,

View File

@ -9,7 +9,6 @@ global
uid 200 uid 200
gid 200 gid 200
chroot /var/empty chroot /var/empty
nbproc 4
daemon daemon
frontend test-proxy frontend test-proxy

View File

@ -27,7 +27,7 @@
* resources on embedded systems. * resources on embedded systems.
*/ */
#ifndef MAX_PROCS #ifndef MAX_PROCS
#define MAX_PROCS LONGBITS #define MAX_PROCS 1
#endif #endif
/* MAX_THREADS defines the highest limit for the global nbthread value. It /* MAX_THREADS defines the highest limit for the global nbthread value. It

View File

@ -24,7 +24,6 @@ syslog Slog_1 -repeat 8 -level info {
haproxy h1 -conf { haproxy h1 -conf {
global global
nbproc 4
tune.ssl.default-dh-param 2048 tune.ssl.default-dh-param 2048
stats bind-process 1 stats bind-process 1
log ${Slog_1_addr}:${Slog_1_port} len 2048 local0 debug err log ${Slog_1_addr}:${Slog_1_port} len 2048 local0 debug err
@ -37,13 +36,11 @@ haproxy h1 -conf {
log global log global
listen http listen http
bind-process 1
bind unix@"${tmpdir}/http.socket" accept-proxy name ssl-offload-http bind unix@"${tmpdir}/http.socket" accept-proxy name ssl-offload-http
option forwardfor option forwardfor
listen ssl-offload-http listen ssl-offload-http
option httplog option httplog
bind-process 2-4
bind "fd@${ssl}" ssl crt ${testdir}/common.pem ssl no-sslv3 alpn h2,http/1.1 bind "fd@${ssl}" ssl crt ${testdir}/common.pem ssl no-sslv3 alpn h2,http/1.1
server http unix@"${tmpdir}/http.socket" send-proxy server http unix@"${tmpdir}/http.socket" send-proxy
} -start } -start

View File

@ -1,4 +1,4 @@
varnishtest "Show info of process 3" varnishtest "Show info of process 1"
#REQUIRE_VERSION=1.9 #REQUIRE_VERSION=1.9
@ -9,8 +9,6 @@ server s1 {
} -start } -start
haproxy h1 -W -S -conf { haproxy h1 -W -S -conf {
global
nbproc 4
defaults defaults
mode http mode http
timeout connect 1s timeout connect 1s
@ -26,6 +24,6 @@ haproxy h1 -W -S -conf {
} -start } -start
haproxy h1 -mcli { haproxy h1 -mcli {
send "@3 show info" send "@1 show info"
expect ~ ".*\nProcess_num: 3\n.*" expect ~ ".*\nProcess_num: 1\n.*"
} -wait } -wait

View File

@ -10,8 +10,6 @@ server s1 {
} -start } -start
haproxy h1 -W -S -conf { haproxy h1 -W -S -conf {
global
nbproc 1
defaults defaults
mode http mode http
timeout connect 1s timeout connect 1s

View File

@ -564,21 +564,9 @@ int cfg_parse_global(const char *file, int linenum, char **args, int kwm)
} }
/* end of user/group name handling*/ /* end of user/group name handling*/
else if (strcmp(args[0], "nbproc") == 0) { else if (strcmp(args[0], "nbproc") == 0) {
if (alertif_too_many_args(1, file, linenum, args, &err_code)) ha_alert("parsing [%s:%d] : nbproc is not supported any more since HAProxy 2.5. Threads will automatically be used on multi-processor machines if available.\n", file, linenum);
goto out; err_code |= ERR_ALERT | ERR_FATAL;
if (*(args[1]) == 0) { goto out;
ha_alert("parsing [%s:%d] : '%s' expects an integer argument.\n", file, linenum, args[0]);
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
}
global.nbproc = atol(args[1]);
all_proc_mask = nbits(global.nbproc);
if (global.nbproc < 1 || global.nbproc > MAX_PROCS) {
ha_alert("parsing [%s:%d] : '%s' must be between 1 and %d (was %d).\n",
file, linenum, args[0], MAX_PROCS, global.nbproc);
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
}
} }
else if (strcmp(args[0], "nbthread") == 0) { else if (strcmp(args[0], "nbthread") == 0) {
if (alertif_too_many_args(1, file, linenum, args, &err_code)) if (alertif_too_many_args(1, file, linenum, args, &err_code))

View File

@ -2646,7 +2646,7 @@ int check_config_validity()
global.nbthread = 1; global.nbthread = 1;
#if defined(USE_THREAD) #if defined(USE_THREAD)
if (global.nbproc == 1) { {
int numa_cores = 0; int numa_cores = 0;
#if defined(__linux__) && defined USE_CPU_AFFINITY #if defined(__linux__) && defined USE_CPU_AFFINITY
if (global.numa_cpu_mapping && !thread_cpu_mask_forced()) if (global.numa_cpu_mapping && !thread_cpu_mask_forced())
@ -2659,12 +2659,6 @@ int check_config_validity()
#endif #endif
} }
if (global.nbproc > 1 && global.nbthread > 1) {
ha_alert("cannot enable multiple processes if multiple threads are configured. Please use either nbproc or nbthread but not both.\n");
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
}
pool_head_requri = create_pool("requri", global.tune.requri_len , MEM_F_SHARED); pool_head_requri = create_pool("requri", global.tune.requri_len , MEM_F_SHARED);
pool_head_capture = create_pool("capture", global.tune.cookie_len, MEM_F_SHARED); pool_head_capture = create_pool("capture", global.tune.cookie_len, MEM_F_SHARED);
@ -2731,10 +2725,6 @@ int check_config_validity()
ha_warning("Proxy '%s': the process specified on the 'bind-process' directive refers to a process number that is higher than global.nbproc. The proxy has been forced to run on process 1 only.\n", curproxy->id); ha_warning("Proxy '%s': the process specified on the 'bind-process' directive refers to a process number that is higher than global.nbproc. The proxy has been forced to run on process 1 only.\n", curproxy->id);
curproxy->bind_proc = 1; curproxy->bind_proc = 1;
} }
else if (!curproxy->bind_proc && nbproc > 1) {
ha_warning("Proxy '%s': all processes specified on the 'bind-process' directive refer to numbers that are all higher than global.nbproc. The directive was ignored and the proxy will run on all processes.\n", curproxy->id);
curproxy->bind_proc = 0;
}
} }
/* check and reduce the bind-proc of each listener */ /* check and reduce the bind-proc of each listener */
@ -3993,22 +3983,6 @@ out_uri_auth_compat:
/* Check multi-process mode compatibility */ /* Check multi-process mode compatibility */
if (global.nbproc > 1 && global.cli_fe) {
list_for_each_entry(bind_conf, &global.cli_fe->conf.bind, by_fe) {
unsigned long mask;
mask = proc_mask(global.cli_fe->bind_proc) && all_proc_mask;
mask &= proc_mask(bind_conf->settings.bind_proc);
/* stop here if more than one process is used */
if (atleast2(mask))
break;
}
if (&bind_conf->by_fe != &global.cli_fe->conf.bind) {
ha_warning("stats socket will not work as expected in multi-process mode (nbproc > 1), you should force process binding globally using 'stats bind-process' or per socket using the 'process' attribute.\n");
}
}
/* Make each frontend inherit bind-process from its listeners when not specified. */ /* Make each frontend inherit bind-process from its listeners when not specified. */
for (curproxy = proxies_list; curproxy; curproxy = curproxy->next) { for (curproxy = proxies_list; curproxy; curproxy = curproxy->next) {
if (curproxy->bind_proc) if (curproxy->bind_proc)

View File

@ -1983,17 +1983,6 @@ static void init(int argc, char **argv)
ha_warning("a master CLI socket was defined, but master-worker mode (-W) is not enabled.\n"); ha_warning("a master CLI socket was defined, but master-worker mode (-W) is not enabled.\n");
} }
if (global.nbproc > 1 && !global.nbthread) {
ha_warning("nbproc is deprecated!\n"
" | For suffering many limitations, the 'nbproc' directive is now deprecated\n"
" | and scheduled for removal in 2.5. Just comment it out: haproxy will use\n"
" | threads and will run on all allocated processors. You may also switch to\n"
" | 'nbthread %d' to keep the same number of processors. If you absolutely\n"
" | want to run in multi-process mode, you can silence this warning by adding\n"
" | 'nbthread 1', but then please report your use case to developers.\n",
global.nbproc);
}
/* defaults sections are not needed anymore */ /* defaults sections are not needed anymore */
proxy_destroy_all_defaults(); proxy_destroy_all_defaults();