From 8fef2b8667f30e4562ac006f992f859d1defdc0e Mon Sep 17 00:00:00 2001 From: Julius Goryavsky Date: Mon, 10 May 2021 04:27:16 +0200 Subject: [PATCH 01/15] MDEV-23580: WSREP_SST: [ERROR] rsync daemon port has been taken This commit contains a large set of further bug fixes and improvements to SST scripts for Galera, continuing the work that was started in MDEV-24962 to make SST scripts work smoothly in different network configurations (especially using ipv6) and with different environment settings: 1) The ipv6 addresses were incorrectly handled in the SST script for rsync (incorrect address substitution for establishing a connection, incorrect address substitution for bind, and so on); 2) Checking the locality of the ip-address in SST scripts did not support ipv6 addresses (such as "[::1]"), which were falsely identified as non-local ip, which further did not allow running two SSTs on different local addresses on the same machine. On the other hand, this bug masked some other errors (related to handling ipv6 addresses); 3) The code for checking the locality of the ip address was different in the SST scripts for rsync and for mysqldump, with individual flaws. This code is now made common and moved to wsrep_sst_common; 4) Waiting for the start of the transport channel (socat, nc, rsync, stunnel) in the wait_for_listen() and check_pid_and_port() functions did not process ipv6 addresses correctly in all cases (not for all branches); 5) Waiting for the start of the transport channel (socat, nc, rsync, stunnel) in the wait_for_listen() and check_pid_and_port() functions for some code branches could give a false positive result due to the textual match of prefixes in the port number and/or PID of the process; 6) Waiting for the start of the transport channel (socat, nc, rsync, stunnel) was supported through different utilities in SST scripts for mariabackup and for rsync, and with various minor flaws in the code. Now the code is still different in these scripts, but it supports a common set of utilities (lsof, ss, sockstat) and is synchronized across patterns that used to check the output of these utilities; 7) In SST via mariabackup, the signal about readiness to receive data is sometimes sent too early - immediately after listen(), and not after accept() (which are called by socat or netcat utility). 8) Checking availability of the some options of some utilities was done using the grep pattern, which easily gives false positives; 9) Common name (CN) for local addresses, if not explicitly specified, is now always replaced to "localhost" to avoid the need to generate many separate certificates for local addresses of one machine and not to depend on which the local address is currently used in test (ipv4 or ipv6, etc.); 10) In tests galera_sst_mariabackup_encrypt_with_key_server and galera_sst_rsync_encrypt_with_key_server the correct certificate is selected to avoid commonname (CN) mismatch problems; 11) Further refactoring to protect against spaces in file names. 12) Further general refactoring to eliminate bash-specific constructs or to improve code readability; 13) The code for setting options for the nc (netcat) utility was different in different scripts for SST - now it is made identical. 14) Fixed long-time broken encryption via xbcrypt in combination with mariabackup and added support for key-based encryption via openssl utility, which is now enabled by default for encrypt=1 mode (this default mode can be changed using a new configuration file option "encypt-format=openssl|xbcrypt", which can be placed in the [mysqld], [sst] or in the [xtrabackup] section) - this change will allow us to use and to test the encypt=1 encryption without installing non-standard third-party utilities. --- ...ariabackup_encrypt_with_key-openssl.result | 3 + ...t_mariabackup_encrypt_with_key-openssl.cnf | 13 + ..._mariabackup_encrypt_with_key-openssl.test | 12 + ...st_mariabackup_encrypt_with_key_server.cnf | 6 +- ...t_mariabackup_encrypt_with_key_server.test | 2 +- .../t/galera_sst_rsync_encrypt_with_key.cnf | 1 - .../galera_sst_rsync_encrypt_with_server.cnf | 5 +- scripts/wsrep_sst_common.sh | 148 ++++-- scripts/wsrep_sst_mariabackup.sh | 478 ++++++++++-------- scripts/wsrep_sst_mysqldump.sh | 26 +- scripts/wsrep_sst_rsync.sh | 156 +++--- scripts/wsrep_sst_xtrabackup-v2.sh | 120 +++-- scripts/wsrep_sst_xtrabackup.sh | 148 +++--- 13 files changed, 645 insertions(+), 473 deletions(-) create mode 100644 mysql-test/suite/galera/r/galera_sst_mariabackup_encrypt_with_key-openssl.result create mode 100644 mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key-openssl.cnf create mode 100644 mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key-openssl.test mode change 100644 => 100755 scripts/wsrep_sst_common.sh diff --git a/mysql-test/suite/galera/r/galera_sst_mariabackup_encrypt_with_key-openssl.result b/mysql-test/suite/galera/r/galera_sst_mariabackup_encrypt_with_key-openssl.result new file mode 100644 index 00000000000..990e0a29506 --- /dev/null +++ b/mysql-test/suite/galera/r/galera_sst_mariabackup_encrypt_with_key-openssl.result @@ -0,0 +1,3 @@ +SELECT 1; +1 +1 diff --git a/mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key-openssl.cnf b/mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key-openssl.cnf new file mode 100644 index 00000000000..63eb47b519d --- /dev/null +++ b/mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key-openssl.cnf @@ -0,0 +1,13 @@ +!include ../galera_2nodes.cnf + +[mysqld] +wsrep_sst_method=mariabackup +wsrep_sst_auth="root:" +wsrep_debug=ON + +[sst] +encrypt-format=openssl +encrypt=1 +encrypt-algo=aes-256-ctr +encrypt-key=4FA92C5873672E20FB163A0BCB2BB4A4 +transferfmt=@ENV.MTR_GALERA_TFMT diff --git a/mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key-openssl.test b/mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key-openssl.test new file mode 100644 index 00000000000..1a78aa22cb3 --- /dev/null +++ b/mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key-openssl.test @@ -0,0 +1,12 @@ +# +# This test checks that encryption with key using openssl with options +# passed to mariabackup via the my.cnf file +# +--source include/galera_cluster.inc +--source include/have_innodb.inc +--source include/have_mariabackup.inc + +SELECT 1; + +--let $wait_condition = SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size'; +--source include/wait_condition.inc diff --git a/mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key_server.cnf b/mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key_server.cnf index 12fca48e065..9abef8820e5 100644 --- a/mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key_server.cnf +++ b/mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key_server.cnf @@ -5,9 +5,9 @@ wsrep_sst_method=mariabackup wsrep_sst_auth="root:" wsrep_debug=ON -ssl-cert=@ENV.MYSQL_TEST_DIR/std_data/client-cert.pem -ssl-key=@ENV.MYSQL_TEST_DIR/std_data/client-key.pem +ssl-cert=@ENV.MYSQL_TEST_DIR/std_data/server-cert.pem +ssl-key=@ENV.MYSQL_TEST_DIR/std_data/server-key.pem ssl-ca=@ENV.MYSQL_TEST_DIR/std_data/cacert.pem [sst] -ssl-mode=VERIFY_CA \ No newline at end of file +ssl-mode=VERIFY_CA diff --git a/mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key_server.test b/mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key_server.test index 19ebd0cf51e..5673dda30cb 100644 --- a/mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key_server.test +++ b/mysql-test/suite/galera/t/galera_sst_mariabackup_encrypt_with_key_server.test @@ -18,7 +18,7 @@ SELECT 1; # Confirm that transfer was SSL-encrypted --let $assert_text = Using openssl based encryption with socat ---let $assert_select = Using openssl based encryption with socat: with key and c +--let $assert_select = Using openssl based encryption with socat: with key and crt --let $assert_count = 1 --let $assert_file = $MYSQLTEST_VARDIR/log/mysqld.1.err --let $assert_only_after = CURRENT_TEST diff --git a/mysql-test/suite/galera/t/galera_sst_rsync_encrypt_with_key.cnf b/mysql-test/suite/galera/t/galera_sst_rsync_encrypt_with_key.cnf index f131088f582..948b52d4bf7 100644 --- a/mysql-test/suite/galera/t/galera_sst_rsync_encrypt_with_key.cnf +++ b/mysql-test/suite/galera/t/galera_sst_rsync_encrypt_with_key.cnf @@ -12,4 +12,3 @@ wsrep_provider_options='base_port=@mysqld.1.#galera_port;gcache.size=1;pc.ignore [mysqld.2] wsrep_provider_options='base_port=@mysqld.2.#galera_port;gcache.size=1;pc.ignore_sb=true' - diff --git a/mysql-test/suite/galera/t/galera_sst_rsync_encrypt_with_server.cnf b/mysql-test/suite/galera/t/galera_sst_rsync_encrypt_with_server.cnf index 8e31e69a590..8ed9348e789 100644 --- a/mysql-test/suite/galera/t/galera_sst_rsync_encrypt_with_server.cnf +++ b/mysql-test/suite/galera/t/galera_sst_rsync_encrypt_with_server.cnf @@ -2,8 +2,8 @@ [mysqld] wsrep_sst_method=rsync -ssl-cert=@ENV.MYSQL_TEST_DIR/std_data/client-cert.pem -ssl-key=@ENV.MYSQL_TEST_DIR/std_data/client-key.pem +ssl-cert=@ENV.MYSQL_TEST_DIR/std_data/server-cert.pem +ssl-key=@ENV.MYSQL_TEST_DIR/std_data/server-key.pem ssl-ca=@ENV.MYSQL_TEST_DIR/std_data/cacert.pem [sst] @@ -14,4 +14,3 @@ wsrep_provider_options='base_port=@mysqld.1.#galera_port;gcache.size=1;pc.ignore [mysqld.2] wsrep_provider_options='base_port=@mysqld.2.#galera_port;gcache.size=1;pc.ignore_sb=true' - diff --git a/scripts/wsrep_sst_common.sh b/scripts/wsrep_sst_common.sh old mode 100644 new mode 100755 index 082873dff3c..d19a0dbfdd5 --- a/scripts/wsrep_sst_common.sh +++ b/scripts/wsrep_sst_common.sh @@ -51,7 +51,7 @@ case "$1" in # # Break address string into host:port/path parts # - case "${WSREP_SST_OPT_ADDR}" in + case "$WSREP_SST_OPT_ADDR" in \[*) # IPv6 # Remove the starting and ending square brackets, if present: @@ -81,7 +81,7 @@ case "$1" in # up to "/" (if present): WSREP_SST_OPT_ADDR_PORT="${remain%%/*}" # If the "/" character is present, then the path is not empty: - if [ "${remain#*/}" != "${remain}" ]; then + if [ "${remain#*/}" != "$remain" ]; then # This operation removes everything up to the "/" character, # effectively removing the port number from the string: readonly WSREP_SST_OPT_PATH="${remain#*/}" @@ -89,10 +89,10 @@ case "$1" in readonly WSREP_SST_OPT_PATH="" fi # The rest of the string is the same as the path (for now): - remain="${WSREP_SST_OPT_PATH}" + remain="$WSREP_SST_OPT_PATH" # If there is one more "/" in the string, then everything before # it will be the module name, otherwise the module name is empty: - if [ "${remain%%/*}" != "${remain}" ]; then + if [ "${remain%%/*}" != "$remain" ]; then # This operation removes the tail after the very first # occurrence of the "/" character (inclusively): readonly WSREP_SST_OPT_MODULE="${remain%%/*}" @@ -103,7 +103,7 @@ case "$1" in remain="${WSREP_SST_OPT_PATH#*/}" # If the rest of the string does not match the original, then there # was something else besides the module name: - if [ "$remain" != "${WSREP_SST_OPT_PATH}" ]; then + if [ "$remain" != "$WSREP_SST_OPT_PATH" ]; then # Extract the part that matches the LSN by removing all # characters starting from the very first "/": readonly WSREP_SST_OPT_LSN="${remain%%/*}" @@ -113,7 +113,7 @@ case "$1" in # If the remainder does not match the original string, # then there is something else (the version number in # our case): - if [ "$remain" != "${WSREP_SST_OPT_LSN}" ]; then + if [ "$remain" != "$WSREP_SST_OPT_LSN" ]; then # Let's extract the version number by removing the tail # after the very first occurence of the "/" character # (inclusively): @@ -535,7 +535,8 @@ readonly WSREP_SST_OPT_ADDR_PORT # try to use my_print_defaults, mysql and mysqldump that come with the sources # (for MTR suite) -SCRIPTS_DIR="$(cd $(dirname "$0"); pwd -P)" +script_binary=$(dirname "$0") +SCRIPTS_DIR=$(cd "$script_binary"; pwd -P) EXTRA_DIR="$SCRIPTS_DIR/../extra" CLIENT_DIR="$SCRIPTS_DIR/../client" @@ -581,30 +582,45 @@ readonly MY_PRINT_DEFAULTS="$MY_PRINT_DEFAULTS $WSREP_SST_OPT_CONF" # parse_cnf() { - local group="$1" + local groups="$1" local var="$2" local reval="" - # normalize the variable names specified in cnf file (user can use _ or - for example log-bin or log_bin) - # then search for needed variable - # finally get the variable value (if variables has been specified multiple time use the last value only) + # normalize the variable names specified in the .cnf file + # (user can use '_' or '-', for example, log-bin or log_bin), + # then search for the last instance of the desired variable + # and finally get the value of that variable (if the variable + # was specified several times - we use only its last instance): - if [ "$group" = '--mysqld' -o \ - "$group" = 'mysqld' ]; then - if [ -n "$WSREP_SST_OPT_SUFFIX_VALUE" ]; then - reval=$($MY_PRINT_DEFAULTS "mysqld$WSREP_SST_OPT_SUFFIX_VALUE" | awk 'BEGIN {OFS=FS="="} {sub(/^--loose/,"-",$0); gsub(/_/,"-",$1); if ($1=="--'"$var"'") lastval=substr($0,length($1)+2)} END {print lastval}') - fi - fi + local pattern='BEGIN {OFS=FS="="} {sub(/^--loose/,"-",$0); gsub(/_/,"-",$1); if ($1=="--'"$var"'") lastval=substr($0,length($1)+2)} END {print lastval}' - if [ -z "$reval" ]; then - reval=$($MY_PRINT_DEFAULTS "$group" | awk 'BEGIN {OFS=FS="="} {sub(/^--loose/,"-",$0); gsub(/_/,"-",$1); if ($1=="--'"$var"'") lastval=substr($0,length($1)+2)} END {print lastval}') - fi + while [ -n "$groups" ]; do + # Remove the largest suffix starting with the '|' character: + local group="${groups%%\|*}" + # Remove the remainder (the group name) from the rest + # of the groups list (as if it were a prefix): + groups="${groups#$group}" + groups="${groups#\|}" + # if the group name is the same as the "[--]mysqld", then + # try to use it together with the group suffix: + if [ "${group#--}" = 'mysqld' -a -n "$WSREP_SST_OPT_SUFFIX_VALUE" ]; then + reval=$($MY_PRINT_DEFAULTS "mysqld$WSREP_SST_OPT_SUFFIX_VALUE" | awk "$pattern") + if [ -n "$reval" ]; then + break + fi + fi + # Let's try to use the group name as it is: + reval=$($MY_PRINT_DEFAULTS "$group" | awk "$pattern") + if [ -n "$reval" ]; then + break + fi + done - # use default if we haven't found a value + # use default if we haven't found a value: if [ -z "$reval" ]; then [ -n "${3:-}" ] && reval="$3" fi - echo $reval + echo "$reval" } # @@ -615,18 +631,37 @@ parse_cnf() # in_config() { - local group="$1" + local groups="$1" local var="$2" local found=0 - if [ "$group" = '--mysqld' -o \ - "$group" = 'mysqld' ]; then - if [ -n "$WSREP_SST_OPT_SUFFIX_VALUE" ]; then - found=$($MY_PRINT_DEFAULTS "mysqld$WSREP_SST_OPT_SUFFIX_VALUE" | awk 'BEGIN {OFS=FS="="; found=0} {sub(/^--loose/,"-",$0); gsub(/_/,"-",$1); if ($1=="--'"$var"'") found=1} END {print found}') - fi - fi - if [ $found -eq 0 ]; then - found=$($MY_PRINT_DEFAULTS "$group" | awk 'BEGIN {OFS=FS="="; found=0} {sub(/^--loose/,"-",$0); gsub(/_/,"-",$1); if ($1=="--'"$var"'") found=1} END {print found}') - fi + + # normalize the variable names specified in the .cnf file + # (user can use '_' or '-', for example, log-bin or log_bin), + # then search for the last instance(s) of the desired variable: + + local pattern='BEGIN {OFS=FS="="; found=0} {sub(/^--loose/,"-",$0); gsub(/_/,"-",$1); if ($1=="--'"$var"'") found=1} END {print found}' + + while [ -n "$groups" ]; do + # Remove the largest suffix starting with the '|' character: + local group="${groups%%\|*}" + # Remove the remainder (the group name) from the rest + # of the groups list (as if it were a prefix): + groups="${groups#$group}" + groups="${groups#\|}" + # if the group name is the same as the "[--]mysqld", then + # try to use it together with the group suffix: + if [ "${group#--}" = 'mysqld' -a -n "$WSREP_SST_OPT_SUFFIX_VALUE" ]; then + found=$($MY_PRINT_DEFAULTS "mysqld$WSREP_SST_OPT_SUFFIX_VALUE" | awk "$pattern") + if [ $found -ne 0 ]; then + break + fi + fi + # Let's try to use the group name as it is: + found=$($MY_PRINT_DEFAULTS "$group" | awk "$pattern") + if [ $found -ne 0 ]; then + break + fi + done echo $found } @@ -793,3 +828,52 @@ wsrep_gen_secret() $RANDOM $RANDOM $RANDOM $RANDOM fi } + +is_local_ip() +{ + [ "$1" = '127.0.0.1' ] && return 0 + [ "$1" = '127.0.0.2' ] && return 0 + [ "$1" = 'localhost' ] && return 0 + [ "$1" = '[::1]' ] && return 0 + [ "$1" = "$(hostname -s)" ] && return 0 + [ "$1" = "$(hostname -f)" ] && return 0 + [ "$1" = "$(hostname -d)" ] && return 0 + + local ip_util="$(command -v ip)" + if [ -x "$ip_util" ]; then + # ip address show ouput format is " inet[6]
/": + "$ip_util" address show \ + | grep -E "^[[:space:]]*inet.? [^[:space:]]+/" -o \ + | grep -F " $1/" >/dev/null && return 0 + else + local ifconfig_util="$(command -v ifconfig)" + if [ -x "$ifconfig_util" ]; then + # ifconfig output format is " inet[6]
...": + "$ifconfig_util" \ + | grep -E "^[[:space:]]*inet.? [^[:space:]]+ " -o \ + | grep -F " $1 " >/dev/null && return 0 + fi + fi + + return 1 +} + +check_sockets_utils() +{ + lsof_available=0 + sockstat_available=0 + ss_available=0 + + [ -x "$(command -v lsof)" ] && lsof_available=1 + [ -x "$(command -v sockstat)" ] && sockstat_available=1 + [ -x "$(command -v ss)" ] && ss_available=1 + + if [ $lsof_available -eq 0 -a \ + $sockstat_available -eq 0 -a \ + $ss_available -eq 0 ] + then + wsrep_log_error "Neither lsof tool, nor ss or sockstat was found in " \ + "the PATH! Make sure you have it installed." + exit 2 # ENOENT + fi +} diff --git a/scripts/wsrep_sst_mariabackup.sh b/scripts/wsrep_sst_mariabackup.sh index 8b05217b2fa..de789dc1728 100644 --- a/scripts/wsrep_sst_mariabackup.sh +++ b/scripts/wsrep_sst_mariabackup.sh @@ -1,6 +1,6 @@ #!/bin/bash -ue -# Copyright (C) 2013 Percona Inc # Copyright (C) 2017-2021 MariaDB +# Copyright (C) 2013 Percona Inc # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -17,14 +17,15 @@ # MA 02110-1335 USA. # Documentation: -# http://www.percona.com/doc/percona-xtradb-cluster/manual/xtrabackup_sst.html +# https://mariadb.com/kb/en/mariabackup-overview/ # Make sure to read that before proceeding! -. $(dirname $0)/wsrep_sst_common +. $(dirname "$0")/wsrep_sst_common wsrep_check_datadir -OS=$(uname) +OS="$(uname)" ealgo="" +eformat="" ekey="" ekeyfile="" encrypt=0 @@ -32,7 +33,7 @@ nproc=1 ecode=0 ssyslog="" ssystag="" -XTRABACKUP_PID="" +MARIABACKUP_PID="" SST_PORT="" REMOTEIP="" tcert="" @@ -47,7 +48,7 @@ lsn="" ecmd="" rlimit="" # Initially -stagemsg="${WSREP_SST_OPT_ROLE}" +stagemsg="$WSREP_SST_OPT_ROLE" cpat="" speciald=1 ib_home_dir="" @@ -59,8 +60,8 @@ strmcmd="" tfmt="" tcmd="" payload=0 -pvformat="-F '%N => Rate:%r Avg:%a Elapsed:%t %e Bytes: %b %p' " -pvopts="-f -i 10 -N $WSREP_SST_OPT_ROLE " +pvformat="-F '%N => Rate:%r Avg:%a Elapsed:%t %e Bytes: %b %p'" +pvopts="-f -i 10 -N $WSREP_SST_OPT_ROLE" STATDIR="" uextra=0 disver="" @@ -79,23 +80,22 @@ readonly SECRET_TAG="secret" # 5.6.21 PXC and later can't donate to an older joiner sst_ver=1 -if pv --help 2>/dev/null | grep -q FORMAT;then - pvopts+=$pvformat +if [ -x "$(command -v pv)" ] && pv --help | grep -qw -- '-F'; then + pvopts="$pvopts $pvformat" fi pcmd="pv $pvopts" declare -a RC set +e MARIABACKUP_BIN="$(command -v mariabackup)" -if [ -z "$MARIABACKUP_BIN" ]; then +if [ ! -x "$MARIABACKUP_BIN" ]; then wsrep_log_error 'mariabackup binary not found in $PATH' exit 42 fi set -e MBSTREAM_BIN=mbstream -XBCRYPT_BIN=xbcrypt # Not available in MariaBackup -DATA="${WSREP_SST_OPT_DATA}" +DATA="$WSREP_SST_OPT_DATA" INFO_FILE="xtrabackup_galera_info" IST_FILE="xtrabackup_ist" MAGIC_FILE="$DATA/$INFO_FILE" @@ -112,7 +112,7 @@ timeit(){ local cmd="$@" local x1 x2 took extcode - if [[ $ttime -eq 1 ]];then + if [ $ttime -eq 1 ]; then x1=$(date +%s) wsrep_log_info "Evaluating $cmd" eval "$cmd" @@ -137,19 +137,21 @@ get_keys() fi if [ $encrypt -eq 0 ]; then - if $MY_PRINT_DEFAULTS xtrabackup | grep -q -- "--encrypt"; then - wsrep_log_error "Unexpected option combination. SST may fail. Refer to http://www.percona.com/doc/percona-xtradb-cluster/manual/xtrabackup_sst.html" + if [ -n "$ealgo" -o -n "$ekey" -o -n "$ekeyfile" ]; then + wsrep_log_error "Options for encryption are specified, " \ + "but encryption itself is disabled. SST may fail." fi return fi if [ $sfmt = 'tar' ]; then - wsrep_log_info "NOTE: Xtrabackup-based encryption - encrypt=1 - cannot be enabled with tar format" + wsrep_log_info "NOTE: key-based encryption (encrypt=1) " \ + "cannot be enabled with tar format" encrypt=-1 return fi - wsrep_log_info "Xtrabackup based encryption enabled in my.cnf - Supported only from Xtrabackup 2.1.4" + wsrep_log_info "Key based encryption enabled in my.cnf" if [ -z "$ealgo" ]; then wsrep_log_error "FATAL: Encryption algorithm empty from my.cnf, bailing out" @@ -161,17 +163,49 @@ get_keys() exit 3 fi - if [ -z "$ekey" ]; then - ecmd="$XBCRYPT_BIN --encrypt-algo='$ealgo' --encrypt-key-file='$ekeyfile'" + if [ "$eformat" = 'openssl' ]; then + get_openssl + if [ -z "$OPENSSL_BINARY" ]; then + wsrep_log_error "If encryption using the openssl is enabled, " \ + "then you need to install openssl" + exit 2 + fi + ecmd="'$OPENSSL_BINARY' enc -$ealgo" + if "$OPENSSL_BINARY" enc -help 2>&1 | grep -qw -- '-pbkdf2'; then + ecmd="$ecmd -pbkdf2" + elif "$OPENSSL_BINARY" enc -help 2>&1 | grep -qw -- '-iter'; then + ecmd="$ecmd -iter 1" + elif "$OPENSSL_BINARY" enc -help 2>&1 | grep -qw -- '-md'; then + ecmd="$ecmd -md sha256" + fi + if [ -z "$ekey" ]; then + ecmd="$ecmd -kfile '$ekeyfile'" + else + ecmd="$ecmd -k '$ekey'" + fi + elif [ "$eformat" = 'xbcrypt' ]; then + if [ ! -x "$(command -v xbcrypt)" ]; then + wsrep_log_error "If encryption using the xbcrypt is enabled, " \ + "then you need to install xbcrypt" + exit 2 + fi + wsrep_log_info "NOTE: xbcrypt-based encryption, " \ + "supported only from Xtrabackup 2.1.4" + if [ -z "$ekey" ]; then + ecmd="xbcrypt --encrypt-algo='$ealgo' --encrypt-key-file='$ekeyfile'" + else + ecmd="xbcrypt --encrypt-algo='$ealgo' --encrypt-key='$ekey'" + fi else - ecmd="$XBCRYPT_BIN --encrypt-algo='$ealgo' --encrypt-key='$ekey'" + wsrep_log_error "Unknown encryption format='$eformat'" + exit 2 fi if [ "$WSREP_SST_OPT_ROLE" = 'joiner' ]; then ecmd="$ecmd -d" fi - stagemsg+="-XB-Encrypted" + stagemsg="$stagemsg-XB-Encrypted" } get_transfer() @@ -179,27 +213,27 @@ get_transfer() TSST_PORT="$SST_PORT" if [ $tfmt = 'nc' ]; then - wsrep_check_programs nc wsrep_log_info "Using netcat as streamer" - + wsrep_check_programs nc + tcmd="nc" if [ "$WSREP_SST_OPT_ROLE" = 'joiner' ]; then - if nc -h 2>&1 | grep -q ncat; then - # Ncat - tcmd="nc -l $TSST_PORT" - elif nc -h 2>&1 | grep -qw -- '-d\>'; then - # Debian netcat + if nc -h 2>&1 | grep -q 'ncat'; then + wsrep_log_info "Using Ncat as streamer" + tcmd="$tcmd -l" + elif nc -h 2>&1 | grep -qw -- '-d'; then + wsrep_log_info "Using Debian netcat as streamer" + tcmd="$tcmd -dl" if [ $WSREP_SST_OPT_HOST_IPv6 -eq 1 ]; then # When host is not explicitly specified (when only the port # is specified) netcat can only bind to an IPv4 address if # the "-6" option is not explicitly specified: - tcmd="nc -dl -6 $TSST_PORT" - else - tcmd="nc -dl $TSST_PORT" + tcmd="$tcmd -6" fi else - # traditional netcat - tcmd="nc -l -p $TSST_PORT" + wsrep_log_info "Using traditional netcat as streamer" + tcmd="$tcmd -l -p" fi + tcmd="$tcmd $TSST_PORT" else # Check to see if netcat supports the '-N' flag. # -N Shutdown the network socket after EOF on stdin @@ -208,33 +242,28 @@ get_transfer() # transfer and cause the command to timeout. # Older versions of netcat did not need this flag and will # return an error if the flag is used. - # - tcmd_extra="" - if nc -h 2>&1 | grep -qw -- -N; then - tcmd_extra="-N" + if nc -h 2>&1 | grep -qw -- '-N'; then + tcmd="$tcmd -N" wsrep_log_info "Using nc -N" fi # netcat doesn't understand [] around IPv6 address if nc -h 2>&1 | grep -q ncat; then - # Ncat wsrep_log_info "Using Ncat as streamer" - tcmd="nc $tcmd_extra $WSREP_SST_OPT_HOST_UNESCAPED $TSST_PORT" - elif nc -h 2>&1 | grep -qw -- '-d\>'; then - # Debian netcat + elif nc -h 2>&1 | grep -qw -- '-d'; then wsrep_log_info "Using Debian netcat as streamer" - tcmd="nc $tcmd_extra $WSREP_SST_OPT_HOST_UNESCAPED $TSST_PORT" else - # traditional netcat wsrep_log_info "Using traditional netcat as streamer" - tcmd="nc -q0 $tcmd_extra $WSREP_SST_OPT_HOST_UNESCAPED $TSST_PORT" + tcmd="$tcmd -q0" fi + tcmd="$tcmd $WSREP_SST_OPT_HOST_UNESCAPED $TSST_PORT" fi else tfmt='socat' - wsrep_check_programs socat - wsrep_log_info "Using socat as streamer" - if [[ $encrypt -eq 2 || $encrypt -eq 3 ]] && ! socat -V | grep -q "WITH_OPENSSL 1";then + wsrep_log_info "Using socat as streamer" + wsrep_check_programs socat + + if [ $encrypt -eq 2 -o $encrypt -eq 3 ] && ! socat -V | grep -q -F 'WITH_OPENSSL 1'; then wsrep_log_error "Encryption requested, but socat is not OpenSSL enabled (encrypt=$encrypt)" exit 2 fi @@ -245,7 +274,7 @@ get_transfer() wsrep_log_error "Both PEM and CRT files required" exit 22 fi - stagemsg+="-OpenSSL-Encrypted-2" + stagemsg="$stagemsg-OpenSSL-Encrypted-2" if [ "$WSREP_SST_OPT_ROLE" = 'joiner' ]; then wsrep_log_info "Decrypting with cert=${tpem}, cafile=${tcert}" tcmd="socat -u openssl-listen:$TSST_PORT,reuseaddr,cert='$tpem',cafile='$tcert'$sockopt stdio" @@ -259,7 +288,7 @@ get_transfer() wsrep_log_error "Both certificate and key files required" exit 22 fi - stagemsg+="-OpenSSL-Encrypted-3" + stagemsg="$stagemsg-OpenSSL-Encrypted-3" if [ -z "$tcert" ]; then # no verification if [ "$WSREP_SST_OPT_ROLE" = 'joiner' ]; then @@ -278,6 +307,8 @@ get_transfer() CN_option="" if [ -n "$WSREP_SST_OPT_REMOTE_USER" ]; then CN_option=",commonname='$WSREP_SST_OPT_REMOTE_USER'" + elif is_local_ip "$WSREP_SST_OPT_HOST_UNESCAPED"; then + CN_option=',commonname=localhost' fi wsrep_log_info "Encrypting with cert=${tpem}, key=${tkey}, cafile=${tcert}" tcmd="socat -u stdio openssl-connect:$REMOTEIP:$TSST_PORT,cert='$tpem',key='$tkey',cafile='$tcert'$CN_option$sockopt" @@ -297,13 +328,13 @@ get_footprint() { pushd "$WSREP_SST_OPT_DATA" 1>/dev/null payload=$(find . -regex '.*\.ibd$\|.*\.MYI$\|.*\.MYD$\|.*ibdata1$' -type f -print0 | du --files0-from=- --block-size=1 -c | awk 'END { print $1 }') - if $MY_PRINT_DEFAULTS xtrabackup | grep -q -- "--compress";then + if $MY_PRINT_DEFAULTS xtrabackup | grep -q -- "--compress"; then # QuickLZ has around 50% compression ratio # When compression/compaction used, the progress is only an approximate. payload=$(( payload*1/2 )) fi popd 1>/dev/null - pcmd+=" -s $payload" + pcmd="$pcmd -s $payload" adjust_progress } @@ -320,9 +351,9 @@ adjust_progress() if [ -n "$progress" -a "$progress" != '1' ]; then if [ -e "$progress" ]; then - pcmd+=" 2>>'$progress'" + pcmd="$pcmd 2>>'$progress'" else - pcmd+=" 2>'$progress'" + pcmd="$pcmd 2>'$progress'" fi elif [ -z "$progress" -a -n "$rlimit" ]; then # When rlimit is non-zero @@ -331,25 +362,26 @@ adjust_progress() if [ -n "$rlimit" -a "$WSREP_SST_OPT_ROLE" = 'donor' ]; then wsrep_log_info "Rate-limiting SST to $rlimit" - pcmd+=" -L \$rlimit" + pcmd="$pcmd -L \$rlimit" fi } +encgroups='--mysqld|sst|xtrabackup' + check_server_ssl_config() { - local section="$1" - tcert=$(parse_cnf "$section" 'ssl-ca') - tpem=$(parse_cnf "$section" 'ssl-cert') - tkey=$(parse_cnf "$section" 'ssl-key') + tcert=$(parse_cnf "$encgroups" 'ssl-ca') + tpem=$(parse_cnf "$encgroups" 'ssl-cert') + tkey=$(parse_cnf "$encgroups" 'ssl-key') } read_cnf() { - sfmt=$(parse_cnf sst streamfmt "mbstream") - tfmt=$(parse_cnf sst transferfmt "socat") + sfmt=$(parse_cnf sst streamfmt 'mbstream') + tfmt=$(parse_cnf sst transferfmt 'socat') - encrypt=$(parse_cnf 'sst' 'encrypt' 0) - tmode=$(parse_cnf 'sst' 'ssl-mode' 'DISABLED' | tr [:lower:] [:upper:]) + encrypt=$(parse_cnf "$encgroups" 'encrypt' 0) + tmode=$(parse_cnf "$encgroups" 'ssl-mode' 'DISABLED' | tr [:lower:] [:upper:]) if [ $encrypt -eq 0 -o $encrypt -ge 2 ] then @@ -363,11 +395,7 @@ read_cnf() then # backward-incompatible behavior if [ -z "$tpem" -a -z "$tkey" -a -z "$tcert" ] then # no old-style SSL config in [sst] - check_server_ssl_config 'sst' - if [ -z "$tpem" -a -z "$tkey" -a -z "$tcert" ] - then # no new-stype SSL config in [sst], try server-wide SSL config - check_server_ssl_config '--mysqld' - fi + check_server_ssl_config fi if [ 0 -eq $encrypt -a -n "$tpem" -a -n "$tkey" ] then @@ -380,29 +408,21 @@ read_cnf() [ "${tmode#VERIFY}" != "$tmode" ] || tcert="" fi fi + elif [ $encrypt -eq 1 ]; then + ealgo=$(parse_cnf "$encgroups" 'encrypt-algo') + eformat=$(parse_cnf "$encgroups" 'encrypt-format' 'openssl') + ekey=$(parse_cnf "$encgroups" 'encrypt-key') + ekeyfile=$(parse_cnf "$encgroups" 'encrypt-key-file') fi - if [ $encrypt -eq 1 ]; then - # Refer to http://www.percona.com/doc/percona-xtradb-cluster/manual/xtrabackup_sst.html - ealgo=$(parse_cnf xtrabackup encrypt "") - if [ -z "$ealgo" ]; then - ealgo=$(parse_cnf sst encrypt-algo "") - ekey=$(parse_cnf sst encrypt-key "") - ekeyfile=$(parse_cnf sst encrypt-key-file "") - else - ekey=$(parse_cnf xtrabackup encrypt-key "") - ekeyfile=$(parse_cnf xtrabackup encrypt-key-file "") - fi - fi - - wsrep_log_info "SSL configuration: CA='"$tcert"', CERT='"$tpem"'," \ - "KEY='"$tkey"', MODE='"$tmode"', encrypt="$encrypt + wsrep_log_info "SSL configuration: CA='$tcert', CERT='$tpem'," \ + "KEY='$tkey', MODE='$tmode', encrypt='$encrypt'" sockopt=$(parse_cnf sst sockopt "") progress=$(parse_cnf sst progress "") ttime=$(parse_cnf sst time 0) cpat=$(parse_cnf sst cpat '.*galera\.cache$\|.*sst_in_progress$\|.*\.sst$\|.*gvwstate\.dat$\|.*grastate\.dat$\|.*\.err$\|.*\.log$\|.*RPM_UPGRADE_MARKER$\|.*RPM_UPGRADE_HISTORY$') - [[ $OS == "FreeBSD" ]] && cpat=$(parse_cnf sst cpat '.*galera\.cache$|.*sst_in_progress$|.*\.sst$|.*gvwstate\.dat$|.*grastate\.dat$|.*\.err$|.*\.log$|.*RPM_UPGRADE_MARKER$|.*RPM_UPGRADE_HISTORY$') + [ $OS = 'FreeBSD' ] && cpat=$(parse_cnf sst cpat '.*galera\.cache$|.*sst_in_progress$|.*\.sst$|.*gvwstate\.dat$|.*grastate\.dat$|.*\.err$|.*\.log$|.*RPM_UPGRADE_MARKER$|.*RPM_UPGRADE_HISTORY$') scomp=$(parse_cnf sst compressor "") sdecomp=$(parse_cnf sst decompressor "") @@ -415,26 +435,20 @@ read_cnf() stimeout=$(parse_cnf sst sst-initial-timeout 300) ssyslog=$(parse_cnf sst sst-syslog 0) ssystag=$(parse_cnf mysqld_safe syslog-tag "${SST_SYSLOG_TAG:-}") - ssystag+="-" + ssystag="$ssystag-" sstlogarchive=$(parse_cnf sst sst-log-archive 1) - sstlogarchivedir=$(parse_cnf sst sst-log-archive-dir "/tmp/sst_log_archive") + sstlogarchivedir=$(parse_cnf sst sst-log-archive-dir '/tmp/sst_log_archive') - if [[ $speciald -eq 0 ]];then + if [ $speciald -eq 0 ]; then wsrep_log_error "sst-special-dirs equal to 0 is not supported, falling back to 1" speciald=1 fi - if [[ $ssyslog -ne -1 ]];then - if $MY_PRINT_DEFAULTS mysqld_safe | grep -q -- "--syslog";then + if [ $ssyslog -ne -1 ]; then + if $MY_PRINT_DEFAULTS mysqld_safe | grep -q -- "--syslog"; then ssyslog=1 fi fi - - if [[ $encrypt -eq 1 ]]; then - wsrep_log_error "Xtrabackup-based encryption is currently not" \ - "supported with MariaBackup" - exit 2 - fi } get_stream() @@ -461,7 +475,7 @@ get_proc() { set +e nproc=$(grep -c processor /proc/cpuinfo) - [[ -z $nproc || $nproc -eq 0 ]] && nproc=1 + [ -z $nproc -o $nproc -eq 0 ] && nproc=1 set -e } @@ -477,7 +491,7 @@ cleanup_joiner() local estatus=$? if [ $estatus -ne 0 ]; then wsrep_log_error "Cleanup after exit with status:$estatus" - elif [ "${WSREP_SST_OPT_ROLE}" = 'joiner' ]; then + elif [ "$WSREP_SST_OPT_ROLE" = 'joiner' ]; then wsrep_log_info "Removing the sst_in_progress file" wsrep_cleanup_progress_file fi @@ -495,10 +509,10 @@ cleanup_joiner() # This means no setsid done in mysqld. # We don't want to kill mysqld here otherwise. - if [[ $$ -eq $pgid ]];then + if [ $$ -eq $pgid ]; then # This means a signal was delivered to the process. # So, more cleanup. - if [[ $estatus -ge 128 ]];then + if [ $estatus -ge 128 ]; then kill -KILL -$$ || true fi fi @@ -509,7 +523,7 @@ cleanup_joiner() check_pid() { local pid_file="$1" - [ -r "$pid_file" ] && ps -p $(cat "$pid_file") >/dev/null 2>&1 + [ -r "$pid_file" ] && ps -p $(cat "$pid_file") 2>&1 >/dev/null } cleanup_donor() @@ -520,11 +534,11 @@ cleanup_donor() wsrep_log_error "Cleanup after exit with status:$estatus" fi - if [ -n "$XTRABACKUP_PID" ]; then - if check_pid $XTRABACKUP_PID + if [ -n "$MARIABACKUP_PID" ]; then + if check_pid $MARIABACKUP_PID then - wsrep_log_error "xtrabackup process is still running. Killing..." - kill_xtrabackup + wsrep_log_error "mariabackup process is still running. Killing..." + kill_mariabackup fi fi @@ -550,10 +564,10 @@ cleanup_donor() # This means no setsid done in mysqld. # We don't want to kill mysqld here otherwise. - if [[ $$ -eq $pgid ]];then + if [ $$ -eq $pgid ]; then # This means a signal was delivered to the process. # So, more cleanup. - if [[ $estatus -ge 128 ]];then + if [ $estatus -ge 128 ]; then kill -KILL -$$ || true fi fi @@ -561,24 +575,57 @@ cleanup_donor() exit $estatus } -kill_xtrabackup() +kill_mariabackup() { - local PID=$(cat "$XTRABACKUP_PID") + local PID=$(cat "$MARIABACKUP_PID") [ -n "$PID" -a "0" != "$PID" ] && kill $PID && (kill $PID && kill -9 $PID) || : - wsrep_log_info "Removing xtrabackup pid file $XTRABACKUP_PID" - rm -f "$XTRABACKUP_PID" || true + wsrep_log_info "Removing mariabackup pid file ($MARIABACKUP_PID)" + rm -f "$MARIABACKUP_PID" || true } setup_ports() { SST_PORT="$WSREP_SST_OPT_PORT" - if [ "$WSREP_SST_OPT_ROLE" = "donor" ]; then - REMOTEIP="${WSREP_SST_OPT_HOST}" - lsn="${WSREP_SST_OPT_LSN}" - sst_ver="${WSREP_SST_OPT_SST_VER}" + if [ "$WSREP_SST_OPT_ROLE" = 'donor' ]; then + REMOTEIP="$WSREP_SST_OPT_HOST" + lsn="$WSREP_SST_OPT_LSN" + sst_ver="$WSREP_SST_OPT_SST_VER" fi } +check_port() +{ + local PORT="$1" + local UTILS="$2" + + local port_info is_util + + if [ $lsof_available -ne 0 ]; then + port_info=$(lsof -i ":$PORT" -Pn 2>/dev/null | \ + grep -F '(LISTEN)') + is_util=$(echo "$port_info" | \ + grep -E "^($UTILS)[^[:space:]]*[[:space:]]+[0-9]+[[:space:]]+") + elif [ $sockstat_available -ne 0 ]; then + port_info=$(sockstat -p "$PORT" 2>/dev/null | \ + grep -F 'LISTEN') + is_util=$(echo "$port_info" | \ + grep -E "[[:space:]]+($UTILS)[^[:space:]]*[[:space:]]+[0-9]+[[:space:]]+") + elif [ $ss_available -ne 0 ]; then + port_info=$(ss -H -p -n -l "( sport = :$PORT )" 2>/dev/null) + is_util=$(echo "$port_info" | \ + grep -E "users:\\(.*\\(\"($UTILS)[^[:space:]]*\".*\.*\\)") + else + wsrep_log_error "unknown sockets utility" + exit 2 # ENOENT + fi + + if [ -z "$is_util" ]; then + return 1 + fi + + return 0 +} + # waits ~10 seconds for nc to open the port and then reports ready # (regardless of timeout) wait_for_listen() @@ -586,16 +633,16 @@ wait_for_listen() local PORT="$1" local ADDR="$2" local MODULE="$3" + for i in {1..50} do - if [ "$OS" = "FreeBSD" ];then - sockstat -46lp $PORT | grep -qE "^[^ ]* *(socat|nc) *[^ ]* *[^ ]* *[^ ]* *[^ ]*:$PORT" && break - else - ss -p state listening "( sport = :$PORT )" | grep -qE 'socat|nc' && break + if check_port "$PORT" 'socat|nc' + then + break fi sleep 0.2 done - echo "ready ${ADDR}/${MODULE}//$sst_ver" + echo "ready $ADDR/$MODULE//$sst_ver" } check_extra() @@ -606,10 +653,10 @@ check_extra() if [ "$thread_handling" = 'pool-of-threads' ]; then local eport=$(parse_cnf '--mysqld' 'extra-port') if [ -n "$eport" ]; then - # Xtrabackup works only locally. - # Hence, setting host to 127.0.0.1 unconditionally. + # mariabackup works only locally, hence, + # setting host to 127.0.0.1 unconditionally: wsrep_log_info "SST through extra_port $eport" - INNOEXTRA+=" --host=127.0.0.1 --port=$eport" + INNOEXTRA="$INNOEXTRA --host=127.0.0.1 --port=$eport" use_socket=0 else wsrep_log_error "Extra port $eport null, failing" @@ -620,7 +667,7 @@ check_extra() fi fi if [ $use_socket -eq 1 -a -n "$WSREP_SST_OPT_SOCKET" ]; then - INNOEXTRA+=" --socket='$WSREP_SST_OPT_SOCKET'" + INNOEXTRA="$INNOEXTRA --socket='$WSREP_SST_OPT_SOCKET'" fi } @@ -630,7 +677,7 @@ recv_joiner() local msg="$2" local tmt=$3 local checkf=$4 - local ltcmd + local wait=$5 if [ ! -d "$dir" ]; then # This indicates that IST is in progress @@ -640,28 +687,34 @@ recv_joiner() pushd "$dir" 1>/dev/null set +e - if [ $tmt -gt 0 -a -x "$(command -v timeout)" ]; then - if timeout --help | grep -q -- '-k'; then - ltcmd="timeout -k $(( tmt+10 )) $tmt $tcmd" - else - ltcmd="timeout -s9 $tmt $tcmd" + local ltcmd="$tcmd" + if [ $tmt -gt 0 ]; then + if [ -x "$(command -v timeout)" ]; then + if timeout --help | grep -qw -- '-k'; then + ltcmd="timeout -k $(( tmt+10 )) $tmt $tcmd" + else + ltcmd="timeout -s9 $tmt $tcmd" + fi fi - timeit "$msg" "$ltcmd | $strmcmd; RC=( "\${PIPESTATUS[@]}" )" - else - timeit "$msg" "$tcmd | $strmcmd; RC=( "\${PIPESTATUS[@]}" )" fi + if [ $wait -ne 0 ]; then + wait_for_listen "$SST_PORT" "$ADDR" "$MODULE" & + fi + + timeit "$msg" "$ltcmd | $strmcmd; RC=( "\${PIPESTATUS[@]}" )" + set -e popd 1>/dev/null - if [[ ${RC[0]} -eq 124 ]];then + if [ ${RC[0]} -eq 124 ]; then wsrep_log_error "Possible timeout in receiving first data from " \ "donor in gtid stage: exit codes: ${RC[@]}" exit 32 fi - for ecode in "${RC[@]}";do - if [[ $ecode -ne 0 ]];then + for ecode in "${RC[@]}"; do + if [ $ecode -ne 0 ]; then wsrep_log_error "Error while getting data from donor node: " \ "exit codes: ${RC[@]}" exit 32 @@ -672,14 +725,14 @@ recv_joiner() if [ ! -r "$MAGIC_FILE" ]; then # this message should cause joiner to abort wsrep_log_error "receiving process ended without creating " \ - "'${MAGIC_FILE}'" + "'$MAGIC_FILE'" wsrep_log_info "Contents of datadir" - wsrep_log_info "$(ls -l ${dir}/*)" + wsrep_log_info $(ls -l "$dir/"*) exit 32 fi # check donor supplied secret - SECRET=$(grep "$SECRET_TAG " "$MAGIC_FILE" 2>/dev/null | cut -d ' ' -f 2) + SECRET=$(grep -- "$SECRET_TAG " "$MAGIC_FILE" 2>/dev/null | cut -d ' ' -f 2) if [ "$SECRET" != "$MY_SECRET" ]; then wsrep_log_error "Donor does not know my secret!" wsrep_log_info "Donor:'$SECRET', my:'$MY_SECRET'" @@ -687,7 +740,7 @@ recv_joiner() fi # remove secret from magic file - grep -v "$SECRET_TAG " "$MAGIC_FILE" > "$MAGIC_FILE.new" + grep -v -- "$SECRET_TAG " "$MAGIC_FILE" > "$MAGIC_FILE.new" mv "$MAGIC_FILE.new" "$MAGIC_FILE" fi } @@ -703,8 +756,8 @@ send_donor() set -e popd 1>/dev/null - for ecode in "${RC[@]}";do - if [[ $ecode -ne 0 ]];then + for ecode in "${RC[@]}"; do + if [ $ecode -ne 0 ]; then wsrep_log_error "Error while sending data to joiner node: " \ "exit codes: ${RC[@]}" exit 32 @@ -717,11 +770,11 @@ monitor_process() local sst_stream_pid=$1 while true ; do - if ! ps -p "${WSREP_SST_OPT_PARENT}" &>/dev/null; then + if ! ps -p "$WSREP_SST_OPT_PARENT" &>/dev/null; then wsrep_log_error "Parent mysqld process (PID:${WSREP_SST_OPT_PARENT}) terminated unexpectedly." exit 32 fi - if ! ps -p "${sst_stream_pid}" &>/dev/null; then + if ! ps -p "$sst_stream_pid" &>/dev/null; then break fi sleep 0.1 @@ -730,7 +783,7 @@ monitor_process() wsrep_check_programs "$MARIABACKUP_BIN" -rm -f "${MAGIC_FILE}" +rm -f "$MAGIC_FILE" if [ "$WSREP_SST_OPT_ROLE" != 'joiner' -a "$WSREP_SST_OPT_ROLE" != 'donor' ]; then wsrep_log_error "Invalid role ${WSREP_SST_OPT_ROLE}" @@ -740,15 +793,15 @@ fi read_cnf setup_ports -if "${MARIABACKUP_BIN}" --help 2>/dev/null | grep -q -- '--version-check'; then +if "$MARIABACKUP_BIN" --help 2>/dev/null | grep -qw -- '--version-check'; then disver='--no-version-check' fi -iopts+=" --databases-exclude='lost+found'" +iopts="$iopts --databases-exclude='lost+found'" if [ ${FORCE_FTWRL:-0} -eq 1 ]; then wsrep_log_info "Forcing FTWRL due to environment variable FORCE_FTWRL equal to $FORCE_FTWRL" - iopts+=' --no-backup-locks' + iopts="$iopts --no-backup-locks" fi # if no command line argument and INNODB_DATA_HOME_DIR environment variable @@ -769,11 +822,9 @@ fi cd "$OLD_PWD" -if [[ $ssyslog -eq 1 ]];then +if [ $ssyslog -eq 1 ]; then - if [ ! -x "$(command -v logger)" ]; then - wsrep_log_error "logger not in path: $PATH. Ignoring" - else + if [ -x "$(command -v logger)" ]; then wsrep_log_info "Logging all stderr of SST/mariabackup to syslog" exec 2> >(logger -p daemon.err -t ${ssystag}wsrep-sst-$WSREP_SST_OPT_ROLE) @@ -787,6 +838,8 @@ if [[ $ssyslog -eq 1 ]];then { logger -p daemon.info -t ${ssystag}wsrep-sst-$WSREP_SST_OPT_ROLE "$@" } + else + wsrep_log_error "logger not in path: $PATH. Ignoring" fi INNOAPPLY="2>&1 | logger -p daemon.err -t ${ssystag}innobackupex-apply" @@ -795,10 +848,9 @@ if [[ $ssyslog -eq 1 ]];then else -if [[ "$sstlogarchive" -eq 1 ]] +if [ $sstlogarchive -eq 1 ] then ARCHIVETIMESTAMP=$(date "+%Y.%m.%d-%H.%M.%S.%N") - newfile="" if [ -n "$sstlogarchivedir" ] then @@ -812,11 +864,12 @@ then then if [ -n "$sstlogarchivedir" ] then - newfile="$sstlogarchivedir/$(basename '$INNOAPPLYLOG').$ARCHIVETIMESTAMP" + newfile=$(basename "$INNOAPPLYLOG") + newfile="$sstlogarchivedir/$newfile.$ARCHIVETIMESTAMP" else newfile="$INNOAPPLYLOG.$ARCHIVETIMESTAMP" fi - wsrep_log_info "Moving ${INNOAPPLYLOG} to ${newfile}" + wsrep_log_info "Moving '$INNOAPPLYLOG' to '$newfile'" mv "$INNOAPPLYLOG" "$newfile" gzip "$newfile" fi @@ -825,11 +878,12 @@ then then if [ -n "$sstlogarchivedir" ] then - newfile="$sstlogarchivedir/$(basename '$INNOMOVELOG').$ARCHIVETIMESTAMP" + newfile=$(basename "$INNOMOVELOG") + newfile="$sstlogarchivedir/$newfile.$ARCHIVETIMESTAMP" else newfile="$INNOMOVELOG.$ARCHIVETIMESTAMP" fi - wsrep_log_info "Moving ${INNOMOVELOG} to ${newfile}" + wsrep_log_info "Moving '$INNOMOVELOG' to '$newfile'" mv "$INNOMOVELOG" "$newfile" gzip "$newfile" fi @@ -838,11 +892,12 @@ then then if [ -n "$sstlogarchivedir" ] then - newfile="$sstlogarchivedir/$(basename '$INNOBACKUPLOG').$ARCHIVETIMESTAMP" + newfile=$(basename "$INNOBACKUPLOG") + newfile="$sstlogarchivedir/$newfile.$ARCHIVETIMESTAMP" else newfile="$INNOBACKUPLOG.$ARCHIVETIMESTAMP" fi - wsrep_log_info "Moving ${INNOBACKUPLOG} to ${newfile}" + wsrep_log_info "Moving '$INNOBACKUPLOG' to '$newfile'" mv "$INNOBACKUPLOG" "$newfile" gzip "$newfile" fi @@ -868,7 +923,7 @@ setup_commands() get_stream get_transfer -if [ "$WSREP_SST_OPT_ROLE" = "donor" ] +if [ "$WSREP_SST_OPT_ROLE" = 'donor' ] then trap cleanup_donor EXIT @@ -881,18 +936,18 @@ then exit 93 fi - if [ -z "$(parse_cnf --mysqld tmpdir)" -a \ - -z "$(parse_cnf xtrabackup tmpdir)" ]; then - xtmpdir=$(mktemp -d) + tmpdir=$(parse_cnf "$encgroups" 'tmpdir') + if [ -z "$tmpdir" ]; then + xtmpdir="$(mktemp -d)" tmpopts="--tmpdir='$xtmpdir'" - wsrep_log_info "Using $xtmpdir as xtrabackup temporary directory" + wsrep_log_info "Using $xtmpdir as mariabackup temporary directory" fi - itmpdir=$(mktemp -d) + itmpdir="$(mktemp -d)" wsrep_log_info "Using $itmpdir as mariabackup temporary directory" if [ -n "$WSREP_SST_OPT_USER" ]; then - INNOEXTRA+=" --user='$WSREP_SST_OPT_USER'" + INNOEXTRA="$INNOEXTRA --user='$WSREP_SST_OPT_USER'" usrst=1 fi @@ -927,10 +982,11 @@ then tcmd="$ecmd | $tcmd" fi - send_donor "$DATA" "${stagemsg}-gtid" + send_donor "$DATA" "$stagemsg-gtid" tcmd="$ttcmd" + # Restore the transport commmand to its original state if [ -n "$progress" ]; then get_footprint tcmd="$pcmd | $tcmd" @@ -944,26 +1000,32 @@ then wsrep_log_info "Streaming the backup to joiner at ${REMOTEIP}:${SST_PORT}" + # Add compression to the head of the stream (if specified) if [ -n "$scomp" ]; then tcmd="$scomp | $tcmd" fi + # Add encryption to the head of the stream (if specified) + if [ $encrypt -eq 1 ]; then + tcmd="$ecmd | $tcmd" + fi + setup_commands set +e - timeit "${stagemsg}-SST" "$INNOBACKUP | $tcmd; RC=( "\${PIPESTATUS[@]}" )" + timeit "$stagemsg-SST" "$INNOBACKUP | $tcmd; RC=( "\${PIPESTATUS[@]}" )" set -e if [ ${RC[0]} -ne 0 ]; then wsrep_log_error "${MARIABACKUP_BIN} finished with error: ${RC[0]}. " \ "Check syslog or ${INNOBACKUPLOG} for details" exit 22 - elif [[ ${RC[$(( ${#RC[@]}-1 ))]} -eq 1 ]]; then + elif [ ${RC[$(( ${#RC[@]}-1 ))]} -eq 1 ]; then wsrep_log_error "$tcmd finished with error: ${RC[1]}" exit 22 fi # mariabackup implicitly writes PID to fixed location in $xtmpdir - XTRABACKUP_PID="$xtmpdir/xtrabackup_pid" + MARIABACKUP_PID="$xtmpdir/xtrabackup_pid" else # BYPASS FOR IST @@ -984,19 +1046,19 @@ then tcmd="$ecmd | $tcmd" fi - strmcmd+=" '$IST_FILE'" + strmcmd="$strmcmd '$IST_FILE'" - send_donor "$DATA" "${stagemsg}-IST" + send_donor "$DATA" "$stagemsg-IST" fi - echo "done ${WSREP_SST_OPT_GTID}" + echo "done $WSREP_SST_OPT_GTID" wsrep_log_info "Total time on donor: $totime seconds" -elif [ "${WSREP_SST_OPT_ROLE}" = "joiner" ] +elif [ "$WSREP_SST_OPT_ROLE" = 'joiner' ] then - [[ -e "$SST_PROGRESS_FILE" ]] && wsrep_log_info "Stale sst_in_progress file: $SST_PROGRESS_FILE" - [[ -n "$SST_PROGRESS_FILE" ]] && touch "$SST_PROGRESS_FILE" + [ -e "$SST_PROGRESS_FILE" ] && wsrep_log_info "Stale sst_in_progress file: $SST_PROGRESS_FILE" + [ -n "$SST_PROGRESS_FILE" ] && touch "$SST_PROGRESS_FILE" ib_home_dir="$INNODB_DATA_HOME_DIR" @@ -1015,7 +1077,7 @@ then ib_undo_dir="$INNODB_UNDO_DIR" - stagemsg="Joiner-Recv" + stagemsg='Joiner-Recv' sencrypted=1 nthreads=1 @@ -1041,42 +1103,41 @@ then exit 42 fi CN=$("$OPENSSL_BINARY" x509 -noout -subject -in "$tpem" | \ - tr "," "\n" | grep "CN =" | cut -d= -f2 | sed s/^\ // | \ + tr "," "\n" | grep -F 'CN =' | cut -d= -f2 | sed s/^\ // | \ sed s/\ %//) fi - MY_SECRET=$(wsrep_gen_secret) + MY_SECRET="$(wsrep_gen_secret)" # Add authentication data to address ADDR="$CN:$MY_SECRET@$ADDR" else MY_SECRET="" # for check down in recv_joiner() fi - wait_for_listen "$SST_PORT" "$ADDR" "$MODULE" & - trap sig_joiner_cleanup HUP PIPE INT TERM trap cleanup_joiner EXIT if [ -n "$progress" ]; then adjust_progress - tcmd+=" | $pcmd" + tcmd="$tcmd | $pcmd" fi get_keys if [ $encrypt -eq 1 -a $sencrypted -eq 1 ]; then - if [ -n "$sdecomp" ]; then - strmcmd="$sdecomp | $ecmd | $strmcmd" - else - strmcmd="$ecmd | $strmcmd" - fi - elif [ -n "$sdecomp" ]; then - strmcmd="$sdecomp | $strmcmd" + strmcmd="$ecmd | $strmcmd" fi - STATDIR=$(mktemp -d) - MAGIC_FILE="$STATDIR/$INFO_FILE" - recv_joiner "$STATDIR" "${stagemsg}-gtid" $stimeout 1 + if [ -n "$sdecomp" ]; then + strmcmd="$sdecomp | $strmcmd" + fi - if ! ps -p ${WSREP_SST_OPT_PARENT} &>/dev/null + check_sockets_utils + + STATDIR="$(mktemp -d)" + MAGIC_FILE="$STATDIR/$INFO_FILE" + + recv_joiner "$STATDIR" "$stagemsg-gtid" $stimeout 1 1 + + if ! ps -p "$WSREP_SST_OPT_PARENT" &>/dev/null then wsrep_log_error "Parent mysqld process (PID:${WSREP_SST_OPT_PARENT}) terminated unexpectedly." exit 32 @@ -1090,12 +1151,12 @@ then rm -rf "$DATA/.sst" fi mkdir -p "$DATA/.sst" - (recv_joiner "$DATA/.sst" "${stagemsg}-SST" 0 0) & + (recv_joiner "$DATA/.sst" "$stagemsg-SST" 0 0 0) & jpid=$! wsrep_log_info "Proceeding with SST" wsrep_log_info "Cleaning the existing datadir and innodb-data/log directories" - if [ "${OS}" = "FreeBSD" ]; then + if [ "$OS" = 'FreeBSD' ]; then find -E ${ib_home_dir:+"$ib_home_dir"} \ ${ib_undo_dir:+"$ib_undo_dir"} \ ${ib_log_dir:+"$ib_log_dir"} \ @@ -1128,13 +1189,13 @@ then get_proc - if [[ ! -s "$DATA/xtrabackup_checkpoints" ]];then + if [ ! -s "$DATA/xtrabackup_checkpoints" ]; then wsrep_log_error "xtrabackup_checkpoints missing, failed mariabackup/SST on donor" exit 2 fi # Compact backups are not supported by mariabackup - if grep -q 'compact = 1' "$DATA/xtrabackup_checkpoints"; then + if grep -q -F 'compact = 1' "$DATA/xtrabackup_checkpoints"; then wsrep_log_info "Index compaction detected" wsrel_log_error "Compact backups are not supported by mariabackup" exit 2 @@ -1149,13 +1210,12 @@ then exit 22 fi - if [[ -n "$progress" ]] && pv --help | grep -q 'line-mode';then + if [ -n "$progress" ] && pv --help | grep -qw -- '--line-mode'; then count=$(find "$DATA" -type f -name '*.qp' | wc -l) count=$(( count*2 )) - if pv --help | grep -q FORMAT;then - pvopts="-f -s $count -l -N Decompression -F '%N => Rate:%r Elapsed:%t %e Progress: [%b/$count]'" - else - pvopts="-f -s $count -l -N Decompression" + pvopts="-f -s $count -l -N Decompression" + if pv --help | grep -qw -- '-F'; then + pvopts="$pvopts -F '%N => Rate:%r Elapsed:%t %e Progress: [%b/$count]'" fi pcmd="pv $pvopts" adjust_progress @@ -1169,10 +1229,10 @@ then timeit "Joiner-Decompression" "find '$DATA' -type f -name '*.qp' -printf '%p\n%h\n' | $dcmd" extcode=$? - if [[ $extcode -eq 0 ]];then + if [ $extcode -eq 0 ]; then wsrep_log_info "Removing qpress files after decompression" find "$DATA" -type f -name '*.qp' -delete - if [[ $? -ne 0 ]];then + if [ $? -ne 0 ]; then wsrep_log_error "Something went wrong with deletion of qpress files. Investigate" fi else @@ -1199,7 +1259,7 @@ then wsrep_log_info "Preparing the backup at ${DATA}" setup_commands - timeit "Xtrabackup prepare stage" "$INNOAPPLY" + timeit "mariabackup prepare stage" "$INNOAPPLY" if [ $? -ne 0 ]; then wsrep_log_error "${MARIABACKUP_BIN} apply finished with errors. Check syslog or ${INNOAPPLYLOG} for details" @@ -1208,8 +1268,8 @@ then MAGIC_FILE="$TDATA/$INFO_FILE" wsrep_log_info "Moving the backup to ${TDATA}" - timeit "Xtrabackup move stage" "$INNOMOVE" - if [[ $? -eq 0 ]];then + timeit "mariabackup move stage" "$INNOMOVE" + if [ $? -eq 0 ]; then wsrep_log_info "Move successful, removing ${DATA}" rm -rf "$DATA" DATA="$TDATA" @@ -1229,7 +1289,9 @@ then wsrep_log_error "SST magic file ${MAGIC_FILE} not found/readable" exit 2 fi - wsrep_log_info "Galera co-ords from recovery: $(cat '${MAGIC_FILE}')" + + coords=$(cat "$MAGIC_FILE") + wsrep_log_info "Galera co-ords from recovery: $coords" cat "$MAGIC_FILE" # Output : UUID:seqno wsrep_gtid_domain_id wsrep_log_info "Total time on joiner: $totime seconds" diff --git a/scripts/wsrep_sst_mysqldump.sh b/scripts/wsrep_sst_mysqldump.sh index 3cc52398caa..e227a888baf 100644 --- a/scripts/wsrep_sst_mysqldump.sh +++ b/scripts/wsrep_sst_mysqldump.sh @@ -18,35 +18,18 @@ # This is a reference script for mysqldump-based state snapshot tansfer -. $(dirname $0)/wsrep_sst_common +. $(dirname "$0")/wsrep_sst_common PATH=$PATH:/usr/sbin:/usr/bin:/sbin:/bin EINVAL=22 -local_ip() -{ - [ "$1" = "127.0.0.1" ] && return 0 - [ "$1" = "127.0.0.2" ] && return 0 - [ "$1" = "localhost" ] && return 0 - [ "$1" = "[::1]" ] && return 0 - [ "$1" = "$(hostname -s)" ] && return 0 - [ "$1" = "$(hostname -f)" ] && return 0 - [ "$1" = "$(hostname -d)" ] && return 0 - - # Now if ip program is not found in the path, we can't return 0 since - # it would block any address. Thankfully grep should fail in this case - ip route get "$1" | grep local >/dev/null && return 0 - - return 1 -} - if test -z "$WSREP_SST_OPT_HOST"; then wsrep_log_error "HOST cannot be nil"; exit $EINVAL; fi if test -z "$WSREP_SST_OPT_PORT"; then wsrep_log_error "PORT cannot be nil"; exit $EINVAL; fi if test -z "$WSREP_SST_OPT_LPORT"; then wsrep_log_error "LPORT cannot be nil"; exit $EINVAL; fi if test -z "$WSREP_SST_OPT_SOCKET";then wsrep_log_error "SOCKET cannot be nil";exit $EINVAL; fi if test -z "$WSREP_SST_OPT_GTID"; then wsrep_log_error "GTID cannot be nil"; exit $EINVAL; fi -if local_ip $WSREP_SST_OPT_HOST && \ +if is_local_ip "$WSREP_SST_OPT_HOST_UNESCAPED" && \ [ "$WSREP_SST_OPT_PORT" = "$WSREP_SST_OPT_LPORT" ] then wsrep_log_error \ @@ -111,7 +94,7 @@ then fi MYSQL="$MYSQL_CLIENT $WSREP_SST_OPT_CONF "\ -"$AUTH -h${WSREP_SST_OPT_HOST_UNESCAPED} "\ +"$AUTH -h$WSREP_SST_OPT_HOST_UNESCAPED "\ "-P$WSREP_SST_OPT_PORT --disable-reconnect --connect_timeout=10" # Check if binary logging is enabled on the joiner node. @@ -139,7 +122,7 @@ then # executed to erase binary logs (if any). Binary logging should also be # turned off for the session so that gtid state does not get altered while # the dump gets replayed on joiner. - if [[ "$LOG_BIN" == 'ON' ]]; then + if [ "$LOG_BIN" = 'ON' ]; then RESET_MASTER="SET GLOBAL wsrep_on=OFF; RESET MASTER; SET GLOBAL wsrep_on=ON;" SET_GTID_BINLOG_STATE="SET GLOBAL wsrep_on=OFF; SET @@global.gtid_binlog_state='$GTID_BINLOG_STATE'; SET GLOBAL wsrep_on=ON;" SQL_LOG_BIN_OFF="SET @@session.sql_log_bin=OFF;" @@ -164,7 +147,6 @@ $MYSQL -e "$STOP_WSREP SET GLOBAL SLOW_QUERY_LOG=OFF" RESTORE_GENERAL_LOG="SET GLOBAL GENERAL_LOG=$GENERAL_LOG_OPT;" RESTORE_SLOW_QUERY_LOG="SET GLOBAL SLOW_QUERY_LOG=$SLOW_LOG_OPT;" - if [ $WSREP_SST_OPT_BYPASS -eq 0 ] then (echo $STOP_WSREP && echo $RESET_MASTER && \ diff --git a/scripts/wsrep_sst_rsync.sh b/scripts/wsrep_sst_rsync.sh index 8db5dbb5e40..70e4a3326a1 100644 --- a/scripts/wsrep_sst_rsync.sh +++ b/scripts/wsrep_sst_rsync.sh @@ -23,13 +23,13 @@ RSYNC_PID= # rsync pid file RSYNC_CONF= # rsync configuration file RSYNC_REAL_PID= # rsync process id -OS=$(uname) +OS="$(uname)" [ "$OS" = 'Darwin' ] && export -n LD_LIBRARY_PATH # Setting the path for lsof on CentOS export PATH="/usr/sbin:/sbin:$PATH" -. $(dirname $0)/wsrep_sst_common +. $(dirname "$0")/wsrep_sst_common wsrep_check_datadir wsrep_check_programs rsync @@ -48,7 +48,7 @@ cleanup_joiner() rm -rf "$MAGIC_FILE" rm -rf "$RSYNC_PID" wsrep_log_info "Joiner cleanup done." - if [ "${WSREP_SST_OPT_ROLE}" = "joiner" ];then + if [ "$WSREP_SST_OPT_ROLE" = 'joiner' ]; then wsrep_cleanup_progress_file fi } @@ -57,68 +57,71 @@ cleanup_joiner() check_pid() { local pid_file="$1" - [ -r "$pid_file" ] && ps -p $(cat "$pid_file") >/dev/null 2>&1 + [ -r "$pid_file" ] && ps -p $(cat "$pid_file") 2>&1 >/dev/null } check_pid_and_port() { local pid_file="$1" local rsync_pid=$2 - local rsync_addr=$3 - local rsync_port=$4 + local rsync_addr="$3" + local rsync_port="$4" - case $OS in - FreeBSD) - local port_info="$(sockstat -46lp ${rsync_port} 2>/dev/null | \ - grep ":${rsync_port}")" - local is_rsync="$(echo $port_info | \ - grep -E '[[:space:]]+(rsync|stunnel)[[:space:]]+'"$rsync_pid" 2>/dev/null)" - ;; - *) - if [ ! -x "$(command -v lsof)" ]; then - wsrep_log_error "lsof tool not found in PATH! Make sure you have it installed." - exit 2 # ENOENT + if [ -z "$rsync_port" -o -z "$rsync_addr" -o -z "$rsync_pid" ]; then + wsrep_log_error "check_pid_and_port(): bad arguments" + exit 2 # ENOENT + fi + + local port_info is_rsync + + if [ $lsof_available -ne 0 ]; then + port_info=$(lsof -i ":$rsync_port" -Pn 2>/dev/null | \ + grep -F '(LISTEN)') + is_rsync=$(echo "$port_info" | \ + grep -E "^(rsync|stunnel)[^[:space:]]*[[:space:]]+$rsync_pid[[:space:]]+") + elif [ $sockstat_available -ne 0 ]; then + port_info=$(sockstat -p "$rsync_port" 2>/dev/null | \ + grep -F 'LISTEN') + is_rsync=$(echo "$port_info" | \ + grep -E "[[:space:]]+(rsync|stunnel)[^[:space:]]*[[:space:]]+$rsync_pid[[:space:]]+") + elif [ $ss_available -ne 0 ]; then + port_info=$(ss -H -p -n -l "( sport = :$rsync_port )" 2>/dev/null) + is_rsync=$(echo "$port_info" | \ + grep -E "users:\\(.*\\(\"(rsync|stunnel)[^[:space:]]*\".*\.*\\)") + else + wsrep_log_error "unknown sockets utility" + exit 2 # ENOENT + fi + + if [ -z "$is_rsync" ]; then + local is_listening_all + if [ $lsof_available -ne 0 ]; then + is_listening_all=$(echo "$port_info" | \ + grep -E "[[:space:]](\\*|\\[?::\\]?):$rsync_port[[:space:]]") + else + if [ $sockstat_available -eq 0 ]; then + port_info=$(echo "$port_info" | grep -q -F 'users:(') + fi + port_info=$(echo "$port_info" | \ + grep -E "[^[:space:]]+[[:space:]]+[^[:space:]]+[[:space:]]+[^[:space:]]+[[:space:]]+[^[:space:]]+[[:space:]]+[^[:space:]]+" -o) + is_listening_all=$(echo "$port_info" | \ + grep -E "[[:space:]](\\*|\\[?::\\]?):$rsync_port\$") fi - local port_info="$(lsof -i :$rsync_port -Pn 2>/dev/null | \ - grep "(LISTEN)")" - local is_rsync="$(echo $port_info | \ - grep -E '^(rsync|stunnel)[[:space:]]+'"$rsync_pid" 2>/dev/null)" - ;; - esac - - local is_listening_all="$(echo $port_info | \ - grep "*:$rsync_port" 2>/dev/null)" - local is_listening_addr="$(echo $port_info | \ - grep -F "$rsync_addr:$rsync_port" 2>/dev/null)" - - if [ ! -z "$is_listening_all" -o ! -z "$is_listening_addr" ]; then - if [ -z "$is_rsync" ]; then - wsrep_log_error "rsync daemon port '$rsync_port' has been taken" + local is_listening_addr=$(echo "$port_info" | \ + grep -w -F -- "$rsync_addr:$rsync_port") + if [ -z "$is_listening_addr" ]; then + is_listening_addr=$(echo "$port_info" | \ + grep -w -F "[$rsync_addr]:$rsync_port") + fi + if [ -n "$is_listening_all" -o -n "$is_listening_addr" ]; then + wsrep_log_error "rsync or stunnel daemon port '$rsync_port' " \ + "has been taken by another program" exit 16 # EBUSY fi + return 1 fi - check_pid "$pid_file" && \ - [ -n "$port_info" ] && [ -n "$is_rsync" ] && \ - [ $(cat "$pid_file") -eq $rsync_pid ] -} -is_local_ip() -{ - local address="$1" - local get_addr_bin="$(command -v ifconfig)" - if [ -z "$get_addr_bin" ] - then - get_addr_bin="$(command -v ip) address show" - # Add an slash at the end, so we don't get false positive : 172.18.0.4 matches 172.18.0.41 - # ip output format is "X.X.X.X/mask" - address="$address/" - else - # Add an space at the end, so we don't get false positive : 172.18.0.4 matches 172.18.0.41 - # ifconfig output format is "X.X.X.X " - address="$address " - fi - - $get_addr_bin | grep -F "$address" > /dev/null + check_pid "$pid_file" && [ $(cat "$pid_file") -eq $rsync_pid ] } STUNNEL_CONF="$WSREP_SST_OPT_DATA/stunnel.conf" @@ -296,7 +299,7 @@ foreground = yes pid = $STUNNEL_PID debug = warning client = yes -connect = ${WSREP_SST_OPT_ADDR%/*} +connect = $WSREP_SST_OPT_HOST_UNESCAPED:$WSREP_SST_OPT_PORT TIMEOUTclose = 0 ${VERIFY_OPT} EOF @@ -322,7 +325,7 @@ EOF # (b) Cluster state ID & wsrep_gtid_domain_id to be written to the file, OR # (c) ERROR file, in case flush tables operation failed. - while [ ! -r "$FLUSHED" ] && ! grep -q ':' "$FLUSHED" >/dev/null 2>&1 + while [ ! -r "$FLUSHED" ] && ! grep -q -F ':' "$FLUSHED" >/dev/null 2>&1 do # Check whether ERROR file exists. if [ -f "$ERROR" ] @@ -368,12 +371,11 @@ EOF eval rsync "'${STUNNEL:+--rsh=$STUNNEL}'" \ --owner --group --perms --links --specials \ --ignore-times --inplace --dirs --delete --quiet \ - $WHOLE_FILE_OPT ${FILTER} "'$WSREP_SST_OPT_DATA/'" \ + $WHOLE_FILE_OPT $FILTER "'$WSREP_SST_OPT_DATA/'" \ "'rsync://$WSREP_SST_OPT_ADDR'" >&2 || RC=$? if [ $RC -ne 0 ]; then wsrep_log_error "rsync returned code $RC:" - case $RC in 12) RC=71 # EPROTO wsrep_log_error \ @@ -414,21 +416,23 @@ EOF exit 255 # unknown error fi - # then, we parallelize the transfer of database directories, use . so that pathconcatenation works + # then, we parallelize the transfer of database directories, + # use . so that path concatenation works: + cd "$WSREP_SST_OPT_DATA" count=1 - [ "$OS" = "Linux" ] && count=$(grep -c processor /proc/cpuinfo) - [ "$OS" = "Darwin" -o "$OS" = "FreeBSD" ] && count=$(sysctl -n hw.ncpu) + [ "$OS" = 'Linux' ] && count=$(grep -c processor /proc/cpuinfo) + [ "$OS" = 'Darwin' -o "$OS" = 'FreeBSD' ] && count=$(sysctl -n hw.ncpu) - find . -maxdepth 1 -mindepth 1 -type d -not -name "lost+found" -not -name ".zfs" \ - -print0 | xargs -I{} -0 -P $count \ + find . -maxdepth 1 -mindepth 1 -type d -not -name 'lost+found' \ + -not -name '.zfs' -print0 | xargs -I{} -0 -P $count \ rsync ${STUNNEL:+--rsh="$STUNNEL"} \ --owner --group --perms --links --specials \ --ignore-times --inplace --recursive --delete --quiet \ $WHOLE_FILE_OPT --exclude '*/ib_logfile*' --exclude '*/aria_log.*' \ - --exclude '*/aria_log_control' "$WSREP_SST_OPT_DATA/"{}"/" \ - "rsync://$WSREP_SST_OPT_ADDR/"{} >&2 || RC=$? + --exclude '*/aria_log_control' "$WSREP_SST_OPT_DATA/{}/" \ + "rsync://$WSREP_SST_OPT_ADDR/{}" >&2 || RC=$? cd "$OLD_PWD" @@ -463,7 +467,7 @@ EOF elif [ "$WSREP_SST_OPT_ROLE" = 'joiner' ] then - wsrep_check_programs lsof + check_sockets_utils touch "$SST_PROGRESS_FILE" MYSQLD_PID="$WSREP_SST_OPT_PARENT" @@ -490,6 +494,7 @@ then ADDR="$WSREP_SST_OPT_ADDR" RSYNC_PORT="$WSREP_SST_OPT_PORT" RSYNC_ADDR="$WSREP_SST_OPT_HOST" + RSYNC_ADDR_UNESCAPED="$WSREP_SST_OPT_HOST_UNESCAPED" trap "exit 32" HUP PIPE trap "exit 3" INT TERM ABRT @@ -521,10 +526,10 @@ EOF # rm -rf "$DATA"/ib_logfile* # we don't want old logs around # If the IP is local listen only in it - if is_local_ip "$RSYNC_ADDR" + if is_local_ip "$RSYNC_ADDR_UNESCAPED" then - RSYNC_EXTRA_ARGS="--address $RSYNC_ADDR" - STUNNEL_ACCEPT="$RSYNC_ADDR:$RSYNC_PORT" + RSYNC_EXTRA_ARGS="--address $RSYNC_ADDR_UNESCAPED" + STUNNEL_ACCEPT="$RSYNC_ADDR_UNESCAPED:$RSYNC_PORT" else # Not local, possibly a NAT, listen on all interfaces RSYNC_EXTRA_ARGS="" @@ -535,7 +540,7 @@ EOF if [ -z "$STUNNEL" ] then - rsync --daemon --no-detach --port "$RSYNC_PORT" --config "$RSYNC_CONF" ${RSYNC_EXTRA_ARGS} & + rsync --daemon --no-detach --port "$RSYNC_PORT" --config "$RSYNC_CONF" $RSYNC_EXTRA_ARGS & RSYNC_REAL_PID=$! else cat << EOF > "$STUNNEL_CONF" @@ -545,6 +550,7 @@ ${CAFILE_OPT} foreground = yes pid = $STUNNEL_PID debug = warning +debug = 6 client = no [rsync] accept = $STUNNEL_ACCEPT @@ -556,7 +562,7 @@ EOF RSYNC_PID="$STUNNEL_PID" fi - until check_pid_and_port "$RSYNC_PID" "$RSYNC_REAL_PID" "$RSYNC_ADDR" "$RSYNC_PORT" + until check_pid_and_port "$RSYNC_PID" "$RSYNC_REAL_PID" "$RSYNC_ADDR_UNESCAPED" "$RSYNC_PORT" do sleep 0.2 done @@ -573,10 +579,10 @@ EOF exit 42 fi CN=$("$OPENSSL_BINARY" x509 -noout -subject -in "$SSTCERT" | \ - tr "," "\n" | grep "CN =" | cut -d= -f2 | sed s/^\ // | \ + tr "," "\n" | grep -F 'CN =' | cut -d= -f2 | sed s/^\ // | \ sed s/\ %//) fi - MY_SECRET=$(wsrep_gen_secret) + MY_SECRET="$(wsrep_gen_secret)" # Add authentication data to address ADDR="$CN:$MY_SECRET@$WSREP_SST_OPT_HOST" else @@ -626,7 +632,7 @@ EOF if [ -r "$MAGIC_FILE" ] then # check donor supplied secret - SECRET=$(grep "$SECRET_TAG " "$MAGIC_FILE" 2>/dev/null | cut -d ' ' -f 2) + SECRET=$(grep -F -- "$SECRET_TAG " "$MAGIC_FILE" 2>/dev/null | cut -d ' ' -f 2) if [ "$SECRET" != "$MY_SECRET" ]; then wsrep_log_error "Donor does not know my secret!" wsrep_log_info "Donor:'$SECRET', my:'$MY_SECRET'" @@ -634,7 +640,7 @@ EOF fi # remove secret from magic file - grep -v "$SECRET_TAG " "$MAGIC_FILE" > "$MAGIC_FILE.new" + grep -v -F -- "$SECRET_TAG " "$MAGIC_FILE" > "$MAGIC_FILE.new" mv "$MAGIC_FILE.new" "$MAGIC_FILE" # UUID:seqno & wsrep_gtid_domain_id is received here. @@ -645,7 +651,7 @@ EOF fi wsrep_cleanup_progress_file -# cleanup_joiner +# cleanup_joiner else wsrep_log_error "Unrecognized role: '$WSREP_SST_OPT_ROLE'" exit 22 # EINVAL diff --git a/scripts/wsrep_sst_xtrabackup-v2.sh b/scripts/wsrep_sst_xtrabackup-v2.sh index c1c9a41bdf2..7718e52184a 100644 --- a/scripts/wsrep_sst_xtrabackup-v2.sh +++ b/scripts/wsrep_sst_xtrabackup-v2.sh @@ -16,10 +16,11 @@ # Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston # MA 02110-1335 USA. -# Documentation: http://www.percona.com/doc/percona-xtradb-cluster/manual/xtrabackup_sst.html +# Documentation: +# http://www.percona.com/doc/percona-xtradb-cluster/manual/xtrabackup_sst.html # Make sure to read that before proceeding! -. $(dirname $0)/wsrep_sst_common +. $(dirname "$0")/wsrep_sst_common wsrep_check_datadir ealgo="" @@ -72,13 +73,13 @@ ssl_cert="" ssl_ca="" ssl_key="" -if which pv &>/dev/null && pv --help | grep -q FORMAT;then +if [ -x "$(command -v pv)" ] && pv --help | grep -qw -- '-F'; then pvopts+=$pvformat fi pcmd="pv $pvopts" declare -a RC -INNOBACKUPEX_BIN=innobackupex +INNOBACKUPEX_BIN='innobackupex' DATA="${WSREP_SST_OPT_DATA}" INFO_FILE="xtrabackup_galera_info" IST_FILE="xtrabackup_ist" @@ -87,9 +88,9 @@ MAGIC_FILE="${DATA}/${INFO_FILE}" # Setting the path for ss and ip export PATH="/usr/sbin:/sbin:$PATH" -OS=$(uname) +OS="$(uname)" -if ! which lsof > /dev/null; then +if [ ! -x "$(command -v lsof)" ]; then wsrep_log_error "lsof tool not found in PATH! Make sure you have it installed." exit 2 # ENOENT fi @@ -131,7 +132,7 @@ get_keys() return fi - if [[ $sfmt == 'tar' ]];then + if [[ "$sfmt" == 'tar' ]];then wsrep_log_info "NOTE: Xtrabackup-based encryption - encrypt=1 - cannot be enabled with tar format" encrypt=-1 return @@ -251,10 +252,8 @@ get_transfer() TSST_PORT="$WSREP_SST_OPT_PORT" if [[ $tfmt == 'nc' ]];then - if [[ ! -x `which nc` ]];then - wsrep_log_error "nc(netcat) not found in path: $PATH" - exit 2 - fi + wsrep_log_info "Using netcat as streamer" + wsrep_check_programs nc if [[ $encrypt -eq 2 || $encrypt -eq 3 || $encrypt -eq 4 ]]; then wsrep_log_error "******** FATAL ERROR *********************** " @@ -264,25 +263,25 @@ get_transfer() exit 22 fi - wsrep_log_info "Using netcat as streamer" - if [[ "$WSREP_SST_OPT_ROLE" == "joiner" ]];then - if nc -h 2>&1 | grep -q ncat;then - # Ncat - tcmd="nc -l ${TSST_PORT}" - elif nc -h 2>&1 | grep -qw -- '-d\>';then - # Debian netcat - if [ $WSREP_SST_OPT_HOST_IPv6 -eq 1 ];then + tcmd="nc" + if [ "$WSREP_SST_OPT_ROLE" = 'joiner' ]; then + if nc -h 2>&1 | grep -q 'ncat'; then + wsrep_log_info "Using Ncat as streamer" + tcmd="$tcmd -l" + elif nc -h 2>&1 | grep -qw -- '-d'; then + wsrep_log_info "Using Debian netcat as streamer" + tcmd="$tcmd -dl" + if [ $WSREP_SST_OPT_HOST_IPv6 -eq 1 ]; then # When host is not explicitly specified (when only the port # is specified) netcat can only bind to an IPv4 address if # the "-6" option is not explicitly specified: - tcmd="nc -dl -6 ${TSST_PORT}" - else - tcmd="nc -dl ${TSST_PORT}" + tcmd="$tcmd -6" fi else - # traditional netcat - tcmd="nc -l -p ${TSST_PORT}" + wsrep_log_info "Using traditional netcat as streamer" + tcmd="$tcmd -l -p" fi + tcmd="$tcmd $TSST_PORT" else # Check to see if netcat supports the '-N' flag. # -N Shutdown the network socket after EOF on stdin @@ -291,30 +290,25 @@ get_transfer() # transfer and cause the command to timeout. # Older versions of netcat did not need this flag and will # return an error if the flag is used. - # - tcmd_extra="" - if nc -h 2>&1 | grep -qw -- -N;then - tcmd_extra+="-N" + if nc -h 2>&1 | grep -qw -- '-N'; then + tcmd="$tcmd -N" + wsrep_log_info "Using nc -N" fi # netcat doesn't understand [] around IPv6 address - if nc -h 2>&1 | grep -q ncat;then - # Ncat - tcmd="nc ${tcmd_extra} ${WSREP_SST_OPT_HOST_UNESCAPED} ${TSST_PORT}" - elif nc -h 2>&1 | grep -qw -- '-d\>';then - # Debian netcat - tcmd="nc ${tcmd_extra} ${WSREP_SST_OPT_HOST_UNESCAPED} ${TSST_PORT}" + if nc -h 2>&1 | grep -q ncat; then + wsrep_log_info "Using Ncat as streamer" + elif nc -h 2>&1 | grep -qw -- '-d'; then + wsrep_log_info "Using Debian netcat as streamer" else - # traditional netcat - tcmd="nc -q0 ${tcmd_extra} ${WSREP_SST_OPT_HOST_UNESCAPED} ${TSST_PORT}" + wsrep_log_info "Using traditional netcat as streamer" + tcmd="$tcmd -q0" fi + tcmd="$tcmd $WSREP_SST_OPT_HOST_UNESCAPED $TSST_PORT" fi else tfmt='socat' wsrep_log_info "Using socat as streamer" - if [[ ! -x `which socat` ]];then - wsrep_log_error "socat not found in path: $PATH" - exit 2 - fi + wsrep_check_programs socat donor_extra="" joiner_extra="" @@ -438,7 +432,7 @@ get_footprint() adjust_progress() { - if [[ ! -x `which pv` ]];then + if [ ! -x "$(command -v pv)" ]; then wsrep_log_error "pv not found in path: $PATH" wsrep_log_error "Disabling all progress/rate-limiting" pcmd="" @@ -529,7 +523,7 @@ get_stream() { if [[ $sfmt == 'xbstream' ]];then wsrep_log_info "Streaming with xbstream" - if [[ "$WSREP_SST_OPT_ROLE" == "joiner" ]];then + if [[ "$WSREP_SST_OPT_ROLE" == "joiner" ]];then strmcmd="xbstream -x" else strmcmd="xbstream -c '${INFO_FILE}'" @@ -537,7 +531,7 @@ get_stream() else sfmt="tar" wsrep_log_info "Streaming with tar" - if [[ "$WSREP_SST_OPT_ROLE" == "joiner" ]];then + if [[ "$WSREP_SST_OPT_ROLE" == "joiner" ]];then strmcmd="tar xfi -" else strmcmd="tar cf - '${INFO_FILE}'" @@ -720,8 +714,8 @@ recv_joiner() pushd "${dir}" 1>/dev/null set +e - if [[ $tmt -gt 0 && -x `which timeout` ]];then - if timeout --help | grep -q -- '-k';then + if [ $tmt -gt 0 -a -x "$(command -v timeout)" ]; then + if timeout --help | grep -qw -- '-k';then ltcmd="timeout -k $(( tmt+10 )) $tmt $tcmd" else ltcmd="timeout -s9 $tmt $tcmd" @@ -751,7 +745,7 @@ recv_joiner() # this message should cause joiner to abort wsrep_log_error "xtrabackup process ended without creating '${MAGIC_FILE}'" wsrep_log_info "Contents of datadir" - wsrep_log_info "$(ls -l ${dir}/*)" + wsrep_log_info $(ls -l "$dir/"*) exit 32 fi } @@ -802,8 +796,8 @@ normalize_version() # Returns 1 (failure) if $1 >= $2, 0 (success) otherwise check_for_version() { - local local_version_str="$( normalize_version $1 )" - local required_version_str="$( normalize_version $2 )" + local local_version_str=$(normalize_version "$1") + local required_version_str=$(normalize_version "$2") if [[ "$local_version_str" < "$required_version_str" ]]; then return 1 @@ -833,7 +827,8 @@ monitor_process() done } -if [[ ! -x `which $INNOBACKUPEX_BIN` ]];then +innobackup=$(command -v "$INNOBACKUPEX_BIN") +if [ ! -x "$innobackup" ]; then wsrep_log_error "innobackupex not in path: $PATH" exit 2 fi @@ -848,21 +843,21 @@ if [[ -z "$XB_VERSION" ]]; then exit 2 fi -if ! check_for_version $XB_VERSION $XB_REQUIRED_VERSION; then +if ! check_for_version "$XB_VERSION" "$XB_REQUIRED_VERSION"; then wsrep_log_error "FATAL: The $INNOBACKUPEX_BIN version is $XB_VERSION. Needs xtrabackup-$XB_REQUIRED_VERSION or higher to perform SST" exit 2 fi rm -f "${MAGIC_FILE}" -if [[ ! ${WSREP_SST_OPT_ROLE} == 'joiner' && ! ${WSREP_SST_OPT_ROLE} == 'donor' ]];then +if [[ ! "${WSREP_SST_OPT_ROLE}" == 'joiner' && ! "${WSREP_SST_OPT_ROLE}" == 'donor' ]];then wsrep_log_error "Invalid role ${WSREP_SST_OPT_ROLE}" exit 22 fi read_cnf -if ${INNOBACKUPEX_BIN} /tmp --help 2>/dev/null | grep -q -- '--version-check'; then +if ${INNOBACKUPEX_BIN} /tmp --help 2>/dev/null | grep -qw -- '--version-check'; then disver='--no-version-check' fi @@ -874,7 +869,7 @@ if [ ${FORCE_FTWRL:-0} -eq 1 ]; then fi if [[ $ssyslog -eq 1 ]];then - if [[ ! -x `which logger` ]];then + if [ ! -x "$(command -v logger)" ]; then wsrep_log_error "logger not in path: $PATH. Ignoring" else wsrep_log_info "Logging all stderr of SST/Innobackupex to syslog" @@ -891,8 +886,8 @@ if [[ $ssyslog -eq 1 ]];then logger -p daemon.info -t ${ssystag}wsrep-sst-$WSREP_SST_OPT_ROLE "$@" } - INNOAPPLY="2>&1 | logger -p daemon.err -t ${ssystag}innobackupex-apply " - INNOMOVE="2>&1 | logger -p daemon.err -t ${ssystag}innobackupex-move " + INNOAPPLY="2>&1 | logger -p daemon.err -t ${ssystag}innobackupex-apply" + INNOMOVE="2>&1 | logger -p daemon.err -t ${ssystag}innobackupex-move" INNOBACKUP="2> >(logger -p daemon.err -t ${ssystag}innobackupex-backup)" fi else @@ -1008,7 +1003,7 @@ then # Add encryption to the head of the stream (if specified) if [[ $encrypt -eq 1 ]]; then - tcmd=" \$ecmd | $tcmd " + tcmd="\$ecmd | $tcmd" fi setup_commands @@ -1182,15 +1177,15 @@ then if [ -n "$qpfiles" ]; then wsrep_log_info "Compressed qpress files found" - if [[ ! -x `which qpress` ]];then + if [ ! -x "$(command -v qpress)" ]; then wsrep_log_error "qpress not found in path: $PATH" exit 22 fi - if [[ -n "$progress" ]] && pv --help | grep -q 'line-mode';then + if [[ -n "$progress" ]] && pv --help | grep -qw '--line-mode';then count=$(find "${DATA}" -type f -name '*.qp' | wc -l) count=$(( count*2 )) - if pv --help | grep -q FORMAT;then + if pv --help | grep -qw -F '-F';then pvopts="-f -s $count -l -N Decompression -F '%N => Rate:%r Elapsed:%t %e Progress: [%b/$count]'" else pvopts="-f -s $count -l -N Decompression" @@ -1269,8 +1264,11 @@ then wsrep_log_error "SST magic file ${MAGIC_FILE} not found/readable" exit 2 fi - wsrep_log_info "Galera co-ords from recovery: $(cat ${MAGIC_FILE})" - cat "${MAGIC_FILE}" # Output : UUID:seqno wsrep_gtid_domain_id + + coords=$(cat "$MAGIC_FILE") + wsrep_log_info "Galera co-ords from recovery: $coords" + cat "$MAGIC_FILE" # Output : UUID:seqno wsrep_gtid_domain_id + wsrep_log_info "Total time on joiner: $totime seconds" fi diff --git a/scripts/wsrep_sst_xtrabackup.sh b/scripts/wsrep_sst_xtrabackup.sh index b8ca7f43ed6..b9fedaadd3a 100644 --- a/scripts/wsrep_sst_xtrabackup.sh +++ b/scripts/wsrep_sst_xtrabackup.sh @@ -16,10 +16,11 @@ # Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston # MA 02110-1335 USA. -# Optional dependencies and options documented here: http://www.percona.com/doc/percona-xtradb-cluster/manual/xtrabackup_sst.html +# Optional dependencies and options documented here: +# http://www.percona.com/doc/percona-xtradb-cluster/manual/xtrabackup_sst.html # Make sure to read that before proceeding! -. $(dirname $0)/wsrep_sst_common +. $(dirname "$0")/wsrep_sst_common wsrep_check_datadir ealgo="" @@ -51,13 +52,13 @@ pvformat="-F '%N => Rate:%r Avg:%a Elapsed:%t %e Bytes: %b %p' " pvopts="-f -i 10 -N $WSREP_SST_OPT_ROLE " uextra=0 -if which pv &>/dev/null && pv --help | grep -q FORMAT;then +if [ -x "$(command -v pv)" ] && pv --help | grep -qw -- '-F'; then pvopts+=$pvformat fi pcmd="pv $pvopts" declare -a RC -INNOBACKUPEX_BIN=innobackupex +INNOBACKUPEX_BIN='innobackupex' DATA="${WSREP_SST_OPT_DATA}" INFO_FILE="xtrabackup_galera_info" IST_FILE="xtrabackup_ist" @@ -67,7 +68,7 @@ MAGIC_FILE="${DATA}/${INFO_FILE}" export PATH="/usr/sbin:/sbin:$PATH" timeit(){ - local stage=$1 + local stage="$1" shift local cmd="$@" local x1 x2 took extcode @@ -102,7 +103,7 @@ get_keys() return fi - if [[ $sfmt == 'tar' ]];then + if [[ "$sfmt" == 'tar' ]];then wsrep_log_info "NOTE: Xtrabackup-based encryption - encrypt=1 - cannot be enabled with tar format" encrypt=0 return @@ -110,17 +111,17 @@ get_keys() wsrep_log_info "Xtrabackup based encryption enabled in my.cnf - Supported only from Xtrabackup 2.1.4" - if [[ -z $ealgo ]];then + if [[ -z "$ealgo" ]];then wsrep_log_error "FATAL: Encryption algorithm empty from my.cnf, bailing out" exit 3 fi - if [[ -z $ekey && ! -r $ekeyfile ]];then + if [[ -z "$ekey" && ! -r "$ekeyfile" ]];then wsrep_log_error "FATAL: Either key or keyfile must be readable" exit 3 fi - if [[ -z $ekey ]];then + if [[ -z "$ekey" ]];then ecmd="xbcrypt --encrypt-algo=$ealgo --encrypt-key-file=$ekeyfile" else ecmd="xbcrypt --encrypt-algo=$ealgo --encrypt-key=$ekey" @@ -136,43 +137,56 @@ get_transfer() TSST_PORT=$WSREP_SST_OPT_PORT if [[ $tfmt == 'nc' ]];then - if [[ ! -x `which nc` ]];then - wsrep_log_error "nc(netcat) not found in path: $PATH" - exit 2 - fi wsrep_log_info "Using netcat as streamer" - if [[ "$WSREP_SST_OPT_ROLE" == "joiner" ]];then - if nc -h 2>&1 | grep -q ncat;then - # Ncat - tcmd="nc -l ${TSST_PORT}" - elif nc -h 2>&1 | grep -q -- '-d\>';then - # Debian netcat - tcmd="nc -dl ${TSST_PORT}" + wsrep_check_programs nc + tcmd="nc" + if [ "$WSREP_SST_OPT_ROLE" = 'joiner' ]; then + if nc -h 2>&1 | grep -q 'ncat'; then + wsrep_log_info "Using Ncat as streamer" + tcmd="$tcmd -l" + elif nc -h 2>&1 | grep -qw -- '-d'; then + wsrep_log_info "Using Debian netcat as streamer" + tcmd="$tcmd -dl" + if [ $WSREP_SST_OPT_HOST_IPv6 -eq 1 ]; then + # When host is not explicitly specified (when only the port + # is specified) netcat can only bind to an IPv4 address if + # the "-6" option is not explicitly specified: + tcmd="$tcmd -6" + fi else - # traditional netcat - tcmd="nc -l -p ${TSST_PORT}" + wsrep_log_info "Using traditional netcat as streamer" + tcmd="$tcmd -l -p" fi + tcmd="$tcmd $TSST_PORT" else - if nc -h 2>&1 | grep -q ncat;then - # Ncat - tcmd="nc ${REMOTEIP} ${TSST_PORT}" - elif nc -h 2>&1 | grep -q -- '-d\>';then - # Debian netcat - tcmd="nc ${REMOTEIP} ${TSST_PORT}" - else - # traditional netcat - tcmd="nc -q0 ${REMOTEIP} ${TSST_PORT}" + # Check to see if netcat supports the '-N' flag. + # -N Shutdown the network socket after EOF on stdin + # If it supports the '-N' flag, then we need to use the '-N' + # flag, otherwise the transfer will stay open after the file + # transfer and cause the command to timeout. + # Older versions of netcat did not need this flag and will + # return an error if the flag is used. + if nc -h 2>&1 | grep -qw -- '-N'; then + tcmd="$tcmd -N" + wsrep_log_info "Using nc -N" fi + # netcat doesn't understand [] around IPv6 address + if nc -h 2>&1 | grep -q ncat; then + wsrep_log_info "Using Ncat as streamer" + elif nc -h 2>&1 | grep -qw -- '-d'; then + wsrep_log_info "Using Debian netcat as streamer" + else + wsrep_log_info "Using traditional netcat as streamer" + tcmd="$tcmd -q0" + fi + tcmd="$tcmd $WSREP_SST_OPT_HOST_UNESCAPED $TSST_PORT" fi else tfmt='socat' wsrep_log_info "Using socat as streamer" - if [[ ! -x `which socat` ]];then - wsrep_log_error "socat not found in path: $PATH" - exit 2 - fi + wsrep_check_programs socat - if [[ $encrypt -eq 2 ]] && ! socat -V | grep -q OPENSSL;then + if [ $encrypt -eq 2 ] && ! socat -V | grep -q -F 'OPENSSL';then wsrep_log_info "NOTE: socat is not openssl enabled, falling back to plain transfer" encrypt=0 fi @@ -202,7 +216,7 @@ get_transfer() get_footprint() { - pushd $WSREP_SST_OPT_DATA 1>/dev/null + pushd "$WSREP_SST_OPT_DATA" 1>/dev/null payload=$(find . -regex '.*\.ibd$\|.*\.MYI$\|.*\.MYD$\|.*ibdata1$' -type f -print0 | du --files0-from=- --block-size=1 -c -s | awk 'END { print $1 }') if $MY_PRINT_DEFAULTS xtrabackup | grep -q -- "--compress";then # QuickLZ has around 50% compression ratio @@ -216,18 +230,18 @@ get_footprint() adjust_progress() { - if [[ -n $progress && $progress != '1' ]];then - if [[ -e $progress ]];then + if [[ -n "$progress" && "$progress" != '1' ]];then + if [[ -e "$progress" ]];then pcmd+=" 2>>$progress" else pcmd+=" 2>$progress" fi - elif [[ -z $progress && -n $rlimit ]];then + elif [[ -z "$progress" && -n "$rlimit" ]];then # When rlimit is non-zero pcmd="pv -q" fi - if [[ -n $rlimit && "$WSREP_SST_OPT_ROLE" == "donor" ]];then + if [[ -n "$rlimit" && "$WSREP_SST_OPT_ROLE" == "donor" ]];then wsrep_log_info "Rate-limiting SST to $rlimit" pcmd+=" -L \$rlimit" fi @@ -304,7 +318,7 @@ cleanup_joiner() wsrep_log_info "Removing the sst_in_progress file" wsrep_cleanup_progress_file fi - if [[ -n $progress && -p $progress ]];then + if [[ -n "$progress" && -p "$progress" ]];then wsrep_log_info "Cleaning up fifo file $progress" rm $progress fi @@ -324,18 +338,18 @@ cleanup_donor() wsrep_log_error "Cleanup after exit with status:$estatus" fi - if [[ -n $XTRABACKUP_PID ]];then - if check_pid $XTRABACKUP_PID + if [[ -n "$XTRABACKUP_PID" ]];then + if check_pid "$XTRABACKUP_PID" then wsrep_log_error "xtrabackup process is still running. Killing... " kill_xtrabackup fi - rm -f $XTRABACKUP_PID + rm -f "$XTRABACKUP_PID" fi - rm -f ${DATA}/${IST_FILE} + rm -f "${DATA}/${IST_FILE}" - if [[ -n $progress && -p $progress ]];then + if [[ -n "$progress" && -p "$progress" ]];then wsrep_log_info "Cleaning up fifo file $progress" rm $progress fi @@ -343,8 +357,8 @@ cleanup_donor() kill_xtrabackup() { - local PID=$(cat $XTRABACKUP_PID) - [ -n "$PID" -a "0" != "$PID" ] && kill $PID && (kill $PID && kill -9 $PID) || : + local PID=$(cat "$XTRABACKUP_PID") + [ -n "$PID" -a $PID -ne 0 ] && kill $PID && (kill $PID && kill -9 $PID) || : rm -f "$XTRABACKUP_PID" } @@ -392,7 +406,7 @@ check_extra() fi } -if [[ ! -x `which innobackupex` ]];then +if [ ! -x "$(command -v innobackupex)" ]; then wsrep_log_error "innobackupex not in path: $PATH" exit 2 fi @@ -507,10 +521,10 @@ then elif [ "${WSREP_SST_OPT_ROLE}" = "joiner" ] then - [[ -e $SST_PROGRESS_FILE ]] && wsrep_log_info "Stale sst_in_progress file: $SST_PROGRESS_FILE" - touch $SST_PROGRESS_FILE + [[ -e "$SST_PROGRESS_FILE" ]] && wsrep_log_info "Stale sst_in_progress file: $SST_PROGRESS_FILE" + touch "$SST_PROGRESS_FILE" - if [[ ! -e ${DATA}/ibdata1 ]];then + if [[ ! -e "${DATA}/ibdata1" ]];then incremental=0 fi @@ -527,11 +541,11 @@ then MODULE="xtrabackup_sst" # May need xtrabackup_checkpoints later on - rm -f ${DATA}/xtrabackup_binary ${DATA}/xtrabackup_galera_info ${DATA}/xtrabackup_logfile + rm -f "${DATA}/xtrabackup_binary" "${DATA}/xtrabackup_galera_info" "${DATA}/xtrabackup_logfile" ADDR="${WSREP_SST_OPT_HOST}:${WSREP_SST_OPT_PORT}" - wait_for_listen ${WSREP_SST_OPT_PORT} ${ADDR} ${MODULE} & + wait_for_listen "${WSREP_SST_OPT_PORT}" "${ADDR}" "${MODULE}" & trap sig_joiner_cleanup HUP PIPE INT TERM trap cleanup_joiner EXIT @@ -543,14 +557,14 @@ then if [[ $incremental -eq 1 ]];then BDATA=$DATA - DATA=$(mktemp -d) + DATA="$(mktemp -d)" MAGIC_FILE="${DATA}/${INFO_FILE}" fi get_keys set +e if [[ $encrypt -eq 1 && $sencrypted -eq 1 ]];then - strmcmd=" $ecmd | $strmcmd" + strmcmd="$ecmd | $strmcmd" fi pushd ${DATA} 1>/dev/null @@ -584,7 +598,7 @@ then # this message should cause joiner to abort wsrep_log_error "xtrabackup process ended without creating '${MAGIC_FILE}'" wsrep_log_info "Contents of datadir" - wsrep_log_info "$(ls -l ${DATA}/**/*)" + wsrep_log_info $(ls -l "${DATA}"/**/*) exit 32 fi @@ -600,9 +614,9 @@ then wsrep_log_info "Removing existing ib_logfile files" if [[ $incremental -ne 1 ]];then - rm -f ${DATA}/ib_logfile* + rm -f "${DATA}/ib_logfile"* else - rm -f ${BDATA}/ib_logfile* + rm -f "${BDATA}/ib_logfile"* fi get_proc @@ -619,19 +633,19 @@ then rebuildcmd="--rebuild-indexes --rebuild-threads=$nthreads" fi - if test -n "$(find ${DATA} -maxdepth 1 -type f -name '*.qp' -print -quit)";then + if test -n $(find "${DATA}" -maxdepth 1 -type f -name '*.qp' -print -quit); then wsrep_log_info "Compressed qpress files found" - if [[ ! -x `which qpress` ]];then + if [ ! -x "$(command -v qpress)" ]; then wsrep_log_error "qpress not found in path: $PATH" exit 22 fi - if [[ -n $progress ]] && pv --help | grep -q 'line-mode';then - count=$(find ${DATA} -type f -name '*.qp' | wc -l) + if [[ -n $progress ]] && pv --help | grep -qw -- '--line-mode';then + count=$(find "${DATA}" -type f -name '*.qp' | wc -l) count=$(( count*2 )) - if pv --help | grep -q FORMAT;then + if pv --help | grep -qw -F '-F';then pvopts="-f -s $count -l -N Decompression -F '%N => Rate:%r Elapsed:%t %e Progress: [%b/$count]'" else pvopts="-f -s $count -l -N Decompression" @@ -644,11 +658,11 @@ then fi wsrep_log_info "Removing existing ibdata1 file" - rm -f ${DATA}/ibdata1 + rm -f "${DATA}/ibdata1" # Decompress the qpress files wsrep_log_info "Decompression with $nproc threads" - timeit "Decompression" "find ${DATA} -type f -name '*.qp' -printf '%p\n%h\n' | $dcmd" + timeit "Decompression" "find '${DATA}' -type f -name '*.qp' -printf '%p\n%h\n' | $dcmd" extcode=$? if [[ $extcode -eq 0 ]];then From 089d82a74be3a96ecaa3e703fb64f1440749d5a6 Mon Sep 17 00:00:00 2001 From: Daniel Bartholomew Date: Mon, 10 May 2021 09:50:56 -0400 Subject: [PATCH 02/15] bump the VERSION --- VERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/VERSION b/VERSION index 4619cee3174..07913307754 100644 --- a/VERSION +++ b/VERSION @@ -1,3 +1,3 @@ MYSQL_VERSION_MAJOR=10 MYSQL_VERSION_MINOR=2 -MYSQL_VERSION_PATCH=38 +MYSQL_VERSION_PATCH=39 From 3cf57aae9f0b0fa3e20b351477fe24b27d86ec8f Mon Sep 17 00:00:00 2001 From: Julius Goryavsky Date: Tue, 11 May 2021 10:04:52 +0200 Subject: [PATCH 03/15] MDEV-23580 addendum: normal operation in configurations where stunnel is not available --- scripts/wsrep_sst_common.sh | 0 scripts/wsrep_sst_rsync.sh | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) mode change 100755 => 100644 scripts/wsrep_sst_common.sh diff --git a/scripts/wsrep_sst_common.sh b/scripts/wsrep_sst_common.sh old mode 100755 new mode 100644 diff --git a/scripts/wsrep_sst_rsync.sh b/scripts/wsrep_sst_rsync.sh index 70e4a3326a1..f32689a9e43 100644 --- a/scripts/wsrep_sst_rsync.sh +++ b/scripts/wsrep_sst_rsync.sh @@ -368,7 +368,7 @@ EOF # first, the normal directories, so that we can detect incompatible protocol RC=0 - eval rsync "'${STUNNEL:+--rsh=$STUNNEL}'" \ + eval rsync ${STUNNEL:+--rsh=\"$STUNNEL\"} \ --owner --group --perms --links --specials \ --ignore-times --inplace --dirs --delete --quiet \ $WHOLE_FILE_OPT $FILTER "'$WSREP_SST_OPT_DATA/'" \ From ec348f555b674f6af083923cbdd1bb0847b35493 Mon Sep 17 00:00:00 2001 From: Sergei Golubchik Date: Tue, 11 May 2021 20:08:40 +0200 Subject: [PATCH 04/15] mtr: --gdb mode, also quote ";", not only " " --- mysql-test/lib/My/Debugger.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mysql-test/lib/My/Debugger.pm b/mysql-test/lib/My/Debugger.pm index 5e04c070e7f..cc151b233d5 100644 --- a/mysql-test/lib/My/Debugger.pm +++ b/mysql-test/lib/My/Debugger.pm @@ -139,7 +139,7 @@ sub do_args($$$$$) { my $v = $debuggers{$k}; # on windows mtr args are quoted (for system), otherwise not (for exec) - sub quote($) { $_[0] =~ / / ? "\"$_[0]\"" : $_[0] } + sub quote($) { $_[0] =~ /[; ]/ ? "\"$_[0]\"" : $_[0] } sub unquote($) { $_[0] =~ s/^"(.*)"$/$1/; $_[0] } sub quote_from_mtr($) { IS_WINDOWS() ? $_[0] : quote($_[0]) } sub unquote_for_mtr($) { IS_WINDOWS() ? $_[0] : unquote($_[0]) } From 3616640a3149b318e0d5602dd39f05e309514dbb Mon Sep 17 00:00:00 2001 From: Andrei Elkin Date: Fri, 17 Jan 2020 20:26:14 +0200 Subject: [PATCH 05/15] MDEV-20821 parallel slave server shutdown hang Parallel slave server shutdown found to be hanging in close_connections() triggered by shutdown due to a slave worker thread would not be notified to exit in case the worker was sitting idle. Fixed with destroying the worker pool earlier that is in slave_prepare_for_shutdown() when all their driver threads have already left. A test file is added to simulate the bug condition as well as check multi-sourced and not-idle worker cases. --- .../rpl/r/rpl_slave_shutdown_mdev20821.result | 79 +++++++++ .../rpl/t/rpl_slave_shutdown_mdev20821.cnf | 19 ++ .../rpl/t/rpl_slave_shutdown_mdev20821.test | 165 ++++++++++++++++++ sql/slave.cc | 3 + 4 files changed, 266 insertions(+) create mode 100644 mysql-test/suite/rpl/r/rpl_slave_shutdown_mdev20821.result create mode 100644 mysql-test/suite/rpl/t/rpl_slave_shutdown_mdev20821.cnf create mode 100644 mysql-test/suite/rpl/t/rpl_slave_shutdown_mdev20821.test diff --git a/mysql-test/suite/rpl/r/rpl_slave_shutdown_mdev20821.result b/mysql-test/suite/rpl/r/rpl_slave_shutdown_mdev20821.result new file mode 100644 index 00000000000..f90d2126103 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_slave_shutdown_mdev20821.result @@ -0,0 +1,79 @@ +include/rpl_init.inc [topology=1->3] +connection server_3; +set default_master_connection = ''; +include/start_slave.inc +Warnings: +Note 1254 Slave is already running +set default_master_connection = 'm2'; +change master to master_host='127.0.0.1', master_port=SERVER_MYPORT_2, master_user='root', master_use_gtid=slave_pos; +include/start_slave.inc +select @@global.slave_parallel_workers as two; +two +2 +connection server_3; +SHUTDOWN; +connection server_3; +connection server_3; +connection server_1; +create table t1 (i int primary key) engine=Innodb; +connection server_2; +create table t2 (i int primary key) engine=Innodb; +connection server_3; +set default_master_connection = ''; +include/start_slave.inc +Warnings: +Note 1254 Slave is already running +set default_master_connection = 'm2'; +include/start_slave.inc +Warnings: +Note 1254 Slave is already running +connection server_2; +insert into t2 values (1); +connection server_3; +connection server_1; +insert into t1 values (1); +connection server_3; +connection server_3; +SHUTDOWN; +connection server_3; +connection server_3; +connection server_3; +set default_master_connection = ''; +include/start_slave.inc +Warnings: +Note 1254 Slave is already running +set default_master_connection = 'm2'; +include/start_slave.inc +Warnings: +Note 1254 Slave is already running +connect conn_block_server3, 127.0.0.1, root,, test, $SERVER_MYPORT_3,; +begin; +insert into t1 values (2); +insert into t2 values (2); +connection server_1; +insert into t1 values (2); +connection server_2; +insert into t2 values (2); +connection server_3; +SHUTDOWN; +connection server_3; +connection server_3; +connection server_3; +set default_master_connection = ''; +include/start_slave.inc +Warnings: +Note 1254 Slave is already running +set default_master_connection = 'm2'; +include/start_slave.inc +Warnings: +Note 1254 Slave is already running +connection server_1; +drop table t1; +connection server_2; +drop table t2; +connection server_3; +set default_master_connection = 'm2'; +include/stop_slave.inc +RESET SLAVE ALL; +set default_master_connection = ''; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_slave_shutdown_mdev20821.cnf b/mysql-test/suite/rpl/t/rpl_slave_shutdown_mdev20821.cnf new file mode 100644 index 00000000000..1e7cdee510b --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_slave_shutdown_mdev20821.cnf @@ -0,0 +1,19 @@ +!include suite/rpl/rpl_1slave_base.cnf +!include include/default_client.cnf + +[mysqld.1] +log-slave-updates +gtid-domain-id=1 + +[mysqld.2] +log-slave-updates +gtid-domain-id=2 + +[mysqld.3] +log-slave-updates +gtid-domain-id=3 +slave_parallel_threads=2 + +[ENV] +SERVER_MYPORT_3= @mysqld.3.port +SERVER_MYSOCK_3= @mysqld.3.socket diff --git a/mysql-test/suite/rpl/t/rpl_slave_shutdown_mdev20821.test b/mysql-test/suite/rpl/t/rpl_slave_shutdown_mdev20821.test new file mode 100644 index 00000000000..563533bb104 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_slave_shutdown_mdev20821.test @@ -0,0 +1,165 @@ +# MDEV-20821 parallel slave server shutdown hang +# +# Test the bug condition of a parallel slave server shutdown +# hang when the parallel workers were idle. +# The bug reported scenario is extented to cover the multi-sources case as well as +# checking is done for both the idle and busy workers cases. + +--source include/have_innodb.inc +--source include/have_binlog_format_mixed.inc +--let $rpl_topology= 1->3 +--source include/rpl_init.inc + +# +# A. idle workers. +# +--connection server_3 +set default_master_connection = ''; +--source include/start_slave.inc + +set default_master_connection = 'm2'; +--replace_result $SERVER_MYPORT_2 SERVER_MYPORT_2 +eval change master to master_host='127.0.0.1', master_port=$SERVER_MYPORT_2, master_user='root', master_use_gtid=slave_pos; +--source include/start_slave.inc + +select @@global.slave_parallel_workers as two; + +# At this point worker threads have no assignement. +# Shutdown must not hang. + +--connection server_3 +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.3.expect +wait +EOF +--send SHUTDOWN +--reap +--source include/wait_until_disconnected.inc + +--connection server_3 +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.3.expect +restart +EOF + +# No hang is *proved* to occur when this point is reached. +--connection server_3 +--enable_reconnect +--source include/wait_until_connected_again.inc + +# +# B. resting workers after some busy time +# +--connection server_1 +create table t1 (i int primary key) engine=Innodb; + +--connection server_2 +create table t2 (i int primary key) engine=Innodb; + +--connection server_3 +set default_master_connection = ''; +--source include/start_slave.inc + +set default_master_connection = 'm2'; +--source include/start_slave.inc + +--connection server_2 +insert into t2 values (1); +--save_master_pos + +--connection server_3 +--sync_with_master 0,'m2' + +--connection server_1 +insert into t1 values (1); +--save_master_pos + +--connection server_3 +--sync_with_master 0,'' + +# At this point worker threads have no assignement. +# Shutdown must not hang. + +--connection server_3 +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.3.expect +wait +EOF +--send SHUTDOWN +--reap +--source include/wait_until_disconnected.inc + +--connection server_3 +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.3.expect +restart +EOF + +# No hang is *proved* to occur when this point is reached. +--connection server_3 +--enable_reconnect +--source include/wait_until_connected_again.inc + +# +# C. busy workers +# +--connection server_3 +set default_master_connection = ''; +--source include/start_slave.inc + +set default_master_connection = 'm2'; +--source include/start_slave.inc + +--connect (conn_block_server3, 127.0.0.1, root,, test, $SERVER_MYPORT_3,) +begin; + insert into t1 values (2); + insert into t2 values (2); + +--connection server_1 +insert into t1 values (2); +--connection server_2 +insert into t2 values (2); + + +# At this point there's a good chance the worker threads are busy. +# SHUTDOWN must proceed without any delay as above. +--connection server_3 +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.3.expect +wait +EOF +--send SHUTDOWN +--reap +--source include/wait_until_disconnected.inc + +--connection server_3 +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.3.expect +restart +EOF + +# No hang is *proved* to occur when this point is reached. +--connection server_3 +--enable_reconnect +--source include/wait_until_connected_again.inc + + +# Cleanup + +--connection server_3 +set default_master_connection = ''; +--source include/start_slave.inc + +set default_master_connection = 'm2'; +--source include/start_slave.inc + +--connection server_1 +drop table t1; + +--connection server_2 +drop table t2; +--save_master_pos + +# (!) The following block is critical to avoid check-mysqld_3.reject by mtr: +--connection server_3 +--sync_with_master 0,'m2' +set default_master_connection = 'm2'; +--source include/stop_slave.inc +RESET SLAVE ALL; +set default_master_connection = ''; + +--source include/rpl_end.inc diff --git a/sql/slave.cc b/sql/slave.cc index 3124b2d10ab..8e26301d926 100644 --- a/sql/slave.cc +++ b/sql/slave.cc @@ -975,6 +975,9 @@ void slave_prepare_for_shutdown() mysql_mutex_lock(&LOCK_active_mi); master_info_index->free_connections(); mysql_mutex_unlock(&LOCK_active_mi); + // It's safe to destruct worker pool now when + // all driver threads are gone. + global_rpl_thread_pool.destroy(); } /* From 355dc74b760414fe03e23929507fadc80029edc9 Mon Sep 17 00:00:00 2001 From: Sachin Kumar Date: Wed, 14 Apr 2021 11:23:38 +0100 Subject: [PATCH 06/15] MDEV-22370 safe_mutex: Trying to lock uninitialized mutex at /data/src/10.4-bug/sql/rpl_parallel.cc, line 470 upon shutdown during FTWRL Problem:- When we issue FTWRL with shutdown in parallel, there is race between FTWRL and shutdown. Shutdown might destroy the mutex (pool->LOCK_rpl_thread_pool) before FTWRL can lock it. So we can get crash on FTWRL thread Solution:- mysql_mutex_destroy(pool->LOCK_rpl_thread_pool) should wait for FTWRL thread to complete its work , and then destroy. So slave_prepare_for_shutdown will just deactivate the pool, and mutex is destroyed later in end_slave() --- mysql-test/r/mdev_22370.result | 4 ++++ mysql-test/t/mdev_22370.test | 17 +++++++++++++++++ sql/rpl_parallel.cc | 15 +++++++++++++++ sql/rpl_parallel.h | 2 ++ sql/slave.cc | 2 +- 5 files changed, 39 insertions(+), 1 deletion(-) create mode 100644 mysql-test/r/mdev_22370.result create mode 100644 mysql-test/t/mdev_22370.test diff --git a/mysql-test/r/mdev_22370.result b/mysql-test/r/mdev_22370.result new file mode 100644 index 00000000000..d422ee6e81f --- /dev/null +++ b/mysql-test/r/mdev_22370.result @@ -0,0 +1,4 @@ +connect con1,localhost,root,,; +SET DEBUG_DBUG='+d,mark_busy_mdev_22370'; +FLUSH TABLES WITH READ LOCK; +connection default; diff --git a/mysql-test/t/mdev_22370.test b/mysql-test/t/mdev_22370.test new file mode 100644 index 00000000000..86bc527ebc0 --- /dev/null +++ b/mysql-test/t/mdev_22370.test @@ -0,0 +1,17 @@ +# +# MDEV-22370 safe_mutex: Trying to lock uninitialized mutex at +# /data/src/10.4-bug/sql/rpl_parallel.cc, line 470 upon shutdown during FTWRL +# +# Purpose of this test case to test crash while FTWRL and shutdown is in race +# condition +# Shutdown can execute first and destroy the mutex making mutex_lock in pool_mark_busy +# to crash + +--source include/have_debug.inc +--connect (con1,localhost,root,,) +SET DEBUG_DBUG='+d,mark_busy_mdev_22370'; +--send + FLUSH TABLES WITH READ LOCK; + +--connection default +--source include/restart_mysqld.inc diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc index 869640fd46f..4503f6ed9be 100644 --- a/sql/rpl_parallel.cc +++ b/sql/rpl_parallel.cc @@ -455,6 +455,7 @@ pool_mark_busy(rpl_parallel_thread_pool *pool, THD *thd) So we protect the infrequent operations of FLUSH TABLES WITH READ LOCK and pool size changes with this condition wait. */ + DBUG_EXECUTE_IF("mark_busy_mdev_22370",my_sleep(1000000);); mysql_mutex_lock(&pool->LOCK_rpl_thread_pool); if (thd) { @@ -2001,10 +2002,24 @@ rpl_parallel_thread_pool::init(uint32 size) void rpl_parallel_thread_pool::destroy() +{ + deactivate(); + destroy_cond_mutex(); +} + +void +rpl_parallel_thread_pool::deactivate() { if (!inited) return; rpl_parallel_change_thread_count(this, 0, 1); +} + +void +rpl_parallel_thread_pool::destroy_cond_mutex() +{ + if (!inited) + return; mysql_mutex_destroy(&LOCK_rpl_thread_pool); mysql_cond_destroy(&COND_rpl_thread_pool); inited= false; diff --git a/sql/rpl_parallel.h b/sql/rpl_parallel.h index a0faeae815c..fba4c7b3fd7 100644 --- a/sql/rpl_parallel.h +++ b/sql/rpl_parallel.h @@ -240,6 +240,8 @@ struct rpl_parallel_thread_pool { rpl_parallel_thread_pool(); int init(uint32 size); void destroy(); + void deactivate(); + void destroy_cond_mutex(); struct rpl_parallel_thread *get_thread(rpl_parallel_thread **owner, rpl_parallel_entry *entry); void release_thread(rpl_parallel_thread *rpt); diff --git a/sql/slave.cc b/sql/slave.cc index 8e26301d926..761fdbe807a 100644 --- a/sql/slave.cc +++ b/sql/slave.cc @@ -977,7 +977,7 @@ void slave_prepare_for_shutdown() mysql_mutex_unlock(&LOCK_active_mi); // It's safe to destruct worker pool now when // all driver threads are gone. - global_rpl_thread_pool.destroy(); + global_rpl_thread_pool.deactivate(); } /* From e607f3398c69147299884d3814cf063d2e7516ce Mon Sep 17 00:00:00 2001 From: Sachin Kumar Date: Wed, 14 Apr 2021 10:56:12 +0100 Subject: [PATCH 07/15] MDEV-25336 Parallel replication causes failed assert while restarting Problem:- When slave is shutdown, we will get this assertion failure sql/sql_list.h:642: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed. Solution:- In close_connections when we call threads.get() it resets to prev and next to NULL. And in parallel worker thread(handle_rpl_parallel_thread) calls unlink_not_visible_thd() which assert on prev and next being not NULL. .unlink_not_visible_thd() should be always called first before threads.get() is called. To make sure worker calls unlink_not_visible_thd() in slave_prepare_for_shutdown() we are deactivating the worker thread pool which in turn will close all worker threads. Since this is already done in 10.4 and 10.5 I am backPorting MDEV-20821 and MDEV-22370 to 10.2. Mdev-22370 is improving the MDEV-20821 patch. --- mysql-test/suite/rpl/t/rpl_slave_shutdown_mdev20821.test | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/mysql-test/suite/rpl/t/rpl_slave_shutdown_mdev20821.test b/mysql-test/suite/rpl/t/rpl_slave_shutdown_mdev20821.test index 563533bb104..6f73de984d3 100644 --- a/mysql-test/suite/rpl/t/rpl_slave_shutdown_mdev20821.test +++ b/mysql-test/suite/rpl/t/rpl_slave_shutdown_mdev20821.test @@ -4,6 +4,10 @@ # hang when the parallel workers were idle. # The bug reported scenario is extented to cover the multi-sources case as well as # checking is done for both the idle and busy workers cases. +# +# MDEV-25336 Parallel replication causes failed assert while restarting +# Since this test case involves slave restart this will help in testing +# Mdev-25336 too. --source include/have_innodb.inc --source include/have_binlog_format_mixed.inc @@ -26,7 +30,7 @@ select @@global.slave_parallel_workers as two; # At this point worker threads have no assignement. # Shutdown must not hang. - +# In 10.2/10.3 there should not be any assert failure `prev != 0 && next != 0' --connection server_3 --write_file $MYSQLTEST_VARDIR/tmp/mysqld.3.expect wait @@ -75,6 +79,7 @@ insert into t1 values (1); --connection server_3 --sync_with_master 0,'' +# In 10.2/10.3 there should not be any assert failure `prev != 0 && next != 0' # At this point worker threads have no assignement. # Shutdown must not hang. @@ -117,6 +122,7 @@ insert into t1 values (2); insert into t2 values (2); +# In 10.2/10.3 there should not be any assert failure `prev != 0 && next != 0' # At this point there's a good chance the worker threads are busy. # SHUTDOWN must proceed without any delay as above. --connection server_3 From 677f1ef6f00793b3ad2a42b4e6f0fcbb7cd0e39d Mon Sep 17 00:00:00 2001 From: Igor Babaev Date: Fri, 14 May 2021 16:43:36 -0700 Subject: [PATCH 08/15] MDEV-25682 Explain shows an execution plan different from actually executed If a select query contained an ORDER BY clause that followed a LIMIT clause or an ORDER BY clause or ORDER BY with LIMIT the EXPLAIN output for the query showed an execution plan different from that was actually executed. Approved by Roman Nozdrin --- mysql-test/r/order_by.result | 25 +++++++++++++++++++++++++ mysql-test/t/order_by.test | 16 ++++++++++++++++ sql/sql_select.cc | 2 +- 3 files changed, 42 insertions(+), 1 deletion(-) diff --git a/mysql-test/r/order_by.result b/mysql-test/r/order_by.result index b1441013bc8..39b4e25d670 100644 --- a/mysql-test/r/order_by.result +++ b/mysql-test/r/order_by.result @@ -3460,4 +3460,29 @@ SET max_length_for_sort_data=@save_max_length_for_sort_data; SET max_sort_length= @save_max_sort_length; SET sql_select_limit= @save_sql_select_limit; DROP TABLE t1; +# +# MDEV-25682: EXPLAIN for SELECT with ORDER BY after [ORDER BY] LIMIT +# +create table t1 (a int); +insert into t1 values (3), (7), (1); +explain (select a from t1 limit 2) order by a desc; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ALL NULL NULL NULL NULL 3 +NULL UNION RESULT ALL NULL NULL NULL NULL NULL Using filesort +(select a from t1 limit 2) order by a desc; +a +7 +3 +create table t2 (a int, b int); +insert into t2 values (3,70), (7,10), (1,40), (4,30); +explain (select b,a from t2 order by a limit 3) order by b desc; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ALL NULL NULL NULL NULL 4 Using filesort +NULL UNION RESULT ALL NULL NULL NULL NULL NULL Using filesort +(select b,a from t2 order by a limit 3) order by b desc; +b a +70 3 +40 1 +30 4 +drop table t1,t2; # End of 10.2 tests diff --git a/mysql-test/t/order_by.test b/mysql-test/t/order_by.test index 36c25ed37fb..4e50fc5b3c6 100644 --- a/mysql-test/t/order_by.test +++ b/mysql-test/t/order_by.test @@ -2293,4 +2293,20 @@ SET max_sort_length= @save_max_sort_length; SET sql_select_limit= @save_sql_select_limit; DROP TABLE t1; +--echo # +--echo # MDEV-25682: EXPLAIN for SELECT with ORDER BY after [ORDER BY] LIMIT +--echo # + +create table t1 (a int); +insert into t1 values (3), (7), (1); +explain (select a from t1 limit 2) order by a desc; +(select a from t1 limit 2) order by a desc; + +create table t2 (a int, b int); +insert into t2 values (3,70), (7,10), (1,40), (4,30); +explain (select b,a from t2 order by a limit 3) order by b desc; +(select b,a from t2 order by a limit 3) order by b desc; + +drop table t1,t2; + --echo # End of 10.2 tests diff --git a/sql/sql_select.cc b/sql/sql_select.cc index b85bd31e23c..ce706209017 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -25332,7 +25332,7 @@ bool mysql_explain_union(THD *thd, SELECT_LEX_UNIT *unit, select_result *result) sl->options|= SELECT_DESCRIBE; } - if (unit->is_union()) + if (unit->is_union() || unit->fake_select_lex) { if (unit->union_needs_tmp_table() && unit->fake_select_lex) { From 4675febb7a49b269eff408f52d8f4d9c5569e079 Mon Sep 17 00:00:00 2001 From: Julius Goryavsky Date: Thu, 13 May 2021 12:23:11 +0200 Subject: [PATCH 09/15] Added missing connection lines to some tests --- .../r/galera_join_with_cc_A.result | 39 +++++++++++++++++ .../r/galera_join_with_cc_B.result | 39 +++++++++++++++++ .../r/galera_join_with_cc_C.result | 42 +++++++++++++++++++ 3 files changed, 120 insertions(+) diff --git a/mysql-test/suite/galera_3nodes/r/galera_join_with_cc_A.result b/mysql-test/suite/galera_3nodes/r/galera_join_with_cc_A.result index 0461f1f1feb..a7234aa9778 100644 --- a/mysql-test/suite/galera_3nodes/r/galera_join_with_cc_A.result +++ b/mysql-test/suite/galera_3nodes/r/galera_join_with_cc_A.result @@ -1,40 +1,79 @@ +connection node_1; CREATE TABLE t1 (pk INT PRIMARY KEY, node INT) ENGINE=innodb; INSERT INTO t1 VALUES (1, 1); +connection node_2; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=1'; +connection node_2; +connection node_3; INSERT INTO t1 VALUES (2, 3); +connection node_2; +connect node_1a, 127.0.0.1, root, , test, $NODE_MYPORT_1; SET wsrep_sync_wait = 0; SET wsrep_on = OFF; SET GLOBAL wsrep_provider_options = 'dbug=d,after_shift_to_joining'; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=0'; +connection node_2; INSERT INTO t1 VALUES (3, 2); +connection node_1a; SET SESSION wsrep_on = 0; SET SESSION wsrep_on = 0; +connection node_3; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=1'; +connection node_2; +connection node_3; INSERT INTO t1 VALUES (4, 3); +connection node_2; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=0'; +connection node_2; INSERT INTO t1 VALUES (5, 2); +connection node_3; +connection node_1a; SET GLOBAL wsrep_provider_options = 'dbug=d,before_send_state_request'; SET GLOBAL wsrep_provider_options = 'signal=after_shift_to_joining'; SET SESSION wsrep_on = 0; SET SESSION wsrep_on = 0; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=1'; +connection node_2; +connection node_3; INSERT INTO t1 VALUES (6, 3); +connection node_2; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=0'; +connection node_2; INSERT INTO t1 VALUES (7, 2); +connection node_3; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=1'; +connection node_2; +connection node_3; INSERT INTO t1 VALUES (8, 3); +connection node_2; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=0'; +connection node_2; +connection node_1a; SET GLOBAL wsrep_provider_options = 'dbug=d,process_primary_configuration'; SET GLOBAL wsrep_provider_options = 'signal=before_send_state_request'; SET SESSION wsrep_on = 0; SET SESSION wsrep_on = 0; +connection node_2; INSERT INTO t1 VALUES (9, 2); +connection node_3; +connection node_1a; SET GLOBAL wsrep_provider_options = 'signal=process_primary_configuration'; SET SESSION wsrep_on = 0; SET SESSION wsrep_on = 0; SET GLOBAL wsrep_provider_options = 'signal=process_primary_configuration'; SET GLOBAL wsrep_provider_options = 'dbug='; +connection node_1; DROP TABLE t1; call mtr.add_suppression("WSREP: Rejecting JOIN message from \(.*\): new State Transfer required."); +connection node_2; call mtr.add_suppression("WSREP: Rejecting JOIN message from \(.*\): new State Transfer required."); +connection node_3; call mtr.add_suppression("WSREP: Rejecting JOIN message from \(.*\): new State Transfer required."); diff --git a/mysql-test/suite/galera_3nodes/r/galera_join_with_cc_B.result b/mysql-test/suite/galera_3nodes/r/galera_join_with_cc_B.result index d878f60ca6b..e3ca22ef01e 100644 --- a/mysql-test/suite/galera_3nodes/r/galera_join_with_cc_B.result +++ b/mysql-test/suite/galera_3nodes/r/galera_join_with_cc_B.result @@ -1,18 +1,37 @@ +connection node_1; CREATE TABLE t1 (pk INT PRIMARY KEY, node INT) ENGINE=innodb; INSERT INTO t1 VALUES (1, 1); +connection node_2; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=1'; +connection node_2; +connection node_3; INSERT INTO t1 VALUES (2, 3); +connection node_2; +connect node_1a, 127.0.0.1, root, , test, $NODE_MYPORT_1; SET wsrep_sync_wait = 0; SET wsrep_on = OFF; SET GLOBAL wsrep_provider_options = 'dbug=d,after_shift_to_joining'; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=0'; +connection node_2; INSERT INTO t1 VALUES (3, 2); +connection node_1a; SET SESSION wsrep_on = 0; SET SESSION wsrep_on = 0; +connection node_3; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=1'; +connection node_2; +connection node_3; INSERT INTO t1 VALUES (4, 3); +connection node_2; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=0'; +connection node_2; INSERT INTO t1 VALUES (5, 2); +connection node_3; +connection node_1a; SET GLOBAL wsrep_provider_options = 'dbug=d,before_send_state_request'; SET GLOBAL wsrep_provider_options = 'signal=after_shift_to_joining'; SET SESSION wsrep_on = 0; @@ -24,18 +43,35 @@ SET SESSION wsrep_on = 0; SELECT * FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_debug_sync_waiters'; VARIABLE_NAME VARIABLE_VALUE WSREP_DEBUG_SYNC_WAITERS after_shift_to_joining +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=1'; +connection node_2; +connection node_3; INSERT INTO t1 VALUES (6, 3); +connection node_2; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=0'; +connection node_2; INSERT INTO t1 VALUES (7, 2); +connection node_3; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=1'; +connection node_2; +connection node_3; INSERT INTO t1 VALUES (8, 3); +connection node_2; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=0'; +connection node_2; +connection node_1a; SET GLOBAL wsrep_provider_options = 'dbug=d,process_primary_configuration'; SET GLOBAL wsrep_provider_options = 'signal=after_shift_to_joining'; SET SESSION wsrep_on = 0; SET SESSION wsrep_on = 0; +connection node_2; INSERT INTO t1 VALUES (9, 2); +connection node_3; +connection node_1a; SET GLOBAL wsrep_provider_options = 'signal=process_primary_configuration'; SET SESSION wsrep_on = 0; SET SESSION wsrep_on = 0; @@ -44,7 +80,10 @@ VARIABLE_NAME VARIABLE_VALUE WSREP_DEBUG_SYNC_WAITERS process_primary_configuration SET GLOBAL wsrep_provider_options = 'signal=process_primary_configuration'; SET GLOBAL wsrep_provider_options = 'dbug='; +connection node_1; DROP TABLE t1; call mtr.add_suppression("WSREP: Rejecting JOIN message from \(.*\): new State Transfer required."); +connection node_2; call mtr.add_suppression("WSREP: Rejecting JOIN message from \(.*\): new State Transfer required."); +connection node_3; call mtr.add_suppression("WSREP: Rejecting JOIN message from \(.*\): new State Transfer required."); diff --git a/mysql-test/suite/galera_3nodes/r/galera_join_with_cc_C.result b/mysql-test/suite/galera_3nodes/r/galera_join_with_cc_C.result index df0a924029c..c37b8837900 100644 --- a/mysql-test/suite/galera_3nodes/r/galera_join_with_cc_C.result +++ b/mysql-test/suite/galera_3nodes/r/galera_join_with_cc_C.result @@ -1,44 +1,83 @@ +connection node_1; CREATE TABLE t1 (pk INT PRIMARY KEY, node INT) ENGINE=innodb; INSERT INTO t1 VALUES (1, 1); +connection node_2; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=1'; +connection node_2; +connection node_3; INSERT INTO t1 VALUES (2, 3); +connection node_2; +connect node_1a, 127.0.0.1, root, , test, $NODE_MYPORT_1; SET wsrep_sync_wait = 0; SET wsrep_on = OFF; SET GLOBAL wsrep_provider_options = 'dbug=d,after_shift_to_joining'; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=0'; +connection node_2; INSERT INTO t1 VALUES (3, 2); +connection node_1a; SET SESSION wsrep_on = 0; SET SESSION wsrep_on = 0; +connection node_3; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=1'; +connection node_2; +connection node_3; INSERT INTO t1 VALUES (4, 3); +connection node_2; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=0'; +connection node_2; INSERT INTO t1 VALUES (5, 2); +connection node_3; +connection node_1a; SET GLOBAL wsrep_provider_options = 'dbug=d,before_send_state_request'; SET GLOBAL wsrep_provider_options = 'signal=after_shift_to_joining'; SET SESSION wsrep_on = 0; SET SESSION wsrep_on = 0; SET GLOBAL wsrep_provider_options = 'dbug='; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=1'; +connection node_2; +connection node_3; +connection node_1a; SET GLOBAL wsrep_provider_options = 'dbug=d,after_shift_to_joining'; SET GLOBAL wsrep_provider_options = 'signal=before_send_state_request'; 4 SELECT * FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_debug_sync_waiters'; VARIABLE_NAME VARIABLE_VALUE WSREP_DEBUG_SYNC_WAITERS +connection node_3; INSERT INTO t1 VALUES (6, 3); +connection node_2; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=0'; +connection node_2; INSERT INTO t1 VALUES (7, 2); +connection node_3; +connection node_1a; SET SESSION wsrep_on = 0; SET SESSION wsrep_on = 0; SET GLOBAL wsrep_provider_options = 'dbug='; SET GLOBAL wsrep_provider_options = 'dbug=d,process_primary_configuration'; SET GLOBAL wsrep_provider_options = 'signal=after_shift_to_joining'; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=1'; +connection node_2; +connection node_3; INSERT INTO t1 VALUES (8, 3); +connection node_2; +connection node_1; SET GLOBAL wsrep_provider_options='gmcast.isolate=0'; +connection node_2; +connection node_1a; SET SESSION wsrep_on = 0; SET SESSION wsrep_on = 0; +connection node_2; INSERT INTO t1 VALUES (9, 2); +connection node_3; +connection node_1a; SET GLOBAL wsrep_provider_options = 'signal=process_primary_configuration'; SET SESSION wsrep_on = 0; SET SESSION wsrep_on = 0; @@ -48,8 +87,11 @@ SET SESSION wsrep_on = 0; SET SESSION wsrep_on = 0; SET GLOBAL wsrep_provider_options = 'dbug='; SET GLOBAL wsrep_provider_options = 'signal=after_shift_to_joining'; +connection node_1; DROP TABLE t1; call mtr.add_suppression("WSREP: Send action {\(.*\), STATE_REQUEST} returned -107 \\(Transport endpoint is not connected\\)"); call mtr.add_suppression("WSREP: Rejecting JOIN message from \(.*\): new State Transfer required."); +connection node_2; call mtr.add_suppression("WSREP: Rejecting JOIN message from \(.*\): new State Transfer required."); +connection node_3; call mtr.add_suppression("WSREP: Rejecting JOIN message from \(.*\): new State Transfer required."); From 6811ed3e106027e1a721027db9b9804c824dfd0c Mon Sep 17 00:00:00 2001 From: Julius Goryavsky Date: Fri, 14 May 2021 12:51:36 +0200 Subject: [PATCH 10/15] MDEV-25669: SST scripts should check all server groups in config files 1) This commit implements reading all sections from configuration files while looking for the current value of any server variable, which were previously only read from the [mysqld.suffix] group and from [mysqld], but not from other groups such as [mariadb.suffix], [mariadb] or, for example, [server]. 2) This commit also fixes misrecognition of some parameters when parsing a command line containing a special marker for the end of the list of options ("--") or when short option names (such as "-s", "-a" and "-h arg") chained together (like a "-sah arg"). Such parameters can be passed to the SST script in the list of arguments after "--mysqld-args" if the server is started with a complex set of options - this was revealed during manual testing of changes to read configuration files. 3) The server-side preparation code for the "--mysqld-args" option list has also been simplified to make it easier to change in the future (if needed), and has been improved to properly handle the special backquote ("`") character in the argument values. --- scripts/wsrep_sst_common.sh | 159 +++++++++++++++------- scripts/wsrep_sst_mariabackup.sh | 14 +- scripts/wsrep_sst_rsync.sh | 20 ++- scripts/wsrep_sst_xtrabackup-v2.sh | 21 ++- scripts/wsrep_sst_xtrabackup.sh | 6 +- sql/wsrep_sst.cc | 209 +++++++++++------------------ 6 files changed, 217 insertions(+), 212 deletions(-) mode change 100644 => 100755 scripts/wsrep_sst_common.sh diff --git a/scripts/wsrep_sst_common.sh b/scripts/wsrep_sst_common.sh old mode 100644 new mode 100755 index d19a0dbfdd5..c98b388a1e2 --- a/scripts/wsrep_sst_common.sh +++ b/scripts/wsrep_sst_common.sh @@ -240,44 +240,108 @@ case "$1" in original_cmd="" shift while [ $# -gt 0 ]; do - # check if the argument is the short option - # (starting with "-" instead of "--"): - if [ "${1#--}" = "$1" -a "${1#-}" != "$1" ]; then - option="${1#-}" - value="" - # check that the option value follows the name, - # without a space: - if [ ${#option} -gt 1 ]; then - # let's separate the first character as the option name, - # and the subsequent characters consider its value: - value="${1#-?}" - option="${1%$value}" - # check that the option name consists of one letter - # and there are the following arguments: - elif [ ${#option} -eq 1 -a $# -gt 1 ]; then - # if the next argument does not start with a "-" character, - # then this is the value of the current option: - if [ "${2#-}" = "$2" ]; then - value="$2" + lname="${1#--}" + # "--" is interpreted as the end of the list of options: + if [ -z "$lname" ]; then + shift + if [ $# -gt 0 ]; then + # copy "--" to the output string: + original_cmd="$original_cmd --" + # All other arguments must be copied unchanged: + while [ $# -gt 0 ]; do + original_cmd="$original_cmd '$1'" shift + done + fi + break; + fi + # Make sure the argument does not start with "--", otherwise it + # is a long option, which is processed after this "if": + if [ "$lname" = "$1" ]; then + # Check if the argument is the short option or the short + # options list, starting with "-": + options="${1#-}" + if [ "$options" != "$1" -a -n "$options" ]; then + slist="" + while [ -n "$options" ]; do + # Let's separate the first character as the current + # option name: + if [ -n "$BASH_VERSION" ]; then + option="${options:0:1}" + else + # If it's not bash, then we need to use slow + # external utilities: + option=$(echo "$options" | cut -c1-1) + fi + # And the subsequent characters consider option value: + value="" + if [ ${#options} -gt 0 ]; then + value="${options#?}" + fi + # Check for options without argument: + if [ "$option" != '?' -a \ + "$option" != 'a' -a \ + "$option" != 's' -a \ + "$option" != 'v' ] + then + # If the option value is absent, then check + # the following argument: + if [ -z "$value" -a $# -gt 1 ]; then + # if the next argument does not start with + # the "-" character, then next argument is + # the current option value: + if [ "${2#-}" = "$2" ]; then + shift + value="$1" + fi + fi + if [ $option == 'h' ]; then + if [ -z "$WSREP_SST_OPT_DATA" ]; then + MYSQLD_OPT_DATADIR="${value%/}" + fi + elif [ $option != 'u' -a \ + $option != 'P' ] + then + if [ -z "$value" ]; then + slist="$slist$option" + elif [ -z "$slist" ]; then + slist="$option '$value'" + else + slist="$slist -$option '$value'" + fi + fi + break + + else + slist="$slist$option" + fi + options="$value" + done + if [ -n "$slist" ]; then + original_cmd="$original_cmd -$slist" fi + elif [ -z "$options" ]; then + # We found an equal sign without any characters after it: + original_cmd="$original_cmd -" + else + # We found a value that does not start with a minus - + # it is a positional argument or the value of previous + # option. Copy it to output string (as is): + original_cmd="$original_cmd '$1'" fi shift - if [ "$option" = 'h' ]; then - if [ -z "$WSREP_SST_OPT_DATA" ]; then - MYSQLD_OPT_DATADIR="${value%/}" - fi - elif [ "$option" != 'u' -a \ - "$option" != 'P' ]; then - if [ -z "$original_cmd" ]; then - original_cmd="'-$option$value'" - else - original_cmd="$original_cmd '-$option$value'" - fi - fi continue; fi + # Now we are sure that we are working with an option + # that has a "long" name, so remove all characters after + # the first equal sign: option="${1%%=*}" + # The "--loose-" prefix should not affect the recognition + # of the option name: + if [ "${option#--loose-}" != "$option" ]; then + option="--${option#--loose-}" + fi + # Some options just need to be removed from the list: if [ "$option" != '--defaults-file' -a \ "$option" != '--defaults-extra-file' -a \ "$option" != '--defaults-group-suffix' -a \ @@ -340,22 +404,17 @@ case "$1" in ;; esac if [ $skip_mysqld_arg -eq 0 ]; then - if [ -z "$original_cmd" ]; then - original_cmd="'$1'" - else - original_cmd="$original_cmd '$1'" - fi + original_cmd="$original_cmd '$1'" fi - fi - shift + fi + shift done - WSREP_SST_OPT_MYSQLD="$original_cmd" + WSREP_SST_OPT_MYSQLD="${original_cmd# *}" break ;; - *) # must be command - # usage - # exit 1 - ;; + *) # Must be command usage + # exit 1 + ;; esac shift done @@ -601,9 +660,9 @@ parse_cnf() # of the groups list (as if it were a prefix): groups="${groups#$group}" groups="${groups#\|}" - # if the group name is the same as the "[--]mysqld", then - # try to use it together with the group suffix: - if [ "${group#--}" = 'mysqld' -a -n "$WSREP_SST_OPT_SUFFIX_VALUE" ]; then + # If the group name is the same as the "mysqld" without "--" prefix, + # then try to use it together with the group suffix: + if [ "$group" = 'mysqld' -a -n "$WSREP_SST_OPT_SUFFIX_VALUE" ]; then reval=$($MY_PRINT_DEFAULTS "mysqld$WSREP_SST_OPT_SUFFIX_VALUE" | awk "$pattern") if [ -n "$reval" ]; then break @@ -616,7 +675,7 @@ parse_cnf() fi done - # use default if we haven't found a value: + # Use default if we haven't found a value: if [ -z "$reval" ]; then [ -n "${3:-}" ] && reval="$3" fi @@ -648,9 +707,9 @@ in_config() # of the groups list (as if it were a prefix): groups="${groups#$group}" groups="${groups#\|}" - # if the group name is the same as the "[--]mysqld", then - # try to use it together with the group suffix: - if [ "${group#--}" = 'mysqld' -a -n "$WSREP_SST_OPT_SUFFIX_VALUE" ]; then + # If the group name is the same as the "mysqld" without "--" prefix, + # then try to use it together with the group suffix: + if [ "$group" = 'mysqld' -a -n "$WSREP_SST_OPT_SUFFIX_VALUE" ]; then found=$($MY_PRINT_DEFAULTS "mysqld$WSREP_SST_OPT_SUFFIX_VALUE" | awk "$pattern") if [ $found -ne 0 ]; then break diff --git a/scripts/wsrep_sst_mariabackup.sh b/scripts/wsrep_sst_mariabackup.sh index de789dc1728..899f3eb4f3c 100644 --- a/scripts/wsrep_sst_mariabackup.sh +++ b/scripts/wsrep_sst_mariabackup.sh @@ -404,7 +404,6 @@ read_cnf() # avoid CA verification if not set explicitly: # nodes may happen to have different CA if self-generated # zeroing up tcert does the trick - local mode=$(parse_cnf 'sst' 'ssl-mode') [ "${tmode#VERIFY}" != "$tmode" ] || tcert="" fi fi @@ -421,8 +420,9 @@ read_cnf() sockopt=$(parse_cnf sst sockopt "") progress=$(parse_cnf sst progress "") ttime=$(parse_cnf sst time 0) - cpat=$(parse_cnf sst cpat '.*galera\.cache$\|.*sst_in_progress$\|.*\.sst$\|.*gvwstate\.dat$\|.*grastate\.dat$\|.*\.err$\|.*\.log$\|.*RPM_UPGRADE_MARKER$\|.*RPM_UPGRADE_HISTORY$') - [ $OS = 'FreeBSD' ] && cpat=$(parse_cnf sst cpat '.*galera\.cache$|.*sst_in_progress$|.*\.sst$|.*gvwstate\.dat$|.*grastate\.dat$|.*\.err$|.*\.log$|.*RPM_UPGRADE_MARKER$|.*RPM_UPGRADE_HISTORY$') + cpat='.*galera\.cache$\|.*sst_in_progress$\|.*\.sst$\|.*gvwstate\.dat$\|.*grastate\.dat$\|.*\.err$\|.*\.log$\|.*RPM_UPGRADE_MARKER$\|.*RPM_UPGRADE_HISTORY$' + [ "$OS" = 'FreeBSD' ] && cpat=$(echo "$cpat" | sed 's/\\|/|/g') + cpat=$(parse_cnf sst cpat "$cpat") scomp=$(parse_cnf sst compressor "") sdecomp=$(parse_cnf sst decompressor "") @@ -445,9 +445,7 @@ read_cnf() fi if [ $ssyslog -ne -1 ]; then - if $MY_PRINT_DEFAULTS mysqld_safe | grep -q -- "--syslog"; then - ssyslog=1 - fi + ssyslog=$(in_config 'mysqld_safe' 'syslog') fi } @@ -771,7 +769,7 @@ monitor_process() while true ; do if ! ps -p "$WSREP_SST_OPT_PARENT" &>/dev/null; then - wsrep_log_error "Parent mysqld process (PID:${WSREP_SST_OPT_PARENT}) terminated unexpectedly." + wsrep_log_error "Parent mysqld process (PID: $WSREP_SST_OPT_PARENT) terminated unexpectedly." exit 32 fi if ! ps -p "$sst_stream_pid" &>/dev/null; then @@ -1139,7 +1137,7 @@ then if ! ps -p "$WSREP_SST_OPT_PARENT" &>/dev/null then - wsrep_log_error "Parent mysqld process (PID:${WSREP_SST_OPT_PARENT}) terminated unexpectedly." + wsrep_log_error "Parent mysqld process (PID: $WSREP_SST_OPT_PARENT) terminated unexpectedly." exit 32 fi diff --git a/scripts/wsrep_sst_rsync.sh b/scripts/wsrep_sst_rsync.sh index f32689a9e43..4f39835e15d 100644 --- a/scripts/wsrep_sst_rsync.sh +++ b/scripts/wsrep_sst_rsync.sh @@ -218,23 +218,21 @@ SSTKEY=$(parse_cnf 'sst' 'tkey') SSTCERT=$(parse_cnf 'sst' 'tcert') SSTCA=$(parse_cnf 'sst' 'tca') +SST_SECTIONS="--mysqld|sst" + check_server_ssl_config() { - local section="$1" - SSTKEY=$(parse_cnf "$section" 'ssl-key') - SSTCERT=$(parse_cnf "$section" 'ssl-cert') - SSTCA=$(parse_cnf "$section" 'ssl-ca') + SSTKEY=$(parse_cnf "$SST_SECTIONS" 'ssl-key') + SSTCERT=$(parse_cnf "$SST_SECTIONS" 'ssl-cert') + SSTCA=$(parse_cnf "$SST_SECTIONS" 'ssl-ca') } -SSLMODE=$(parse_cnf 'sst' 'ssl-mode' | tr [:lower:] [:upper:]) +SSLMODE=$(parse_cnf "$SST_SECTIONS" 'ssl-mode' | tr [:lower:] [:upper:]) +# no old-style SSL config in [sst], check for new one: if [ -z "$SSTKEY" -a -z "$SSTCERT" -a -z "$SSTCA" ] then - # no old-style SSL config in [sst], check for new one - check_server_ssl_config 'sst' - if [ -z "$SSTKEY" -a -z "$SSTCERT" -a -z "$SSTCA" ]; then - check_server_ssl_config '--mysqld' - fi + check_server_ssl_config fi if [ -z "$SSLMODE" ]; then @@ -602,7 +600,7 @@ EOF if ! ps -p $MYSQLD_PID >/dev/null then wsrep_log_error \ - "Parent mysqld process (PID:$MYSQLD_PID) terminated unexpectedly." + "Parent mysqld process (PID: $MYSQLD_PID) terminated unexpectedly." kill -- -$MYSQLD_PID sleep 1 exit 32 diff --git a/scripts/wsrep_sst_xtrabackup-v2.sh b/scripts/wsrep_sst_xtrabackup-v2.sh index 7718e52184a..6f26cd3e287 100644 --- a/scripts/wsrep_sst_xtrabackup-v2.sh +++ b/scripts/wsrep_sst_xtrabackup-v2.sh @@ -470,11 +470,8 @@ read_cnf() progress=$(parse_cnf sst progress "") rebuild=$(parse_cnf sst rebuild 0) ttime=$(parse_cnf sst time 0) - if [ "${OS}" = "FreeBSD" ]; then - cpat=$(parse_cnf sst cpat '.*\.pem$|.*init\.ok$|.*galera\.cache$|.*sst_in_progress$|.*\.sst$|.*gvwstate\.dat$|.*grastate\.dat$|.*\.err$|.*\.log$|.*RPM_UPGRADE_MARKER$|.*RPM_UPGRADE_HISTORY$') - else - cpat=$(parse_cnf sst cpat '.*\.pem$\|.*init\.ok$\|.*galera\.cache$\|.*sst_in_progress$\|.*\.sst$\|.*gvwstate\.dat$\|.*grastate\.dat$\|.*\.err$\|.*\.log$\|.*RPM_UPGRADE_MARKER$\|.*RPM_UPGRADE_HISTORY$') - fi + cpat='.*\.pem$\|.*init\.ok$\|.*galera\.cache$\|.*sst_in_progress$\|.*\.sst$\|.*gvwstate\.dat$\|.*grastate\.dat$\|.*\.err$\|.*\.log$\|.*RPM_UPGRADE_MARKER$\|.*RPM_UPGRADE_HISTORY$' + [ "$OS" = 'FreeBSD' ] && cpat=$(echo "$cpat" | sed 's/\\|/|/g') ealgo=$(parse_cnf xtrabackup encrypt "") ekey=$(parse_cnf xtrabackup encrypt-key "") ekeyfile=$(parse_cnf xtrabackup encrypt-key-file "") @@ -512,10 +509,8 @@ read_cnf() ssystag=$(parse_cnf mysqld_safe syslog-tag "${SST_SYSLOG_TAG:-}") ssystag+="-" - if [[ $ssyslog -ne -1 ]];then - if $MY_PRINT_DEFAULTS mysqld_safe | grep -q -- "--syslog";then - ssyslog=1 - fi + if [ $ssyslog -ne -1 ]; then + ssyslog=$(in_config 'mysqld_safe' 'syslog') fi } @@ -813,7 +808,7 @@ monitor_process() while true ; do if ! ps --pid "${WSREP_SST_OPT_PARENT}" &>/dev/null; then - wsrep_log_error "Parent mysqld process (PID:${WSREP_SST_OPT_PARENT}) terminated unexpectedly." + wsrep_log_error "Parent mysqld process (PID: $WSREP_SST_OPT_PARENT) terminated unexpectedly." kill -- -"${WSREP_SST_OPT_PARENT}" exit 32 fi @@ -936,8 +931,8 @@ then exit 93 fi - if [ -z "$(parse_cnf --mysqld tmpdir)" -a \ - -z "$(parse_cnf xtrabackup tmpdir)" ]; then + tmpdir=$(parse_cnf "--mysqld|sst|xtrabackup" 'tmpdir') + if [ -z "$tmpdir" ]; then xtmpdir=$(mktemp -d) tmpopts=" --tmpdir='$xtmpdir'" wsrep_log_info "Using $xtmpdir as xtrabackup temporary directory" @@ -1112,7 +1107,7 @@ then if ! ps -p ${WSREP_SST_OPT_PARENT} &>/dev/null then - wsrep_log_error "Parent mysqld process (PID:${WSREP_SST_OPT_PARENT}) terminated unexpectedly." + wsrep_log_error "Parent mysqld process (PID: $WSREP_SST_OPT_PARENT) terminated unexpectedly." exit 32 fi diff --git a/scripts/wsrep_sst_xtrabackup.sh b/scripts/wsrep_sst_xtrabackup.sh index b9fedaadd3a..716865ce9c5 100644 --- a/scripts/wsrep_sst_xtrabackup.sh +++ b/scripts/wsrep_sst_xtrabackup.sh @@ -385,7 +385,8 @@ check_extra() { local use_socket=1 if [[ $uextra -eq 1 ]];then - if [ $(parse_cnf --mysqld thread-handling) = 'pool-of-threads'];then + local thread_handling=$(parse_cnf '--mysqld' 'thread-handling') + if [ "$thread_handling" = 'pool-of-threads' ]; then local eport=$(parse_cnf --mysqld extra-port) if [[ -n $eport ]];then # Xtrabackup works only locally. @@ -530,7 +531,6 @@ then if [[ $incremental -eq 1 ]];then wsrep_log_info "Incremental SST enabled" - #lsn=$(/pxc/bin/mysqld $WSREP_SST_OPT_CONF --basedir=/pxc --wsrep-recover 2>&1 | grep -o 'log sequence number .*' | cut -d " " -f 4 | head -1) lsn=$(grep to_lsn xtrabackup_checkpoints | cut -d= -f2 | tr -d ' ') wsrep_log_info "Recovered LSN: $lsn" fi @@ -604,7 +604,7 @@ then if ! ps -p ${WSREP_SST_OPT_PARENT} &>/dev/null then - wsrep_log_error "Parent mysqld process (PID:${WSREP_SST_OPT_PARENT}) terminated unexpectedly." + wsrep_log_error "Parent mysqld process (PID: $WSREP_SST_OPT_PARENT) terminated unexpectedly." exit 32 fi diff --git a/sql/wsrep_sst.cc b/sql/wsrep_sst.cc index 38560c7c663..e844086f5e3 100644 --- a/sql/wsrep_sst.cc +++ b/sql/wsrep_sst.cc @@ -664,6 +664,49 @@ static int sst_append_env_var(wsp::env& env, return -env.error(); } +#ifdef __WIN__ +/* + Space, single quote, ampersand, backquote, I/O redirection + characters, caret, all brackets, plus, exclamation and comma + characters require text to be enclosed in double quotes: +*/ +#define IS_SPECIAL(c) \ + (isspace(c) || c == '\'' || c == '&' || c == '`' || c == '|' || \ + c == '>' || c == '<' || c == ';' || c == '^' || \ + c == '[' || c == ']' || c == '{' || c == '}' || \ + c == '(' || c == ')' || c == '+' || c == '!' || \ + c == ',') +/* + Inside values, equals character are interpreted as special + character and requires quotation: +*/ +#define IS_SPECIAL_V(c) (IS_SPECIAL(c) || c == '=') +/* + Double quotation mark and percent characters require escaping: +*/ +#define IS_REQ_ESCAPING(c) (c == '""' || c == '%') +#else +/* + Space, single quote, ampersand, backquote, and I/O redirection + characters require text to be enclosed in double quotes. The + semicolon is used to separate shell commands, so it must be + enclosed in double quotes as well: +*/ +#define IS_SPECIAL(c) \ + (isspace(c) || c == '\'' || c == '&' || c == '`' || c == '|' || \ + c == '>' || c == '<' || c == ';') +/* + Inside values, characters are interpreted as in parameter names: +*/ +#define IS_SPECIAL_V(c) IS_SPECIAL(c) +/* + Double quotation mark and backslash characters require + backslash prefixing, the dollar symbol is used to substitute + a variable value, therefore it also requires escaping: +*/ +#define IS_REQ_ESCAPING(c) (c == '"' || c == '\\' || c == '$') +#endif + static size_t estimate_cmd_len (bool* extra_args) { /* @@ -688,23 +731,17 @@ static size_t estimate_cmd_len (bool* extra_args) char c; while ((c = *arg++) != 0) { - /* - Space, single quote, ampersand, and I/O redirection characters - require text to be enclosed in double quotes: - */ - if (isspace(c) || c == '\'' || c == '&' || c == '|' || -#ifdef __WIN__ - c == '>' || c == '<') -#else - /* - The semicolon is used to separate shell commands, so it must be - enclosed in double quotes as well: - */ - c == '>' || c == '<' || c == ';') -#endif + if (IS_SPECIAL(c)) { quotation= true; } + else if (IS_REQ_ESCAPING(c)) + { + cmd_len++; +#ifdef __WIN__ + quotation= true; +#endif + } /* If the equals symbol is encountered, then we need to separately process the right side: @@ -723,58 +760,20 @@ static size_t estimate_cmd_len (bool* extra_args) } while ((c = *arg++) != 0) { - /* - Space, single quote, ampersand, and I/O redirection characters - require text to be enclosed in double quotes: - */ - if (isspace(c) || c == '\'' || c == '&' || c == '|' || -#ifdef __WIN__ - c == '>' || c == '<') -#else - /* - The semicolon is used to separate shell commands, so it must be - enclosed in double quotes as well: - */ - c == '>' || c == '<' || c == ';') -#endif + if (IS_SPECIAL_V(c)) { quotation= true; } - /* - Double quotation mark or backslash symbol requires backslash - prefixing: - */ -#ifdef __WIN__ - else if (c == '"' || c == '\\') -#else - /* - The dollar symbol is used to substitute a variable, therefore - it also requires escaping: - */ - else if (c == '"' || c == '\\' || c == '$') -#endif + else if (IS_REQ_ESCAPING(c)) { cmd_len++; +#ifdef __WIN__ + quotation= true; +#endif } } break; } - /* - Double quotation mark or backslash symbol requires backslash - prefixing: - */ -#ifdef __WIN__ - else if (c == '"' || c == '\\') -#else - /* - The dollar symbol is used to substitute a variable, therefore - it also requires escaping: - */ - else if (c == '"' || c == '\\' || c == '$') -#endif - { - cmd_len++; - } } /* Perhaps we need to quote the entire argument or its right part: */ if (quotation) @@ -817,23 +816,17 @@ static void copy_orig_argv (char* cmd_str) char c; while ((c = *arg_scan++) != 0) { - /* - Space, single quote, ampersand, and I/O redirection characters - require text to be enclosed in double quotes: - */ - if (isspace(c) || c == '\'' || c == '&' || c == '|' || -#ifdef __WIN__ - c == '>' || c == '<') -#else - /* - The semicolon is used to separate shell commands, so it must be - enclosed in double quotes as well: - */ - c == '>' || c == '<' || c == ';') -#endif + if (IS_SPECIAL(c)) { quotation= true; } + else if (IS_REQ_ESCAPING(c)) + { + plain= false; +#ifdef __WIN__ + quotation= true; +#endif + } /* If the equals symbol is encountered, then we need to separately process the right side: @@ -868,13 +861,13 @@ static void copy_orig_argv (char* cmd_str) while (m) { c = *arg++; -#ifdef __WIN__ - if (c == '"' || c == '\\') -#else - if (c == '"' || c == '\\' || c == '$') -#endif + if (IS_REQ_ESCAPING(c)) { +#ifdef __WIN__ + *cmd_str++ = c; +#else *cmd_str++ = '\\'; +#endif } *cmd_str++ = c; m--; @@ -903,58 +896,20 @@ static void copy_orig_argv (char* cmd_str) /* Let's deal with the left side of the expression: */ while ((c = *arg_scan++) != 0) { - /* - Space, single quote, ampersand, and I/O redirection characters - require text to be enclosed in double quotes: - */ - if (isspace(c) || c == '\'' || c == '&' || c == '|' || -#ifdef __WIN__ - c == '>' || c == '<') -#else - /* - The semicolon is used to separate shell commands, so it must be - enclosed in double quotes as well: - */ - c == '>' || c == '<' || c == ';') -#endif + if (IS_SPECIAL_V(c)) { quotation= true; } - /* - Double quotation mark or backslash symbol requires backslash - prefixing: - */ -#ifdef __WIN__ - else if (c == '"' || c == '\\') -#else - /* - The dollar symbol is used to substitute a variable, therefore - it also requires escaping: - */ - else if (c == '"' || c == '\\' || c == '$') -#endif + else if (IS_REQ_ESCAPING(c)) { plain= false; +#ifdef __WIN__ + quotation= true; +#endif } } break; } - /* - Double quotation mark or backslash symbol requires backslash - prefixing: - */ -#ifdef __WIN__ - else if (c == '"' || c == '\\') -#else - /* - The dollar symbol is used to substitute a variable, therefore - it also requires escaping: - */ - else if (c == '"' || c == '\\' || c == '$') -#endif - { - plain= false; - } } if (n) { @@ -977,13 +932,13 @@ static void copy_orig_argv (char* cmd_str) { while ((c = *arg++) != 0) { -#ifdef __WIN__ - if (c == '"' || c == '\\') -#else - if (c == '"' || c == '\\' || c == '$') -#endif + if (IS_REQ_ESCAPING(c)) { +#ifdef __WIN__ + *cmd_str++ = c; +#else *cmd_str++ = '\\'; +#endif } *cmd_str++ = c; } @@ -1352,7 +1307,7 @@ static int sst_donate_mysqldump (const char* addr, WSREP_SST_OPT_GTID " '%s:%lld' " WSREP_SST_OPT_GTID_DOMAIN_ID " '%d'" "%s", - addr, port, mysqld_port, mysqld_unix_port, + addr, port, mysqld_port, mysqld_unix_port, wsrep_defaults_file, uuid_str, (long long)seqno, wsrep_gtid_domain_id, bypass ? " " WSREP_SST_OPT_BYPASS : ""); @@ -1476,7 +1431,7 @@ static int sst_flush_tables(THD* thd) WSREP_WARN("Current client character set is non-supported parser character set: %s", current_charset->csname); thd->variables.character_set_client = &my_charset_latin1; WSREP_WARN("For SST temporally setting character set to : %s", - my_charset_latin1.csname); + my_charset_latin1.csname); } if (run_sql_command(thd, "FLUSH TABLES WITH READ LOCK")) @@ -1544,7 +1499,7 @@ static void sst_disallow_writes (THD* thd, bool yes) WSREP_WARN("Current client character set is non-supported parser character set: %s", current_charset->csname); thd->variables.character_set_client = &my_charset_latin1; WSREP_WARN("For SST temporally setting character set to : %s", - my_charset_latin1.csname); + my_charset_latin1.csname); } snprintf (query_str, query_max, "SET GLOBAL innodb_disallow_writes=%d", From 80ae3677e1b90a7f5d9dfc55ba0612e065b8ce97 Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Mon, 17 May 2021 09:39:19 +1000 Subject: [PATCH 11/15] MDEV-25681: --relay-log{,-index} missing warning No longer a MySQL server, "his" is the wrong pronoun for a server. Thanks Michael Newton for highlighting these problems Also changed slave -> replica. --- sql/rpl_rli.cc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc index 5273b33c728..3a882f6d416 100644 --- a/sql/rpl_rli.cc +++ b/sql/rpl_rli.cc @@ -203,8 +203,8 @@ a file name for --relay-log-index option", opt_relaylog_index_name); */ sql_print_warning("Neither --relay-log nor --relay-log-index were used;" " so replication " - "may break when this MySQL server acts as a " - "slave and has his hostname changed!! Please " + "may break when this MariaDB server acts as a " + "replica and has its hostname changed. Please " "use '--log-basename=#' or '--relay-log=%s' to avoid " "this problem.", ln); name_warning_sent= 1; From 410e3c1a9a364219481392a408f12a432d83b2f2 Mon Sep 17 00:00:00 2001 From: Sujatha Date: Wed, 12 May 2021 18:00:06 +0530 Subject: [PATCH 12/15] MDEV-17515: GTID Replication in optimistic mode deadlock Problem: ======= In slave_parallel_mode=optimistic configuration, when admin commands and DML operation on the same table are scheduled simultaneously for execution, it results in lock conflict and slave server either hangs due to deadlock or goes down with an assert. Analysis: ======== Admin commands OPTIMIZE, REPAIR and ANALYZE are written to binary log as ordinary transactions. When 'slave_parallel_mode' is 'optimistic' DMLs are allowed to run in parallel. But these locks are not detected by parallel replication deadlock detection-and-handling mechanism. At times they result in deadlock or assertion. Fix: === Flag admin commands as DDL in Gtid_log_event at the time of writing to binary log. Add a new bit EXECUTED_TABLE_ADMIN_CMD to 'm_unsafe_rollback_flags'. During 'mysql_admin_table' command execution it accepts a list of tables to be processed and executes them in a loop. Upon successful execution enable 'EXECUTED_TABLE_ADMIN_CMD' bit in thd->transaction.stmt_unsafe_rollback_flags. Gtid_log_event constructor will notice this flag and mark the current transaction with 'FL_DDL' flag. Gtid_log_events marked as FL_DDL will not be scheduled parallel execution, on the slave. They will execute in isolation to prevent deadlocks. Note: Removed the call to 'trans_commit_implicit' from 'mysql_admin_table' function as 'mysql_execute_command' will take care of invoking 'trans_commit_implicit'. --- mysql-test/include/commit.inc | 2 +- mysql-test/r/commit_1innodb.result | 2 +- .../rpl/r/rpl_mark_optimize_tbl_ddl.result | 77 ++++++++++ .../rpl/t/rpl_mark_optimize_tbl_ddl.test | 142 ++++++++++++++++++ sql/handler.h | 10 ++ sql/log_event.cc | 4 +- sql/sql_admin.cc | 55 +++---- sql/sql_class.h | 3 +- 8 files changed, 257 insertions(+), 38 deletions(-) create mode 100644 mysql-test/suite/rpl/r/rpl_mark_optimize_tbl_ddl.result create mode 100644 mysql-test/suite/rpl/t/rpl_mark_optimize_tbl_ddl.test diff --git a/mysql-test/include/commit.inc b/mysql-test/include/commit.inc index a28d3e5f3d1..1844a5320f7 100644 --- a/mysql-test/include/commit.inc +++ b/mysql-test/include/commit.inc @@ -770,7 +770,7 @@ call p_verify_status_increment(2, 0, 2, 0); commit; call p_verify_status_increment(0, 0, 0, 0); check table t1, t2, t3; -call p_verify_status_increment(6, 0, 6, 0); +call p_verify_status_increment(4, 0, 4, 0); commit; call p_verify_status_increment(0, 0, 0, 0); drop view v1; diff --git a/mysql-test/r/commit_1innodb.result b/mysql-test/r/commit_1innodb.result index 1adba7b4c4c..24d0f79325a 100644 --- a/mysql-test/r/commit_1innodb.result +++ b/mysql-test/r/commit_1innodb.result @@ -873,7 +873,7 @@ Table Op Msg_type Msg_text test.t1 check status OK test.t2 check status OK test.t3 check status OK -call p_verify_status_increment(6, 0, 6, 0); +call p_verify_status_increment(4, 0, 4, 0); SUCCESS commit; diff --git a/mysql-test/suite/rpl/r/rpl_mark_optimize_tbl_ddl.result b/mysql-test/suite/rpl/r/rpl_mark_optimize_tbl_ddl.result new file mode 100644 index 00000000000..a39dad85244 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_mark_optimize_tbl_ddl.result @@ -0,0 +1,77 @@ +include/rpl_init.inc [topology=1->2] +connection server_1; +FLUSH TABLES; +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +connection server_2; +SET @save_slave_parallel_threads= @@GLOBAL.slave_parallel_threads; +SET @save_slave_parallel_mode= @@GLOBAL.slave_parallel_mode; +include/stop_slave.inc +SET GLOBAL slave_parallel_threads=2; +SET GLOBAL slave_parallel_mode=optimistic; +include/start_slave.inc +connection server_1; +CREATE TABLE t1(a INT) ENGINE=INNODB; +OPTIMIZE TABLE t1; +Table Op Msg_type Msg_text +test.t1 optimize note Table does not support optimize, doing recreate + analyze instead +test.t1 optimize status OK +INSERT INTO t1 VALUES(1); +INSERT INTO t1 SELECT 1+a FROM t1; +INSERT INTO t1 SELECT 2+a FROM t1; +connection server_2; +# +# Verify that following admin commands are marked as ddl +# 'OPTIMIZE TABLE', 'REPAIR TABLE' and 'ANALYZE TABLE' +# +connection server_1; +OPTIMIZE TABLE t1; +Table Op Msg_type Msg_text +test.t1 optimize note Table does not support optimize, doing recreate + analyze instead +test.t1 optimize status OK +REPAIR TABLE t1; +Table Op Msg_type Msg_text +test.t1 repair note The storage engine for the table doesn't support repair +ANALYZE TABLE t1; +Table Op Msg_type Msg_text +test.t1 analyze status OK +FLUSH LOGS; +FOUND 1 /GTID 0-1-8 ddl/ in mysqlbinlog.out +FOUND 1 /GTID 0-1-9 ddl/ in mysqlbinlog.out +FOUND 1 /GTID 0-1-10 ddl/ in mysqlbinlog.out +# +# Clean up +# +DROP TABLE t1; +connection server_2; +FLUSH LOGS; +# +# Check that ALTER TABLE commands with ANALYZE, OPTIMIZE and REPAIR on +# partitions will be marked as DDL in binary log. +# +connection server_1; +CREATE TABLE t1(id INT) PARTITION BY RANGE (id) (PARTITION p0 VALUES LESS THAN (100), +PARTITION pmax VALUES LESS THAN (MAXVALUE)); +INSERT INTO t1 VALUES (1), (10), (100), (1000); +ALTER TABLE t1 ANALYZE PARTITION p0; +Table Op Msg_type Msg_text +test.t1 analyze status OK +ALTER TABLE t1 OPTIMIZE PARTITION p0; +Table Op Msg_type Msg_text +test.t1 optimize status OK +ALTER TABLE t1 REPAIR PARTITION p0; +Table Op Msg_type Msg_text +test.t1 repair status OK +FLUSH LOGS; +FOUND 1 /GTID 0-1-14 ddl/ in mysqlbinlog.out +FOUND 1 /GTID 0-1-15 ddl/ in mysqlbinlog.out +FOUND 1 /GTID 0-1-16 ddl/ in mysqlbinlog.out +# +# Clean up +# +DROP TABLE t1; +connection server_2; +include/stop_slave.inc +SET GLOBAL slave_parallel_threads= @save_slave_parallel_threads; +SET GLOBAL slave_parallel_mode= @save_slave_parallel_mode; +include/start_slave.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_mark_optimize_tbl_ddl.test b/mysql-test/suite/rpl/t/rpl_mark_optimize_tbl_ddl.test new file mode 100644 index 00000000000..6d66e3fd088 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_mark_optimize_tbl_ddl.test @@ -0,0 +1,142 @@ +# ==== Purpose ==== +# +# Test verifies that there is no deadlock or assertion in +# slave_parallel_mode=optimistic configuration while applying admin command +# like 'OPTIMIZE TABLE', 'REPAIR TABLE' and 'ANALYZE TABLE'. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create a table, execute OPTIMIZE TABLE command on the table followed +# by some DMLS. +# 1 - No assert should happen on slave server. +# 2 - Assert that 'OPTIMIZE TABLE', 'REPAIR TABLE' and 'ANALYZE TABLE' are +# marked as 'DDL' in the binary log. +# +# ==== References ==== +# +# MDEV-17515: GTID Replication in optimistic mode deadlock +# +--source include/have_partition.inc +--source include/have_innodb.inc +--let $rpl_topology=1->2 +--source include/rpl_init.inc + +--connection server_1 +FLUSH TABLES; +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; + +--connection server_2 +SET @save_slave_parallel_threads= @@GLOBAL.slave_parallel_threads; +SET @save_slave_parallel_mode= @@GLOBAL.slave_parallel_mode; +--source include/stop_slave.inc +SET GLOBAL slave_parallel_threads=2; +SET GLOBAL slave_parallel_mode=optimistic; +--source include/start_slave.inc + +--connection server_1 +CREATE TABLE t1(a INT) ENGINE=INNODB; +OPTIMIZE TABLE t1; +INSERT INTO t1 VALUES(1); +INSERT INTO t1 SELECT 1+a FROM t1; +INSERT INTO t1 SELECT 2+a FROM t1; +--save_master_pos + +--connection server_2 +--sync_with_master + +--echo # +--echo # Verify that following admin commands are marked as ddl +--echo # 'OPTIMIZE TABLE', 'REPAIR TABLE' and 'ANALYZE TABLE' +--echo # +--connection server_1 + +OPTIMIZE TABLE t1; +--let optimize_gtid= `SELECT @@GLOBAL.gtid_binlog_pos` + +REPAIR TABLE t1; +--let repair_gtid= `SELECT @@GLOBAL.gtid_binlog_pos` + +ANALYZE TABLE t1; +--let analyze_gtid= `SELECT @@GLOBAL.gtid_binlog_pos` + +let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1); +FLUSH LOGS; + +--let $MYSQLD_DATADIR= `select @@datadir` +--exec $MYSQL_BINLOG $MYSQLD_DATADIR/$binlog_file > $MYSQLTEST_VARDIR/tmp/mysqlbinlog.out + +--let SEARCH_PATTERN= GTID $optimize_gtid ddl +--let SEARCH_FILE= $MYSQLTEST_VARDIR/tmp/mysqlbinlog.out +--source include/search_pattern_in_file.inc + +--let SEARCH_PATTERN= GTID $repair_gtid ddl +--let SEARCH_FILE= $MYSQLTEST_VARDIR/tmp/mysqlbinlog.out +--source include/search_pattern_in_file.inc + +--let SEARCH_PATTERN= GTID $analyze_gtid ddl +--let SEARCH_FILE= $MYSQLTEST_VARDIR/tmp/mysqlbinlog.out +--source include/search_pattern_in_file.inc + +--echo # +--echo # Clean up +--echo # +DROP TABLE t1; +--remove_file $MYSQLTEST_VARDIR/tmp/mysqlbinlog.out +--save_master_pos + +--connection server_2 +--sync_with_master +FLUSH LOGS; + +--echo # +--echo # Check that ALTER TABLE commands with ANALYZE, OPTIMIZE and REPAIR on +--echo # partitions will be marked as DDL in binary log. +--echo # +--connection server_1 +CREATE TABLE t1(id INT) PARTITION BY RANGE (id) (PARTITION p0 VALUES LESS THAN (100), + PARTITION pmax VALUES LESS THAN (MAXVALUE)); +INSERT INTO t1 VALUES (1), (10), (100), (1000); + +ALTER TABLE t1 ANALYZE PARTITION p0; +--let analyze_gtid= `SELECT @@GLOBAL.gtid_binlog_pos` + +ALTER TABLE t1 OPTIMIZE PARTITION p0; +--let optimize_gtid= `SELECT @@GLOBAL.gtid_binlog_pos` + +ALTER TABLE t1 REPAIR PARTITION p0; +--let repair_gtid= `SELECT @@GLOBAL.gtid_binlog_pos` + +let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1); +FLUSH LOGS; + +--exec $MYSQL_BINLOG $MYSQLD_DATADIR/$binlog_file > $MYSQLTEST_VARDIR/tmp/mysqlbinlog.out + +--let SEARCH_PATTERN= GTID $analyze_gtid ddl +--let SEARCH_FILE= $MYSQLTEST_VARDIR/tmp/mysqlbinlog.out +--source include/search_pattern_in_file.inc + +--let SEARCH_PATTERN= GTID $optimize_gtid ddl +--let SEARCH_FILE= $MYSQLTEST_VARDIR/tmp/mysqlbinlog.out +--source include/search_pattern_in_file.inc + +--let SEARCH_PATTERN= GTID $repair_gtid ddl +--let SEARCH_FILE= $MYSQLTEST_VARDIR/tmp/mysqlbinlog.out +--source include/search_pattern_in_file.inc + +--echo # +--echo # Clean up +--echo # +DROP TABLE t1; +--remove_file $MYSQLTEST_VARDIR/tmp/mysqlbinlog.out +--save_master_pos + +--connection server_2 +--sync_with_master + +--source include/stop_slave.inc +SET GLOBAL slave_parallel_threads= @save_slave_parallel_threads; +SET GLOBAL slave_parallel_mode= @save_slave_parallel_mode; +--source include/start_slave.inc + +--source include/rpl_end.inc diff --git a/sql/handler.h b/sql/handler.h index e0e0604176d..f618c1d6469 100644 --- a/sql/handler.h +++ b/sql/handler.h @@ -1481,7 +1481,17 @@ struct THD_TRANS static unsigned int const DROPPED_TEMP_TABLE= 0x04; static unsigned int const DID_WAIT= 0x08; static unsigned int const DID_DDL= 0x10; + static unsigned int const EXECUTED_TABLE_ADMIN_CMD= 0x20; + void mark_executed_table_admin_cmd() + { + DBUG_PRINT("debug", ("mark_executed_table_admin_cmd")); + m_unsafe_rollback_flags|= EXECUTED_TABLE_ADMIN_CMD; + } + bool trans_executed_admin_cmd() + { + return (m_unsafe_rollback_flags & EXECUTED_TABLE_ADMIN_CMD) != 0; + } void mark_created_temp_table() { DBUG_PRINT("debug", ("mark_created_temp_table")); diff --git a/sql/log_event.cc b/sql/log_event.cc index 94b2af20354..04577be4f6f 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -7561,8 +7561,10 @@ Gtid_log_event::Gtid_log_event(THD *thd_arg, uint64 seq_no_arg, flags2|= FL_WAITED; if (thd_arg->transaction.stmt.trans_did_ddl() || thd_arg->transaction.stmt.has_created_dropped_temp_table() || + thd_arg->transaction.stmt.trans_executed_admin_cmd() || thd_arg->transaction.all.trans_did_ddl() || - thd_arg->transaction.all.has_created_dropped_temp_table()) + thd_arg->transaction.all.has_created_dropped_temp_table() || + thd_arg->transaction.all.trans_executed_admin_cmd()) flags2|= FL_DDL; else if (is_transactional && !is_tmp_table) flags2|= FL_TRANSACTIONAL; diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc index 17eede61337..2b3593388bb 100644 --- a/sql/sql_admin.cc +++ b/sql/sql_admin.cc @@ -434,7 +434,8 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, int (handler::*operator_func)(THD *, HA_CHECK_OPT *), int (view_operator_func)(THD *, TABLE_LIST*, - HA_CHECK_OPT *)) + HA_CHECK_OPT *), + bool is_cmd_replicated) { TABLE_LIST *table; List field_list; @@ -1147,6 +1148,13 @@ send_result_message: break; } } + /* + Admin commands acquire table locks and these locks are not detected by + parallel replication deadlock detection-and-handling mechanism. Hence + they must be marked as DDL so that they are not scheduled in parallel + with conflicting DMLs resulting in deadlock. + */ + thd->transaction.stmt.mark_executed_table_admin_cmd(); if (table->table && !table->view) { if (table->table->s->tmp_table) @@ -1182,9 +1190,7 @@ send_result_message: } else { - if (trans_commit_stmt(thd) || - (stmt_causes_implicit_commit(thd, CF_IMPLICIT_COMMIT_END) && - trans_commit_implicit(thd))) + if (trans_commit_stmt(thd)) goto err; } close_thread_tables(thd); @@ -1209,6 +1215,11 @@ send_result_message: if (protocol->write()) goto err; } + if (is_cmd_replicated && !thd->lex->no_write_to_binlog) + { + if (write_bin_log(thd, TRUE, thd->query(), thd->query_length())) + goto err; + } my_eof(thd); thd->resume_subsequent_commits(suspended_wfc); @@ -1270,7 +1281,7 @@ bool mysql_assign_to_keycache(THD* thd, TABLE_LIST* tables, check_opt.key_cache= key_cache; DBUG_RETURN(mysql_admin_table(thd, tables, &check_opt, "assign_to_keycache", TL_READ_NO_INSERT, 0, 0, - 0, 0, &handler::assign_to_keycache, 0)); + 0, 0, &handler::assign_to_keycache, 0, false)); } @@ -1297,7 +1308,7 @@ bool mysql_preload_keys(THD* thd, TABLE_LIST* tables) */ DBUG_RETURN(mysql_admin_table(thd, tables, 0, "preload_keys", TL_READ_NO_INSERT, 0, 0, 0, 0, - &handler::preload_keys, 0)); + &handler::preload_keys, 0, false)); } @@ -1315,15 +1326,7 @@ bool Sql_cmd_analyze_table::execute(THD *thd) WSREP_TO_ISOLATION_BEGIN_WRTCHK(NULL, NULL, first_table); res= mysql_admin_table(thd, first_table, &m_lex->check_opt, "analyze", lock_type, 1, 0, 0, 0, - &handler::ha_analyze, 0); - /* ! we write after unlocking the table */ - if (!res && !m_lex->no_write_to_binlog) - { - /* - Presumably, ANALYZE and binlog writing doesn't require synchronization - */ - res= write_bin_log(thd, TRUE, thd->query(), thd->query_length()); - } + &handler::ha_analyze, 0, true); m_lex->select_lex.table_list.first= first_table; m_lex->query_tables= first_table; @@ -1346,7 +1349,7 @@ bool Sql_cmd_check_table::execute(THD *thd) goto error; /* purecov: inspected */ res= mysql_admin_table(thd, first_table, &m_lex->check_opt, "check", lock_type, 0, 0, HA_OPEN_FOR_REPAIR, 0, - &handler::ha_check, &view_check); + &handler::ha_check, &view_check, false); m_lex->select_lex.table_list.first= first_table; m_lex->query_tables= first_table; @@ -1371,15 +1374,7 @@ bool Sql_cmd_optimize_table::execute(THD *thd) mysql_recreate_table(thd, first_table, true) : mysql_admin_table(thd, first_table, &m_lex->check_opt, "optimize", TL_WRITE, 1, 0, 0, 0, - &handler::ha_optimize, 0); - /* ! we write after unlocking the table */ - if (!res && !m_lex->no_write_to_binlog) - { - /* - Presumably, OPTIMIZE and binlog writing doesn't require synchronization - */ - res= write_bin_log(thd, TRUE, thd->query(), thd->query_length()); - } + &handler::ha_optimize, 0, true); m_lex->select_lex.table_list.first= first_table; m_lex->query_tables= first_table; @@ -1404,16 +1399,8 @@ bool Sql_cmd_repair_table::execute(THD *thd) TL_WRITE, 1, MY_TEST(m_lex->check_opt.sql_flags & TT_USEFRM), HA_OPEN_FOR_REPAIR, &prepare_for_repair, - &handler::ha_repair, &view_repair); + &handler::ha_repair, &view_repair, true); - /* ! we write after unlocking the table */ - if (!res && !m_lex->no_write_to_binlog) - { - /* - Presumably, REPAIR and binlog writing doesn't require synchronization - */ - res= write_bin_log(thd, TRUE, thd->query(), thd->query_length()); - } m_lex->select_lex.table_list.first= first_table; m_lex->query_tables= first_table; diff --git a/sql/sql_class.h b/sql/sql_class.h index e08bb3e6358..890cce7bcb2 100644 --- a/sql/sql_class.h +++ b/sql/sql_class.h @@ -4530,7 +4530,8 @@ public: transaction.all.m_unsafe_rollback_flags|= (transaction.stmt.m_unsafe_rollback_flags & (THD_TRANS::DID_WAIT | THD_TRANS::CREATED_TEMP_TABLE | - THD_TRANS::DROPPED_TEMP_TABLE | THD_TRANS::DID_DDL)); + THD_TRANS::DROPPED_TEMP_TABLE | THD_TRANS::DID_DDL | + THD_TRANS::EXECUTED_TABLE_ADMIN_CMD)); } /* Reset current_linfo From 88c7a58ecf231865492729f892f6aafe72cafdad Mon Sep 17 00:00:00 2001 From: Sujatha Date: Wed, 12 May 2021 18:03:45 +0530 Subject: [PATCH 13/15] MDEV-22530: Aborting OPTIMIZE TABLE still logs in binary log and replicates to the Slave server. Problem: ======== Aborting OPTIMIZE TABLE still logs in binary logs and replicates to the Slave server. "Optimize table" command under execution, is killed by using "Ctrl-C" as shown below. MariaDB [test]> optimize table t2; ^CCtrl-C -- query killed. Continuing normally. In spite of query execution being interrupted the query gets written to binary log. Analysis: ======== Admin command execution logic is not handling KILL command, hence it ignores the KILL command and completes its execution. Fix: === Check for thread killed notification, during admin command execution and handle it. If thread kill occurs prior to any table modification the query will not be written to binary log. If kill happens after at least one table is modified then the query will be written to binary log. Ex: command in execution is 'OPTIMIZE TABLE t1,t2' and the thread kill happens after t1 table is modified then 'OPTIMIZE TABLE t1,t2' will be written to binary log as admin commands will not make the slave to diverge from master. --- .../binlog/r/binlog_admin_cmd_kill.result | 40 ++++++++ .../suite/binlog/t/binlog_admin_cmd_kill.test | 95 +++++++++++++++++++ sql/sql_admin.cc | 10 +- 3 files changed, 144 insertions(+), 1 deletion(-) create mode 100644 mysql-test/suite/binlog/r/binlog_admin_cmd_kill.result create mode 100644 mysql-test/suite/binlog/t/binlog_admin_cmd_kill.test diff --git a/mysql-test/suite/binlog/r/binlog_admin_cmd_kill.result b/mysql-test/suite/binlog/r/binlog_admin_cmd_kill.result new file mode 100644 index 00000000000..a2eb7ee5c0a --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_admin_cmd_kill.result @@ -0,0 +1,40 @@ +# +# Kill OPTIMIZE command prior to table modification +# +RESET MASTER; +CREATE TABLE t1 (f INT) ENGINE=INNODB; +CREATE TABLE t2 (f INT) ENGINE=INNODB; +connect con1,127.0.0.1,root,,test,$MASTER_MYPORT,; +connection con1; +SET debug_sync='admin_command_kill_before_modify SIGNAL ready_to_be_killed WAIT_FOR master_cont'; +OPTIMIZE TABLE t1,t2; +connection default; +SET debug_sync='now WAIT_FOR ready_to_be_killed'; +KILL THD_ID; +SET debug_sync = 'reset'; +disconnect con1; +include/show_binlog_events.inc +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE TABLE t1 (f INT) ENGINE=INNODB +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE TABLE t2 (f INT) ENGINE=INNODB +DROP TABLE t1,t2; +FLUSH LOGS; +# +# Kill OPTIMIZE command after table modification +# +CREATE TABLE t1 (f INT) ENGINE=INNODB; +CREATE TABLE t2 (f INT) ENGINE=INNODB; +connect con1,127.0.0.1,root,,test,$MASTER_MYPORT,; +connection con1; +SET debug_sync='admin_command_kill_after_modify SIGNAL ready_to_be_killed WAIT_FOR master_cont'; +OPTIMIZE TABLE t1,t2; +connection default; +SET debug_sync='now WAIT_FOR ready_to_be_killed'; +KILL THD_ID; +SET debug_sync = 'reset'; +disconnect con1; +DROP TABLE t1,t2; +FLUSH LOGS; +FOUND 1 /OPTIMIZE TABLE t1,t2/ in mysqlbinlog.out diff --git a/mysql-test/suite/binlog/t/binlog_admin_cmd_kill.test b/mysql-test/suite/binlog/t/binlog_admin_cmd_kill.test new file mode 100644 index 00000000000..9b248097ae2 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_admin_cmd_kill.test @@ -0,0 +1,95 @@ +# ==== Purpose ==== +# +# Test verifies that when an admin command execution is interrupted by KILL +# command it should stop its execution. The admin command in binary log should +# contain only the list of tables which have successfully executed admin +# command prior to kill. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create two table t1,t2. +# 1 - Execute OPTIMIZE TABLE t1,t2 command. +# 2 - Using debug sync mechanism kill OPTIMIZE TABLE command at a stage +# where it has not optimized any table. +# 3 - Check that OPTIMIZE TABLE command is not written to binary log. +# 4 - Using debug sync mechanism hold the execution of OPTIMIZE TABLE after +# t1 table optimization. Now kill the OPTIMIZE TABLE command. +# 5 - Observe the binlog output, the OPTIMIZE TABLE command should display `t1,t2`. +# 6 - Please note that, we binlog the entire query even if at least one +# table is modified as admin commands are safe to replicate and they will +# not make the slave to diverge. +# +# ==== References ==== +# +# MDEV-22530: Aborting OPTIMIZE TABLE still logs in binary log and replicates to the Slave server. +# +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_innodb.inc + +--echo # +--echo # Kill OPTIMIZE command prior to table modification +--echo # +RESET MASTER; + +CREATE TABLE t1 (f INT) ENGINE=INNODB; +CREATE TABLE t2 (f INT) ENGINE=INNODB; + +--connect(con1,127.0.0.1,root,,test,$MASTER_MYPORT,) +--connection con1 +SET debug_sync='admin_command_kill_before_modify SIGNAL ready_to_be_killed WAIT_FOR master_cont'; +--send OPTIMIZE TABLE t1,t2 + +--connection default +SET debug_sync='now WAIT_FOR ready_to_be_killed'; +--let $thd_id= `SELECT ID FROM INFORMATION_SCHEMA.PROCESSLIST WHERE INFO LIKE '%OPTIMIZE TABLE %'` + +# Now kill. +--replace_result $thd_id THD_ID +eval KILL $thd_id; + +SET debug_sync = 'reset'; +--disconnect con1 + +--source include/show_binlog_events.inc +DROP TABLE t1,t2; + +FLUSH LOGS; + +--echo # +--echo # Kill OPTIMIZE command after table modification +--echo # + +CREATE TABLE t1 (f INT) ENGINE=INNODB; +CREATE TABLE t2 (f INT) ENGINE=INNODB; + +--connect(con1,127.0.0.1,root,,test,$MASTER_MYPORT,) +--connection con1 +SET debug_sync='admin_command_kill_after_modify SIGNAL ready_to_be_killed WAIT_FOR master_cont'; +--send OPTIMIZE TABLE t1,t2 + +--connection default +SET debug_sync='now WAIT_FOR ready_to_be_killed'; +--let $thd_id= `SELECT ID FROM INFORMATION_SCHEMA.PROCESSLIST WHERE INFO LIKE '%OPTIMIZE TABLE %'` + +# Now kill. +--replace_result $thd_id THD_ID +eval KILL $thd_id; + +SET debug_sync = 'reset'; +--disconnect con1 + +DROP TABLE t1,t2; +let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1); +FLUSH LOGS; + +--let $MYSQLD_DATADIR= `select @@datadir` +--exec $MYSQL_BINLOG $MYSQLD_DATADIR/$binlog_file > $MYSQLTEST_VARDIR/tmp/mysqlbinlog.out + +--let SEARCH_PATTERN= OPTIMIZE TABLE t1,t2 +--let SEARCH_FILE= $MYSQLTEST_VARDIR/tmp/mysqlbinlog.out +--source include/search_pattern_in_file.inc + +--remove_file $MYSQLTEST_VARDIR/tmp/mysqlbinlog.out diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc index 2b3593388bb..21afa1154f6 100644 --- a/sql/sql_admin.cc +++ b/sql/sql_admin.cc @@ -446,6 +446,7 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, int compl_result_code; bool need_repair_or_alter= 0; wait_for_commit* suspended_wfc; + bool is_table_modified= false; DBUG_ENTER("mysql_admin_table"); DBUG_PRINT("enter", ("extra_open_options: %u", extra_open_options)); @@ -496,6 +497,10 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, bool open_for_modify= org_open_for_modify; DBUG_PRINT("admin", ("table: '%s'.'%s'", table->db, table->table_name)); + DEBUG_SYNC(thd, "admin_command_kill_before_modify"); + + if (thd->is_killed()) + break; strxmov(table_name, db, ".", table->table_name, NullS); thd->open_options|= extra_open_options; table->lock_type= lock_type; @@ -1192,6 +1197,8 @@ send_result_message: { if (trans_commit_stmt(thd)) goto err; + if (!is_table_modified) + is_table_modified= true; } close_thread_tables(thd); thd->release_transactional_locks(); @@ -1214,8 +1221,9 @@ send_result_message: if (protocol->write()) goto err; + DEBUG_SYNC(thd, "admin_command_kill_after_modify"); } - if (is_cmd_replicated && !thd->lex->no_write_to_binlog) + if (is_table_modified && is_cmd_replicated && !thd->lex->no_write_to_binlog) { if (write_bin_log(thd, TRUE, thd->query(), thd->query_length())) goto err; From 23cad4d8c5a24f3d86ddfeef900512ac4714efcf Mon Sep 17 00:00:00 2001 From: Julius Goryavsky Date: Mon, 17 May 2021 18:59:26 +0200 Subject: [PATCH 14/15] wsrep_sst_common.sh: file mode changed back to 664 --- scripts/wsrep_sst_common.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100755 => 100644 scripts/wsrep_sst_common.sh diff --git a/scripts/wsrep_sst_common.sh b/scripts/wsrep_sst_common.sh old mode 100755 new mode 100644 From b9a2e4609f93f2cab253f02bc7535960852de3c5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marko=20M=C3=A4kel=C3=A4?= Date: Tue, 18 May 2021 08:37:24 +0300 Subject: [PATCH 15/15] MDEV-25594: Assertion failure in DeadlockChecker::check_and_resolve() ha_innobase::index_read(): If an autocommit non-locking transaction was already started, refuse to access a SPATIAL INDEX. Once a non-locking autocommit transaction has started, it must remain in that mode (not acquire any locks). This should fix one cause of the assertion failure that would occur in DeadlockChecker::check_and_resolve() under heavy load, presumably due to concurrent execution of trx_commit_in_memory(). --- storage/innobase/handler/ha_innodb.cc | 8 ++++++-- storage/innobase/include/dict0mem.h | 20 ++++++++++++++++---- 2 files changed, 22 insertions(+), 6 deletions(-) diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index e431b3f3595..8bdc0a9e478 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -9399,8 +9399,12 @@ ha_innobase::index_read( /* For R-Tree index, we will always place the page lock to pages being searched */ - if (dict_index_is_spatial(index)) { - ++m_prebuilt->trx->will_lock; + if (index->is_spatial() && !m_prebuilt->trx->will_lock) { + if (trx_is_started(m_prebuilt->trx)) { + DBUG_RETURN(HA_ERR_READ_ONLY_TRANSACTION); + } else { + m_prebuilt->trx->will_lock = true; + } } /* Note that if the index for which the search template is built is not diff --git a/storage/innobase/include/dict0mem.h b/storage/innobase/include/dict0mem.h index ae7f00c01bf..5424b38a927 100644 --- a/storage/innobase/include/dict0mem.h +++ b/storage/innobase/include/dict0mem.h @@ -2,7 +2,7 @@ Copyright (c) 1996, 2017, Oracle and/or its affiliates. All Rights Reserved. Copyright (c) 2012, Facebook Inc. -Copyright (c) 2013, 2020, MariaDB Corporation. +Copyright (c) 2013, 2021, MariaDB Corporation. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software @@ -1036,6 +1036,21 @@ struct dict_index_t{ return DICT_CLUSTERED == (type & (DICT_CLUSTERED | DICT_IBUF)); } + /** @return whether this is a generated clustered index */ + bool is_gen_clust() const { return type == DICT_CLUSTERED; } + + /** @return whether this is a clustered index */ + bool is_clust() const { return type & DICT_CLUSTERED; } + + /** @return whether this is a unique index */ + bool is_unique() const { return type & DICT_UNIQUE; } + + /** @return whether this is a spatial index */ + bool is_spatial() const { return UNIV_UNLIKELY(type & DICT_SPATIAL); } + + /** @return whether this is the change buffer */ + bool is_ibuf() const { return UNIV_UNLIKELY(type & DICT_IBUF); } + /** @return whether the index includes virtual columns */ bool has_virtual() const { return type & DICT_VIRTUAL; } @@ -1075,9 +1090,6 @@ struct dict_index_t{ } } - /** @return whether this is the change buffer */ - bool is_ibuf() const { return UNIV_UNLIKELY(type & DICT_IBUF); } - /** Assign the number of new column to be added as a part of the index @param n_vcol number of virtual columns to be added */