1berry/MariaDB-server
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

MDEV-24962: Galera SST innobackupex-move ignores Environment settings

Browse Source 
After switching to the new mariabackup interface (instead of
the outdated innobackupex interface, which is supported for
compatibility), we need to explicitly pass a path to the datadir
directory as a parameter, since in the new interface the value
of this option is not automatically set in such a way that it
always matches the SST/IST logic. This commit adds passing this
option as an explicit parameter to mariabackup. This commit also
removed unnecessary options that are not used and not supported
by mariabackup.

Also, numerous flaws in the common wsrep_sst_common script have
been fixed:

 1) There are many bash-specific constructs in the script that
    may not be supported by other interpreters, which can lead
    to the most unexpected errors during SST, because failures
    in the interpretation of bash-specific constructs lead to
    incorrect parsing of arguments;
 2) There is parse_cnf() function which is often called by other
    scripts for the "mysqld" or "--mysqld" group, but it does not
    take into account the default group suffix, which leads to
    reading values only from the default group, which then leads
    to errors due to reading the default values instead of the
    values for a specific group;
 3) Some options such as --user, --innodb-data-home-dir or --datadir
    are not removed from the --mysqld-args list, although they are
    processed inside scripts (and passing of these options funther
    may cause problems for mariabackup);
 4) If an argument that the script understands is present in
    the --mysqld-args list twice, then this causes SST to fail,
    instead of reading the most recent value;
 5) The "--host" parameter is technically still supported among
    the arguments of the SST scripts, but in reality scripts do not
    work with it as expected, especially if it has an IPv6 address;
 6) If the port number is absent in the --address parameter value,
    but the port number is explicitly passed through the --port
    argument, then the scripts for mariabackup and xtrabackup-v2
    fail;
 7) If a new address interface is used (with the --address parameter),
    then automatic default port substitution is not performed, although
    it is supported for the legacy --host/--port interface.
 8) If there are spaces in the parameter values after --mysqld_args,
    then their further transfer does not occur correctly, which
    causes mariabackup to fail during SST - the space splits
    the argument in such a way that it breaks the parsing of the
    following parameters;
 9) If most of the parameters that are names or paths to the files
    or directories contain spaces, then SST scripts fail in an
    unpredictable way due to incorrect variable substitutions;
10) If the --log-bin option is passed among the arguments of myqlds
    (--mysqld-args) without a parameter, and the --binlog option
    is not specified, then the script cannot substitute the default
    name for binlog and cannot construct binlog name using the
    --log-basename argument (which is against server specifications);
11) Tail slashes are not removed from the directory names, which,
    upon further substitution, leads to the appearance of a double
    slash in the file paths;
12) The explicit --binlog parameter (which is now always transmitted
    from the server side) and the "hidden" --log-bin parameter in the
    list of arguments after --mysqld-args are perceived as two different
    parameters in different parts of the scripts, and if they are do not
    match for some reason, this will lead to failures during SST;

Also, all new changes from the 10.6 branch have been migrated here,
including the latest pull requests for authentication (only the part
that concerns SST scripts).

It also fixes dozens of other bugs in all SST scripts.
This commit is contained in:
Julius Goryavsky 2021-04-28 01:39:31 +02:00
parent e0324bf300
commit 1ae7673aae
15 changed files with 1390 additions and 860 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
mysql-test/suite/galera/disabled.def
View File

@ -5,7 +5,7 @@
# Separate the test case name and the comment with ':'.
#
# <testcasename> : MDEV-<xxxx> <comment>
#
#
# Do not use any TAB characters for whitespace.
#
##############################################################################
@ -17,7 +17,7 @@ galera_as_slave_replication_bundle : MDEV-15785 OPTION_GTID_BEGIN is set in Gtid
galera_concurrent_ctas : MDEV-24842 Galera test failure on galera_concurrent_ctas
galera_gcache_recover_manytrx : MDEV-18834 Galera test failure
galera_mdl_race : MDEV-21524: galera.galera_mdl_race MTR failed: query 'reap' succeeded - should have failed with errno 1213
galera_parallel_simple : MDEV-20318 galera.galera_parallel_simple fails
galera_parallel_simple : MDEV-20318 galera.galera_parallel_simple fails
galera_partition : MDEV-21806: galera.galera_partition MTR failed: failed to recover from DONOR state
galera_shutdown_nonprim : MDEV-21493 galera.galera_shutdown_nonprim
galera_var_node_address : MDEV-20485 Galera test failure
@ -28,3 +28,6 @@ sql_log_bin : MDEV-21491 galera.sql_log_bin
versioning_trx_id: MDEV-18590: galera.versioning_trx_id: Test failure: mysqltest: Result content mismatch
galera_wsrep_provider_unset_set: wsrep_provider is read-only for security reasons
pxc-421: wsrep_provider is read-only for security reasons
galera_sst_xtrabackup-v2: Test fails due to innodb issues
galera_sst_xtrabackup-v2_encrypt_with_key: Test fails due to innodb issues
galera_sst_xtrabackup-v2_data_dir: Test fails due to innodb issues

13
mysql-test/suite/galera/r/galera_ist_innodb_flush_logs,debug.rdiff
View File

@ -1,11 +1,12 @@
--- r/galera_ist_innodb_flush_logs.result 2018-09-05 10:34:36.192439933 +0300
+++ r/galera_ist_innodb_flush_logs.reject 2018-09-17 10:20:06.039150838 +0300
@@ -86,3 +86,100 @@
@@ -86,3 +86,111 @@
DROP TABLE t1;
COMMIT;
SET AUTOCOMMIT=ON;
+Performing State Transfer on a server that has been killed and restarted
+while a DDL was in progress on it
+connection node_1;
+CREATE TABLE t1 (f1 CHAR(255)) ENGINE=InnoDB;
+SET AUTOCOMMIT=OFF;
+START TRANSACTION;
@ -14,6 +15,7 @@
+INSERT INTO t1 VALUES ('node1_committed_before');
+INSERT INTO t1 VALUES ('node1_committed_before');
+INSERT INTO t1 VALUES ('node1_committed_before');
+connection node_2;
+START TRANSACTION;
+INSERT INTO t1 VALUES ('node2_committed_before');
+INSERT INTO t1 VALUES ('node2_committed_before');
@ -22,9 +24,12 @@
+INSERT INTO t1 VALUES ('node2_committed_before');
+COMMIT;
+SET GLOBAL debug_dbug = 'd,sync.alter_opened_table';
+connection node_1;
+ALTER TABLE t1 ADD COLUMN f2 INTEGER;
+connection node_2;
+SET wsrep_sync_wait = 0;
+Killing server ...
+connection node_1;
+SET AUTOCOMMIT=OFF;
+START TRANSACTION;
+INSERT INTO t1 (f1) VALUES ('node1_committed_during');
@ -39,6 +44,7 @@
+INSERT INTO t1 (f1) VALUES ('node1_to_be_committed_after');
+INSERT INTO t1 (f1) VALUES ('node1_to_be_committed_after');
+INSERT INTO t1 (f1) VALUES ('node1_to_be_committed_after');
+connect node_1a_galera_st_kill_slave_ddl, 127.0.0.1, root, , test, $NODE_MYPORT_1;
+SET AUTOCOMMIT=OFF;
+START TRANSACTION;
+INSERT INTO t1 (f1) VALUES ('node1_to_be_rollbacked_after');
@ -46,7 +52,9 @@
+INSERT INTO t1 (f1) VALUES ('node1_to_be_rollbacked_after');
+INSERT INTO t1 (f1) VALUES ('node1_to_be_rollbacked_after');
+INSERT INTO t1 (f1) VALUES ('node1_to_be_rollbacked_after');
+connection node_2;
+Performing --wsrep-recover ...
+connection node_2;
+Starting server ...
+Using --wsrep-start-position when starting mysqld ...
+SET AUTOCOMMIT=OFF;
@ -57,6 +65,7 @@
+INSERT INTO t1 (f1) VALUES ('node2_committed_after');
+INSERT INTO t1 (f1) VALUES ('node2_committed_after');
+COMMIT;
+connection node_1;
+INSERT INTO t1 (f1) VALUES ('node1_to_be_committed_after');
+INSERT INTO t1 (f1) VALUES ('node1_to_be_committed_after');
+INSERT INTO t1 (f1) VALUES ('node1_to_be_committed_after');
@ -71,6 +80,7 @@
+INSERT INTO t1 (f1) VALUES ('node1_committed_after');
+INSERT INTO t1 (f1) VALUES ('node1_committed_after');
+COMMIT;
+connection node_1a_galera_st_kill_slave_ddl;
+INSERT INTO t1 (f1) VALUES ('node1_to_be_rollbacked_after');
+INSERT INTO t1 (f1) VALUES ('node1_to_be_rollbacked_after');
+INSERT INTO t1 (f1) VALUES ('node1_to_be_rollbacked_after');
@ -88,6 +98,7 @@
+1
+COMMIT;
+SET AUTOCOMMIT=ON;
+connection node_1;
+SELECT COUNT(*) = 2 FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = 't1';
+COUNT(*) = 2
+1

2
mysql-test/suite/galera/r/galera_ist_xtrabackup-v2.result
View File

@ -21,7 +21,7 @@ INSERT INTO t1 VALUES ('node2_committed_before');
INSERT INTO t1 VALUES ('node2_committed_before');
COMMIT;
Unloading wsrep provider ...
SET GLOBAL wsrep_provider = 'none';
SET GLOBAL wsrep_cluster_address = '';
connection node_1;
SET AUTOCOMMIT=OFF;
START TRANSACTION;

17
mysql-test/suite/galera/r/galera_ssl_upgrade.result
View File

@ -1,4 +1,11 @@
call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown");
connection node_1;
call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown.*");
connection node_2;
call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown.*");
connection node_1;
connection node_2;
connection node_1;
connection node_2;
SELECT VARIABLE_VALUE = 'Synced' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_local_state_comment';
VARIABLE_VALUE = 'Synced'
1
@ -6,12 +13,6 @@ SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_N
VARIABLE_VALUE = 2
1
connection node_1;
call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown.*");
connection node_2;
call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown.*");
connection node_1;
connection node_2;
connection node_1;
SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
VARIABLE_VALUE = 2
1
@ -23,3 +24,5 @@ connection node_1;
SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
VARIABLE_VALUE = 2
1
disconnect node_2;
disconnect node_1;

2
mysql-test/suite/galera/t/galera_ist_innodb_flush_logs.cnf
View File

@ -1,7 +1,7 @@
!include ../galera_2nodes.cnf
[mysqld]
wsrep_sst_method=xtrabackup-v2
wsrep_sst_method=mariabackup
wsrep_sst_auth=root:
innodb_safe_truncate=OFF

2
mysql-test/suite/galera/t/galera_ist_innodb_flush_logs.test
View File

@ -7,7 +7,7 @@
--source include/big_test.inc
--source include/galera_cluster.inc
--source include/have_innodb.inc
--source include/have_xtrabackup.inc
--source include/have_mariabackup.inc
--source suite/galera/include/galera_st_kill_slave.inc
--source suite/galera/include/galera_st_kill_slave_ddl.inc

22
mysql-test/suite/galera/t/galera_ssl_upgrade.test
View File

@ -8,16 +8,16 @@
--source include/have_innodb.inc
--source include/have_ssl_communication.inc
call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown");
SELECT VARIABLE_VALUE = 'Synced' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_local_state_comment';
SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
--connection node_1
call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown.*");
--connection node_2
call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown.*");
# Save original auto_increment_offset values.
--let $node_1=node_1
--let $node_2=node_2
--source include/auto_increment_offset_save.inc
# Setup galera ports
--connection node_1
--source suite/galera/include/galera_base_port.inc
@ -27,6 +27,9 @@ call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown.*");
--source suite/galera/include/galera_base_port.inc
--let $NODE_GALERAPORT_2 = $_NODE_GALERAPORT
SELECT VARIABLE_VALUE = 'Synced' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_local_state_comment';
SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
# 2. Restart node #1 with a socket.ssl_ca that includes both the new and the old certificate
--connection node_1
@ -40,7 +43,7 @@ call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown.*");
--source include/wait_condition.inc
SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
# 3. Restart node #2 with the new socket.ssl_ca , socket.ssl_cert and socket.ssl_key
# 3. Restart node #2 with the new socket.ssl_ca , socket.ssl_cert and socket.ssl_key
--connection node_2
--source include/shutdown_mysqld.inc
@ -52,7 +55,7 @@ SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_N
--source include/wait_condition.inc
SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
# 4. Restart node #1 with the new socket.ssl_ca , socket.ssl_cert and socket.ssl_key
# 4. Restart node #1 with the new socket.ssl_ca , socket.ssl_cert and socket.ssl_key
--connection node_1
--source include/shutdown_mysqld.inc
@ -65,3 +68,8 @@ SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_N
SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
# Upgrade complete. Both nodes now use the new key and certificate
# Restore original auto_increment_offset values.
--source include/auto_increment_offset_restore.inc
--source include/galera_end.inc

4
mysql-test/suite/galera/t/galera_sst_xtrabackup-v2_encrypt_with_key.cnf
View File

@ -7,7 +7,7 @@ wsrep_debug=ON
innodb_safe_truncate=OFF
[SST]
tkey=@ENV.MYSQL_TEST_DIR/std_data/galera-key.pem
tcert=@ENV.MYSQL_TEST_DIR/std_data/galera-cert.pem
tkey=@ENV.MYSQL_TEST_DIR/std_data/cakey.pem
tcert=@ENV.MYSQL_TEST_DIR/std_data/cacert.pem
encrypt=3
transferfmt=@ENV.MTR_GALERA_TFMT

6
mysql-test/suite/galera_3nodes/disabled.def
View File

@ -5,17 +5,13 @@
# Separate the test case name and the comment with ':'.
#
# <testcasename> : MDEV-<xxxx> <comment>
#
#
# Do not use any TAB characters for whitespace.
#
##############################################################################
GAL-501 : MDEV-24645 galera_3nodes.GAL-501 MTR failed: failed to open gcomm backend connection: 110
galera_gtid_2_cluster : MDEV-23775 Galera test failure on galera_3nodes.galera_gtid_2_cluster
galera_ipv6_mariabackup : MDEV-24440: galera_3nodes.galera_ipv6_mariabackup MTR fails sporadically: Failed to read from: wsrep_sst_mariabackup --role 'donor' --address '[::1]:16028/xtrabackup_sst//1'
galera_ipv6_mariabackup_section : MDEV-22195: galera_3nodes.galera_ipv6_mariabackup_section MTR failed: assert_grep.inc failed
galera_ipv6_mysqldump : MDEV-24036: galera_3nodes.galera_ipv6_mysqldump: rare random crashes during shutdown
galera_ipv6_rsync_section : MDEV-23580: galera_3nodes.galera_ipv6_rsync_section MTR failed: WSREP_SST: [ERROR] rsync daemon port '16008' has been taken
galera_ist_gcache_rollover : MDEV-23578 WSREP: exception caused by message: {v=0,t=1,ut=255,o=4,s=0,sr=0,as=1,f=6,src=50524cfe,srcvid=view_id(REG,50524cfe,4),insvid=view_id(UNKNOWN,00000000,0),ru=00000000,r=[-1,-1],fs=75,nl=(}
galera_load_data_ist : MDEV-24639 galera_3nodes.galera_load_data_ist MTR failed with SIGABRT: query 'reap' failed: 2013: Lost connection to MySQL server during query
galera_load_data_ist : MDEV-24639 galera_3nodes.galera_load_data_ist MTR failed with SIGABRT: query 'reap' failed: 2013: Lost connection to MySQL server during query

629
scripts/wsrep_sst_common.sh
View File

@ -1,4 +1,5 @@
# Copyright (C) 2012-2015 Codership Oy
# Copyright (C) 2017-2021 MariaDB
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@ -21,24 +22,32 @@ set -u
WSREP_SST_OPT_BYPASS=0
WSREP_SST_OPT_BINLOG=""
WSREP_SST_OPT_BINLOG_INDEX=""
WSREP_SST_OPT_LOG_BASENAME=""
WSREP_SST_OPT_DATA=""
WSREP_SST_OPT_AUTH=${WSREP_SST_OPT_AUTH:-}
WSREP_SST_OPT_USER=${WSREP_SST_OPT_USER:-}
WSREP_SST_OPT_PSWD=${WSREP_SST_OPT_PSWD:-}
WSREP_SST_OPT_AUTH="${WSREP_SST_OPT_AUTH:-}"
WSREP_SST_OPT_USER="${WSREP_SST_OPT_USER:-}"
WSREP_SST_OPT_PSWD="${WSREP_SST_OPT_PSWD:-}"
WSREP_SST_OPT_REMOTE_AUTH="${WSREP_SST_OPT_REMOTE_AUTH:-}"
WSREP_SST_OPT_DEFAULT=""
WSREP_SST_OPT_EXTRA_DEFAULT=""
WSREP_SST_OPT_SUFFIX_DEFAULT=""
WSREP_SST_OPT_SUFFIX_VALUE=""
WSREP_SST_OPT_MYSQLD=""
INNODB_DATA_HOME_DIR_ARG=""
INNODB_LOG_GROUP_HOME_ARG=""
INNODB_UNDO_DIR_ARG=""
LOG_BIN_ARG=""
WSREP_SST_OPT_PORT=""
WSREP_SST_OPT_ADDR=""
WSREP_SST_OPT_ADDR_PORT=""
WSREP_SST_OPT_HOST=""
WSREP_SST_OPT_HOST_UNESCAPED=""
WSREP_SST_OPT_HOST_ESCAPED=""
INNODB_DATA_HOME_DIR="${INNODB_DATA_HOME_DIR:-}"
INNODB_LOG_GROUP_HOME="${INNODB_LOG_GROUP_HOME:-}"
INNODB_UNDO_DIR="${INNODB_UNDO_DIR:-}"
INNOEXTRA=""
while [ $# -gt 0 ]; do
case "$1" in
'--address')
readonly WSREP_SST_OPT_ADDR="$2"
WSREP_SST_OPT_ADDR="$2"
#
# Break address string into host:port/path parts
#
@ -46,60 +55,61 @@ case "$1" in
\[*)
# IPv6
# Remove the starting and ending square brackets, if present:
addr_no_bracket=${WSREP_SST_OPT_ADDR#\[}
readonly WSREP_SST_OPT_HOST_UNESCAPED=${addr_no_bracket%%\]*}
# Square brackets are needed in most cases:
readonly WSREP_SST_OPT_HOST="[${WSREP_SST_OPT_HOST_UNESCAPED}]"
addr_no_bracket="${WSREP_SST_OPT_ADDR#\[}"
# Some utilities and subsequent code require an address
# without square brackets:
readonly WSREP_SST_OPT_HOST_UNESCAPED="${addr_no_bracket%%\]*}"
# Square brackets are needed in most cases:
readonly WSREP_SST_OPT_HOST="[${WSREP_SST_OPT_HOST_UNESCAPED}]"
readonly WSREP_SST_OPT_HOST_ESCAPED="\\[${WSREP_SST_OPT_HOST_UNESCAPED}\\]"
# Mark this address as IPv6:
readonly WSREP_SST_OPT_HOST_IPv6=1
;;
*)
readonly WSREP_SST_OPT_HOST=${WSREP_SST_OPT_ADDR%%[:/]*}
readonly WSREP_SST_OPT_HOST_UNESCAPED=$WSREP_SST_OPT_HOST
readonly WSREP_SST_OPT_HOST_ESCAPED=$WSREP_SST_OPT_HOST
readonly WSREP_SST_OPT_HOST="${WSREP_SST_OPT_ADDR%%[:/]*}"
readonly WSREP_SST_OPT_HOST_UNESCAPED="$WSREP_SST_OPT_HOST"
readonly WSREP_SST_OPT_HOST_ESCAPED="$WSREP_SST_OPT_HOST"
readonly WSREP_SST_OPT_HOST_IPv6=0
;;
esac
# Let's remove the leading part that contains the host address:
remain=${WSREP_SST_OPT_ADDR#${WSREP_SST_OPT_HOST_ESCAPED}}
remain="${WSREP_SST_OPT_ADDR#$WSREP_SST_OPT_HOST_ESCAPED}"
# Let's remove the ":" character that separates the port number
# from the hostname:
remain=${remain#:}
remain="${remain#:}"
# Extract the port number from the address - all characters
# up to "/" (if present):
readonly WSREP_SST_OPT_ADDR_PORT=${remain%%/*}
WSREP_SST_OPT_ADDR_PORT="${remain%%/*}"
# If the "/" character is present, then the path is not empty:
if [ "${remain#*/}" != "${remain}" ]; then
# This operation removes everything up to the "/" character,
# effectively removing the port number from the string:
readonly WSREP_SST_OPT_PATH=${remain#*/}
readonly WSREP_SST_OPT_PATH="${remain#*/}"
else
readonly WSREP_SST_OPT_PATH=""
fi
# The rest of the string is the same as the path (for now):
remain=${WSREP_SST_OPT_PATH}
remain="${WSREP_SST_OPT_PATH}"
# If there is one more "/" in the string, then everything before
# it will be the module name, otherwise the module name is empty:
if [ "${remain%%/*}" != "${remain}" ]; then
# This operation removes the tail after the very first
# occurrence of the "/" character (inclusively):
readonly WSREP_SST_OPT_MODULE=${remain%%/*}
readonly WSREP_SST_OPT_MODULE="${remain%%/*}"
else
readonly WSREP_SST_OPT_MODULE=""
fi
# Remove the module name part from the string, which ends with "/":
remain=${WSREP_SST_OPT_PATH#*/}
remain="${WSREP_SST_OPT_PATH#*/}"
# If the rest of the string does not match the original, then there
# was something else besides the module name:
if [ "$remain" != "${WSREP_SST_OPT_PATH}" ]; then
# Extract the part that matches the LSN by removing all
# characters starting from the very first "/":
readonly WSREP_SST_OPT_LSN=${remain%%/*}
readonly WSREP_SST_OPT_LSN="${remain%%/*}"
# Exctract everything after the first occurrence of
# the "/" character in the string:
remain=${remain#*/}
remain="${remain#*/}"
# If the remainder does not match the original string,
# then there is something else (the version number in
# our case):
@ -107,7 +117,7 @@ case "$1" in
# Let's extract the version number by removing the tail
# after the very first occurence of the "/" character
# (inclusively):
readonly WSREP_SST_OPT_SST_VER=${remain%%/*}
readonly WSREP_SST_OPT_SST_VER="${remain%%/*}"
else
readonly WSREP_SST_OPT_SST_VER=""
fi
@ -121,23 +131,23 @@ case "$1" in
WSREP_SST_OPT_BYPASS=1
;;
'--datadir')
readonly WSREP_SST_OPT_DATA="$2"
# Let's remove the trailing slash:
readonly WSREP_SST_OPT_DATA="${2%/}"
shift
;;
'--innodb-data-home-dir')
readonly INNODB_DATA_HOME_DIR_ARG="$2"
# Let's remove the trailing slash:
readonly INNODB_DATA_HOME_DIR="${2%/}"
shift
;;
'--innodb-log-group-home-dir')
readonly INNODB_LOG_GROUP_HOME_ARG="$2"
# Let's remove the trailing slash:
readonly INNODB_LOG_GROUP_HOME="${2%/}"
shift
;;
'--innodb-undo-directory')
readonly INNODB_UNDO_DIR_ARG="$2"
shift
;;
'--log-bin')
readonly LOG_BIN_ARG="$2"
# Let's remove the trailing slash:
readonly INNODB_UNDO_DIR="${2%/}"
shift
;;
'--defaults-file')
@ -154,7 +164,28 @@ case "$1" in
shift
;;
'--host')
readonly WSREP_SST_OPT_HOST="$2"
case "$2" in
\[*)
# IPv6
# Remove the starting and ending square brackets, if present:
addr_no_bracket="${2#\[}"
# Some utilities and subsequent code require an address
# without square brackets:
readonly WSREP_SST_OPT_HOST_UNESCAPED="${addr_no_bracket%%\]*}"
# Square brackets are needed in most cases:
readonly WSREP_SST_OPT_HOST="[${WSREP_SST_OPT_HOST_UNESCAPED}]"
readonly WSREP_SST_OPT_HOST_ESCAPED="\\[${WSREP_SST_OPT_HOST_UNESCAPED}\\]"
# Mark this address as IPv6:
readonly WSREP_SST_OPT_HOST_IPv6=1
;;
*)
readonly WSREP_SST_OPT_HOST="$2"
readonly WSREP_SST_OPT_HOST_UNESCAPED="$2"
readonly WSREP_SST_OPT_HOST_ESCAPED="$2"
readonly WSREP_SST_OPT_HOST_IPv6=0
;;
esac
WSREP_SST_OPT_ADDR="$WSREP_SST_OPT_HOST"
shift
;;
'--local-port')
@ -189,12 +220,16 @@ case "$1" in
readonly WSREP_SST_OPT_GTID="$2"
shift
;;
'--binlog')
WSREP_SST_OPT_BINLOG="$2"
'--binlog'|'--log-bin')
readonly WSREP_SST_OPT_BINLOG="$2"
shift
;;
'--binlog-index')
WSREP_SST_OPT_BINLOG_INDEX="$2"
'--binlog-index'|'--log-bin-index')
readonly WSREP_SST_OPT_BINLOG_INDEX="$2"
shift
;;
'--log-basename')
readonly WSREP_SST_OPT_LOG_BASENAME="$2"
shift
;;
'--gtid-domain-id')
@ -205,45 +240,114 @@ case "$1" in
original_cmd=""
shift
while [ $# -gt 0 ]; do
option=${1%%=*}
if [[ "$option" != "--defaults-file" && \
"$option" != "--defaults-extra-file" && \
"$option" != "--defaults-group-suffix" && \
"$option" != "--port" && \
"$option" != "--socket" ]]; then
value=${1#*=}
if [ "$value" == "$1" ]; then
value=""
fi
case "$option" in
'--innodb-data-home-dir')
if [ -z "$INNODB_DATA_HOME_DIR_ARG" ]; then
readonly INNODB_DATA_HOME_DIR_ARG="$value"
fi
;;
'--innodb-log-group-home-dir')
if [ -z "$INNODB_LOG_GROUP_HOME_ARG" ]; then
readonly INNODB_LOG_GROUP_HOME_ARG="$value"
fi
;;
'--innodb-undo-directory')
if [ -z "$INNODB_UNDO_DIR_ARG" ]; then
readonly INNODB_UNDO_DIR_ARG="$value"
fi
;;
'--log-bin')
if [ -z "$LOG_BIN_ARG" ]; then
readonly LOG_BIN_ARG="$value"
fi
;;
esac
if [ -z "$original_cmd" ]; then
original_cmd="$1"
else
original_cmd="$original_cmd $1"
fi
# check if the argument is the short option
# (starting with "-" instead of "--"):
if [ "${1#--}" = "$1" -a "${1#-}" != "$1" ]; then
option="${1#-}"
value=""
# check that the option value follows the name,
# without a space:
if [ ${#option} -gt 1 ]; then
# let's separate the first character as the option name,
# and the subsequent characters consider its value:
value="${1#-?}"
option="${1%$value}"
# check that the option name consists of one letter
# and there are the following arguments:
elif [ ${#option} -eq 1 -a $# -gt 1 ]; then
# if the next argument does not start with a "-" character,
# then this is the value of the current option:
if [ "${2#-}" = "$2" ]; then
value="$2"
shift
fi
fi
shift
if [ "$option" = 'h' ]; then
if [ -z "$WSREP_SST_OPT_DATA" ]; then
MYSQLD_OPT_DATADIR="${value%/}"
fi
elif [ "$option" != 'u' -a \
"$option" != 'P' ]; then
if [ -z "$original_cmd" ]; then
original_cmd="'-$option$value'"
else
original_cmd="$original_cmd '-$option$value'"
fi
fi
continue;
fi
shift
option="${1%%=*}"
if [ "$option" != '--defaults-file' -a \
"$option" != '--defaults-extra-file' -a \
"$option" != '--defaults-group-suffix' -a \
"$option" != '--user' -a \
"$option" != '--port' -a \
"$option" != '--socket' ]; then
value="${1#*=}"
if [ "$value" = "$1" ]; then
value=""
fi
# Let's fill in the variables containing important paths
# that might not have been passed through explicit parameters
# (+ removing the trailing slash in these paths). Many of these
# options are processed internally within scripts or passed
# explicitly to other programs, so we need to remove them
# from mysqld's argument list:
skip_mysqld_arg=0
case "$option" in
'--innodb-data-home-dir')
if [ -z "$INNODB_DATA_HOME_DIR" ]; then
MYSQLD_OPT_INNODB_DATA_HOME_DIR="${value%/}"
fi
skip_mysqld_arg=1
;;
'--innodb-log-group-home-dir')
if [ -z "$INNODB_LOG_GROUP_HOME" ]; then
MYSQLD_OPT_INNODB_LOG_GROUP_HOME="${value%/}"
fi
skip_mysqld_arg=1
;;
'--innodb-undo-directory')
if [ -z "$INNODB_UNDO_DIR" ]; then
MYSQLD_OPT_INNODB_UNDO_DIR="${value%/}"
fi
skip_mysqld_arg=1
;;
'--log-bin')
if [ -z "$WSREP_SST_OPT_BINLOG" ]; then
MYSQLD_OPT_LOG_BIN="$value"
fi
skip_mysqld_arg=1
;;
'--log-bin-index')
if [ -z "$WSREP_SST_OPT_BINLOG_INDEX" ]; then
MYSQLD_OPT_LOG_BIN_INDEX="$value"
fi
skip_mysqld_arg=1
;;
'--log-basename')
if [ -z "$WSREP_SST_OPT_LOG_BASENAME" ]; then
MYSQLD_OPT_LOG_BASENAME="$value"
fi
skip_mysqld_arg=1
;;
'--datadir')
if [ -z "$WSREP_SST_OPT_DATA" ]; then
MYSQLD_OPT_DATADIR="${value%/}"
fi
skip_mysqld_arg=1
;;
esac
if [ $skip_mysqld_arg -eq 0 ]; then
if [ -z "$original_cmd" ]; then
original_cmd="'$1'"
else
original_cmd="$original_cmd '$1'"
fi
fi
fi
shift
done
readonly WSREP_SST_OPT_MYSQLD="$original_cmd"
break
@ -256,19 +360,168 @@ esac
shift
done
readonly WSREP_SST_OPT_BYPASS
readonly WSREP_SST_OPT_BINLOG
readonly WSREP_SST_OPT_BINLOG_INDEX
if [ -n "${WSREP_SST_OPT_ADDR_PORT:-}" ]; then
if [ -n "${WSREP_SST_OPT_PORT:-}" ]; then
if [ "$WSREP_SST_OPT_PORT" != "$WSREP_SST_OPT_ADDR_PORT" ]; then
echo "WSREP_SST: [ERROR] port in --port=$WSREP_SST_OPT_PORT differs from port in --address=$WSREP_SST_OPT_ADDR" >&2
exit 2
fi
else
readonly WSREP_SST_OPT_PORT="$WSREP_SST_OPT_ADDR_PORT"
fi
# The same argument can be present on the command line several
# times, in this case we must take its last value:
if [ -n "${MYSQLD_OPT_INNODB_DATA_HOME_DIR:-}" -a \
-z "$INNODB_DATA_HOME_DIR" ]; then
readonly INNODB_DATA_HOME_DIR="$MYSQLD_OPT_INNODB_DATA_HOME_DIR"
fi
if [ -n "${MYSQLD_OPT_INNODB_LOG_GROUP_HOME:-}" -a \
-z "$INNODB_LOG_GROUP_HOME" ]; then
readonly INNODB_LOG_GROUP_HOME="$MYSQLD_OPT_INNODB_LOG_GROUP_HOME"
fi
if [ -n "${MYSQLD_OPT_INNODB_UNDO_DIR:-}" -a \
-z "$INNODB_UNDO_DIR" ]; then
readonly INNODB_UNDO_DIR="$MYSQLD_OPT_INNODB_UNDO_DIR"
fi
if [ -n "${MYSQLD_OPT_LOG_BIN:-}" -a \
-z "$WSREP_SST_OPT_BINLOG" ]; then
readonly WSREP_SST_OPT_BINLOG="$MYSQLD_OPT_LOG_BIN"
fi
if [ -n "${MYSQLD_OPT_LOG_BIN_INDEX:-}" -a \
-z "$WSREP_SST_OPT_BINLOG_INDEX" ]; then
readonly WSREP_SST_OPT_BINLOG_INDEX="$MYSQLD_OPT_LOG_BIN_INDEX"
fi
if [ -n "${MYSQLD_OPT_DATADIR:-}" -a \
-z "$WSREP_SST_OPT_DATA" ]; then
readonly WSREP_SST_OPT_DATA="$MYSQLD_OPT_DATADIR"
fi
if [ -n "${MYSQLD_OPT_LOG_BASENAME:-}" -a \
-z "$WSREP_SST_OPT_LOG_BASENAME" ]; then
readonly WSREP_SST_OPT_LOG_BASENAME="$MYSQLD_OPT_LOG_BASENAME"
fi
# If the --log-bin option is present without a value, then
# setting WSREP_SST_OPT_BINLOG by using other arguments:
if [ -z "$WSREP_SST_OPT_BINLOG" -a -n "${MYSQLD_OPT_LOG_BIN+x}" ]; then
if [ -n "$WSREP_SST_OPT_LOG_BASENAME" ]; then
# If the WSREP_SST_OPT_BINLOG variable is not set, but
# --log-basename is present among the arguments to mysqld,
# then set WSREP_SST_OPT_BINLOG equal to the base name with
# the "-bin" suffix:
readonly WSREP_SST_OPT_BINLOG="$WSREP_SST_OPT_LOG_BASENAME-bin"
else
# Take the default name:
readonly WSREP_SST_OPT_BINLOG='mysql-bin'
fi
fi
# Reconstructing the command line arguments that control the innodb
# and binlog options:
if [ -n "$WSREP_SST_OPT_LOG_BASENAME" ]; then
INNOEXTRA="$INNOEXTRA --log-basename='$WSREP_SST_OPT_LOG_BASENAME'"
fi
if [ -n "$INNODB_DATA_HOME_DIR" ]; then
INNOEXTRA="$INNOEXTRA --innodb-data-home-dir='$INNODB_DATA_HOME_DIR'"
fi
if [ -n "$INNODB_LOG_GROUP_HOME" ]; then
INNOEXTRA="$INNOEXTRA --innodb-log-group-home-dir='$INNODB_LOG_GROUP_HOME'"
fi
if [ -n "$INNODB_UNDO_DIR" ]; then
INNOEXTRA="$INNOEXTRA --innodb-undo-directory='$INNODB_UNDO_DIR'"
fi
if [ -n "$WSREP_SST_OPT_BINLOG" ]; then
INNOEXTRA="$INNOEXTRA --log-bin='$WSREP_SST_OPT_BINLOG'"
if [ -n "$WSREP_SST_OPT_BINLOG_INDEX" ]; then
INNOEXTRA="$INNOEXTRA --log-bin-index='$WSREP_SST_OPT_BINLOG_INDEX'"
fi
fi
get_binlog()
{
# if no command line argument and WSREP_SST_OPT_BINLOG is not set,
# try to get it from my.cnf:
if [ -z "$WSREP_SST_OPT_BINLOG" ]; then
WSREP_SST_OPT_BINLOG=$(parse_cnf '--mysqld' 'log-bin')
fi
# if no command line argument and WSREP_SST_OPT_BINLOG_INDEX is not set,
# try to get it from my.cnf:
if [ -z "$WSREP_SST_OPT_BINLOG_INDEX" ]; then
WSREP_SST_OPT_BINLOG_INDEX=$(parse_cnf '--mysqld' 'log-bin-index')
fi
# if no command line argument and WSREP_SST_OPT_LOG_BASENAME is not set,
# try to get it from my.cnf:
if [ -z "$WSREP_SST_OPT_LOG_BASENAME" ]; then
WSREP_SST_OPT_LOG_BASENAME=$(parse_cnf '--mysqld' 'log-basename')
fi
if [ -z "$WSREP_SST_OPT_BINLOG" ]; then
# If the --log-bin option is specified without a parameter,
# then we need to build the name of the index file according
# to the rules described in the server documentation:
if [ -n "${MYSQLD_OPT_LOG_BIN+x}" -o \
$(in_config '--mysqld' 'log-bin') -eq 1 ]
then
if [ -n "$WSREP_SST_OPT_LOG_BASENAME" ]; then
# If the WSREP_SST_OPT_BINLOG variable is not set, but
# --log-basename is present among the arguments of mysqld,
# then set WSREP_SST_OPT_BINLOG equal to the base name with
# the "-bin" suffix:
readonly WSREP_SST_OPT_BINLOG="$WSREP_SST_OPT_LOG_BASENAME-bin"
else
# If the --log-bin option is present without a value, then
# we take the default name:
readonly WSREP_SST_OPT_BINLOG='mysql-bin'
fi
fi
fi
if [ -n "$WSREP_SST_OPT_BINLOG" ]; then
# If the name of the index file is not specified, then we will build
# it according to the specifications for the server:
if [ -z "$WSREP_SST_OPT_BINLOG_INDEX" ]; then
if [ -n "$WSREP_SST_OPT_LOG_BASENAME" ]; then
# If the WSREP_SST_OPT_BINLOG variable is not set, but
# --log-basename is present among the arguments of mysqld,
# then set WSREP_SST_OPT_BINLOG equal to the base name with
# the "-bin" suffix:
readonly WSREP_SST_OPT_BINLOG_INDEX="$WSREP_SST_OPT_LOG_BASENAME-bin.index"
else
# If the base name not specified, then we take
# the default name:
readonly WSREP_SST_OPT_BINLOG_INDEX='mysql-bin.index'
fi
fi
fi
}
# Check the presence of the port value and, if necessary, transfer
# the port number from the address to the WSREP_SST_OPT_PORT variable
# or vice versa, and also, if necessary, substitute the missing port
# value into the address value:
if [ -n "$WSREP_SST_OPT_ADDR_PORT" ]; then
if [ -n "$WSREP_SST_OPT_PORT" ]; then
if [ "$WSREP_SST_OPT_PORT" != "$WSREP_SST_OPT_ADDR_PORT" ]; then
echo "WSREP_SST: [ERROR] port in --port=$WSREP_SST_OPT_PORT differs from port in --address=$WSREP_SST_OPT_ADDR" >&2
exit 2
fi
else
# If the address contains a port number, assign it to
# the corresponding variable:
readonly WSREP_SST_OPT_PORT="$WSREP_SST_OPT_ADDR_PORT"
fi
elif [ -n "$WSREP_SST_OPT_ADDR" ]; then
# If the port is missing, take the default port:
if [ -z "$WSREP_SST_OPT_PORT" ]; then
readonly WSREP_SST_OPT_PORT=4444
fi
WSREP_SST_OPT_ADDR_PORT="$WSREP_SST_OPT_PORT"
# Let's remove the leading part that contains the host address:
remain="${WSREP_SST_OPT_ADDR#$WSREP_SST_OPT_HOST_ESCAPED}"
# Let's remove the ":" character that separates the port number
# from the hostname:
remain="${remain#:}"
# Let's remove all characters upto first "/" character that
# separates the hostname with port number from the path:
remain="${remain#/}"
# Let's construct a new value for the address with the port:
WSREP_SST_OPT_ADDR="$WSREP_SST_OPT_HOST:$WSREP_SST_OPT_PORT"
if [ -n "$remain" ]; then
WSREP_SST_OPT_ADDR="$WSREP_SST_OPT_ADDR/$remain"
fi
fi
readonly WSREP_SST_OPT_ADDR
readonly WSREP_SST_OPT_ADDR_PORT
# try to use my_print_defaults, mysql and mysqldump that come with the sources
# (for MTR suite)
@ -279,13 +532,13 @@ CLIENT_DIR="$SCRIPTS_DIR/../client"
if [ -x "$CLIENT_DIR/mysql" ]; then
MYSQL_CLIENT="$CLIENT_DIR/mysql"
else
MYSQL_CLIENT=$(which mysql)
MYSQL_CLIENT="$(command -v mysql)"
fi
if [ -x "$CLIENT_DIR/mysqldump" ]; then
MYSQLDUMP="$CLIENT_DIR/mysqldump"
else
MYSQLDUMP=$(which mysqldump)
MYSQLDUMP="$(command -v mysqldump)"
fi
if [ -x "$SCRIPTS_DIR/my_print_defaults" ]; then
@ -293,7 +546,7 @@ if [ -x "$SCRIPTS_DIR/my_print_defaults" ]; then
elif [ -x "$EXTRA_DIR/my_print_defaults" ]; then
MY_PRINT_DEFAULTS="$EXTRA_DIR/my_print_defaults"
else
MY_PRINT_DEFAULTS=$(which my_print_defaults)
MY_PRINT_DEFAULTS="$(command -v my_print_defaults)"
fi
wsrep_defaults="$WSREP_SST_OPT_DEFAULT"
@ -307,29 +560,133 @@ fi
readonly WSREP_SST_OPT_CONF="$wsrep_defaults$WSREP_SST_OPT_SUFFIX_DEFAULT"
readonly MY_PRINT_DEFAULTS="$MY_PRINT_DEFAULTS $WSREP_SST_OPT_CONF"
wsrep_auth_not_set()
#
# User can specify mariabackup specific settings that will be used during sst
# process like encryption, etc. Parse such configuration option.
#
# 1st parameter: group (config file section like sst) or
# my_print_defaults argument (like --mysqld)
# 2nd parameter: var : name of the variable in the section, e.g. server-id
# 3rd parameter: default value for the parameter
#
parse_cnf()
{
[ -z "$WSREP_SST_OPT_AUTH" -o "$WSREP_SST_OPT_AUTH" = "(null)" ]
local group="$1"
local var="$2"
local reval=""
# normalize the variable names specified in cnf file (user can use _ or - for example log-bin or log_bin)
# then search for needed variable
# finally get the variable value (if variables has been specified multiple time use the last value only)
if [ "$group" = '--mysqld' -o \
"$group" = 'mysqld' ]; then
if [ -n "$WSREP_SST_OPT_SUFFIX_VALUE" ]; then
reval=$($MY_PRINT_DEFAULTS "mysqld$WSREP_SST_OPT_SUFFIX_VALUE" | awk 'BEGIN {OFS=FS="="} {sub(/^--loose/,"-",$0); gsub(/_/,"-",$1); if ($1=="--'"$var"'") lastval=substr($0,length($1)+2)} END {print lastval}')
fi
fi
if [ -z "$reval" ]; then
reval=$($MY_PRINT_DEFAULTS "$group" | awk 'BEGIN {OFS=FS="="} {sub(/^--loose/,"-",$0); gsub(/_/,"-",$1); if ($1=="--'"$var"'") lastval=substr($0,length($1)+2)} END {print lastval}')
fi
# use default if we haven't found a value
if [ -z "$reval" ]; then
[ -n "${3:-}" ] && reval="$3"
fi
echo $reval
}
# State Snapshot Transfer authentication password was displayed in the ps output. Bug fixed #1200727.
if $MY_PRINT_DEFAULTS sst | grep -q "wsrep_sst_auth"; then
if wsrep_auth_not_set; then
WSREP_SST_OPT_AUTH=$($MY_PRINT_DEFAULTS sst | grep -- "--wsrep_sst_auth" | cut -d= -f2)
#
# This function simply checks for the presence of the parameter
# in the config file, but does not return its value. It returns "1"
# (true) even if the parameter is present in the configuration file
# without a value:
#
in_config()
{
local group="$1"
local var="$2"
local found=0
if [ "$group" = '--mysqld' -o \
"$group" = 'mysqld' ]; then
if [ -n "$WSREP_SST_OPT_SUFFIX_VALUE" ]; then
found=$($MY_PRINT_DEFAULTS "mysqld$WSREP_SST_OPT_SUFFIX_VALUE" | awk 'BEGIN {OFS=FS="="; found=0} {sub(/^--loose/,"-",$0); gsub(/_/,"-",$1); if ($1=="--'"$var"'") found=1} END {print found}')
fi
fi
fi
readonly WSREP_SST_OPT_AUTH
if [ $found -eq 0 ]; then
found=$($MY_PRINT_DEFAULTS "$group" | awk 'BEGIN {OFS=FS="="; found=0} {sub(/^--loose/,"-",$0); gsub(/_/,"-",$1); if ($1=="--'"$var"'") found=1} END {print found}')
fi
echo $found
}
# Splitting AUTH into potential user:password pair
wsrep_auth_not_set()
{
[ -z "$WSREP_SST_OPT_AUTH" ]
}
# Get rid of incorrect values resulting from substitution
# in programs external to the script:
if [ "$WSREP_SST_OPT_USER" = '(null)' ]; then
WSREP_SST_OPT_USER=""
fi
if [ "$WSREP_SST_OPT_PSWD" = '(null)' ]; then
WSREP_SST_OPT_PSWD=""
fi
if [ "$WSREP_SST_OPT_AUTH" = '(null)' ]; then
WSREP_SST_OPT_AUTH=""
fi
# Let's read the value of the authentication string from the
# configuration file so that it does not go to the command line
# and does not appear in the ps output:
if wsrep_auth_not_set; then
WSREP_SST_OPT_AUTH=$(parse_cnf 'sst' 'wsrep-sst-auth')
fi
# Splitting WSREP_SST_OPT_AUTH as "user:password" pair:
if ! wsrep_auth_not_set
then
WSREP_SST_OPT_USER="${WSREP_SST_OPT_AUTH%%:*}"
WSREP_SST_OPT_PSWD="${WSREP_SST_OPT_AUTH##*:}"
# Extract username as shortest prefix up to first ':' character:
WSREP_SST_OPT_AUTH_USER="${WSREP_SST_OPT_AUTH%%:*}"
if [ -z "$WSREP_SST_OPT_USER" ]; then
# if the username is not in the command line arguments,
# set the username and password using WSREP_SST_OPT_AUTH
# from the environment:
WSREP_SST_OPT_USER="$WSREP_SST_OPT_AUTH_USER"
WSREP_SST_OPT_PSWD="${WSREP_SST_OPT_AUTH#*:}"
elif [ "$WSREP_SST_OPT_USER" = "$WSREP_SST_OPT_AUTH_USER" ]; then
# If the username in the command line arguments and in
# the environment variable are the same, set the password
# if it was not specified in the command line:
if [ -z "$WSREP_SST_OPT_PSWD" ]; then
WSREP_SST_OPT_PSWD="${WSREP_SST_OPT_AUTH#*:}"
fi
else
# The username is passed through the command line and does
# not match the username in the environment variable - ignore
# the environment and rebuild the authentication parameters:
WSREP_SST_OPT_AUTH="$WSREP_SST_OPT_USER:$WSREP_SST_OPT_PSWD"
fi
fi
readonly WSREP_SST_OPT_USER
readonly WSREP_SST_OPT_PSWD
readonly WSREP_SST_OPT_AUTH
if [ -n "${WSREP_SST_OPT_DATA:-}" ]
if [ -n "$WSREP_SST_OPT_REMOTE_AUTH" ]
then
# Split auth string at the last ':'
readonly WSREP_SST_OPT_REMOTE_USER="${WSREP_SST_OPT_REMOTE_AUTH%%:*}"
readonly WSREP_SST_OPT_REMOTE_PSWD="${WSREP_SST_OPT_REMOTE_AUTH#*:}"
else
readonly WSREP_SST_OPT_REMOTE_USER=
readonly WSREP_SST_OPT_REMOTE_PSWD=
fi
readonly WSREP_SST_OPT_REMOTE_AUTH
if [ -n "$WSREP_SST_OPT_DATA" ]
then
SST_PROGRESS_FILE="$WSREP_SST_OPT_DATA/sst_in_progress"
else
@ -361,15 +718,14 @@ wsrep_log_info()
wsrep_cleanup_progress_file()
{
[ -n "${SST_PROGRESS_FILE:-}" ] && rm -f "$SST_PROGRESS_FILE" 2>/dev/null || true
[ -n "$SST_PROGRESS_FILE" ] && rm -f "$SST_PROGRESS_FILE" 2>/dev/null || true
}
wsrep_check_program()
{
local prog=$1
if ! which $prog >/dev/null
then
local prog="$1"
local cmd=$(command -v "$prog")
if [ ! -x "$cmd" ]; then
echo "'$prog' not found in PATH"
return 2 # no such file or directory
fi
@ -388,29 +744,42 @@ wsrep_check_programs()
return $ret
}
#
# user can specify mariabackup specific settings that will be used during sst
# process like encryption, etc.....
# parse such configuration option. (group for xb settings is [sst] in my.cnf
#
# 1st param: group (config file section like sst) or my_print_defaults argument (like --mysqld)
# 2nd param: var : name of the variable in the section, e.g. server-id
# 3rd param: - : default value for the param
parse_cnf()
wsrep_check_datadir()
{
local group=$1
local var=$2
local reval=""
# normalize the variable names specified in cnf file (user can use _ or - for example log-bin or log_bin)
# then search for needed variable
# finally get the variable value (if variables has been specified multiple time use the last value only)
reval=$($MY_PRINT_DEFAULTS "${group}" | awk -v var="${var}" 'BEGIN { OFS=FS="=" } { gsub(/_/,"-",$1); if ( $1=="--"var) lastval=substr($0,length($1)+2) } END { print lastval}')
# use default if we haven't found a value
if [ -z "$reval" ]; then
[ -n "$3" ] && reval=$3
if [ -z "$WSREP_SST_OPT_DATA" ]
then
wsrep_log_error "The '--datadir' parameter must be passed to the SST script"
exit 2
fi
}
get_openssl()
{
# If the OPENSSL_BINARY variable is already defined, just return:
if [ -n "${OPENSSL_BINARY+x}" ]; then
return
fi
# Let's look for openssl:
OPENSSL_BINARY="$(command -v openssl)"
if [ ! -x "$OPENSSL_BINARY" ]; then
OPENSSL_BINARY='/usr/bin/openssl'
if [ ! -x "$OPENSSL_BINARY" ]; then
OPENSSL_BINARY=""
fi
fi
readonly OPENSSL_BINARY
}
# Generate a string equivalent to 16 random bytes
wsrep_gen_secret()
{
get_openssl
if [ -n "$OPENSSL_BINARY" ]
then
echo $("$OPENSSL_BINARY" rand -hex 16)
else
printf "%04x%04x%04x%04x%04x%04x%04x%04x" \
$RANDOM $RANDOM $RANDOM $RANDOM \
$RANDOM $RANDOM $RANDOM $RANDOM
fi
echo $reval
}

787
scripts/wsrep_sst_mariabackup.sh
View File

File diff suppressed because it is too large Load Diff

6
scripts/wsrep_sst_mysqldump.sh
View File

@ -1,5 +1,6 @@
#!/bin/bash -ue
# Copyright (C) 2009-2015 Codership Oy
# Copyright (C) 2017-2021 MariaDB
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@ -25,6 +26,7 @@ EINVAL=22
local_ip()
{
[ "$1" = "127.0.0.1" ] && return 0
[ "$1" = "127.0.0.2" ] && return 0
[ "$1" = "localhost" ] && return 0
[ "$1" = "[::1]" ] && return 0
[ "$1" = "$(hostname -s)" ] && return 0
@ -138,8 +140,8 @@ then
# turned off for the session so that gtid state does not get altered while
# the dump gets replayed on joiner.
if [[ "$LOG_BIN" == 'ON' ]]; then
RESET_MASTER="RESET MASTER;"
SET_GTID_BINLOG_STATE="SET @@global.gtid_binlog_state='$GTID_BINLOG_STATE';"
RESET_MASTER="SET GLOBAL wsrep_on=OFF; RESET MASTER; SET GLOBAL wsrep_on=ON;"
SET_GTID_BINLOG_STATE="SET GLOBAL wsrep_on=OFF; SET @@global.gtid_binlog_state='$GTID_BINLOG_STATE'; SET GLOBAL wsrep_on=ON;"
SQL_LOG_BIN_OFF="SET @@session.sql_log_bin=OFF;"
fi
fi

339
scripts/wsrep_sst_rsync.sh
View File

@ -1,6 +1,7 @@
#!/bin/bash -ue
# Copyright (C) 2010-2014 Codership Oy
# Copyright (C) 2017-2021 MariaDB
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@ -23,12 +24,13 @@ RSYNC_CONF= # rsync configuration file
RSYNC_REAL_PID= # rsync process id
OS=$(uname)
[ "$OS" = "Darwin" ] && export -n LD_LIBRARY_PATH
[ "$OS" = 'Darwin' ] && export -n LD_LIBRARY_PATH
# Setting the path for lsof on CentOS
export PATH="/usr/sbin:/sbin:$PATH"
. $(dirname $0)/wsrep_sst_common
wsrep_check_datadir
wsrep_check_programs rsync
@ -54,13 +56,13 @@ cleanup_joiner()
# Check whether rsync process is still running.
check_pid()
{
local pid_file=$1
[ -r "$pid_file" ] && ps -p $(cat $pid_file) >/dev/null 2>&1
local pid_file="$1"
[ -r "$pid_file" ] && ps -p $(cat "$pid_file") >/dev/null 2>&1
}
check_pid_and_port()
{
local pid_file=$1
local pid_file="$1"
local rsync_pid=$2
local rsync_addr=$3
local rsync_port=$4
@ -73,11 +75,10 @@ check_pid_and_port()
grep -E '[[:space:]]+(rsync|stunnel)[[:space:]]+'"$rsync_pid" 2>/dev/null)"
;;
*)
if ! which lsof > /dev/null; then
if [ ! -x "$(command -v lsof)" ]; then
wsrep_log_error "lsof tool not found in PATH! Make sure you have it installed."
exit 2 # ENOENT
fi
local port_info="$(lsof -i :$rsync_port -Pn 2>/dev/null | \
grep "(LISTEN)")"
local is_rsync="$(echo $port_info | \
@ -96,22 +97,21 @@ check_pid_and_port()
exit 16 # EBUSY
fi
fi
check_pid $pid_file && \
check_pid "$pid_file" && \
[ -n "$port_info" ] && [ -n "$is_rsync" ] && \
[ $(cat $pid_file) -eq $rsync_pid ]
[ $(cat "$pid_file") -eq $rsync_pid ]
}
is_local_ip()
{
local address="$1"
local get_addr_bin=`which ifconfig`
local get_addr_bin="$(command -v ifconfig)"
if [ -z "$get_addr_bin" ]
then
get_addr_bin=`which ip`
get_addr_bin="$get_addr_bin address show"
get_addr_bin="$(command -v ip) address show"
# Add an slash at the end, so we don't get false positive : 172.18.0.4 matches 172.18.0.41
# ip output format is "X.X.X.X/mask"
address="${address}/"
address="$address/"
else
# Add an space at the end, so we don't get false positive : 172.18.0.4 matches 172.18.0.41
# ifconfig output format is "X.X.X.X "
@ -134,64 +134,64 @@ BINLOG_TAR_FILE="$WSREP_SST_OPT_DATA/wsrep_sst_binlog.tar"
BINLOG_N_FILES=1
rm -f "$BINLOG_TAR_FILE" || :
if ! [ -z $WSREP_SST_OPT_BINLOG ]
then
BINLOG_DIRNAME=$(dirname $WSREP_SST_OPT_BINLOG)
BINLOG_FILENAME=$(basename $WSREP_SST_OPT_BINLOG)
BINLOG_INDEX_DIRNAME=$(dirname $WSREP_SST_OPT_BINLOG)
BINLOG_INDEX_FILENAME=$(basename $WSREP_SST_OPT_BINLOG)
get_binlog
if [ -n "$WSREP_SST_OPT_BINLOG" ]; then
BINLOG_DIRNAME=$(dirname "$WSREP_SST_OPT_BINLOG")
BINLOG_FILENAME=$(basename "$WSREP_SST_OPT_BINLOG")
fi
if ! [ -z $WSREP_SST_OPT_BINLOG_INDEX ]
then
BINLOG_INDEX_DIRNAME=$(dirname $WSREP_SST_OPT_BINLOG_INDEX)
BINLOG_INDEX_FILENAME=$(basename $WSREP_SST_OPT_BINLOG_INDEX)
fi
WSREP_LOG_DIR=${WSREP_LOG_DIR:-""}
# Try to set WSREP_LOG_DIR from the command line:
if [ ! -z "$INNODB_LOG_GROUP_HOME_ARG" ]; then
WSREP_LOG_DIR=$INNODB_LOG_GROUP_HOME_ARG
fi
# if no command line arg and WSREP_LOG_DIR is not set,
# if no command line argument and INNODB_LOG_GROUP_HOME is not set,
# try to get it from my.cnf:
if [ -z "$WSREP_LOG_DIR" ]; then
WSREP_LOG_DIR=$(parse_cnf mysqld$WSREP_SST_OPT_SUFFIX_VALUE innodb-log-group-home-dir '')
fi
if [ -z "$WSREP_LOG_DIR" ]; then
WSREP_LOG_DIR=$(parse_cnf --mysqld innodb-log-group-home-dir '')
if [ -z "$INNODB_LOG_GROUP_HOME" ]; then
INNODB_LOG_GROUP_HOME=$(parse_cnf '--mysqld' 'innodb-log-group-home-dir')
fi
OLD_PWD="$(pwd)"
WSREP_LOG_DIR="$INNODB_LOG_GROUP_HOME"
if [ -n "$WSREP_LOG_DIR" ]; then
# handle both relative and absolute paths
WSREP_LOG_DIR=$(cd $WSREP_SST_OPT_DATA; mkdir -p "$WSREP_LOG_DIR"; cd $WSREP_LOG_DIR; pwd -P)
WSREP_LOG_DIR=$(cd "$WSREP_SST_OPT_DATA"; mkdir -p "$WSREP_LOG_DIR"; cd "$WSREP_LOG_DIR"; pwd -P)
else
# default to datadir
WSREP_LOG_DIR=$(cd $WSREP_SST_OPT_DATA; pwd -P)
WSREP_LOG_DIR=$(cd "$WSREP_SST_OPT_DATA"; pwd -P)
fi
INNODB_DATA_HOME_DIR=${INNODB_DATA_HOME_DIR:-""}
# Try to set INNODB_DATA_HOME_DIR from the command line:
if [ ! -z "$INNODB_DATA_HOME_DIR_ARG" ]; then
INNODB_DATA_HOME_DIR=$INNODB_DATA_HOME_DIR_ARG
fi
# if no command line arg and INNODB_DATA_HOME_DIR environment variable
cd "$OLD_PWD"
# if no command line argument and INNODB_DATA_HOME_DIR environment variable
# is not set, try to get it from my.cnf:
if [ -z "$INNODB_DATA_HOME_DIR" ]; then
INNODB_DATA_HOME_DIR=$(parse_cnf mysqld$WSREP_SST_OPT_SUFFIX_VALUE innodb-data-home-dir '')
fi
if [ -z "$INNODB_DATA_HOME_DIR" ]; then
INNODB_DATA_HOME_DIR=$(parse_cnf --mysqld innodb-data-home-dir '')
INNODB_DATA_HOME_DIR=$(parse_cnf '--mysqld' 'innodb-data-home-dir')
fi
if [ -n "$INNODB_DATA_HOME_DIR" ]; then
# handle both relative and absolute paths
INNODB_DATA_HOME_DIR=$(cd $WSREP_SST_OPT_DATA; mkdir -p "$INNODB_DATA_HOME_DIR"; cd $INNODB_DATA_HOME_DIR; pwd -P)
INNODB_DATA_HOME_DIR=$(cd "$WSREP_SST_OPT_DATA"; mkdir -p "$INNODB_DATA_HOME_DIR"; cd "$INNODB_DATA_HOME_DIR"; pwd -P)
else
# default to datadir
INNODB_DATA_HOME_DIR=$(cd $WSREP_SST_OPT_DATA; pwd -P)
INNODB_DATA_HOME_DIR=$(cd "$WSREP_SST_OPT_DATA"; pwd -P)
fi
cd "$OLD_PWD"
# if no command line argument then try to get it from my.cnf:
if [ -z "$INNODB_UNDO_DIR" ]; then
INNODB_UNDO_DIR=$(parse_cnf '--mysqld' 'innodb-undo-directory')
fi
if [ -n "$INNODB_UNDO_DIR" ]; then
# handle both relative and absolute paths
INNODB_UNDO_DIR=$(cd "$WSREP_SST_OPT_DATA"; mkdir -p "$INNODB_UNDO_DIR"; cd "$INNODB_UNDO_DIR"; pwd -P)
else
# default to datadir
INNODB_UNDO_DIR=$(cd "$WSREP_SST_OPT_DATA"; pwd -P)
fi
cd "$OLD_PWD"
# Old filter - include everything except selected
# FILTER=(--exclude '*.err' --exclude '*.pid' --exclude '*.sock' \
# --exclude '*.conf' --exclude core --exclude 'galera.*' \
@ -206,30 +206,99 @@ FILTER="-f '- /lost+found'
-f '+ /wsrep_sst_binlog.tar'
-f '- $INNODB_DATA_HOME_DIR/ib_lru_dump'
-f '- $INNODB_DATA_HOME_DIR/ibdata*'
-f '+ /undo*'
-f '+ $INNODB_UNDO_DIR/undo*'
-f '+ /*/'
-f '- /*'"
SSTKEY=$(parse_cnf sst tkey "")
SSTCERT=$(parse_cnf sst tcert "")
STUNNEL=""
if [ -f "$SSTKEY" ] && [ -f "$SSTCERT" ] && wsrep_check_programs stunnel
# old-style SSL config
SSTKEY=$(parse_cnf 'sst' 'tkey')
SSTCERT=$(parse_cnf 'sst' 'tcert')
SSTCA=$(parse_cnf 'sst' 'tca')
check_server_ssl_config()
{
local section="$1"
SSTKEY=$(parse_cnf "$section" 'ssl-key')
SSTCERT=$(parse_cnf "$section" 'ssl-cert')
SSTCA=$(parse_cnf "$section" 'ssl-ca')
}
SSLMODE=$(parse_cnf 'sst' 'ssl-mode' | tr [:lower:] [:upper:])
if [ -z "$SSTKEY" -a -z "$SSTCERT" ]
then
STUNNEL="stunnel ${STUNNEL_CONF}"
# no old-style SSL config in [sst], check for new one
check_server_ssl_config 'sst'
if [ -z "$SSTKEY" -a -z "$SSTCERT" ]; then
check_server_ssl_config '--mysqld'
fi
fi
if [ "$WSREP_SST_OPT_ROLE" = "donor" ]
if [ -z "$SSLMODE" ]; then
# Implicit verification if CA is set and the SSL mode
# is not specified by user:
if [ -n "$SSTCA" ]; then
if [ -x "$(command -v stunnel)" ]; then
SSLMODE='VERIFY_CA'
fi
# Require SSL by default if SSL key and cert are present:
elif [ -n "$SSTKEY" -a -n "$SSTCERT" ]; then
SSLMODE='REQUIRED'
fi
fi
if [ -n "$SSTCA" ]
then
CAFILE_OPT="CAfile = $SSTCA"
else
CAFILE_OPT=""
fi
if [ "${SSLMODE#VERIFY}" != "$SSLMODE" ]
then
case "$SSLMODE" in
'VERIFY_IDENTITY')
VERIFY_OPT='verifyPeer = yes'
;;
'VERIFY_CA')
VERIFY_OPT='verifyChain = yes'
;;
*)
wsrep_log_error "Unrecognized ssl-mode option: '$SSLMODE'"
exit 22 # EINVAL
esac
if [ -z "$CAFILE_OPT" ]
then
wsrep_log_error "Can't have ssl-mode=$SSLMODE without CA file"
exit 22 # EINVAL
fi
else
VERIFY_OPT=""
fi
STUNNEL=""
if [ -n "$SSLMODE" -a "$SSLMODE" != 'DISABLED' ] && wsrep_check_programs stunnel
then
wsrep_log_info "Using stunnel for SSL encryption: CAfile: $SSTCA, SSLMODE: $SSLMODE"
STUNNEL="stunnel $STUNNEL_CONF"
fi
readonly SECRET_TAG="secret"
if [ "$WSREP_SST_OPT_ROLE" = 'donor' ]
then
cat << EOF > "$STUNNEL_CONF"
CApath = ${SSTCERT%/*}
key = $SSTKEY
cert = $SSTCERT
${CAFILE_OPT}
foreground = yes
pid = $STUNNEL_PID
debug = warning
client = yes
connect = ${WSREP_SST_OPT_ADDR%/*}
TIMEOUTclose = 0
verifyPeer = yes
${VERIFY_OPT}
EOF
if [ $WSREP_SST_OPT_BYPASS -eq 0 ]
@ -242,7 +311,7 @@ EOF
rm -rf "$ERROR"
# Use deltaxfer only for WAN
inv=$(basename $0)
inv=$(basename "$0")
[ "$inv" = "wsrep_sst_rsync_wan" ] && WHOLE_FILE_OPT="" \
|| WHOLE_FILE_OPT="--whole-file"
@ -266,48 +335,43 @@ EOF
sleep 0.2
done
STATE="$(cat $FLUSHED)"
STATE=$(cat "$FLUSHED")
rm -rf "$FLUSHED"
sync
if ! [ -z $WSREP_SST_OPT_BINLOG ]
if [ -n "$WSREP_SST_OPT_BINLOG" ]
then
# Prepare binlog files
OLD_PWD="$(pwd)"
cd $BINLOG_DIRNAME
cd "$BINLOG_DIRNAME"
if ! [ -z $WSREP_SST_OPT_BINLOG_INDEX ]
binlog_files_full=$(tail -n $BINLOG_N_FILES ${BINLOG_FILENAME}.index)
then
cd $BINLOG_INDEX_DIRNAME
binlog_files_full=$(tail -n $BINLOG_N_FILES ${BINLOG_INDEX_FILENAME}.index)
fi
binlog_files_full=$(tail -n $BINLOG_N_FILES "${WSREP_SST_OPT_BINLOG_INDEX%.index}.index")
cd $BINLOG_DIRNAME
binlog_files=""
for ii in $binlog_files_full
do
binlog_files="$binlog_files $(basename $ii)"
binlog_file=$(basename "$ii")
binlog_files="$binlog_files $binlog_file"
done
if ! [ -z "$binlog_files" ]
if [ -n "$binlog_files" ]
then
wsrep_log_info "Preparing binlog files for transfer:"
tar -cvf $BINLOG_TAR_FILE $binlog_files >&2
tar -cvf "$BINLOG_TAR_FILE" $binlog_files >&2
fi
cd "$OLD_PWD"
fi
# first, the normal directories, so that we can detect incompatible protocol
RC=0
eval rsync ${STUNNEL:+--rsh="$STUNNEL"} \
eval rsync ${STUNNEL:+--rsh=\"$STUNNEL\"} \
--owner --group --perms --links --specials \
--ignore-times --inplace --dirs --delete --quiet \
$WHOLE_FILE_OPT ${FILTER} "$WSREP_SST_OPT_DATA/" \
rsync://$WSREP_SST_OPT_ADDR >&2 || RC=$?
if [ "$RC" -ne 0 ]; then
if [ $RC -ne 0 ]; then
wsrep_log_error "rsync returned code $RC:"
case $RC in
@ -341,7 +405,7 @@ EOF
rsync ${STUNNEL:+--rsh="$STUNNEL"} \
--owner --group --perms --links --specials \
--ignore-times --inplace --dirs --delete --quiet \
$WHOLE_FILE_OPT -f '+ /ib_logfile[0-9]*' -f '- **' "$WSREP_LOG_DIR/" \
$WHOLE_FILE_OPT -f '+ /ib_logfile[0-9]*' -f '+ /aria_log.*' -f '+ /aria_log_control' -f '- **' "$WSREP_LOG_DIR/" \
rsync://$WSREP_SST_OPT_ADDR-log_dir >&2 || RC=$?
if [ $RC -ne 0 ]; then
@ -350,8 +414,7 @@ EOF
fi
# then, we parallelize the transfer of database directories, use . so that pathconcatenation works
OLD_PWD="$(pwd)"
cd $WSREP_SST_OPT_DATA
cd "$WSREP_SST_OPT_DATA"
count=1
[ "$OS" = "Linux" ] && count=$(grep -c processor /proc/cpuinfo)
@ -362,7 +425,7 @@ EOF
rsync ${STUNNEL:+--rsh="$STUNNEL"} \
--owner --group --perms --links --specials \
--ignore-times --inplace --recursive --delete --quiet \
$WHOLE_FILE_OPT --exclude '*/ib_logfile*' "$WSREP_SST_OPT_DATA"/{}/ \
$WHOLE_FILE_OPT --exclude '*/ib_logfile*' --exclude "*/aria_log.*" --exclude "*/aria_log_control" "$WSREP_SST_OPT_DATA"/{}/ \
rsync://$WSREP_SST_OPT_ADDR/{} >&2 || RC=$?
cd "$OLD_PWD"
@ -385,50 +448,46 @@ EOF
echo "continue" # now server can resume updating data
echo "$STATE" > "$MAGIC_FILE"
if [ -n "$WSREP_SST_OPT_REMOTE_PSWD" ]; then
# Let joiner know that we know its secret
echo "$SECRET_TAG $WSREP_SST_OPT_REMOTE_PSWD" >> "$MAGIC_FILE"
fi
rsync ${STUNNEL:+--rsh="$STUNNEL"} \
--archive --quiet --checksum "$MAGIC_FILE" rsync://$WSREP_SST_OPT_ADDR
echo "done $STATE"
elif [ "$WSREP_SST_OPT_ROLE" = "joiner" ]
elif [ "$WSREP_SST_OPT_ROLE" = 'joiner' ]
then
wsrep_check_programs lsof
touch $SST_PROGRESS_FILE
MYSQLD_PID=$WSREP_SST_OPT_PARENT
touch "$SST_PROGRESS_FILE"
MYSQLD_PID="$WSREP_SST_OPT_PARENT"
MODULE="rsync_sst"
RSYNC_PID="$WSREP_SST_OPT_DATA/$MODULE.pid"
# give some time for lingering rsync from previous SST to complete
check_round=0
while check_pid $RSYNC_PID && [ $check_round -lt 10 ]
while check_pid "$RSYNC_PID" && [ $check_round -lt 10 ]
do
wsrep_log_info "lingering rsync daemon found at startup, waiting for it to exit"
check_round=$(( check_round + 1 ))
sleep 1
done
if check_pid $RSYNC_PID
if check_pid "$RSYNC_PID"
then
wsrep_log_error "rsync daemon already running."
exit 114 # EALREADY
fi
rm -rf "$RSYNC_PID"
ADDR=$WSREP_SST_OPT_ADDR
if [ "${ADDR#\[}" != "$ADDR" ]; then
RSYNC_PORT=$(echo $ADDR | awk -F '\\]:' '{ print $2 }')
RSYNC_ADDR=$(echo $ADDR | awk -F '\\]:' '{ print $1 }')"]"
else
RSYNC_PORT=$(echo $ADDR | awk -F ':' '{ print $2 }')
RSYNC_ADDR=$(echo $ADDR | awk -F ':' '{ print $1 }')
fi
if [ -z "$RSYNC_PORT" ]
then
RSYNC_PORT=4444
ADDR="$RSYNC_ADDR:$RSYNC_PORT"
fi
ADDR="$WSREP_SST_OPT_ADDR"
RSYNC_PORT="$WSREP_SST_OPT_PORT"
RSYNC_ADDR="$WSREP_SST_OPT_HOST"
trap "exit 32" HUP PIPE
trap "exit 3" INT TERM ABRT
@ -436,7 +495,7 @@ then
RSYNC_CONF="$WSREP_SST_OPT_DATA/$MODULE.conf"
if [ -n "${MYSQL_TMP_DIR:-}" ] ; then
if [ -n "${MYSQL_TMP_DIR:-}" ]; then
SILENT="log file = $MYSQL_TMP_DIR/rsyncd.log"
else
SILENT=""
@ -459,7 +518,6 @@ EOF
# rm -rf "$DATA"/ib_logfile* # we don't want old logs around
readonly RSYNC_PORT=${WSREP_SST_OPT_PORT:-4444}
# If the IP is local listen only in it
if is_local_ip "$RSYNC_ADDR"
then
@ -481,18 +539,19 @@ EOF
cat << EOF > "$STUNNEL_CONF"
key = $SSTKEY
cert = $SSTCERT
${CAFILE_OPT}
foreground = yes
pid = $STUNNEL_PID
debug = warning
client = no
[rsync]
accept = $STUNNEL_ACCEPT
exec = $(which rsync)
execargs = rsync --server --daemon --config=$RSYNC_CONF .
exec = $(command -v rsync)
execargs = rsync --server --daemon --config='$RSYNC_CONF' .
EOF
stunnel "$STUNNEL_CONF" &
RSYNC_REAL_PID=$!
RSYNC_PID=$STUNNEL_PID
RSYNC_PID="$STUNNEL_PID"
fi
until check_pid_and_port "$RSYNC_PID" "$RSYNC_REAL_PID" "$RSYNC_ADDR" "$RSYNC_PORT"
@ -500,7 +559,30 @@ EOF
sleep 0.2
done
echo "ready $WSREP_SST_OPT_HOST:$RSYNC_PORT/$MODULE"
if [ "${SSLMODE#VERIFY}" != "$SSLMODE" ]
then # backward-incompatible behavior
CN=""
if [ -n "$SSTCERT" ]
then
# find out my Common Name
get_openssl
if [ -z "$OPENSSL_BINARY" ]; then
wsrep_log_error 'openssl not found but it is required for authentication'
exit 42
fi
CN=$("$OPENSSL_BINARY" x509 -noout -subject -in "$SSTCERT" | \
tr "," "\n" | grep "CN =" | cut -d= -f2 | sed s/^\ // | \
sed s/\ %//)
fi
MY_SECRET=$(wsrep_gen_secret)
# Add authentication data to address
ADDR="$CN:$MY_SECRET@$WSREP_SST_OPT_HOST"
else
MY_SECRET="" # for check down in recv_joiner()
ADDR=$WSREP_SST_OPT_HOST
fi
echo "ready $ADDR:$RSYNC_PORT/$MODULE"
# wait for SST to complete by monitoring magic file
while [ ! -r "$MAGIC_FILE" ] && check_pid "$RSYNC_PID" && \
@ -513,37 +595,46 @@ EOF
then
wsrep_log_error \
"Parent mysqld process (PID:$MYSQLD_PID) terminated unexpectedly."
kill -- -"${MYSQLD_PID}"
kill -- -$MYSQLD_PID
sleep 1
exit 32
fi
if ! [ -z $WSREP_SST_OPT_BINLOG ]
then
OLD_PWD="$(pwd)"
cd $BINLOG_DIRNAME
if [ -n "$WSREP_SST_OPT_BINLOG" ]; then
if [ -f "$BINLOG_TAR_FILE" ]; then
cd "$BINLOG_DIRNAME"
binlog_index="${WSREP_SST_OPT_BINLOG_INDEX%.index}.index"
if [ -f $BINLOG_TAR_FILE ]
then
# Clean up old binlog files first
rm -f ${BINLOG_FILENAME}.*
wsrep_log_info "Extracting binlog files:"
tar -xvf $BINLOG_TAR_FILE >&2
for ii in $(ls -1 ${BINLOG_FILENAME}.*)
do
if ! [ -z $WSREP_SST_OPT_BINLOG_INDEX ]
echo ${BINLOG_DIRNAME}/${ii} >> ${BINLOG_FILENAME}.index
then
echo ${BINLOG_DIRNAME}/${ii} >> ${BINLOG_INDEX_DIRNAME}/${BINLOG_INDEX_FILENAME}.index
fi
done
fi
rm -f "$BINLOG_FILENAME".[0-9]*
[ -f "$binlog_index" ] && rm "$binlog_index"
cd "$OLD_PWD"
wsrep_log_info "Extracting binlog files:"
tar -xvf "$BINLOG_TAR_FILE" >> _binlog_tmp_files_$!
while read bin_file; do
echo "$BINLOG_DIRNAME/$bin_file" >> "$binlog_index"
done < _binlog_tmp_files_$!
rm -f _binlog_tmp_files_$!
cd "$OLD_PWD"
fi
fi
if [ -r "$MAGIC_FILE" ]
then
# check donor supplied secret
SECRET=$(grep "$SECRET_TAG " "$MAGIC_FILE" 2>/dev/null | cut -d ' ' -f 2)
if [ "$SECRET" != "$MY_SECRET" ]; then
wsrep_log_error "Donor does not know my secret!"
wsrep_log_info "Donor:'$SECRET', my:'$MY_SECRET'"
exit 32
fi
# remove secret from magic file
grep -v "$SECRET_TAG " "$MAGIC_FILE" > "$MAGIC_FILE.new"
mv "$MAGIC_FILE.new" "$MAGIC_FILE"
# UUID:seqno & wsrep_gtid_domain_id is received here.
cat "$MAGIC_FILE" # Output : UUID:seqno wsrep_gtid_domain_id
else
@ -558,6 +649,6 @@ else
exit 22 # EINVAL
fi
rm -f $BINLOG_TAR_FILE || :
rm -f "$BINLOG_TAR_FILE" || :
exit 0

399
scripts/wsrep_sst_xtrabackup-v2.sh
View File

@ -1,5 +1,6 @@
#!/bin/bash -ue
# Copyright (C) 2013 Percona Inc
# Copyright (C) 2017-2021 MariaDB
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@ -19,6 +20,7 @@
# Make sure to read that before proceeding!
. $(dirname $0)/wsrep_sst_common
wsrep_check_datadir
ealgo=""
ekey=""
@ -93,7 +95,7 @@ if ! which lsof > /dev/null; then
fi
timeit(){
local stage=$1
local stage="$1"
shift
local cmd="$@"
local x1 x2 took extcode
@ -137,17 +139,17 @@ get_keys()
wsrep_log_info "Xtrabackup based encryption enabled in my.cnf - Supported only from Xtrabackup 2.1.4"
if [[ -z $ealgo ]];then
if [[ -z "$ealgo" ]];then
wsrep_log_error "FATAL: Encryption algorithm empty from my.cnf, bailing out"
exit 3
fi
if [[ -z $ekey && ! -r $ekeyfile ]];then
if [[ -z "$ekey" && ! -r "$ekeyfile" ]];then
wsrep_log_error "FATAL: Either key or keyfile must be readable"
exit 3
fi
if [[ -z $ekey ]];then
if [[ -z "$ekey" ]];then
ecmd="xbcrypt --encrypt-algo=$ealgo --encrypt-key-file=$ekeyfile"
else
wsrep_log_warning "Using the 'encrypt-key' option causes the encryption key"
@ -199,8 +201,8 @@ check_for_dhparams()
#
verify_cert_matches_key()
{
local cert_path=$1
local key_path=$2
local cert_path="$1"
local key_path="$2"
wsrep_check_programs openssl diff
@ -226,9 +228,9 @@ verify_cert_matches_key()
#
verify_file_exists()
{
local file_path=$1
local error_message1=$2
local error_message2=$3
local file_path="$1"
local error_message1="$2"
local error_message2="$3"
if ! [[ -r "$file_path" ]]; then
wsrep_log_error "******** FATAL ERROR ************************* "
@ -246,7 +248,7 @@ verify_file_exists()
get_transfer()
{
TSST_PORT=${WSREP_SST_OPT_PORT:-4444}
TSST_PORT="$WSREP_SST_OPT_PORT"
if [[ $tfmt == 'nc' ]];then
if [[ ! -x `which nc` ]];then
@ -264,26 +266,46 @@ get_transfer()
wsrep_log_info "Using netcat as streamer"
if [[ "$WSREP_SST_OPT_ROLE" == "joiner" ]];then
if nc -h 2>&1 | grep -q ncat; then
# Ncat
tcmd="nc $sockopt -l ${TSST_PORT}"
elif nc -h 2>&1 | grep -q -- '-d\>';then
# Debian netcat
tcmd="nc $sockopt -dl ${TSST_PORT}"
else
# traditional netcat
tcmd="nc $sockopt -l -p ${TSST_PORT}"
fi
else
if nc -h 2>&1 | grep -q ncat;then
# Ncat
tcmd="nc ${WSREP_SST_OPT_HOST_UNESCAPED} ${TSST_PORT}"
elif nc -h 2>&1 | grep -q -- '-d\>';then
tcmd="nc -l ${TSST_PORT}"
elif nc -h 2>&1 | grep -qw -- '-d\>';then
# Debian netcat
tcmd="nc ${WSREP_SST_OPT_HOST_UNESCAPED} ${TSST_PORT}"
if [ $WSREP_SST_OPT_HOST_IPv6 -eq 1 ];then
# When host is not explicitly specified (when only the port
# is specified) netcat can only bind to an IPv4 address if
# the "-6" option is not explicitly specified:
tcmd="nc -dl -6 ${TSST_PORT}"
else
tcmd="nc -dl ${TSST_PORT}"
fi
else
# traditional netcat
tcmd="nc -q0 ${WSREP_SST_OPT_HOST_UNESCAPED} ${TSST_PORT}"
tcmd="nc -l -p ${TSST_PORT}"
fi
else
# Check to see if netcat supports the '-N' flag.
# -N Shutdown the network socket after EOF on stdin
# If it supports the '-N' flag, then we need to use the '-N'
# flag, otherwise the transfer will stay open after the file
# transfer and cause the command to timeout.
# Older versions of netcat did not need this flag and will
# return an error if the flag is used.
#
tcmd_extra=""
if nc -h 2>&1 | grep -qw -- -N;then
tcmd_extra+="-N"
fi
# netcat doesn't understand [] around IPv6 address
if nc -h 2>&1 | grep -q ncat;then
# Ncat
tcmd="nc ${tcmd_extra} ${WSREP_SST_OPT_HOST_UNESCAPED} ${TSST_PORT}"
elif nc -h 2>&1 | grep -qw -- '-d\>';then
# Debian netcat
tcmd="nc ${tcmd_extra} ${WSREP_SST_OPT_HOST_UNESCAPED} ${TSST_PORT}"
else
# traditional netcat
tcmd="nc -q0 ${tcmd_extra} ${WSREP_SST_OPT_HOST_UNESCAPED} ${TSST_PORT}"
fi
fi
else
@ -323,7 +345,7 @@ get_transfer()
if [[ "$WSREP_SST_OPT_ROLE" == "joiner" ]]; then
# dhparams check (will create ssl_dhparams if needed)
check_for_dhparams
joiner_extra=",dhparam=$ssl_dhparams"
joiner_extra=",dhparam='$ssl_dhparams'"
fi
fi
if check_for_version "$SOCAT_VERSION" "1.7.3"; then
@ -343,10 +365,10 @@ get_transfer()
stagemsg+="-OpenSSL-Encrypted-2"
if [[ "$WSREP_SST_OPT_ROLE" == "joiner" ]];then
wsrep_log_info "Decrypting with CERT: $tcert, CA: $tca"
tcmd="socat -u openssl-listen:${TSST_PORT},reuseaddr,cert=${tcert},cafile=${tca}${joiner_extra}${sockopt} stdio"
tcmd="socat -u openssl-listen:${TSST_PORT},reuseaddr,cert='${tcert}',cafile='${tca}'${joiner_extra}${sockopt} stdio"
else
wsrep_log_info "Encrypting with CERT: $tcert, CA: $tca"
tcmd="socat -u stdio openssl-connect:${WSREP_SST_OPT_HOST}:${TSST_PORT},cert=${tcert},cafile=${tca}${donor_extra}${sockopt}"
tcmd="socat -u stdio openssl-connect:${WSREP_SST_OPT_HOST}:${TSST_PORT},cert='${tcert}',cafile='${tca}'${donor_extra}${sockopt}"
fi
elif [[ $encrypt -eq 3 ]];then
wsrep_log_warning "**** WARNING **** encrypt=3 is deprecated and will be removed in a future release"
@ -360,10 +382,10 @@ get_transfer()
stagemsg+="-OpenSSL-Encrypted-3"
if [[ "$WSREP_SST_OPT_ROLE" == "joiner" ]];then
wsrep_log_info "Decrypting with CERT: $tcert, KEY: $tkey"
tcmd="socat -u openssl-listen:${TSST_PORT},reuseaddr,cert=${tcert},key=${tkey},verify=0${joiner_extra}${sockopt} stdio"
tcmd="socat -u openssl-listen:${TSST_PORT},reuseaddr,cert='${tcert}',key='${tkey}',verify=0${joiner_extra}${sockopt} stdio"
else
wsrep_log_info "Encrypting with CERT: $tcert, KEY: $tkey"
tcmd="socat -u stdio openssl-connect:${WSREP_SST_OPT_HOST}:${TSST_PORT},cert=${tcert},key=${tkey},verify=0${sockopt}"
tcmd="socat -u stdio openssl-connect:${WSREP_SST_OPT_HOST}:${TSST_PORT},cert='${tcert}',key='${tkey}',verify=0${sockopt}"
fi
elif [[ $encrypt -eq 4 ]]; then
wsrep_log_info "Using openssl based encryption with socat: with key, crt, and ca"
@ -381,10 +403,10 @@ get_transfer()
stagemsg+="-OpenSSL-Encrypted-4"
if [[ "$WSREP_SST_OPT_ROLE" == "joiner" ]]; then
wsrep_log_info "Decrypting with CERT: $ssl_cert, KEY: $ssl_key, CA: $ssl_ca"
tcmd="socat -u openssl-listen:${TSST_PORT},reuseaddr,cert=${ssl_cert},key=${ssl_key},cafile=${ssl_ca},verify=1${joiner_extra}${sockopt} stdio"
tcmd="socat -u openssl-listen:${TSST_PORT},reuseaddr,cert='${ssl_cert}',key='${ssl_key}',cafile='${ssl_ca}',verify=1${joiner_extra}${sockopt} stdio"
else
wsrep_log_info "Encrypting with CERT: $ssl_cert, KEY: $ssl_key, CA: $ssl_ca"
tcmd="socat -u stdio openssl-connect:${WSREP_SST_OPT_HOST}:${TSST_PORT},cert=${ssl_cert},key=${ssl_key},cafile=${ssl_ca},verify=1${donor_extra}${sockopt}"
tcmd="socat -u stdio openssl-connect:${WSREP_SST_OPT_HOST}:${TSST_PORT},cert='${ssl_cert}',key='${ssl_key}',cafile='${ssl_ca}',verify=1${donor_extra}${sockopt}"
fi
else
if [[ $encrypt -eq 1 ]]; then
@ -402,7 +424,7 @@ get_transfer()
get_footprint()
{
pushd $WSREP_SST_OPT_DATA 1>/dev/null
pushd "$WSREP_SST_OPT_DATA" 1>/dev/null
payload=$(find . -regex '.*\.ibd$\|.*\.MYI$\|.*\.MYD$\|.*ibdata1$' -type f -print0 | du --files0-from=- --block-size=1 -c -s | awk 'END { print $1 }')
if $MY_PRINT_DEFAULTS xtrabackup | grep -q -- "--compress";then
# QuickLZ has around 50% compression ratio
@ -425,18 +447,18 @@ adjust_progress()
return
fi
if [[ -n $progress && $progress != '1' ]];then
if [[ -e $progress ]];then
if [[ -n "$progress" && "$progress" != '1' ]];then
if [[ -e "$progress" ]];then
pcmd+=" 2>>$progress"
else
pcmd+=" 2>$progress"
fi
elif [[ -z $progress && -n $rlimit ]];then
elif [[ -z "$progress" && -n "$rlimit" ]];then
# When rlimit is non-zero
pcmd="pv -q"
fi
if [[ -n $rlimit && "$WSREP_SST_OPT_ROLE" == "donor" ]];then
if [[ -n "$rlimit" && "$WSREP_SST_OPT_ROLE" == "donor" ]];then
wsrep_log_info "Rate-limiting SST to $rlimit"
pcmd+=" -L \$rlimit"
fi
@ -466,7 +488,7 @@ read_cnf()
sdecomp=$(parse_cnf sst decompressor "")
# Refer to http://www.percona.com/doc/percona-xtradb-cluster/manual/xtrabackup_sst.html
if [[ -z $ealgo ]];then
if [[ -z "$ealgo" ]];then
ealgo=$(parse_cnf sst encrypt-algo "")
ekey=$(parse_cnf sst encrypt-key "")
ekeyfile=$(parse_cnf sst encrypt-key-file "")
@ -510,15 +532,15 @@ get_stream()
if [[ "$WSREP_SST_OPT_ROLE" == "joiner" ]];then
strmcmd="xbstream -x"
else
strmcmd="xbstream -c \${INFO_FILE}"
strmcmd="xbstream -c '${INFO_FILE}'"
fi
else
sfmt="tar"
wsrep_log_info "Streaming with tar"
if [[ "$WSREP_SST_OPT_ROLE" == "joiner" ]];then
strmcmd="tar xfi - "
strmcmd="tar xfi -"
else
strmcmd="tar cf - \${INFO_FILE} "
strmcmd="tar cf - '${INFO_FILE}'"
fi
fi
}
@ -547,13 +569,12 @@ cleanup_joiner()
wsrep_log_info "Removing the sst_in_progress file"
wsrep_cleanup_progress_file
fi
if [[ -n $progress && -p $progress ]];then
if [[ -n "$progress" && -p "$progress" ]];then
wsrep_log_info "Cleaning up fifo file $progress"
rm $progress
fi
if [[ -n ${STATDIR:-} ]];then
[[ -d $STATDIR ]] && rm -rf $STATDIR
if [[ -n "${STATDIR:-}" ]];then
[[ -d "$STATDIR" ]] && rm -rf "$STATDIR"
fi
# Final cleanup
@ -586,8 +607,8 @@ cleanup_donor()
wsrep_log_error "Cleanup after exit with status:$estatus"
fi
if [[ -n ${XTRABACKUP_PID:-} ]];then
if check_pid $XTRABACKUP_PID
if [[ -n "${XTRABACKUP_PID:-}" ]];then
if check_pid "$XTRABACKUP_PID"
then
wsrep_log_error "xtrabackup process is still running. Killing... "
kill_xtrabackup
@ -595,21 +616,21 @@ cleanup_donor()
fi
rm -f ${DATA}/${IST_FILE} || true
rm -f "${DATA}/${IST_FILE}" || true
if [[ -n $progress && -p $progress ]];then
if [[ -n "$progress" && -p "$progress" ]];then
wsrep_log_info "Cleaning up fifo file $progress"
rm -f $progress || true
rm -f "$progress" || true
fi
wsrep_log_info "Cleaning up temporary directories"
if [[ -n $xtmpdir ]];then
[[ -d $xtmpdir ]] && rm -rf $xtmpdir || true
if [[ -n "$xtmpdir" ]];then
[[ -d "$xtmpdir" ]] && rm -rf "$xtmpdir" || true
fi
if [[ -n $itmpdir ]];then
[[ -d $itmpdir ]] && rm -rf $itmpdir || true
if [[ -n "$itmpdir" ]];then
[[ -d "$itmpdir" ]] && rm -rf "$itmpdir" || true
fi
# Final cleanup
@ -632,7 +653,7 @@ cleanup_donor()
kill_xtrabackup()
{
local PID=$(cat $XTRABACKUP_PID)
local PID=$(cat "$XTRABACKUP_PID")
[ -n "$PID" -a "0" != "$PID" ] && kill $PID && (kill $PID && kill -9 $PID) || :
wsrep_log_info "Removing xtrabackup pid file $XTRABACKUP_PID"
rm -f "$XTRABACKUP_PID" || true
@ -642,9 +663,9 @@ kill_xtrabackup()
# (regardless of timeout)
wait_for_listen()
{
local HOST=$1
local PORT=$2
local MODULE=$3
local HOST="$1"
local PORT="$2"
local MODULE="$3"
local LSOF_OUT
for i in {1..300}
@ -660,10 +681,11 @@ wait_for_listen()
check_extra()
{
local use_socket=1
if [[ $uextra -eq 1 ]];then
if [ $(parse_cnf --mysqld thread-handling) = 'pool-of-threads'];then
local eport=$(parse_cnf --mysqld extra-port)
if [[ -n $eport ]];then
if [ $uextra -eq 1 ]; then
local thread_handling=$(parse_cnf '--mysqld' 'thread-handling')
if [ "$thread_handling" = 'pool-of-threads' ]; then
local eport=$(parse_cnf '--mysqld' 'extra-port')
if [ -n "$eport" ]; then
# Xtrabackup works only locally.
# Hence, setting host to 127.0.0.1 unconditionally.
wsrep_log_info "SST through extra_port $eport"
@ -677,25 +699,25 @@ check_extra()
wsrep_log_info "Thread pool not set, ignore the option use_extra"
fi
fi
if [[ $use_socket -eq 1 ]] && [[ -n "$WSREP_SST_OPT_SOCKET" ]];then
INNOEXTRA+=" --socket=$WSREP_SST_OPT_SOCKET"
if [ $use_socket -eq 1 -a -n "$WSREP_SST_OPT_SOCKET" ]; then
INNOEXTRA+=" --socket='$WSREP_SST_OPT_SOCKET'"
fi
}
recv_joiner()
{
local dir=$1
local msg=$2
local dir="$1"
local msg="$2"
local tmt=$3
local checkf=$4
local ltcmd
if [[ ! -d ${dir} ]];then
if [[ ! -d "${dir}" ]];then
# This indicates that IST is in progress
return
fi
pushd ${dir} 1>/dev/null
pushd "${dir}" 1>/dev/null
set +e
if [[ $tmt -gt 0 && -x `which timeout` ]];then
@ -736,10 +758,10 @@ recv_joiner()
send_donor()
{
local dir=$1
local msg=$2
local dir="$1"
local msg="$2"
pushd ${dir} 1>/dev/null
pushd "${dir}" 1>/dev/null
set +e
timeit "$msg" "$strmcmd | $tcmd; RC=( "\${PIPESTATUS[@]}" )"
set -e
@ -747,7 +769,7 @@ send_donor()
for ecode in "${RC[@]}";do
if [[ $ecode -ne 0 ]];then
wsrep_log_error "Error while getting data from donor node: " \
wsrep_log_error "Error while sending data to joiner node: " \
"exit codes: ${RC[@]}"
exit 32
fi
@ -821,7 +843,7 @@ fi
XB_REQUIRED_VERSION="2.3.5"
XB_VERSION=`$INNOBACKUPEX_BIN --version 2>&1 | grep -oe '[0-9]\.[0-9][\.0-9]*' | head -n1`
if [[ -z $XB_VERSION ]]; then
if [[ -z "$XB_VERSION" ]]; then
wsrep_log_error "FATAL: Cannot determine the $INNOBACKUPEX_BIN version. Needs xtrabackup-$XB_REQUIRED_VERSION or higher to perform SST"
exit 2
fi
@ -841,18 +863,17 @@ fi
read_cnf
if ${INNOBACKUPEX_BIN} /tmp --help 2>/dev/null | grep -q -- '--version-check'; then
disver="--no-version-check"
disver='--no-version-check'
fi
if [[ ${FORCE_FTWRL:-0} -eq 1 ]];then
iopts+=" --databases-exclude='lost+found'"
if [ ${FORCE_FTWRL:-0} -eq 1 ]; then
wsrep_log_info "Forcing FTWRL due to environment variable FORCE_FTWRL equal to $FORCE_FTWRL"
iopts+=" --no-backup-locks "
iopts+=' --no-backup-locks'
fi
INNOEXTRA=""
if [[ $ssyslog -eq 1 ]];then
if [[ ! -x `which logger` ]];then
wsrep_log_error "logger not in path: $PATH. Ignoring"
else
@ -874,44 +895,38 @@ if [[ $ssyslog -eq 1 ]];then
INNOMOVE="2>&1 | logger -p daemon.err -t ${ssystag}innobackupex-move "
INNOBACKUP="2> >(logger -p daemon.err -t ${ssystag}innobackupex-backup)"
fi
else
INNOAPPLY="&>\${DATA}/innobackup.prepare.log"
INNOMOVE="&>\${DATA}/innobackup.move.log"
INNOBACKUP="2>\${DATA}/innobackup.backup.log"
INNOAPPLY="&>'$DATA/innobackup.prepare.log'"
INNOMOVE="&>'$DATA/innobackup.move.log'"
INNOBACKUP="2>'$DATA/innobackup.backup.log'"
fi
get_stream
get_transfer
INNODB_DATA_HOME_DIR=${INNODB_DATA_HOME_DIR:-""}
# Try to set INNODB_DATA_HOME_DIR from the command line:
if [ ! -z "$INNODB_DATA_HOME_DIR_ARG" ]; then
INNODB_DATA_HOME_DIR=$INNODB_DATA_HOME_DIR_ARG
fi
# if no command line arg and INNODB_DATA_HOME_DIR environment variable
# if no command line argument and INNODB_DATA_HOME_DIR environment variable
# is not set, try to get it from my.cnf:
if [ -z "$INNODB_DATA_HOME_DIR" ]; then
INNODB_DATA_HOME_DIR=$(parse_cnf mysqld$WSREP_SST_OPT_SUFFIX_VALUE innodb-data-home-dir '')
fi
if [ -z "$INNODB_DATA_HOME_DIR" ]; then
INNODB_DATA_HOME_DIR=$(parse_cnf --mysqld innodb-data-home-dir '')
fi
if [ ! -z "$INNODB_DATA_HOME_DIR" ]; then
INNOEXTRA+=" --innodb-data-home-dir=$INNODB_DATA_HOME_DIR"
INNODB_DATA_HOME_DIR=$(parse_cnf '--mysqld' 'innodb-data-home-dir')
fi
OLD_PWD="$(pwd)"
if [ -n "$INNODB_DATA_HOME_DIR" ]; then
# handle both relative and absolute paths
INNODB_DATA_HOME_DIR=$(cd $DATA; mkdir -p "$INNODB_DATA_HOME_DIR"; cd $INNODB_DATA_HOME_DIR; pwd -P)
INNODB_DATA_HOME_DIR=$(cd "$DATA"; mkdir -p "$INNODB_DATA_HOME_DIR"; cd "$INNODB_DATA_HOME_DIR"; pwd -P)
else
# default to datadir
INNODB_DATA_HOME_DIR=$(cd $DATA; pwd -P)
INNODB_DATA_HOME_DIR=$(cd "$DATA"; pwd -P)
fi
INNOAPPLY="${INNOBACKUPEX_BIN} $disver $iapts \$INNOEXTRA --apply-log \$rebuildcmd \${DATA} ${INNOAPPLY}"
INNOMOVE="${INNOBACKUPEX_BIN} ${WSREP_SST_OPT_CONF} $disver $impts --move-back --force-non-empty-directories \${DATA} ${INNOMOVE}"
INNOBACKUP="${INNOBACKUPEX_BIN} ${WSREP_SST_OPT_CONF} $disver $iopts \$tmpopts \$INNOEXTRA --galera-info --stream=\$sfmt \$itmpdir ${INNOBACKUP}"
cd "$OLD_PWD"
setup_commands () {
INNOAPPLY="$INNOBACKUPEX_BIN $disver $iapts $INNOEXTRA --apply-log $rebuildcmd '$DATA' $INNOAPPLY"
INNOMOVE="$INNOBACKUPEX_BIN $WSREP_SST_OPT_CONF $disver $impts --move-back --force-non-empty-directories '$DATA' $INNOMOVE"
INNOBACKUP="$INNOBACKUPEX_BIN $WSREP_SST_OPT_CONF $disver $iopts $tmpopts $INNOEXTRA --galera-info --stream='$sfmt' '$itmpdir' $INNOBACKUP"
}
if [ "$WSREP_SST_OPT_ROLE" = "donor" ]
then
@ -920,31 +935,30 @@ then
if [ $WSREP_SST_OPT_BYPASS -eq 0 ]
then
usrst=0
if [[ -z $WSREP_SST_OPT_SST_VER ]];then
if [ -z "$WSREP_SST_OPT_SST_VER" ]; then
wsrep_log_error "Upgrade joiner to 5.6.21 or higher for backup locks support"
wsrep_log_error "The joiner is not supported for this version of donor"
exit 93
fi
if [[ -z $(parse_cnf mysqld$WSREP_SST_OPT_SUFFIX_VALUE tmpdir "") && \
-z $(parse_cnf --mysqld tmpdir "") && \
-z $(parse_cnf xtrabackup tmpdir "") ]]; then
if [ -z "$(parse_cnf --mysqld tmpdir)" -a \
-z "$(parse_cnf xtrabackup tmpdir)" ]; then
xtmpdir=$(mktemp -d)
tmpopts=" --tmpdir=$xtmpdir"
tmpopts=" --tmpdir='$xtmpdir'"
wsrep_log_info "Using $xtmpdir as xtrabackup temporary directory"
fi
itmpdir=$(mktemp -d)
wsrep_log_info "Using $itmpdir as innobackupex temporary directory"
if [[ -n "${WSREP_SST_OPT_USER:-}" && "$WSREP_SST_OPT_USER" != "(null)" ]]; then
INNOEXTRA+=" --user=$WSREP_SST_OPT_USER"
if [ -n "$WSREP_SST_OPT_USER" ]; then
INNOEXTRA+=" --user='$WSREP_SST_OPT_USER'"
usrst=1
fi
if [ -n "${WSREP_SST_OPT_PSWD:-}" ]; then
export MYSQL_PWD=$WSREP_SST_OPT_PSWD
elif [[ $usrst -eq 1 ]];then
if [ -n "$WSREP_SST_OPT_PSWD" ]; then
export MYSQL_PWD="$WSREP_SST_OPT_PSWD"
elif [ $usrst -eq 1 ]; then
# Empty password, used for testing, debugging etc.
unset MYSQL_PWD
fi
@ -960,24 +974,24 @@ then
ttcmd="$tcmd"
if [[ $encrypt -eq 1 ]];then
if [[ -n $scomp ]];then
tcmd=" \$ecmd | $scomp | $tcmd "
if [ $encrypt -eq 1 ]; then
if [ -n "$scomp" ]; then
tcmd="\$ecmd | $scomp | $tcmd"
else
tcmd=" \$ecmd | $tcmd "
tcmd="\$ecmd | $tcmd"
fi
elif [[ -n $scomp ]];then
tcmd=" $scomp | $tcmd "
elif [ -n "$scomp" ]; then
tcmd="$scomp | $tcmd"
fi
send_donor $DATA "${stagemsg}-gtid"
# Restore the transport commmand to its original state
tcmd="$ttcmd"
if [[ -n $progress ]];then
if [ -n "$progress" ]; then
get_footprint
tcmd="$pcmd | $tcmd"
elif [[ -n $rlimit ]];then
elif [ -n "$rlimit" ]; then
adjust_progress
tcmd="$pcmd | $tcmd"
fi
@ -985,10 +999,10 @@ then
wsrep_log_info "Sleeping before data transfer for SST"
sleep 10
wsrep_log_info "Streaming the backup to joiner at ${WSREP_SST_OPT_HOST} ${WSREP_SST_OPT_PORT:-4444}"
wsrep_log_info "Streaming the backup to joiner at ${WSREP_SST_OPT_HOST}:${WSREP_SST_OPT_PORT}"
# Add compression to the head of the stream (if specified)
if [[ -n $scomp ]]; then
if [[ -n "$scomp" ]]; then
tcmd="$scomp | $tcmd"
fi
@ -997,6 +1011,7 @@ then
tcmd=" \$ecmd | $tcmd "
fi
setup_commands
set +e
timeit "${stagemsg}-SST" "$INNOBACKUP | $tcmd; RC=( "\${PIPESTATUS[@]}" )"
set -e
@ -1023,18 +1038,18 @@ then
echo "${WSREP_SST_OPT_GTID} ${WSREP_SST_OPT_GTID_DOMAIN_ID}" > "${MAGIC_FILE}"
echo "1" > "${DATA}/${IST_FILE}"
get_keys
if [[ $encrypt -eq 1 ]];then
if [[ -n $scomp ]];then
tcmd=" \$ecmd | $scomp | $tcmd "
if [ $encrypt -eq 1 ]; then
if [ -n "$scomp" ]; then
tcmd="\$ecmd | $scomp | $tcmd"
else
tcmd=" \$ecmd | $tcmd "
tcmd="\$ecmd | $tcmd"
fi
elif [[ -n $scomp ]];then
tcmd=" $scomp | $tcmd "
elif [ -n "$scomp" ]; then
tcmd="$scomp | $tcmd"
fi
strmcmd+=" \${IST_FILE}"
send_donor $DATA "${stagemsg}-IST"
send_donor "$DATA" "${stagemsg}-IST"
fi
@ -1043,29 +1058,26 @@ then
elif [ "${WSREP_SST_OPT_ROLE}" = "joiner" ]
then
[[ -e $SST_PROGRESS_FILE ]] && wsrep_log_info "Stale sst_in_progress file: $SST_PROGRESS_FILE"
[[ -n $SST_PROGRESS_FILE ]] && touch $SST_PROGRESS_FILE
[[ -e "$SST_PROGRESS_FILE" ]] && wsrep_log_info "Stale sst_in_progress file: $SST_PROGRESS_FILE"
[[ -n "$SST_PROGRESS_FILE" ]] && touch "$SST_PROGRESS_FILE"
ib_home_dir=$INNODB_DATA_HOME_DIR
ib_home_dir="$INNODB_DATA_HOME_DIR"
# Try to set ib_log_dir from the command line:
ib_log_dir=$INNODB_LOG_GROUP_HOME_ARG
if [ -z "$ib_log_dir" ]; then
ib_log_dir=$(parse_cnf mysqld$WSREP_SST_OPT_SUFFIX_VALUE innodb-log-group-home-dir "")
fi
if [ -z "$ib_log_dir" ]; then
ib_log_dir=$(parse_cnf --mysqld innodb-log-group-home-dir "")
# if no command line argument and INNODB_LOG_GROUP_HOME is not set,
# try to get it from my.cnf:
if [ -z "$INNODB_LOG_GROUP_HOME" ]; then
INNODB_LOG_GROUP_HOME=$(parse_cnf '--mysqld' 'innodb-log-group-home-dir')
fi
# Try to set ib_undo_dir from the command line:
ib_undo_dir=$INNODB_UNDO_DIR_ARG
if [ -z "$ib_undo_dir" ]; then
ib_undo_dir=$(parse_cnf mysqld$WSREP_SST_OPT_SUFFIX_VALUE innodb-undo-directory "")
fi
if [ -z "$ib_undo_dir" ]; then
ib_undo_dir=$(parse_cnf --mysqld innodb-undo-directory "")
ib_log_dir="$INNODB_LOG_GROUP_HOME"
# if no command line argument then try to get it from my.cnf:
if [ -z "$INNODB_UNDO_DIR" ]; then
INNODB_UNDO_DIR=$(parse_cnf '--mysqld' 'innodb-undo-directory')
fi
ib_undo_dir="$INNODB_UNDO_DIR"
stagemsg="Joiner-Recv"
sencrypted=1
@ -1076,32 +1088,32 @@ then
rm -f "${DATA}/${IST_FILE}"
# May need xtrabackup_checkpoints later on
rm -f ${DATA}/xtrabackup_binary ${DATA}/xtrabackup_galera_info ${DATA}/xtrabackup_logfile
rm -f "${DATA}/xtrabackup_binary" "${DATA}/xtrabackup_galera_info" "${DATA}/xtrabackup_logfile"
wait_for_listen ${WSREP_SST_OPT_HOST} ${WSREP_SST_OPT_PORT:-4444} ${MODULE} &
wait_for_listen ${WSREP_SST_OPT_HOST} ${WSREP_SST_OPT_PORT} ${MODULE} &
trap sig_joiner_cleanup HUP PIPE INT TERM
trap cleanup_joiner EXIT
if [[ -n $progress ]];then
if [[ -n "$progress" ]];then
adjust_progress
tcmd+=" | $pcmd"
fi
get_keys
if [[ $encrypt -eq 1 && $sencrypted -eq 1 ]];then
if [[ -n $sdecomp ]];then
strmcmd=" $sdecomp | \$ecmd | $strmcmd"
if [ $encrypt -eq 1 -a $sencrypted -eq 1 ]; then
if [ -n "$sdecomp" ]; then
strmcmd="$sdecomp | \$ecmd | $strmcmd"
else
strmcmd=" \$ecmd | $strmcmd"
strmcmd="\$ecmd | $strmcmd"
fi
elif [[ -n $sdecomp ]];then
strmcmd=" $sdecomp | $strmcmd"
elif [ -n "$sdecomp" ]; then
strmcmd="$sdecomp | $strmcmd"
fi
STATDIR=$(mktemp -d)
MAGIC_FILE="${STATDIR}/${INFO_FILE}"
recv_joiner $STATDIR "${stagemsg}-gtid" $stimeout 1
recv_joiner "$STATDIR" "${stagemsg}-gtid" $stimeout 1
if ! ps -p ${WSREP_SST_OPT_PARENT} &>/dev/null
then
@ -1112,12 +1124,12 @@ then
if [ ! -r "${STATDIR}/${IST_FILE}" ]
then
if [[ -d ${DATA}/.sst ]];then
if [[ -d "${DATA}/.sst" ]];then
wsrep_log_info "WARNING: Stale temporary SST directory: ${DATA}/.sst from previous state transfer. Removing"
rm -rf ${DATA}/.sst
rm -rf "${DATA}/.sst"
fi
mkdir -p ${DATA}/.sst
(recv_joiner $DATA/.sst "${stagemsg}-SST" 0 0) &
mkdir -p "${DATA}/.sst"
(recv_joiner "$DATA/.sst" "${stagemsg}-SST" 0 0) &
jpid=$!
wsrep_log_info "Proceeding with SST"
@ -1128,25 +1140,19 @@ then
find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -prune -regex $cpat -o -exec rm -rfv {} 1>&2 \+
fi
tempdir=$LOG_BIN_ARG
if [ -z "$tempdir" ]; then
tempdir=$(parse_cnf mysqld$WSREP_SST_OPT_SUFFIX_VALUE log-bin "")
fi
if [ -z "$tempdir" ]; then
tempdir=$(parse_cnf --mysqld log-bin "")
fi
if [[ -n ${tempdir:-} ]];then
binlog_dir=$(dirname $tempdir)
binlog_file=$(basename $tempdir)
if [[ -n ${binlog_dir:-} && $binlog_dir != '.' && $binlog_dir != $DATA ]];then
pattern="$binlog_dir/$binlog_file\.[0-9]+$"
wsrep_log_info "Cleaning the binlog directory $binlog_dir as well"
find $binlog_dir -maxdepth 1 -type f -regex $pattern -exec rm -fv {} 1>&2 \+ || true
rm $binlog_dir/*.index || true
fi
get_binlog
if [ -n "$WSREP_SST_OPT_BINLOG" ]; then
binlog_dir=$(dirname "$WSREP_SST_OPT_BINLOG")
cd "$binlog_dir"
wsrep_log_info "Cleaning the binlog directory $binlog_dir as well"
rm -fv "$WSREP_SST_OPT_BINLOG".[0-9]* 1>&2 \+ || true
binlog_index="${WSREP_SST_OPT_BINLOG_INDEX%.index}.index"
[ -f "$binlog_index" ] && rm -fv "$binlog_index" 1>&2 \+ || true
cd "$OLD_PWD"
fi
TDATA=${DATA}
TDATA="${DATA}"
DATA="${DATA}/.sst"
MAGIC_FILE="${DATA}/${INFO_FILE}"
@ -1155,13 +1161,13 @@ then
get_proc
if [[ ! -s ${DATA}/xtrabackup_checkpoints ]];then
if [[ ! -s "${DATA}/xtrabackup_checkpoints" ]];then
wsrep_log_error "xtrabackup_checkpoints missing, failed innobackupex/SST on donor"
exit 2
fi
# Rebuild indexes for compact backups
if grep -q 'compact = 1' ${DATA}/xtrabackup_checkpoints;then
if grep -q 'compact = 1' "${DATA}/xtrabackup_checkpoints";then
wsrep_log_info "Index compaction detected"
rebuild=1
fi
@ -1169,11 +1175,11 @@ then
if [[ $rebuild -eq 1 ]];then
nthreads=$(parse_cnf xtrabackup rebuild-threads $nproc)
wsrep_log_info "Rebuilding during prepare with $nthreads threads"
rebuildcmd="--rebuild-indexes --rebuild-threads=$nthreads"
rebuildcmd="--rebuild-indexes --rebuild-threads='$nthreads'"
fi
if test -n "$(find ${DATA} -maxdepth 1 -type f -name '*.qp' -print -quit)";then
qpfiles=$(find "${DATA}" -maxdepth 1 -type f -name '*.qp' -print -quit)
if [ -n "$qpfiles" ]; then
wsrep_log_info "Compressed qpress files found"
if [[ ! -x `which qpress` ]];then
@ -1181,8 +1187,8 @@ then
exit 22
fi
if [[ -n $progress ]] && pv --help | grep -q 'line-mode';then
count=$(find ${DATA} -type f -name '*.qp' | wc -l)
if [[ -n "$progress" ]] && pv --help | grep -q 'line-mode';then
count=$(find "${DATA}" -type f -name '*.qp' | wc -l)
count=$(( count*2 ))
if pv --help | grep -q FORMAT;then
pvopts="-f -s $count -l -N Decompression -F '%N => Rate:%r Elapsed:%t %e Progress: [%b/$count]'"
@ -1203,7 +1209,7 @@ then
if [[ $extcode -eq 0 ]];then
wsrep_log_info "Removing qpress files after decompression"
find ${DATA} -type f -name '*.qp' -delete
find "$DATA" -type f -name '*.qp' -delete
if [[ $? -ne 0 ]];then
wsrep_log_error "Something went wrong with deletion of qpress files. Investigate"
fi
@ -1213,23 +1219,24 @@ then
fi
fi
if [[ ! -z $WSREP_SST_OPT_BINLOG ]];then
if [ -n "$WSREP_SST_OPT_BINLOG" ]; then
BINLOG_DIRNAME=$(dirname $WSREP_SST_OPT_BINLOG)
BINLOG_FILENAME=$(basename $WSREP_SST_OPT_BINLOG)
BINLOG_DIRNAME=$(dirname "$WSREP_SST_OPT_BINLOG")
BINLOG_FILENAME=$(basename "$WSREP_SST_OPT_BINLOG")
# To avoid comparing data directory and BINLOG_DIRNAME
mv $DATA/${BINLOG_FILENAME}.* $BINLOG_DIRNAME/ 2>/dev/null || true
mv "$DATA/$BINLOG_FILENAME".* "$BINLOG_DIRNAME/" 2>/dev/null || true
pushd $BINLOG_DIRNAME &>/dev/null
for bfiles in $(ls -1 ${BINLOG_FILENAME}.[0-9]*);do
echo ${BINLOG_DIRNAME}/${bfiles} >> ${BINLOG_FILENAME}.index
cd "$BINLOG_DIRNAME"
for bfile in $(ls -1 "$BINLOG_FILENAME".[0-9]*); do
echo "$BINLOG_DIRNAME/$bfile" >> "${WSREP_SST_OPT_BINLOG_INDEX%.index}.index"
done
popd &> /dev/null
cd "$OLD_PWD"
fi
wsrep_log_info "Preparing the backup at ${DATA}"
setup_commands
timeit "Xtrabackup prepare stage" "$INNOAPPLY"
if [ $? -ne 0 ];
@ -1240,14 +1247,14 @@ then
MAGIC_FILE="${TDATA}/${INFO_FILE}"
set +e
rm $TDATA/innobackup.prepare.log $TDATA/innobackup.move.log
rm "$TDATA/innobackup.prepare.log" "$TDATA/innobackup.move.log"
set -e
wsrep_log_info "Moving the backup to ${TDATA}"
timeit "Xtrabackup move stage" "$INNOMOVE"
if [[ $? -eq 0 ]];then
wsrep_log_info "Move successful, removing ${DATA}"
rm -rf $DATA
DATA=${TDATA}
rm -rf "$DATA"
DATA="${TDATA}"
else
wsrep_log_error "Move failed, keeping ${DATA} for further diagnosis"
wsrep_log_error "Check ${DATA}/innobackup.move.log for details"
@ -1258,7 +1265,7 @@ then
wsrep_log_info "${IST_FILE} received from donor: Running IST"
fi
if [[ ! -r ${MAGIC_FILE} ]];then
if [[ ! -r "${MAGIC_FILE}" ]];then
wsrep_log_error "SST magic file ${MAGIC_FILE} not found/readable"
exit 2
fi

15
scripts/wsrep_sst_xtrabackup.sh
View File

@ -1,5 +1,6 @@
#!/bin/bash -ue
# Copyright (C) 2013 Percona Inc
# Copyright (C) 2017-2021 MariaDB
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@ -19,6 +20,7 @@
# Make sure to read that before proceeding!
. $(dirname $0)/wsrep_sst_common
wsrep_check_datadir
ealgo=""
ekey=""
@ -131,7 +133,7 @@ get_keys()
get_transfer()
{
TSST_PORT=${WSREP_SST_OPT_PORT:-4444}
TSST_PORT=$WSREP_SST_OPT_PORT
if [[ $tfmt == 'nc' ]];then
if [[ ! -x `which nc` ]];then
@ -406,7 +408,6 @@ read_cnf
get_stream
get_transfer
INNOEXTRA=""
INNOAPPLY="${INNOBACKUPEX_BIN} ${WSREP_SST_OPT_CONF} --apply-log \$rebuildcmd \${DATA} &>\${DATA}/innobackup.prepare.log"
INNOBACKUP="${INNOBACKUPEX_BIN} ${WSREP_SST_OPT_CONF} \$INNOEXTRA --galera-info --stream=\$sfmt \${TMPDIR} 2>\${DATA}/innobackup.backup.log"
@ -419,14 +420,14 @@ then
usrst=0
TMPDIR="${TMPDIR:-/tmp}"
if [[ -n "${WSREP_SST_OPT_USER:-}" && "$WSREP_SST_OPT_USER" != "(null)" ]]; then
if [ -n "$WSREP_SST_OPT_USER" ]; then
INNOEXTRA+=" --user=$WSREP_SST_OPT_USER"
usrst=1
fi
if [ -n "${WSREP_SST_OPT_PSWD:-}" ]; then
if [ -n "$WSREP_SST_OPT_PSWD" ]; then
INNOEXTRA+=" --password=$WSREP_SST_OPT_PSWD"
elif [[ $usrst -eq 1 ]];then
elif [ $usrst -eq 1 ]; then
# Empty password, used for testing, debugging etc.
INNOEXTRA+=" --password="
fi
@ -528,9 +529,9 @@ then
# May need xtrabackup_checkpoints later on
rm -f ${DATA}/xtrabackup_binary ${DATA}/xtrabackup_galera_info ${DATA}/xtrabackup_logfile
ADDR="${WSREP_SST_OPT_HOST}:${WSREP_SST_OPT_PORT:-4444}"
ADDR="${WSREP_SST_OPT_HOST}:${WSREP_SST_OPT_PORT}"
wait_for_listen ${WSREP_SST_OPT_PORT:-4444} ${ADDR} ${MODULE} &
wait_for_listen ${WSREP_SST_OPT_PORT} ${ADDR} ${MODULE} &
trap sig_joiner_cleanup HUP PIPE INT TERM
trap cleanup_joiner EXIT