Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Vlad Lesin
MDEV-31956 SSD based InnoDB buffer pool extension

In one of the practical cloud MariaDB setups, a server node accesses its
datadir over the network, but also has a fast local SSD storage for
temporary data. The content of such temporary storage is lost when the
server container is destroyed.

The commit uses this ephemeral fast local storage (SSD) as an extension of
the portion of InnoDB buffer pool (DRAM) that caches persistent data
pages. This cache is separated from the persistent storage of data files
and ib_logfile0 and ignored during backup.

The following system variables were introduced:

innodb_extended_buffer_pool_size - the size of external buffer pool
file, if it equals to 0, external buffer pool will not be used;

innodb_extended_buffer_pool_path - the directory in which the external
buffer pool file is created, the data directory is used if the variable
is not set.

If innodb_extended_buffer_pool_size is not equal to 0, external buffer
pool file will be created on startup of a normal server instance. It is
created as a temporary file, so that the operating system removes it
when the server closes it or exits, and no stale file is left behind
after a crash. For this purpose row_merge_file_create_mode() was
generalized into pfs_create_temp_file() and moved to fil0fil.cc, so that
both the merge sort files and the external buffer pool file are created
by the same function.

Only clean pages will be flushed to external buffer pool file. There is
no need to flush dirty pages, as such pages will become clean after
flushing, and then will be evicted when they reach the tail of LRU list.
Freed pages are not written to the external buffer pool file either,
they are just evicted.

The general idea of this commit is to flush clean pages to external
buffer pool file when they are evicted.

A page can be evicted either by transaction thread or by background
thread of page cleaner. In some cases transaction thread is waiting for
page cleaner thread to finish its job. We can't do flushing in external
buffer pool file when transaction threads are waiting for eviction,
that would hurt performance. That's why the only case for flushing is
when page cleaner thread evicts pages in background and there are no
waiters. For this purpose buf_pool_t::done_flush_list_waiters_count
variable was introduced, we flush evicted clean pages only if the
variable is zeroed.

Clean pages are evicted in buf_flush_LRU_list_batch() to keep some
amount of pages in buffer pool's free list. That's why we flush every
second page to external buffer pool file, otherwise there could be not
enough amount of pages in free list to let transaction threads to
allocate buffer pool pages without page cleaner waiting. This might be
not a good solution, but this is enough for prototyping.

External buffer pool page is introduced to store information in buffer
pool page hash about the certain page can be read from external buffer
pool file. The first several members of such page must be the same as the
members of internal page. External page frame must be equal to the
certain value to distinguish external page from internal one. External
buffer pages are preallocated on startup in external pages array. We
could get rid of the frame in external page, and check if the page's
address belongs to the array to distinguish external and internal pages.

There are also external pages free and LRU lists. When some internal page
is decided to be flushed in external buffer pool file, a new external
page is allocated either from the head of external free list, or from
the tail of external LRU list. Both lists are protected with
buf_pool.mutex. It makes sense, because a page is removed from internal
LRU list during eviction under buf_pool.mutex.

Then internal page is locked and the allocated external page is attached
to io request for external buffer pool file, and when write request is
completed, the internal page is replaced with external one in page hash,
external page is pushed to the head of external LRU list and internal
page is unlocked. After internal page was removed from external free list,
it was not placed in external LRU, and placed there only
after write completion, so the page can't be used by the other threads
until write is completed.

Page hash chain get element function has additional template parameter,
which notifies the function if external pages must be ignored or not. We
don't ignore external pages in page hash in two cases, when some page is
initialized for read and when one is reinitialized for new page creating.

When an internal page is initialized for read and external page with the
same page id is found in page hash, the internal page is locked,
the external page in replaced with newly initialized internal page in the
page hash chain, the external page is removed from external LRU list and
attached to io request to external buffer pool file. When the io request
is completed, external page is returned to external free list,
internal page is unlocked. So during read external page is absent in both
external LRU and free lists and can't be reused.

When an internal page is initialized for new page creating and external
pages with the same page id is found in page hash, we just remove external
page from the page hash chain and external LRU list and push it to the
head of external free list. So the external page can be used for future
flushing.

The external buffer pool file is not represented by a fil_space_t. The
requests to it are issued by fil_system_t::ext_bp_io(), and the external
buffer pool page which a request refers to is stored in IORequest in
place of the fil_node_t. The pages are written to and read from the
external buffer pool file in the same form in which they are written to
their tablespaces, i.e. compressed and encrypted pages stay compressed
and encrypted in external buffer pool file.

If a write to the external buffer pool file fails, a warning is written
to the error log and the external buffer pool is disabled for the rest
of the server lifetime.

Currently the commit passed some local smoke tests, mtr and RQG tests
with external buffer pool turned on.

TODO:
1. Add some monitoring, i.e. how much pages are currently in external
  buffer pool, the percent of hits during reading, take a look at the
  current buffer pool monitoring and implement the general monitoring
  tools for external buffer pool.
2. Think about partial initialization of external pages array, as it was
  done for internal pages.
3. Take a look at compressed LRU list, it looks like currently it's not
  covered with eviction to external buffer pool file (I don't currently
  understand if we need it at all).
4. Think about more suitable algorithm for eviction to external buffer
  pool, currently just every second page is flushed.
Kristian Nielsen
Fix some incorrect uses of off_t

The off_t type is POSIX, not standard C/C++, and generally should not be
used in non-platform specific code. (The main problem is that is is a
32-bit signed integer on 64-bit Windows, potentially causing integer
overflow).

Signed-off-by: Kristian Nielsen <[email protected]>
Andrei Elkin
MDEV-42337 no semi-sync kill of a closed connection

This is MDEV-42004 fixes follow-up.
Repl_semi_sync_slave::kill_connection() reconnected to the master at the IO thread exit even though end_server() had already closed the connection (e.g. STOP SLAVE interrupted a reconnect), so the stop waited rpl_semi_sync_slave_kill_conn_timeout once more.
Return early when mysql->net.vio is NULL.
Andrei Elkin
MDEV-42004 STOP SLAVE waits for slave_net_timeout when master is unreachable

STOP SLAVE could take up to slave_net_timeout to stop the slave IO thread
when the master accepts the connection but does not answer, e.g. when its
host is frozen or its packets are dropped.
The IO thread was blocked in a reconnect
  try_to_reconnect() -> safe_reconnect() -> connect_to_master() ->
  mysql_reconnect() -> mysql_real_connect()
waiting for the greeting of the master, and nothing could wake it up in the
base of this patch, 11.4 at c5272f132d3. The stall is not specific to
semi-sync: an asynchronous slave waits the same, and a semi-sync one waits
in addition for rpl_semi_sync_slave_kill_conn_timeout at the IO thread exit,
which this commit does not change.

The wait was interruptible before MDEV-32567 removed the pthread_kill() of
the slave thread in terminate_slave_thread() (commit 013fc02a23e) in order to
rely on the THD::awake_no_mutex() -> vio_shutdown(active_vio) mechanism.
This bug exposes that it does not cover slave reconnect, as the slave's IO
thread does not bind its THD::active_vio to a new socket vio at the time of
the reconnect attempt.

This commit makes that "tentative" vio reachable for awake_no_mutex().
For that mysql_real_connect() now publishes the vio to the slave thread
right after it is created.
See the new hook slave_io_thread_attach_vio() in the TCP branch (the Unix one
is irrelevant for slave) that pairs an existing slave_io_thread_detach_vio().
So now in the patch when STOP SLAVE shuts the vio down, which wakes up
the poll().
The detach hook is called when the bound to THD::active_vio is about
to be destroyed.

connect_to_master() reports the interrupt to the error log when
log_warnings > 2 and the slave has been stopped after the failed connect:

  Slave I/O thread: reconnect to master '[email protected]:19000' was
  interrupted by the slave stop: Lost connection to server at 'reading
  initial communication packet', system error: 0 ...

Tested with rpl.rpl_stop_slave_unreachable_master.

The test simulates an unresponsive master: it is not shut down but frozen
with SIGSTOP, so the kernel still completes the TCP handshake of a new
connection, while no greeting is sent, and the IO thread gets into its
reconnect. The test waits for that to most probably happen and then
runs STOP SLAVE, which normally (not on an overloaded server) must not wait
for slave_net_timeout.
A shut-down master would not serve for that: its port is closed, so the
connect fails at once with "Connection refused", and the IO thread, which
then sleeps connect_retry in slave_sleep() (it checks io_slave_killed()),
stops promptly without this fix.
The test is not run on Windows, as it needs SIGSTOP.

With --let $debug_test= 1 the test prints how long STOP SLAVE took. Measured
this way, it returned after 3.5 s before the fix and 1.0 s after, with
slave_net_timeout=3 and rpl_semi_sync_slave_kill_conn_timeout=1; the same
test with a plain asynchronous slave took 2.5 s and 0.0 s.
For a reliable run on BuildBot the time is not checked. The fix is proven
by the error text in the note above. A reconnect that waits for the greeting
until the timeout is out fails with "waiting for initial communication
packet ... Connection timed out", while the one whose vio is shut down fails
with "reading initial communication packet". The test greps the slave's
error log for the latter with search_pattern_in_file.inc, which reports

  FOUND 1 /Slave I/O thread: reconnect to master .* was interrupted by the slave stop: .*reading initial communication packet/ in mysqld.2.err
Alexey Yurchenko
MDEV-35940, MDEV-35964 reenable MTR tests

As MDEV-35940, MDEV-35964 have been fixed, reenable tests disabled
because of them
Sergei Golubchik
MDEV-42324 CONNECTION of a MULTIPLE table is used as a format string

For ODBC tables with MULTIPLE=1 TDBODBC::GetFile() replaced the DBQ
value in CONNECTION with "%s" and TDBODBC::SetFile() passed the result
to snprintf() as the format string.

Store the parts of CONNECTION before and after the file name as two
strings and concatenate them with the file name.

Assisted-by: Claude:claude-opus-5-5
Marko Mäkelä
fixup! 4b7f27348c38d44545f528015e2811ee5ee3291e
Marko Mäkelä
MDEV-39091 BACKUP SERVER of ENGINE=RocksDB

RocksDB is based on log structured merge trees (LSM trees)
that are immutable files. A consistent point-in-time snapshot
is obtained at BACKUP_LOCK_COMMIT cheaply via rdb_create_checkpoint(),
which is also what mariadb-backup is invoking via
SET GLOBAL rocksdb_create_checkpoint.

Under BACKUP_LOCK_COMMIT, a backup snapshot is created in #rocksdb under
rocksdb_datadir (by default, #rocksdb/#rocksdb in the server datadir).
This snapshot will be made available in #rocksdb subdirectory of the
backup target. The default rocksdb_datadir="./#rocksdb" makes restore
transparent: just point datadir to the backup directory or a restored
copy of it.

The transfer of the snapshot is deferred to BACKUP_FINISHED, after the
backup lock has been released, and finally the checkpoint is removed
from rocksdb_datadir.

Co-developed by: Thirunarayanan Balathandayuthapani
Kristian Nielsen
MDEV-42237: Fix integer overflow of relay log position

The off_t type is 32-bit signed on Windows, casting the relay log position
to off_t will overflow when >= 2**31.

Signed-off-by: Kristian Nielsen <[email protected]>
Aleksey Midenkov
MDEV-41645 Auto-create of SYSTEM_TIME history partitions checks the partition limit without existing partitions and subpartitions

partition_info::vers_set_hist_part() counts the history partitions to
auto-create for INTERVAL ... AUTO and stopped when the count reached
MAX_PARTITIONS - 2. The limit compared only the number of new partitions
with MAX_PARTITIONS - 2. It ignored the partitions the table already has
and the number of subpartitions, so the real total
((num_parts + create_count) * num_subparts) could be well over
MAX_PARTITIONS while this check passed. The early error was then not
given, and the limit was left to whatever checks the actual partition
creation does.

The fix compares the total of the existing and the new partitions, times
the number of subpartitions, with MAX_PARTITIONS.
Marko Mäkelä
Add my_dir_open() written by Monty
Georgi (Joro) Kodinov
MDEV-42338: Treat openssl,pcre and zlib library headers as "system"

Mark the 3d party library headers as "system" headers so that the
warnings in these files are suppressed for supporting compilers.
Marko Mäkelä
MDEV-39091 BACKUP SERVER of ENGINE=RocksDB

RocksDB is based on log structured merge trees (LSM trees)
that are immutable files. A consistent point-in-time snapshot
is obtained at BACKUP_LOCK_COMMIT cheaply via rdb_create_checkpoint(),
which is also what mariadb-backup is invoking via
SET GLOBAL rocksdb_create_checkpoint.

Under BACKUP_LOCK_COMMIT, a backup snapshot is created in #rocksdb under
rocksdb_datadir (by default, #rocksdb/#rocksdb in the server datadir).
This snapshot will be made available in #rocksdb subdirectory of the
backup target. The default rocksdb_datadir="./#rocksdb" makes restore
transparent: just point datadir to the backup directory or a restored
copy of it.

The transfer of the snapshot is deferred to BACKUP_FINISHED, after the
backup lock has been released, and finally the checkpoint is removed
from rocksdb_datadir.

Co-developed by: Thirunarayanan Balathandayuthapani
Reviewed by: Alessandro Vetere
kjarir
MDEV-10526: Add binary string support to bitwise operators and aggregates

Previously all bitwise operators and BIT_AND/BIT_OR/BIT_XOR converted
their arguments to BIGINT, silently truncating values wider than 64 bits.
This made them useless for INET6 addresses, UUIDs and other binary data.

When the arguments are binary strings (BINARY, VARBINARY, BLOB, but not
hex hybrids like 0xFF), the operators &, |, ^, ~, << and >> and the
aggregate functions BIT_AND, BIT_OR and BIT_XOR now work byte by byte
and return a binary string of the length of the operand(s).

Scalar operators:
- Implemented as Handler_str subclasses of the existing Item_handled_func
  handlers. Func_handler_bit_bin_to_bin and
  Func_handler_bit_op2_bin_to_bin hold the common code.
- Operands of different lengths give ER_INVALID_BITWISE_OPERANDS_SIZE
  as a warning and the result is NULL.
- The result collation is aggregated from the binary string argument(s),
  as for other string functions.

Aggregate functions:
- A binary accumulator initialized to the neutral value (0xFF for
  BIT_AND, 0x00 for BIT_OR and BIT_XOR).
- The result length is the length of the aggregated values, which may
  be shorter than the declared length (e.g. INET6_ATON() of an IPv4
  address). All values of a group must have the same length, otherwise
  ER_INVALID_BITWISE_OPERANDS_SIZE is raised.
- An empty set returns the neutral value of the declared length.
- The operand is limited to 512 bytes,
  ER_INVALID_BITWISE_AGGREGATE_OPERANDS_SIZE otherwise.
- The result data type is VARBINARY. val_int(), val_real(),
  val_decimal() and get_date() convert the binary string the same way
  as for the scalar operators.
- Window functions use per-bit counters, as for the 64-bit mode.
- reset_field()/update_field() support the binary mode for GROUP BY
  with a temporary table. Rows rejected by a FILTER clause (MDEV-24943)
  leave the field unchanged.
- direct_add() for a binary partial result, used by Spider.

MDEV-10526: Add Spider pushdown for bitwise aggregates

Push BIT_AND, BIT_OR and BIT_XOR down to the data nodes and merge the
partial results locally with direct_add(), in both integer and binary
string mode. The functions are associative and commutative, so the
partial results can be combined like for SUM, MIN and MAX.

In integer mode the partial result is parsed with strntoull10rnd():
row->val_int() overflows for values above LONGLONG_MAX.
A partial result that can not be parsed is an error, as any substitute
value would silently corrupt the merged result.

Aggregates with a FILTER clause are not pushed down.

MDEV-10526: Address review: test coverage, Spider empty data nodes

Add tests for the code paths reported as not covered:
- scalar operators: NULL in the first operand, empty strings, shift
  counts inside a byte, across bytes, out of range and negative
- aggregates: the binary result used as an integer, a boolean and a
  date, the integer result as a string, a double, a decimal and a date,
  data types that are not allowed, values of different lengths in a
  temporary table group and in a window, sliding frames removing
  values (also with FILTER), window functions over WITH ROLLUP
- Spider: groups whose partial results come from several data nodes
  (update_field() with direct_add()), empty data nodes, partial
  results that are not an integer, too long or NULL
- out of memory when allocating the result of an operator and the
  bit counters of a window function (debug builds)

Fix the bugs found by the new tests:
- A binary aggregate pushed down to Spider failed with
  ER_INVALID_BITWISE_OPERANDS_SIZE if the first data node had no rows
  and the values were shorter than the declared length. The neutral
  value of the declared length returned by such a node was taken as
  the first value. A neutral partial result is now always skipped,
  and ordinary values are not checked for it any more, so values of
  different lengths are always an error.
- A partial result longer than the declared length of the argument
  (the data node has a wider column) was written past the end of the
  accumulator. It is now rejected.

Remove the code that could not be reached:
- BIT_AND, BIT_OR and BIT_XOR never return NULL, so Item_sum_bit does
  not need to handle a NULL partial result or a NULL temporary table
  field. Spider returns an error for a NULL.
- The binary accumulator and the partial result are now kept in
  buffers inside Item_sum_bit: they are at most 512 bytes long. There
  is no allocation that can fail any more.
- The operators allocate the result in one common method, which also
  handles empty strings.
- Window functions never aggregate values of different lengths, and
  are never copied by copy_or_same(), this is now asserted.

MDEV-10526: Fix Spider bitwise aggregate tests for --ps-protocol

The prepared statement protocol changes the value of the
Spider_direct_aggregate status variable for the GROUP BY query, and
the stored procedure protocol adds a note to SHOW WARNINGS after an
error. Run that query without the prepared statement protocol and
show only the warnings of the query.
Alexey Botchkov
Excessive json_nice() calls eliminated.

Item_json_func descendants now have val_json() doing
the job, returning the valid JSON as it is.
The val_str() gets the val_json() return and trims it with json_nice().
Functions that surely return the nice json just return the val_json()
in the val_str().
Thirunarayanan Balathandayuthapani
MDEV-39061 mariadb-backup compatible wrapper for BACKUP SERVER

This adds a shell script that lets users keep using their existing
mariadb-backup commands while the real work is done by the new
server-side BACKUP SERVER command. The goal is "drop-in": users
should not have to change their backup scripts.

mariadb-backup-server.sh : Understands the usual mariadb-backup
modes and translates each one.

mbstream-server.sh : lets streamed backups be unpacked by
pipelines that expect the mbstream CLI

mariadb-backup-pipe.sh is the stream sink that
BACKUP SERVER itself runs.

All three are documented in README.md.

--backup
========
Connects with the mariadb client and runs
"BACKUP SERVER TO '<dir>'".

Connection options (--user, --host, --port, --socket,
--defaults-file, ssl, ...) are passed through to the client.

--parallel=N becomes "<N+1> CONCURRENT": mariadb-backup
runs a dedicated log_copying_thread() besides its N data-copy
threads, while CONCURRENT counts every BACKUP SERVER worker.
The result is clamped to the 1..256 the parser accepts,
and to a minimum of 2.

After the backup it writes backup-prepare.cnf into the backup
directory, recording what --prepare needs later:
  - where mariadbd lives
  - InnoDB parameters (page size, data file path, undo tablespaces,
                      checksum algorithm, log file size)
  - If the server is encrypted then how to reload the
    encryption key plugin (the file_key_management variables),
    so an encrypted backup can be prepared without extra input.
  - every active provider_* compression plugin, with plugin-dir.

The providers matter because --prepare runs "mariadbd --bootstrap
--defaults-file=backup-prepare.cnf", which suppresses the
/etc/my.cnf.d snippet that normally enables a provider. Recovery has
to decompress a PAGE_COMPRESSED page before it can apply redo to it,
so without the provider it aborts with "Unable to apply log to
corrupted page". mariadb-backup carries the same plugins through
backup-my.cnf (encryption_plugin.cc QUERY_PLUGIN collects the
ENCRYPTION plugin and the provider_* DAEMON plugins together), so
this is parity with it rather than a new capability.

--backup --stream
=================
Runs "BACKUP SERVER WITH 'pipe'".
The server only accepts a bare command name, which it resolves
n the PATH of the mariadbd process after prepending mariadb-backup-,
so uses the installed mariadb-backup-pipe instead.

mariadb-backup-pipe writes the tar into
.mariadb-backup-pipe-<stream>.tar, a named pipe the wrapper
created in the server @@datadir and is already draining to
its own stdout.

The backup therefore reaches the consumer as it is produced and
never lands on local disk. The wrapper appends backup-prepare.cnf
as a final tar afterwards; the server's tar carries no
end-of-archive marker, so that trailing archive supplies the
only one and the whole stream extracts with a plain "tar -x".
The trailing archive is written with "tar -b 1" so that nothing
follows its end-of-archive marker. At the default blocking factor tar
pads the archive to 10240 bytes; the reader stops at the marker and
exits with the padding still unwritten, which on a small pipe is a
short write. tar treats a short write as fatal, so a backup that had
in fact completed was reported as failed.

--parallel is ignored here, with a warning.

Four properties follow from how BACKUP SERVER streams,
all differing from mariadb-backup:
- local: the stream command runs inside the server,
so the wrapper must share its filesystem;
- fifo lives in @@datadir, because mariadb-backup-pipe resolves it
relative to the working directory of mariadbd, so --stream needs
write access there;
- tar only: any --stream=<format> yields tar;
- single-threaded: one worker, so no parallel read either.
--target-dir is optional in stream mode;

mbstream-server.sh maps the mbstream CLI onto a plain
"tar -x"/"tar -c", so existing "mbstream -x"/"-c" pipelines
keep working on the wrapper's stream.

mbstream-only flags (-p/--parallel, ...) are accepted and
ignored; any other unknown option is rejected.

Environment overrides:
MARIADB (client),
MARIADBD (the --prepare bootstrap server) and
TAR (the tar implementation, e.g. TAR=bsdtar) can each be overridden.

To run the bootstrap under rr, put it in MARIADBD and
let rr's own _RR_TRACE_DIR choose the trace location, e.g.
  _RR_TRACE_DIR=/dev/shm/rr MARIADBD='rr record mariadbd'

--prepare
=========
Starts "mariadbd --bootstrap" on the backup directory using
backup-prepare.cnf as its defaults file, replays the archived redo
log between the start and target LSN read from backup.cnf,
then builds a fresh ib_logfile0 so a normal server can start
on the directory.

mariadbd is taken from the path recorded in backup-prepare.cnf
if that binary exists, otherwise by searching
/libexec, /sbin, /bin and the configured install directories.

PATH is not searched; set MARIADBD to point elsewhere.
User --defaults-file/-extra-file and encryption options are
layered onto the bootstrap.

--copy-back / --move-back
=========================
Copy or move a prepared backup into the datadir. The datadir
is created if missing, a non-empty datadir is refused unless
--force-non-empty-directories is given, and a chown
reminder is printed.

If --aria-log-dir-path is given, the Aria logs (aria_log_control,
aria_log.*) are relocated into that directory.

Packaging
=========
The wrapper is not installed by default and never replaces the
real mariadb-backup / mbstream binaries.
1. cmake -DWITH_MARIABACKUP_WRAPPER=ON (default OFF) controls it.
2. When ON, the scripts install as /usr/bin/mariadb-backup-server,
/usr/bin/mbstream-server and /usr/bin/mariadb-backup-pipe, tagged
COMPONENT Backup so they ship in the mariadb-backup package.
mariadb-backup-pipe must end up in the PATH of the mariadbd
process, not merely in the PATH of whoever runs the wrapper.
  mariadb-backup-pipe is generated unconditionally, because the
  mariabackup test suite needs it on the server's PATH even when the
  wrapper is not installed; only its install rule is gated.
3. RPM: nothing extra to do. the component handles it.
4. DEB: not wired. debian/rules uses --fail-missing and does not
enable the option, so the -server binaries are not listed.
To ship via DEB, make a paired change: add
-DWITH_MARIABACKUP_WRAPPER=ON in debian/rules and list all three of
usr/bin/mariadb-backup-server, usr/bin/mbstream-server and
usr/bin/mariadb-backup-pipe in debian/mariadb-backup.install together.
5. The real mariadb-backup/mbstream binaries and the
mariabackup symlink are left untouched; opt in via an alias or a
symlink early in PATH.

Limitations (not supported yet)
===============================
1) Incremental backup & prepare (--incremental-basedir,
  --incremental-dir, --apply-log-only)
2) --rollback-xa
3) Partial backup (--databases, --tables, --tables-file)
4) Output compression and encryption (--compress, --encrypt)
5) --export is accepted but only warns and runs a plain recovery
6) --extra-lsndir is ignored
7) --parallel is ignored with --stream
8) Windows: POSIX sh only, not installed on Windows

Behaviour differences from native mariadb-backup
================================================
- The wrapper needs the mariadb client on PATH for --backup;
--prepare needs mariadbd recorded in backup-prepare.cnf, in a
standard install directory, or named by MARIADBD
- BACKUP SERVER refuses an already-existing target directory
- BACKUP SERVER does copy the data file as raw pages without
checksum validation, so a corrupted table is not detected
at backup time
- --prepare only works on a wrapper-made backup: it
needs backup-prepare.cnf
- --stream is tar, not xbstream, local-only and single-threaded,
and needs write access to @@datadir for the fifo

Tests
=====
include/have_mariabackup_wrapper.inc redirects $XTRABACKUP to
mariadb-backup-server.sh and $XBSTREAM to mbstream-server.sh,
skipping when a wrapper or the mariadb client is unavailable.

include/have_mariabackup_combination.inc runs a test under both the
[CLIENT] mariadb-backup binary and the [SERVER] wrapper.

suite/mariabackup/suite.pm prepends the directory holding
mariadb-backup-pipe to PATH. BACKUP SERVER resolves the command in the
PATH of the mariadbd process, which mtr starts from the suite's
environment, so it has to be set there rather than with a --let at
--exec time. have_mariabackup_wrapper.inc skips the [SERVER]
combination when the helper is not on PATH.
Kristian Nielsen
Fix some incorrect uses of off_t

The off_t type is POSIX, not standard C/C++, and generally should not be
used in non-platform specific code. (The main problem is that is is a
32-bit signed integer on 64-bit Windows, potentially causing integer
overflow).

Signed-off-by: Kristian Nielsen <[email protected]>
Kristian Nielsen
MDEV-41658 fix Galera node crash when a parallel slave worker is bf_killed

A parallel-replication worker would assert in ha_rollback_trans() because
wsrep_after_statement() and ha_commit_trans()'s wsrep error path could
roll back its transaction before retry_event_group() unmarks it as
committing.

Skip mark_start_commit() when using Galera. Galera certification can cause
rollback of parallel replication transactions during commit when there is a
conflicting transaction outside of replication. This can cause to start the
next GCO too early, while the previous GCO still needs to rollback and
retry.

Do not fail replication with an error when wsrep_after_statement() fails due
to a retryable error like brute-force kill.

Signed-off-by: Kristian Nielsen <[email protected]>
Andrei Elkin
MDEV-42337 no semi-sync kill of a closed connection

This is MDEV-42004 fixes follow-up.
Repl_semi_sync_slave::kill_connection() reconnected to the master at
the IO thread exit even though end_server() had already closed the
connection (e.g. STOP SLAVE interrupted a reconnect), so the stop
waited rpl_semi_sync_slave_kill_conn_timeout once more.  Return early
when mysql->net.vio is NULL.
Marko Mäkelä
fixup! 471877aef91b1069b7fa38f10a066f8881eaf0e1
Marko Mäkelä
Replace native API with my_dir_open()
Marko Mäkelä
fixup! 77311fbaf3e42d429c6c2f7ac3de1d9842b71d1b
Marko Mäkelä
fixup! 504234891164fe8a1b6065108f6ddaae7356067e
Marko Mäkelä
fixup! 7cfefd08e195bda8e03b0687632953656128ce7d
Aleksey Midenkov
MDEV-41644 date_add_interval(): signed overflow on huge YEAR/MONTH interval values

INTERVAL::year and INTERVAL::month are unsigned, but date_add_interval()
cast them to long and did signed arithmetic before the range check. A
value above LONG_MAX became negative, and the multiplication or addition
could overflow, which is undefined behaviour. The result could also wrap
into the valid range: the year sum is stored in the 32-bit
MYSQL_TIME::year, so INTERVAL 4294967301 YEAR added 5 years instead of
failing.

The fix checks the year and month magnitude before the arithmetic, as is
already done for days with MAX_DAY_NUMBER: no interval of 10000 years or
more, or of 120000 months or more, gives a valid date.
Marko Mäkelä
Revert "fixup! 7efc89bb39daf1d63015e130f62d139dc307c0ce"

This reverts commit 26c319d5d19cd4b865c88ed01111900a7c31214c.
Marko Mäkelä
Encapsulate all streaming in backup_sink
Marko Mäkelä
Fix Windows once more
Jan Lindström
MDEV-42284 fix galera_sr.mysql-wsrep#215 ER_LOCK_WAIT_TIMEOUT

Node_1's own causality wait (wsrep_sync_wait, on by default) blocked
its plain SELECT on t1 while the test held node_1's applier paused at
the sync.wsrep_apply_cb debug sync point, so the SELECT could not
reach the GTID the paused applier had not yet reached and timed out
with ER_LOCK_WAIT_TIMEOUT instead of returning.

Set wsrep_sync_wait = 0 on node_1 for that SELECT and the INSERT that
follows it, saved and restored around each of the test's three blocks.
Marko Mäkelä
fixup! 4b7f27348c38d44545f528015e2811ee5ee3291e
Aleksey Midenkov
MDEV-41643 Temporal to_string() returns a String with unset length when allocation fails

Temporal_hybrid::to_string() returned the destination String with a stale
length when String::alloc() failed, instead of signalling the failure. The
sibling to_string() methods of Interval_DDhhmmssff, Time, Datetime and Date
shared the identical flaw.

The fix returns NULL on allocation failure in all five, so they report OOM
consistently.

The val_str() implementations that return the to_string() result
directly did not set null_value, so a NULL result would be taken for a
value. They now go through the new Item::check_str(), which sets
null_value from the result, so a NULL does not stick to the item for the
next execution. The time_to_string_oom debug keyword makes
Time::to_string() fail as on out of memory.
Marko Mäkelä
squash! e3994e5a387d754120584b1c52218c2f1374527c

Remove unused my_dir_rewind()
Marko Mäkelä
fixup! 7efc89bb39daf1d63015e130f62d139dc307c0ce
Marko Mäkelä
fixup! 7cfefd08e195bda8e03b0687632953656128ce7d
Marko Mäkelä
squash! 7cfefd08e195bda8e03b0687632953656128ce7d

Move most file-copying functions to mysys
and some file APIs to member functions of
backup_target and backup_sink.
Sergei Golubchik
MDEV-42070 handlersocket crashes after open_index with huge number of return fields

alloca() sizes in do_exec_on_index() and cmd_find_internal() depend
on client-controlled counts (return fields, filters) bounded only by
the request line length.

* make DENA_ALLOCA_ALLOCATE always use malloc
* free key_buf/filter_buf on the early "filterblob" return

Also, a huge statement id in open_index or IN list length in exec
resized a vector to 4G elements, aborting on std::bad_alloc.
Limit both to 1024.

Assisted-by: Claude:claude-opus-5-5
Marko Mäkelä
backup::append_async(): Linux-specific backup::append() of immutable files
Kristian Nielsen
Fix corrupt query string in mysqltest embedded

The mysqltest --send command re-used the same buffer for the query string to
send to each connection thread. This caused a race where a following --send
could overwrite the query before the first connection thread had time to
copy out the string, resulting in executing a corrupt query.

Instead, allocate a new string for each --send and free it in the connection
thread.

This was seen as a sporadic failure in --embeded-server in test
innodb.temp_truncate_freed_race (and probably others).

Signed-off-by: Kristian Nielsen <[email protected]>
Alexey Botchkov
Excessive json_nice() calls eliminated.

Item_json_func descendants now have val_json() doing
the job, returning the valid JSON as it is.
The val_str() gets the val_json() return and trims it with json_nice().