Console View

Categories: connectors experimental galera main

Legend:

Passed

Failed

Warnings

Failed Again

Running

Exception

Offline

No data

connectors

experimental

galera

main

f5c1fba0b81b...

f5c1fba0b81b7c8edd0ea2de406550ddb6a97061

Thirunarayanan Balathandayuthapani

thiru@mariadb.com

MDEV-39061 mariadb-backup compatible wrapper for BACKUP SERVER

This adds a shell script that lets users keep using their existing
mariadb-backup commands while the real work is done by the new
server-side BACKUP SERVER command. The goal is "drop-in": users
should not have to change their backup scripts.

mariadb-backup-server.sh : Understands the usual mariadb-backup
modes and translates each one.

mbstream-server.sh : lets streamed backups be unpacked by
pipelines that expect the mbstream CLI

mariadb-backup-pipe.sh is the stream sink that
BACKUP SERVER itself runs.

All three are documented in README.md.

--backup
========
Connects with the mariadb client and runs
"BACKUP SERVER TO '<dir>'".

Connection options (--user, --host, --port, --socket,
--defaults-file, ssl, ...) are passed through to the client.

--parallel=N becomes "<N+1> CONCURRENT": mariadb-backup
runs a dedicated log_copying_thread() besides its N data-copy
threads, while CONCURRENT counts every BACKUP SERVER worker.
The result is clamped to the 1..256 the parser accepts,
and to a minimum of 2.

After the backup it writes backup-prepare.cnf into the backup
directory, recording what --prepare needs later:
- where mariadbd lives
- InnoDB parameters (page size, data file path, undo tablespaces,
checksum algorithm, log file size)
- If the server is encrypted then how to reload the
encryption key plugin (the file_key_management variables),
so an encrypted backup can be prepared without extra input.

--backup --stream
=================
Runs "BACKUP SERVER WITH 'pipe'".
The server only accepts a bare command name, which it resolves
n the PATH of the mariadbd process after prepending mariadb-backup-,
so uses the installed mariadb-backup-pipe instead.

mariadb-backup-pipe writes the tar into
.mariadb-backup-pipe-<stream>.tar, a named pipe the wrapper
created in the server @@datadir and is already draining to
its own stdout.

The backup therefore reaches the consumer as it is produced and
never lands on local disk. The wrapper appends backup-prepare.cnf
as a final tar afterwards; the server's tar carries no
end-of-archive marker, so that trailing archive supplies the
only one and the whole stream extracts with a plain "tar -x".
--parallel is ignored here, with a warning.

Four properties follow from how BACKUP SERVER streams,
all differing from mariadb-backup:
- local: the stream command runs inside the server,
so the wrapper must share its filesystem;
- fifo lives in @@datadir, because mariadb-backup-pipe resolves it
relative to the working directory of mariadbd, so --stream needs
write access there;
- tar only: any --stream=<format> yields tar;
- single-threaded: one worker, so no parallel read either.
--target-dir is optional in stream mode;

mbstream-server.sh maps the mbstream CLI onto a plain
"tar -x"/"tar -c", so existing "mbstream -x"/"-c" pipelines
keep working on the wrapper's stream.

mbstream-only flags (-p/--parallel, ...) are accepted and
ignored; any other unknown option is rejected.

Environment overrides:
MARIADB (client),
MARIADBD (the --prepare bootstrap server) and
TAR (the tar implementation, e.g. TAR=bsdtar) can each be overridden.

To run the bootstrap under rr, put it in MARIADBD and
let rr's own _RR_TRACE_DIR choose the trace location, e.g.
_RR_TRACE_DIR=/dev/shm/rr MARIADBD='rr record mariadbd'

--prepare
=========
Starts "mariadbd --bootstrap" on the backup directory using
backup-prepare.cnf as its defaults file, replays the archived redo
log between the start and target LSN read from backup.cnf,
then builds a fresh ib_logfile0 so a normal server can start
on the directory.

mariadbd is taken from the path recorded in backup-prepare.cnf
if that binary exists, otherwise by searching
/libexec, /sbin, /bin and the configured install directories.

PATH is not searched; set MARIADBD to point elsewhere.
User --defaults-file/-extra-file and encryption options are
layered onto the bootstrap.

--copy-back / --move-back
=========================
Copy or move a prepared backup into the datadir. The datadir
is created if missing, a non-empty datadir is refused unless
--force-non-empty-directories is given, and a chown
reminder is printed.

If --aria-log-dir-path is given, the Aria logs (aria_log_control,
aria_log.*) are relocated into that directory.

Packaging
=========
The wrapper is not installed by default and never replaces the
real mariadb-backup / mbstream binaries.
1. cmake -DWITH_MARIABACKUP_WRAPPER=ON (default OFF) controls it.
2. When ON, the scripts install as /usr/bin/mariadb-backup-server,
/usr/bin/mbstream-server and /usr/bin/mariadb-backup-pipe, tagged
COMPONENT Backup so they ship in the mariadb-backup package.
mariadb-backup-pipe must end up in the PATH of the mariadbd
process, not merely in the PATH of whoever runs the wrapper.
3. RPM: nothing extra to do. the component handles it.
4. DEB: not wired. debian/rules uses --fail-missing and does not
enable the option, so the -server binaries are not listed.
To ship via DEB, make a paired change: add
-DWITH_MARIABACKUP_WRAPPER=ON in debian/rules and list all three of
usr/bin/mariadb-backup-server, usr/bin/mbstream-server and
usr/bin/mariadb-backup-pipe in debian/mariadb-backup.install together.
5. The real mariadb-backup/mbstream binaries and the
mariabackup symlink are left untouched; opt in via an alias or a
symlink early in PATH.

Limitations (not supported yet)
===============================
1) Incremental backup & prepare (--incremental-basedir,
--incremental-dir, --apply-log-only)
2) --rollback-xa
3) Partial backup (--databases, --tables, --tables-file)
4) Output compression and encryption (--compress, --encrypt)
5) --export is accepted but only warns and runs a plain recovery
6) --extra-lsndir is ignored
7) --parallel is ignored with --stream
8) Windows: POSIX sh only, not installed on Windows

Behaviour differences from native mariadb-backup
================================================
- The wrapper needs the mariadb client on PATH for --backup;
--prepare needs mariadbd recorded in backup-prepare.cnf, in a
standard install directory, or named by MARIADBD
- BACKUP SERVER refuses an already-existing target directory
- BACKUP SERVER does copy the data file as raw pages without
checksum validation, so a corrupted table is not detected
at backup time
- --prepare only works on a wrapper-made backup: it
needs backup-prepare.cnf
- --stream is tar, not xbstream, local-only and single-threaded,
and needs write access to @@datadir for the fifo

Tests
=====
include/have_mariabackup_wrapper.inc redirects $XTRABACKUP to
mariadb-backup-server.sh and $XBSTREAM to mbstream-server.sh,
skipping when a wrapper or the mariadb client is unavailable.

include/have_mariabackup_combination.inc runs a test under both the
[CLIENT] mariadb-backup binary and the [SERVER] wrapper.

ec09018bff67...

ec09018bff67e5b0c32314ffb646d2786c74ae55

Marko Mäkelä

marko.makela@mariadb.com

MDEV-39091 BACKUP SERVER of ENGINE=RocksDB

RocksDB is based on log structured merge trees (LSM trees)
that are immutable files. A consistent point-in-time snapshot
is obtained at BACKUP_LOCK_COMMIT cheaply via rdb_create_checkpoint(),
which is also what mariadb-backup is invoking via
SET GLOBAL rocksdb_create_checkpoint.

The actual hard-linking or copying is deferred to
BACKUP_FINISHED, after the backup MDL has been released,
and the server-side checkpoint is removed at end of FINISH.

A backup snapshot is created in #rocksdb under
rocksdb_datadir (by default, #rocksdb/#rocksdb in the server datadir).
This snapshot will be made available in #rocksdb subdirectory of the
backup target. The default rocksdb_datadir="./#rocksdb" makes restore
transparent: just point datadir to the backup directory or a restored
copy of it.

Co-developed by: Thirunarayanan Balathandayuthapani

e6bba41a5c38...

e6bba41a5c38e66a674226b4afbc94cbf0fa2148

Thirunarayanan Balathandayuthapani

thiru@mariadb.com

MDEV-42037 Assertion failure bpage->oldest_modification() when ALTER TABLE … DISCARD TABLESPACE races with the page cleaner

Problem:
=======
The page cleaner holds the latch of a dirty page while
buf_flush_space() has released buf_pool.mutex. DISCARD TABLESPACE
meanwhile runs buf_flush_remove_pages(), which removes the page from
buf_pool.flush_list without taking the page latch. When the cleaner
resumes, buf_flush_discard_page() asserts oldest_modification()

Solution:
========
buf_flush_discard_page(): remove the page from the flush list only if
oldest_modification() is nonzero. Always release the latch and free
the page, and return whether it was freed.

e1acbf1147db...

e1acbf1147db51247797ef7a6bc0e588eb3b0e68

Yuchen Pei

ycp@mariadb.com

MDEV-25848 do not let get_mm_parts() match cloned key parts

get_claimed_key_part() appended clones at param->key_parts_end, so
every later get_mm_parts() call iterated over them too and cloned each
clone again. N independent predicates on the same RTREE/ARRAY field
thus produced 2^(N-1) key parts, exhausting MAX_KEY at the 8th
predicate (e.g. 8 ORed "x MEMBER OF (js)" lost range access).

Keep clones in a separate region after key_parts_end, tracked by
RANGE_OPT_PARAM::key_part_clones_end. get_mm_parts() only walks
key_parts..key_parts_end, so each predicate creates at most one clone
and N predicates use N key parts.

Assisted-By: Claude Opus 5.5 <[email protected]>

e0462e34f33c...

e0462e34f33c066228601597a45798ab6c725bd6

Oleksandr Byelkin

sanja@mariadb.com

Merge branch '11.4' into 11.8

dcf87aa0491a...

dcf87aa0491a9010c15f55c7b508e791cd34f82e

Alessandro Vetere

iminelink@gmail.com

MDEV-42040 Recovery ignores failure to apply log to a new page

When the redo log initializes a page (INIT_PAGE), buf_read_recover()
does not read the page from the data file.
IORequest::fake_read_complete() builds the page from the log alone by
calling recv_recover_page().

If log_phys_t::apply() returns APPLIED_CORRUPTED, recv_recover_page()
evicts the page with buf_pool_t::corrupted_evict() and returns nullptr.
IORequest::fake_read_complete() ignored the nullptr and set no recovery
error flag. Thus srv_start() continued, a later checkpoint discarded the
log records of the evicted page, and the table stayed corrupted.

IORequest::fake_read_complete(): If recv_recover_page() returns nullptr,
call log_set_corrupt(). The page content comes only from the log, so
the failure is in the log. srv_start() now aborts with DB_CORRUPTION,
and the log is kept.

Assisted-by: Claude:claude-opus-5-5

da1642c7730a...

da1642c7730aa5d9709c01ba27a720a48986d0c6

Georgi (Joro) Kodinov

joro@mariadb.org

MDEV-38486: post-merge fix: set to empty.

d84d14daada7...

d84d14daada75a888a7f70e1945bef2f7355a5d4

Yuchen Pei

ycp@mariadb.com

MDEV-25848 one range slot per ARRAY index

Before, every MEMBER OF / JSON_OVERLAPS / JSON_CONTAINS point got its
own cloned KEY_PART and key slot, and points were combined only through
index_merge (OR) or by picking one slot (AND).

Now all points on an ARRAY key part share the original slot:

* SEL_ARG_ARRAY holds a list of make_key() points ("the array contains
any of them") and the summed records_in_range() estimate and cost,
taken once when the point is built.
* key_or() of two ARRAY ranges takes the union of the points;
key_and() keeps the operand with the lower estimate, the WHERE
condition checks the rest.
* KEY_PART cloning (get_claimed_key_part()) is now RTREE-only, so
ARRAY-only queries size SEL_TREE::keys by param->keys again.
* QUICK_RANGE_SELECT_ARRAY gets one QUICK_RANGE per point and passes
all points to hlindex_read_first(); json_index reads them in turn
and skips trefs it has already returned.
* check_quick_select_array() uses the stored estimate, and rejects
non-ARRAY SEL_ARGs (e.g. IS NULL) on the ARRAY key part.

ORed MEMBER OF and JSON_OVERLAPS now give a single range scan instead
of index_merge sort_union, with no MAX_KEY limit on the number of
points.

ce972cd66c1d...

ce972cd66c1d9e4a8797802b61c5ba38d21924f5

Marko Mäkelä

marko.makela@mariadb.com

MDEV-14992/MDEV-38362/MDEV-39092 BACKUP SERVER

The following SQL statements will be introduced:

BACKUP SERVER TO '/path/to/directory' [ 1 CONCURRENT ];
BACKUP SERVER WITH [ 1 CONCURRENT ] 'command';

In place of the 1, any positive number of threads may be specified.
For the first variant, '/path/to' must exist and '/path/to/directory'
must not exist; that is where the backup will be written to.

For BACKUP SERVER WITH, the string mariadb-backup-
will be prepended to the specified command and quoted.
Path separators or quotes are not allowed in command.
We expect the mariadb-backup-command to be in the
search PATH of the mariadbd process.
For now, we distribute no such script or program;
the user has to write one, something like this:

zstdmt|exec ssh [email protected] "exec cat > $1.tar.zstd"

The standard input of that command will be in a format compatible with
GNU tar --format=oldgnu (and also BSD tar variants that are also part of
Microsoft Windows and Apple macOS). The command is expected to optionally
compress and encrypt the stream and redirect it to a file on a local or
a remote server. The BACKUP SERVER WITH will append an additional argument,
a positive base-ten number in ASCII, starting with 1, to identify the
current thread. In this way, there can be multiple concurrent streams.
The InnoDB write-ahead log will be streamed near the end, because
the stream format requires the file sizes to be declared upfront.

In BACKUP SERVER TO ... 2 CONCURRENT or more, one thread will be
responsible for copying the InnoDB write-ahead log from the beginning
of the backup, running InnoDB_backup::log_track(). Other files will be
copied by other threads.

Note: In single-threaded BACKUP SERVER TO, the parameter
innodb_log_recovery_start that is written out to backup.cnf is
STRICTLY NECESSARY TO AVOID CORRUPTION during recovery! By default,
InnoDB crash recovery starts from the latest available log
checkpoint. However, for restoring a backup, recovery must start from
the checkpoint that was the latest when the backup was
started. Starting recovery from a possible later checkpoint will
result in a corrupted database!

The backup or the first stream will contain a file backup.cnf, which
includes parameters needed for restoring the backup. Currently,
these are innodb_log_recovery_start and innodb_log_recovery_target.
If innodb_log_recovery_target>0, InnoDB will be in read-only mode,
not allowing any writes to persistent files other than via the log
application.

To restore a streaming backup made with BACKUP SERVER WITH, an empty
directory needs to be created and all streams be extracted there using
the standard tar utility of the operating system, optionally after
undoing any encryption or compression that had been added by the
backup command. Then, the backup is prepared or MariaDB server started
up on the extracted directory, similar to as if the BACKUP SERVER TO
statement had been used.

The following will be implemented separately:

MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER
MDEV-40163 Partial backup and restore
MDEV-39091 Back up ENGINE=RocksDB
MDEV-40333 Concurrent DDL for Aria tables in BACKUP SERVER
MDEV-39987 Improve BACKUP SERVER performance for ENGINE=Aria

The implementation introduces a basic multi-threaded driver
Sql_cmd_backup, storage engine interfaces, and basic copying of the
storage engines InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV.

backup_target: A structured data type to represent a target directory.
On Microsoft Windows, we must use directory paths because there is
no variant of CopyFileEx() that would work on file handles.

backup_sink: Wraps a per-thread output stream as well as storage
engine specific context. The member "id" identifies the thread,
between 0 and CONCURRENT-1.

backup_context: Process-wide BACKUP SERVER context.
Handles the directory traversal and copying of
files for built-in storage engines that do not
implement this backup interface.

backup_target_stage: Wrap backup_context.

backup_target_stage::step(),
backup_context::step(): Process a file
from a directory scan, or by invoking
handlerton::backup_step().

handlerton::backup_file(): Check if a file should be
included in the backup. Implemented for ENGINE=Aria
in maria_backup_file().

handlerton::backup_start(), handlerton::backup_end(): Invoked at the
start or end of a backup phase, in the thread that executes a
BACKUP SERVER statement.

handlerton::backup_step(): A backup step that can be invoked from
multiple threads concurrently, between the execution of the corresponding
handlerton::backup_start() and handlerton::backup_end() of the same
phase.

copy_entire_file(): A file copying service for POSIX systems.

clone_file_range(): A wrapper for Linux FICLONERANGE, which
may fail and thus require a fallback to backup::copy().

backup::copy(): A partial or sparse file-copying service.
Note: On Linux we never invoke sendfile(2) for
copying between files, nor copy_file_range(2) for copying
large files, because can be much slower than the alternatives.

backup::copy_or_stream(): Copy or stream a file.

backup::append(): The streaming streaming equivalent of backup::copy()..

backup_stream_zeropad(): Zero-pad the last tar block if needed.

backup_stream_append_async(): A variant of backup::append()
where the source file region is guaranteed to be immutable after the
call returns. Zero-copy Linux sendfile(2) is inherently risky for copying
data files that may be modified in place, because it could introduce
a race condition between a page write that runs concurrently with a
child process that is reading the data from the pipe. This could be
improved with io_uring later.

pread_write(): Allocate a buffer of up to 4 MiB.
This is the "slow path" of copying or streaming files.
For InnoDB data files, BACKUP SERVER TO invokes this on output that
has been opened in unbuffered I/O (O_DIRECT) mode, to avoid polluting
the Linux file system cache (block cache). For the InnoDB log,
regular buffered I/O performed best in our tests.

maria_backup_start(): The entry point for ENGINE=Aria.
Currently, the directory traversal is single-threaded and a memory hog.
This is to be improved in MDEV-39987. The Aria log and data file
copying were implemented by Michael 'Monty' Widenius and
Alessandro Vetere.

maria_backup_log_tail(): Copy the necessary part of the last
ENGINE=Aria log, by making use of block-cloning when possible.
The copy will be trimmed to the logical end of the backup.
The logic was developed by Alessandro Vetere.

InnoDB_backup::init(): Wait for a possible previous BACKUP SERVER
operation to reach the very end of InnoDB_backup::context::cleanup()
so that the context can be safely reused.

InnoDB_backup::queue: Collection of tablespace IDs and payload sizes
at the start of the backup, and the log_sys.first_lsn of log files
that have to be included in the backup. If any data file is created or
extended while the backup is executing, we must have the corresponding
write-ahead-log entries that we are copying since the latest
checkpoint that was completed when the backup started. If any
tablespaces are deleted during the backup, we may or may not copy
them, and the application of a FILE_DELETE record will remove them.
Similarly, applying FILE_RENAME or FILE_CREATE records will rename or
create files during recovery as needed.

InnoDB_backup::context::tracked: Log file queue for the dedicated
log-copying task InnoDB_backup::log_track().

innodb_backup_log_tracking(): Determine if log-tracking backup is
active, which blocks concurrent changes of innodb_log_file_buffering
and innodb_log_file_write_through.

log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this
is to make BACKUP SERVER prevent the concurrent execution of
SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size
when innodb_log_archive=OFF.

log_sys.archived_checkpoint: Keep track of the earliest available
checkpoint, corresponding to log_sys.archived_lsn. This reflects
SET GLOBAL innodb_log_recovery_start (which is settable now), for
incremental backup.

fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed
to be known. To speed up startup, InnoDB does not normally open
all tablespace files.

fil_space_t::create_lsn: Change to Atomic_relaxed and use this to
indicate tablespace creation LSN, in addition to indicate undo
tablespace rebuild LSN.

fil_space_t::backup_end: The first page number that is not being backed up
(by default 0, to indicate that no backup is in progress).

fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be
covered by fil_space_t::backup_end. This is the unit of "page range locking"
during InnoDB backup.

fil_node_t::backup_name: A snapshot of the file name, sampled in
InnoDB_backup::init(). For backing up any files that were created before
the backup start checkpoint LSN, these names will be written to the backup.
On restore, the files will be renamed or deleted as needed, by
applying FILE_RENAME or FILE_DELETE records.

fil_node_t::rename(), fil_node_t::set_backup_name(),
fil_node_t::get_backup_name(), fil_node_t::~fil_node_t(),
fil_node_t::clear_backup_name(): Deal with fil_node_t::backup_name.

buf_page_t::write_fix_try(), buf_page_t::write_unfix_try(): Try to set
or unset a fake "write fix" on a page, to prevent concurrent flush()
during a backup batch. The atomic operations may run concurrently with
set_reinit() and set_freed(). The fake "write fix" does not prevent
any concurrent read or write of the page data in the buffer pool; it
only blocks writes to the underlying data file.

buf_page_t::flush(): Atomically test and set write fix, and
skip the operation if the fake "write fix" was set.

buf_page_t::set_freed(), buf_page_t::set_reinit(): Employ a
compare-and-exchange loop to accommodate for the "write fix".

innodb_backup_batch_wait(): Look up any pages that we are about to
back up. For any dirty pages, invoke buf_page_t::write_fix_try() to
try to set a fake "write fix" lock-free. If the page is currently
write-fixed between buf_page_t::flush() and
buf_page_t::write_complete(), acquire and release a page U-latch to
wait for the conflicting write to complete.

InnoDB_backup::backup_batch_start(),
InnoDB_backup::backup_batch_stop(): Adjust fil_space_t::backup_end and
fake "write fix" of dirty pages to protect the copying of a range of
pages from the underlying file.

InnoDB_backup::commit(): Enqueue the remaining log to be copied.

InnoDB_backup::checkpoint_complete(): If backup is running and
commit() has not been called, add each completed innodb_archive_log=ON
file to the queue. Else, skip or delete, as appropriate.

InnoDB_backup::archive_create(), innodb_backup_archive_create():
Hook for log_t::archive_create().

log_t::backup_start(): If we were running with innodb_log_archive=ON,
ensure that the latest file is a valid recovery starting point.
That is, wait for the latest log checkpoint to be within the file.

buf_flush_list_space(): Check for concurrent backup before writing each
page. This is inefficient, but this function may be invoked from multiple
threads concurrently, and it cannot be changed easily, especially for
fil_crypt_thread().

fil_ibd_create(): Set fil_space_t::create_lsn after the file has been
created.

dict_load_tablespaces(): Determine the size of each file if
upgrade==true. Backup depends on that.

buf_dblwr_t::begin(), buf_dblwr_t::end(), buf_dblwr_t::size():
Accessors to allow BACKUP SERVER to skip the contents of the
doublewrite buffer in the system tablespace. It is only useful for
crash recovery in case a data page had been incompletely written by
the time the server was killed. If the server is killed during a
backup, the backup will be incomplete and unusable anyway.
Furthermore, the page range locking makes page writes and backup
mutually exclusive.

Tested by: Matthias Leich
Tested by: Saahil Alam
Tested by: Rahul Raj
Co-authored by: Michael 'Monty' Widenius
Co-authored by: Alessandro Vetere
Reviewed by: Thirunarayanan Balathandayuthapani
Reviewed by: Marko Mäkelä (the ENGINE=Aria part)

c07ef488d208...

c07ef488d208ff26aed4b9a74e254874a934d899

Sergei Petrunia

sergey@mariadb.com

MDEV-38274: Unexpectedly large pages_read_count in main.analyze_stmt_prefetch_count

The testcase checked the values of pages_prefetch_read_count and
pages_read_count that InnoDB reported for the query.

The numbers depend on how fast the prefetch thread is in comparison with
the query thread. It can also happen that we request to prefetch a page
but then still have to wait for it to be read, which is counted in
pages_read_count.

Replace the checks in the file with ones that will hold in all
circumstances.

b395045b5a55...

b395045b5a55aacac18acda8a9651432d8d8ad1e

rusher

diego.dupin@mariadb.com

Merge branch '3.4-stmt-fetch-loop' into 3.4-perf

af4b453fc427...

af4b453fc427a296d867f1e1a3165176be05e975

Alexey (Holyfoot) Botchkov

holyfoot@mariadb.com

MDEV-36261 XMLTYPE: methods - step 1

Method functions added to the XMLTYPE.
functions added here are:

FUNCTION XMLTYPE.createNonSchemaBasedXML() RETURN XMLType
FUNCTION XMLTYPE.createSchemaBasedXML(schema TEXT DEFAULT NULL) RETURN XMLType
FUNCTION XMLTYPE.getClobVal() RETURN LONGTEXT
FUNCTION XMLTYPE.getNamespace() RETURN LONGTEXT
FUNCTION XMLTYPE.isFragment() RETURN INTEGER
FUNCTION XMLTYPE.getSchemaURL() RETURN LONGTEXT
FUNCTION XMLTYPE.getNumberVal() RETURN DOUBLE
FUNCTION XMLTYPE.getStringVal() RETURN LONGTEXT
FUNCTION XMLTYPE.getRootElement() RETURN LONGTEXT

ab976e5b770e...

ab976e5b770eb81e7493ec907b61ded83ee9ab53

Sergei Golubchik

serg@mariadb.org

skip columnstore better

followup for 550444c3bd4

a2e66eb33abe...

a2e66eb33abeb4da9597a0646bbee0796a648968

rusher

diego.dupin@mariadb.com

Merge branch '3.4-codec-simplify' into 3.4-perf

9ebc3fb398d4...

9ebc3fb398d4eef58bb1ec6d7c051c2d43859cca

rusher

diego.dupin@mariadb.com

Simplify the binary result conversion functions

Make ps_fetch_from_1_to_8_bytes() static inline: it is only called with
a constant byte count, but had external linkage so the compiler kept it
out of line and switched on the byte count at runtime for every integer
column. Each caller now contains only the store and range check for its
width.

In ps_fetch_datetime(), choose the destination MYSQL_TIME first and
decode once instead of calling convert_to_datetime() from every case of
the buffer type switch, where the DATE case duplicated
DATETIME/TIMESTAMP. Being called once, convert_to_datetime() is inlined
as well.

No functional change.

Assisted-by: Claude Fable 5.1

90596d281d17...

90596d281d175dfeb1744a91ed1c3f21f9c1362f

Alexey Yurchenko

alexey.yurchenko@mariadb.com

MDEV-41851 register cache manager with SAVEPOINT handler list

MDEV-41851 happens because by the time SAVEPOINT is called there
exists no connection cache manager yet and it does not care to
create one. As a result the cache manger is lazily created later
but is missing from the SAVEPOINT's handler list and thus is not
fired on ROLLBACK TO SAVEPOINT keeping the event in the cache
which in turn is replicated on subsequent COMMIT and applied on
the replicas.
Make sure to create cache manager in wsrep_register_binlog_handler()
if it does not yet exist.

8eb44f0b1406...

8eb44f0b1406e47dc4d2cfa90e2cc642bf93f775

Abdelrahman Hedia

bodyhadia44@gmail.com

MDEV-38486: No semi-sync ACK for @@skip_replication transactions

A transaction committed with @@skip_replication=1 is marked on the master as
not taking part in replication. Semi-sync did not know about that, so it
registered the transaction for an ACK and the commit waited for one. A slave
running with --replicate-events-marked-for-skip=FILTER_ON_MASTER never receives
such a transaction and so never ACKs it: the commit blocked until
rpl_semi_sync_master_timeout expired, and semi-sync was switched off.

Such a transaction now does not take part in semi-sync either: the master
neither requests nor awaits an ACK for it, whatever each slave then does with
it. Semi-sync is a master-side guarantee, and the transaction has already
opted out of replication on the master. Making it depend on the slaves'
--replicate-events-marked-for-skip would not be consistent anyway, as a
FILTER_ON_SLAVE slave ACKs an event which it then throws away, just like a
FILTER_ON_MASTER slave that never received it.

The whole change is in the semi-sync master code, behind its existing
get_master_enabled() checks, so nothing changes when semi-sync is off.
Repl_semi_sync_master::skips_ack() tests the session's OPTION_SKIP_REPLICATION;
report_binlog_update() then does not register the transaction, and
wait_after_sync() and wait_after_commit() do not wait for it. The option
cannot change inside a transaction or a statement, so registering and waiting
always agree. wait_after_sync() now takes the transaction's THD for that,
because in group commit the waiting thread is the leader, not the thread of
the transaction.

Test: rpl.rpl_semi_sync_skip_replication runs, for every
rpl_semi_sync_master_wait_point and every replicate_events_marked_for_skip
value, @@skip_replication changes through both the direct binlog write and the
group commit paths, and checks that no ACK is requested, granted or missed for
them. It also forces a skip_replication and an ordinary transaction into one
binlog group commit, with either one as the leader, and checks that exactly one
ACK is requested and granted, and that a following ordinary transaction is
still ACKed.

861c6f1eced1...

861c6f1eced1806127eadbdbd7e7a738300af161

Kristian Nielsen

knielsen@knielsen-hq.org

Fix help

Signed-off-by: Kristian Nielsen <[email protected]>

857fa9ae5630...

857fa9ae56304e4087bae6e06bf656615f08af51

Oleksandr Byelkin

sanja@mariadb.com

Merge branch 'bb-11.4-release' into br-11.8-merge

82351e5b863d...

82351e5b863d7b6bac08e58d3065a10eba6d2137

rusher

diego.dupin@mariadb.com

Merge branch '3.4-single-pass-metadata' into 3.4-perf

7cfefd08e195...

7cfefd08e195bda8e03b0687632953656128ce7d

Marko Mäkelä

marko.makela@mariadb.com

707efb615611...

707efb615611d7d56153cee6747d060ab91bf8a3

Andreas Schnederle-Wagner

schnederle@futureweb.at

MDEV-41757 Phrase search misses rows of a FULLTEXT index built over rows with an empty string in an earlier indexed column

When a FULLTEXT index is built over existing rows,
row_merge_fts_doc_tokenize() moves the start position of the next
field by the length of the field plus one byte, also for a field that
holds an empty string. fts_init_recover_doc(), which adds the rows
that were not synced to the cache again after a crash, does the same.
Rows added to an existing index (fts_fetch_doc_from_rec(),
fts_fetch_doc_from_tuple()) and phrase search
(fts_query_fetch_document()) skip an empty field, as they skip a NULL.
So the words after an empty field were stored one byte too far, and
phrase search, which starts to match at the stored position, did not
find the row.

An empty field no longer moves the position. Indexes built before
this fix keep the wrong positions until they are rebuilt.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

520b19798e0e...

520b19798e0efd2424f71e504aef0eab4e238a4a

Oleksandr Byelkin

sanja@mariadb.com

New help

5092a4419d78...

5092a4419d7877c219d773939dc8d72608073296

Oleksandr Byelkin

sanja@mariadb.com

Merge branch '11.8' into 12.3

4da3fcf642ff...

4da3fcf642ff627aec0480c1c3cc51fa049aefa9

Thirunarayanan Balathandayuthapani

thiru@mariadb.com

MDEV-41336 CHECK TABLE EXTENDED crashes with innodb_force_recovery=3

Problem:
=========
After a restart where purge has not run (When
innodb_force_recovery=3), purge_sys.head and purge_sys.tail are both
unset (trx_no=0). trx_lists_init_at_db_start() calls
purge_sys.clone_oldest_view<true>(), which clamped end_view to that
0, so end_view.m_low_limit_id became 0 and changes_visible()
returned false for every transaction id. end_view therefore
claimed that no history had been purged, although undo log records
purged before the restart may already have been freed and reused.

CHECK TABLE EXTENDED relies on end_view to decide whether an old
version's undo log record is still available. It followed the roll
pointer of a purged transaction into a reused undo page and
trx_undo_rec_get_pars() hit the assertion

Solution:
========
In purge_sys_t::clone_oldest_view<true>(), clamp end_view only if
purge has a known position (head.trx_no/tail.trx_no is nonzero).
Otherwise keep the oldest view as is, so that the undo log records
that purge is allowed to remove are reported as
DB_MISSING_HISTORY rather than being read.

412003c47cca...

412003c47cca77fbe392ae687e3967b6cd19f82f

Andreas Schnederle-Wagner

schnederle@futureweb.at

MDEV-41751 A FULLTEXT index added later misses rows when the first one was added to an empty table

When ALTER TABLE rebuilds a table with a hidden FTS_DOC_ID column, as
when adding the first FULLTEXT index or rebuilding a table that has
one, prepare_inplace_alter_table_dict() sets DICT_TF2_FTS_ADD_DOC_ID on
the new table, so that row_merge_build_indexes() assigns new FTS_DOC_ID
values; it clears the flag when it is done. Since MDEV-23805,
inplace_alter_table() skips that phase when the table is empty, so the
flag stayed set in the cached table definition until the table was
loaded again, for example after a restart. A FULLTEXT index added
before that without a rebuild got new document IDs instead of the
FTS_DOC_ID of the rows, and searches on it did not find the rows that
existed when it was created.

Do not set the flag when inplace_alter_table() skips that phase, that
is, when ha_alter_info->mdl_exclusive_after_prepare is set.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

3d1745037127...

3d1745037127c08e253667d3bfed343a0d0a8e7d

Oleksandr Byelkin

sanja@mariadb.com

Merge branch '10.11' into 11.4

32b5c211977d...

32b5c211977dc7030a2193c6eb952e3e75c386c6

Yuchen Pei

ycp@mariadb.com

31cdc1648dfb...

31cdc1648dfb038c8f0e890ef491fd9f3c606abb

rusher

diego.dupin@mariadb.com

[CONC-876] Reduce per-row overhead when parsing result sets

Binary rows, mthd_stmt_fetch_to_bind(): load loop invariant values
(field and bind arrays, bind_result_done, report_data_truncation,
result_callback_type) once per row instead of re-reading them from the
statement for every column, and merge the temporal length check and the
CONC-820 limits into a single switch on the column type. Conversion
functions and result callbacks advance a private copy of the position;
the next column always starts at the already validated column end, so
the row pointer no longer needs to live in memory. row_ptr is no longer
reset for all columns before each row: it is set for every column
during the pass, and cleared for the failing and the following columns
when a row is malformed.

Text rows, mthd_my_read_rows(): allocate the MYSQL_ROWS node and the
row data (column pointers and values) in one memory root allocation
instead of two. The destination bounds check in the column loop is
dropped: a value consumes at least len + 1 packet bytes (length prefix
included) and is stored in len + 1 bytes, so the fields + pkt_len byte
buffer cannot overflow once the value fits in the packet.

Assisted-by: Claude Fable 5.1

2fc546131a53...

2fc546131a535267201890ae021719ae3a7867dc

Pekka Lampio

pekka.lampio@galeracluster.com

MDEV-38999: Galera - stale reads in explicit transactions

wsrep_must_sync_wait() skipped a statement's causal wait unconditionally
once a multi-statement transaction was open, regardless of which
wsrep_sync_wait bit was configured - so later statements could run
against a view no fresher than whatever BEGIN itself happened to check.
A timed-out wait's error also stayed sticky until commit/rollback,
breaking ROLLBACK itself.

Fix: only skip the wait once the transaction has made its first write or
locking read (SELECT ... FOR UPDATE/LOCK IN SHARE MODE, including via a
stored procedure CALL, autocommit statements, and UNION) - after that,
re-waiting risks a self-deadlock against the local applier. Clear a
timed-out wait's error immediately instead of leaving it sticky.

Tests: galera_MDEV-38999*.test cover the bitmask gap, the self-deadlock
risk, stored-procedure/UNION/autocommit locking reads, and the query
cache's independent wait path.

Assisted-by Claude AI

2cd82b6ec68b...

2cd82b6ec68b02a49632f64f545a2cfa71c01170

Yuchen Pei

ycp@mariadb.com

MDEV-25848 Range optimizer support for JSON_OVERLAPS and JSON_CONTAINS

Implemented get_mm_tree for JSON_OVERLAPS and JSON_CONTAINS.

For each JSON_OVERLAPS/JSON_CONTAINS(field, value), parse the array or
scalar element into a list. Map with get_mm_parts over the list and
tree_or/tree_and the results. At leaf level uses a helper extracted
from MEMBER OF.

Also moved JSON_OVERLAPS to inherit from Item_bool_func2_with_rev due
to its symmetry.

230f541c44c2...

230f541c44c2477c5f8296ec9e0d2ccc2b46a372

Alexander Barkov

bar@mariadb.com

MDEV-39563 Implement UPDATE ... RETURNING ... INTO

Adding support for UPDATE .. RETURNING .. INTO queries.

For example:

UPDATE t1 SET a=10,b=20 RETURNING a,b INTO va,vb;
UPDATE t1 SET a=10,b=20 RETURNING a,b INTO @a,@b;

Limitations:
1. These types of queries:
- REPLACE .. RETURNING .. INTO
- DELETE .. RETURNING .. INTO
- INSERT .. RETURNING .. INTO
do not work - they return an error.
They will be implemented separately, when needed.

2. UPDATE..RETURNING..INTO with --binlog_format=statement is not allowed
and an error is raised.

3. Using OLD_VALUE(col) inside UPDATE..RETURNING..INTO is not allowed
and an error is raised.

4. Multi-table updates, as well as single table updates with a subquery
to the same table in WHERE (which get converted to multi-table) do
not work and an error is raised.

Notes:

1. ANALYZE and EXPLAIN
Both
ANALYZE UPDATE .. RETURNING .. INTO ..
EXPLAIN UPDATE .. RETURNING .. INTO ..
return this error:
'RETURNING..INTO' is not allowed in this context

2. Behavior on no data and on multiple rows

Only the rows that were actually changed are processed. A row that matches
the WHERE condition but needs no update is not changed, so it is not
returned by plain RETURNING either.

Situation | RETURNING | RETURNING .. INTO
-----------------------------+------------------+-----------------------------
Empty table, WHERE matches | empty set | Warning ER_SP_FETCH_NO_DATA,
nothing, WHERE 1=0, LIMIT 0 | | target is not changed

One row matches, needs no | empty set | Warning ER_SP_FETCH_NO_DATA,
update | | target is not changed

Several rows match, none | empty set | Warning ER_SP_FETCH_NO_DATA,
needs an update | | target is not changed
| | (no ER_TOO_MANY_ROWS)

One row matches and is | that row | Target is assigned
changed | |

Several rows match, exactly | that row | Target is assigned from
one changes | | that row

Two or more rows change | all changed rows | ER_TOO_MANY_ROWS at the 2nd
| | changed row (not updated)

Notes:
- The warning is never an error: not in the strict mode, not with
sql_mode='', not with IGNORE. The INTO targets keep their old values.
In stored routines the warning activates NOT FOUND handlers.
- This is the same for all storage engines, including those that quickly
know the table is empty and use a degenerated plan (e.g. MyISAM).
- After ER_TOO_MANY_ROWS, transactional engines roll back the statement,
non-transactional engines keep the rows updated before the error.
- The targets can be user variables, SP variables, ROW variables
and ROW fields.

3. Behavior on VIEW with CHECK OPTION

The "too many rows" check is done after the view's CHECK OPTION check.
With UPDATE IGNORE, a row rejected by WITH CHECK OPTION is skipped
(with a warning) and does not count as an updated row, so
UPDATE IGNORE .. RETURNING .. INTO does not raise ER_TOO_MANY_ROWS
if only one row is actually updated.

Helper changes:

1. The grammar in analyze_stmt_command was changed to have
LEX::analyze_stmt set to true earlier, so
LEX::set_returning_into_result() already knows if this
is an ANALYZE statement.

2. The Sql_cmd_update constructor is now called earlier in the grammar,
to be able to call Sql_cmd_update::set_with_old_value_items()
in the SET and RETURNING clauses.

3. Sql_cmd_dml::lex is now set during the constructor time.
It makes things easier:
- Sql_cmd_update::returns_result_set() needs the lex.
- Sql_cmd_delete::orig_multitable and Sql_cmd_update::orig_multitable
are not needed any more.
They were used only in Sql_cmd_delete::sql_command_code() and
Sql_cmd_update::sql_command_code().
Sql_cmd_dml::sql_command_code() now returns lex->sql_command.
The overrides Sql_cmd_delete::sql_command_code() and
Sql_cmd_update::sql_command_code() were removed.

This commit also fixes bugs found during testing:
- MDEV-41262 ER_TOO_MANY_ROWS should be checked after CHECK OPTION in view
while Update Returning Into
- MDEV-41265 SELECT/ UPDATE RETURNING INTO a single row-field fails with
ER_WRONG_NUMBER_OF_COLUMNS_IN_SELECT, even though the column count matches
This bug was also fixed in 12.3.4.
- MDEV-41260 Misleading message text "uses a LIMIT clause" after Update
Returning Into

1b34e9f5e11f...

1b34e9f5e11fdd851bcec9f865a4aa8a07c38a64

rusher

diego.dupin@mariadb.com

[misc] Remove the flags parameter from the internal length decoders

net_field_length_safe() and net_field_length_ll_safe() took a flags argument whose only defined value, MARIADB_LENENC_FORBID_NULL, was passed by every internal caller that does not accept a NULL marker, and all of them already test the result against ARIADB_LENENC_OK.

since this must be the most called function in the whole connector, drop the parameter and the flag validation: 0xFB is always eported as MARIADB_LENENC_NULL, which those callers reject as before.

1abe530c0d8e...

1abe530c0d8e349dd09a2c3f3b5f9b233bfc556a

Yuchen Pei

ycp@mariadb.com

1a4f519624b6...

1a4f519624b682c05f5b373589b20e44602d0d48

Yuchen Pei

ycp@mariadb.com

MDEV-25848 Revert SEL_TREE::keys allocation size for B-tree indexes

0e3425c2872e...

0e3425c2872ea83814cb3abf8c84fe577a45306a

Marko Mäkelä

marko.makela@mariadb.com

0364192a2529...

0364192a2529766174e9d2720552543d3b588ae2

Yuchen Pei

ycp@mariadb.com