Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Sergei Golubchik
send ssl_info_client to prove the authentication was not relayed

A MitM can relay the authentication, let the client fail on ssl_info
and keep using the authenticated connection.

* MARIADB_CLIENT_SSL_INFO_CLIENT: set it whenever TLS is used
* after the authentication OK packet send ssl_info_client, the server
  does not reply (but might disconnect in case of active MitM).
Sergei Golubchik
MDEV-40530 Bind the client authentication response to the server certificate

MDEV-31855 introduced ssl_info that let the client verify
that it uses the correct certificate. But MitM could hijack the
connection after the authentication - the client will detect it
and disconnect but the server won't and will continue talking
to the attacker. Let's introduce ssl_info_client packet that
enables the server to verify that the client uses the correct
certificate.

When require_secure_transport=ON and the connection isn't local:
* the server sets MARIADB_CLIENT_SSL_INFO_CLIENT in the initial handshake
* the client sets it in the reply, if it understands it
* when both sides have it, the client sends ssl_info_client right
  after the OK packet, acl_authenticate() reads it before the session
  is used (init_connect, audit, etc)
* a connection is rejected unless ssl_info_client is valid, either
  for the server certificate fingerprint or for "CA" (the client
  verified the certificate itself)
* an account without a password hash (plugin has no hash_password()
  or empty password) is rejected unless the account REQUIRE a client
  certificate

Assisted-by: Claude:claude-opus-5-5
Oleg Smirnov
MDEV-40439 Evaluate cheap subconditions before expensive ones in WHERE and ON conditions

The arguments of AND and OR are evaluated left to right with
short-circuiting, in the order written in the query. An expensive
subcondition written first is therefore evaluated even for rows that
a cheap one would reject (AND) or accept (OR).

Reorder the arguments of each AND/OR so that the expensive ones come
after the cheap ones, keeping the relative order within each group.
The walk goes down into nested AND/OR and into trigger conditions.
A subcondition is expensive if it contains a subquery, a stored
function call, or an item that marks itself as expensive through
is_expensive_processor() (a UDF, MATCH()).

An AND/OR that has a non-deterministic argument (one with RAND_TABLE_BIT)
keeps its order, to prevent unwanted change of behavior. Nested lists
are still reordered.

The reordering is applied to:
- the conditions attached to tables (select_cond, select->cond,
  on_precond), at the end of make_join_select() and before the
  attached_conditions_summary trace, so the trace shows the final
  order;
- BNL/BNLH scan filters, in make_scan_filter(). They are built from
  the original WHERE/ON, and check_join_cache_usage() may build them
  after make_join_select();
- outer_ref_cond and pseudo_bits_cond, the conditions a subquery
  checks once per execution before running its join.
The steps that later build new conditions from these (index condition
pushdown, removal of duplicate BNL conditions) take subsets and keep
the order.

Not covered yet: HAVING, exec_const_cond, terms added after
make_join_select() (the index condition remainder rejected by an
engine, HAVING moved to a table's condition). Reordering HAVING would
need care with Item_is_not_null_test and Item_ref_null_helper, which
record NULLs for IN->EXISTS.
Alexander Barkov
MDEV-41527 MEDIUMTEXT gets truncated to TINYTEXT in assoc array RECORD field

Adding 13.1 specific tests with package-wide data types.
Marko Mäkelä
Merge 13.1 into main
Oleg Smirnov
MDEV-40439 Evaluate cheap subconditions before expensive ones in WHERE and ON conditions

The arguments of AND and OR are evaluated left to right with
short-circuiting, in the order written in the query. An expensive
subcondition written first is therefore evaluated even for rows that
a cheap one would reject (AND) or accept (OR).

Reorder the arguments of each AND/OR so that the expensive ones come
after the cheap ones, keeping the relative order within each group.
The walk goes down into nested AND/OR and into trigger conditions.
A subcondition is expensive if it contains a subquery, a stored
function call, or an item that marks itself as expensive through
is_expensive_processor() (a UDF, MATCH()).

An AND/OR that has a non-deterministic argument (one with RAND_TABLE_BIT)
keeps its order, to prevent unwanted change of behavior. Nested lists
are still reordered.

The reordering is applied to:
- the conditions attached to tables (select_cond, select->cond,
  on_precond), at the end of make_join_select() and before the
  attached_conditions_summary trace, so the trace shows the final
  order;
- BNL/BNLH scan filters, in make_scan_filter(). They are built from
  the original WHERE/ON, and check_join_cache_usage() may build them
  after make_join_select();
- outer_ref_cond and pseudo_bits_cond, the conditions a subquery
  checks once per execution before running its join.
The steps that later build new conditions from these (index condition
pushdown, removal of duplicate BNL conditions) take subsets and keep
the order.

Not covered yet: HAVING, exec_const_cond, terms added after
make_join_select() (the index condition remainder rejected by an
engine, HAVING moved to a table's condition). Reordering HAVING would
need care with Item_is_not_null_test and Item_ref_null_helper, which
record NULLs for IN->EXISTS.
Marko Mäkelä
MDEV-40935: Infinite loop in SET GLOBAL innodb_log_archive

log_t::set_archive(): Ensure that a log checkpoint will be written
even if no dirty pages exist in the buffer pool. This was reproduced
with an intermittent livelock during the test innodb.log_file_size_online.
Marko Mäkelä
MDEV-42009: Set GLOBAL innodb_log_file_size=ON is not crash safe

log_t::set_archive(): If the log is memory-mapped
(on /dev/shm or on a DAX mounted file system),
zero-fill the rest of the file before updating the header,
so that any old garbage from innodb_log_archive=OFF format
will be properly discarded before recovery.
Oleksandr Byelkin
new columnstore 25.10.8
Sergei Petrunia
MDEV-38274: Unexpectedly large pages_read_count in main.analyze_stmt_prefetch_count

The testcase checked the values of pages_prefetch_read_count and
pages_read_count that InnoDB reported for the query.

The numbers depend on how fast the prefetch thread is in comparison with
the query thread. It can also happen that we request to prefetch a page
but then still have to wait for it to be read, which is counted in
pages_read_count.

Replace the checks in the file with ones that will hold in all
circumstances.
Monty
Removed warning from cmake

- DEPENDS cannot be used with ADD_CUSTOM_COMMAND. Removed it
Vladislav Vaintroub
MDEV-42087 Memory leak in sql_service when a statement fails after it sent its metadata

When a statement that runs through mysql_real_connect_local() fails
after it sent its metadata, the error is stored in the dataset of the
result set. embedded_get_error() freed only the MYSQL_DATA, and not its
MEM_ROOT, so 8192 bytes were lost, for example for a statement that is
aborted by KILL QUERY. Free the dataset with free_rows().

Assisted-by: Claude:claude-sonnet-5-5
rusher
CONC-873: Avoid copying result set metadata on every stmt execution
Sergei Golubchik
MDEV-40530 Bind the client authentication response to the server certificate

MDEV-31855 introduced ssl_info that let the client verify
that it uses the correct certificate. But MitM could hijack the
connection after the authentication - the client will detect it
and disconnect but the server won't and will continue talking
to the attacker. Let's introduce ssl_info_client packet that
enables the server to verify that the client uses the correct
certificate.

When require_secure_transport=ON and the connection isn't local:
* the server sets MARIADB_CLIENT_SSL_INFO_CLIENT in the initial handshake
* the client sets it in the reply, if it understands it
* when both sides have it, the client sends ssl_info_client right
  after the OK packet, acl_authenticate() reads it before the session
  is used (init_connect, audit, etc)
* a connection is rejected unless ssl_info_client is valid, either
  for the server certificate fingerprint or for "CA" (the client
  verified the certificate itself)
* an account without a password hash (plugin has no hash_password()
  or empty password) is rejected unless the account REQUIRE a client
  certificate

Assisted-by: Claude:claude-opus-5-5
rusher
CONC-875: Read packets directly from the read ahead cache

Assisted-by: Claude Fable 5.1
Sergei Petrunia
MDEV-25848: record/replay hlindex records_in_range() in optimizer context

check_quick_select_array() now goes through hook_hl_records_in_range(),
which records or replaces both the row estimate and the cost
(record_hl_records_in_range / infuse_hl_records_in_range). The calls are
stored in a new per-table "hl_records_in_range_calls" array.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Thirunarayanan Balathandayuthapani
MDEV-32067 InnoDB linear read-ahead had better be logical

The traditional linear read-ahead, enabled by
innodb_read_ahead_threshold, only helps when consecutively
accessed pages have adjacent page numbers.

That is rarely true for B-tree leaf pages: after splits,
merges and page reuse, the logical order of leaves has
nothing to do with their physical page numbers. So for a
scan of scattered leaves the old read-ahead either did nothing
or read the wrong pages.

This replaces linear read-ahead of index pages with a logical
read-ahead that prefetches the actual leaf pages a scan is
going to visit. The PAGE_LEVEL=1 page already lists the page
numbers of its leaves in key order, and every scan descends
through it. So the child page numbers are copied from that page
and read asynchronously, first while descending, and then in
a rolling fashion as the scan advances.
After this change, the exact nonzero value of
innodb_read_ahead_threshold matters only for the read-ahead of
undo log pages.

buf_read_ahead_undo(): Renamed from buf_read_ahead_linear(). Now
invoked only for undo log pages, whose page numbers are
sequential. It is no longer called on BLOB pages or index pages.
Resident pages are skipped, and the tablespace reference is
taken only right before a read is issued.

buf_read_ahead_pages(): Read ahead a set of known B-tree leaf
pages, at most buf_pool_t::READ_AHEAD_PAGES of them. Pages that
are already in the buffer pool are skipped. It stops at a page
beyond the end of the file, when no free block is available
(it never waits for one), or when a read cannot be issued.
It does nothing for the temporary tablespace.

buf_read_ahead_one(): Read ahead a single page, by calling
buf_read_ahead_pages(). Used to prefetch the leaf after the
next one when a persistent cursor enters a page.

buf_read_ahead_random(): Split into a decision part (which extent
to read, in buf0buf.cc) and an action part (issue the reads).
A page of the extent that is already resident is skipped, and the
pages that were read are counted.

Read-ahead is skipped when the number of pending reads exceeds
half of the buffer pool size in pages.

fil_space_t::s_lock_try(): Try to acquire the shared allocation
latch without waiting. Undo and system tablespace shrinking holds
fil_space_t::latch exclusively (x_lock_space()), and
buf_read_ahead_undo() holds it shared while it checks the size and
issues the reads. If the latch is not available, the read-ahead is
skipped.

trx_undo_get_prev_rec(), trx_undo_get_prev_rec_from_prev_page():
Take the trx_undo_t object instead of a long parameter list and
always latch shared.

btr_ra_batch: A caller-owned buffer of leaf page numbers. It also
records the PAGE_LEVEL=1 page, the child page number of the last
node pointer collected, and the leaf at which to stop, so that the
harvest can be resumed later.

btr_ra_batch::collect(): Append the child page numbers of the node
pointers of a PAGE_LEVEL=1 page in scan order (forward or
reverse), starting from the given record, until the buffer is full,
the page is exhausted, or the last leaf of the scan was collected.

btr_ra_batch::resume_rec(): Locate the record from which the
harvest should continue, by finding the last collected child by
value. It walks only genuine records, so a concurrent page
reorganization cannot cause an invalid read.

dict_index_t::get_child(): Read the child page number of a node
pointer record without rec_get_offsets().

dict_index_t::get_child_offset(), rec_data_size(): The offset after
the first fields of a record. rec_data_size() is the field walk
that used to be inside rec_fold(), now shared; rec_fold() computes
the CRC-32C of the first rec_data_size() bytes. Both are defined in
btr0sea.cc, outside #ifdef BTR_CUR_HASH_ADAPT because
get_child_offset() is needed also without the adaptive hash index,
and in the same compilation unit as rec_fold(), so that it can be
inlined. btr_node_ptr_set_child_page_no() uses get_child_offset()
instead of rec_get_offsets().

btr_cur_t::search_leaf(), btr_cur_t::open_leaf(): Optionally take a
btr_ra_batch. At PAGE_LEVEL=1, after the descent has located the
child to follow, collect the leaf page numbers starting at that
child in the scan direction. These are exactly the leaves that
the cursor will visit.

btr_pcur_move_to_next_page(): Prefetch the sibling after the
one being entered, with buf_read_ahead_one().

btr_pcur_move_backward_from_page(): Add a no-wait fast path that
latches the previous page if it is resident (validating it against
the current page), and prefetch the one before it.

btr_copy_blob_prefix(): Simplified; no longer relies on
FIL_PAGE_PREV/NEXT, which are not initialized on BLOB pages.

records_in_range() reports the first and last leaf page of a
range. handler::multi_range_read_info_const() takes a page_range
parameter and returns the extent of the whole scan: the first leaf
of the first range and the last leaf of the last range, or
unused_page_range if any range does not report one. The extent is
carried by opt_range.cc through the chosen QUICK_RANGE_SELECT,
whose reset() calls the new handler::advise_page_range() before
multi_range_read_init(). All storage engines that override
multi_range_read_info_const() follow the new signature.
The extent only sizes and bounds the read-ahead; it never decides
which pages are read.

ha_innobase::ReadAhead: The state of the read-ahead of a scan, held
in the data member ha_innobase::readahead. It is reset at the end
of each statement.

ReadAhead::init(), init_full(): Enable read-ahead with a ceiling
(max_window) of at most 64 leaves, which is capped at 1/256 of the
buffer pool (but at least 4 leaves), because every prefetched page
occupies a buffer pool frame as soon as its read is issued.

ReadAhead::advise(), set_limit(), window_for_extent(): Derive the
ceiling from the extent advised by the optimizer, falling back to
a leaf count computed from the LIMIT. One leaf, or a LIMIT of 2
or less, means no read-ahead.

ha_innobase::search_with_readahead(): Position the cursor while
collecting leaf pages, then call ReadAhead::start(). It is used by
index_read() and by the first general_fetch() only when read-ahead
is armed, so the other paths carry no read-ahead locals. The first
batch has 4 leaves if the optimizer advised an extent of several
leaves, and 2 otherwise, so that probes such as MIN() and LIMIT 1
do not read many leaves. It stops at the last leaf of the range.
Lookups of a complete unique key do not use it.

ReadAhead::start(): Prefetch the collected batch and remember the
PAGE_LEVEL=1 page and the last child, the rolling cursor.

ReadAhead::refill(): As the scan advances (general_fetch()), after
about half of the batch was consumed, resume the harvest under an
index S-latch, move on to the sibling PAGE_LEVEL=1 page when one is
exhausted, and prefetch the next batch, which is twice as large as
the previous one, up to the ceiling. The reads are issued after the
latches are released. The harvest ends at the advised last leaf, or
at the end of the index when the extent is unknown, or when the
PAGE_LEVEL=1 page was freed or reused.

Read-ahead is started from index_read() (range and full index
scans), rnd_init()/general_fetch() (full table scans), and, at
one-page depth, from row_merge_read_clustered_index() (OPTIMIZE and
ALTER rebuild). general_fetch() sets active_handler_stats before
issuing read-ahead so prefetched pages are attributed to the query.

dict_stats_analyze_index_for_n_prefix(): Choose the records to dive
below first, prefetch their child pages in one batch with
buf_read_ahead_pages(), and then dive below the same records.
Vladislav Vaintroub
fixup! MDEV-42087 Memory leak in sql_service when a statement fails after it sent its metadata

SQL keywords in upper case in test_sql_service_abort

Assisted-by: Claude:claude-sonnet-5-5
Kristian Nielsen
MDEV-41658 fix Galera node crash when a parallel slave worker is bf_killed

A parallel-replication worker would assert in ha_rollback_trans() because
wsrep_after_statement() and ha_commit_trans()'s wsrep error path could
roll back its transaction before retry_event_group() unmarks it as
committing.

Skip mark_start_commit() when using Galera. Galera certification can cause
rollback of parallel replication transactions during commit when there is a
conflicting transaction outside of replication. This can cause to start the
next GCO too early, while the previous GCO still needs to rollback and
retry.

Do not fail replication with an error when wsrep_after_statement() fails due
to a retryable error like brute-force kill.

Signed-off-by: Kristian Nielsen <[email protected]>
Vladislav Vaintroub
MDEV-38741 Abort the running statement when its client disconnects

The server notices that a client is gone only when it reads from or
writes to the socket. A statement that waits, for example for an InnoDB
row lock, keeps running, and its transaction keeps its locks.

Watch the client socket from the end of login until the connection ends
or the client sends COM_QUIT, with epoll (Linux), kqueue (BSD, macOS),
or on Windows with WSAEventSelect and an I/O completion port, and the
exit of the client process for a named pipe. When the client has gone,
the watcher sets THD::client_gone. If a command that expects an answer
is running, it is aborted with KILL_CONNECTION. If the connection is
idle, the next such command is aborted when it starts. COM_QUIT and
COM_STMT_CLOSE are not aborted, a client can send them and then quit.
The watcher and the connection thread order their accesses with fences,
no lock is taken for a command. The socket is not shut down, the
connection ends after the error is sent, and the reason, "Client
disconnected", is logged once. An idle connection with the disconnect
noted shows that State in the processlist. Galera connections are not
aborted.

The global variable disconnect_watcher, ON by default, can be set OFF,
which also ends the watches that exist. It exists only where there is a
watcher: Linux, Windows, the BSDs and macOS, not the embedded server.

mysqltest dirty_close takes an optional linger or nolinger argument.

New tests: main.disconnect_*, sys_vars.disconnect_watcher_basic and
mysql_client_test test_disconnect_*. Some rpl and innodb tests are
adjusted to pass with the watcher ON and OFF.

Assisted-by: Claude:claude-sonnet-5-5
rusher
[misc] Decode length encoded integers without nested call
Monty
MDEV-41810 Atomic CoR: a crash at the end of CREATE OR REPLACE loses both the old and the new table

The given test case was already in the test suite, but unfortunately I had
missed a failing test in the result file.

Fixed by marking the created table complete before removing the
backup table.
I also extended create_replace_no_binlog.test with a new crash point to
ensure state is tested before and after marking table complete.
Marko Mäkelä
MDEV-14992/MDEV-38362/MDEV-39092 BACKUP SERVER

The following SQL statements will be introduced:

BACKUP SERVER TO '/path/to/directory' [ 1 CONCURRENT ];
BACKUP SERVER WITH [ 1 CONCURRENT ] 'command';

In place of the 1, any positive number of threads may be specified.
For the first variant, '/path/to' must exist and '/path/to/directory'
must not exist; that is where the backup will be written to.

For BACKUP SERVER WITH, the string mariadb-backup-
will be prepended to the specified command and quoted.
Path separators or quotes are not allowed in command.
We expect the mariadb-backup-command to be in the
search PATH of the mariadbd process.
For now, we distribute no such script or program;
the user has to write one, something like this:

zstdmt|exec ssh [email protected] "exec cat > $1.tar.zstd"

The standard input of that command will be in a format compatible with
GNU tar --format=oldgnu (and also BSD tar variants that are also part of
Microsoft Windows and Apple macOS). The command is expected to optionally
compress and encrypt the stream and redirect it to a file on a local or
a remote server. The BACKUP SERVER WITH will append an additional argument,
a positive base-ten number in ASCII, starting with 1, to identify the
current thread. In this way, there can be multiple concurrent streams.
The InnoDB write-ahead log will be streamed near the end, because
the stream format requires the file sizes to be declared upfront.

In BACKUP SERVER TO ... 2 CONCURRENT or more, one thread will be
responsible for copying the InnoDB write-ahead log from the beginning
of the backup, running InnoDB_backup::log_track(). Other files will be
copied by other threads.

Note: In single-threaded BACKUP SERVER TO, the parameter
innodb_log_recovery_start that is written out to backup.cnf is
STRICTLY NECESSARY TO AVOID CORRUPTION during recovery! By default,
InnoDB crash recovery starts from the latest available log
checkpoint. However, for restoring a backup, recovery must start from
the checkpoint that was the latest when the backup was
started. Starting recovery from a possible later checkpoint will
result in a corrupted database!

The backup or the first stream will contain a file backup.cnf, which
includes parameters needed for restoring the backup. Currently,
these are innodb_log_recovery_start and innodb_log_recovery_target.
If innodb_log_recovery_target>0, InnoDB will be in read-only mode,
not allowing any writes to persistent files other than via the log
application.

To restore a streaming backup made with BACKUP SERVER WITH, an empty
directory needs to be created and all streams be extracted there using
the standard tar utility of the operating system, optionally after
undoing any encryption or compression that had been added by the
backup command. Then, the backup is prepared or MariaDB server started
up on the extracted directory, similar to as if the BACKUP SERVER TO
statement had been used.

The following will be implemented separately:

MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER
MDEV-40163 Partial backup and restore
MDEV-39091 Back up ENGINE=RocksDB
MDEV-40333 Concurrent DDL for Aria tables in BACKUP SERVER
MDEV-39987 Improve BACKUP SERVER performance for ENGINE=Aria

The implementation introduces a basic multi-threaded driver
Sql_cmd_backup, storage engine interfaces, and basic copying of the
storage engines InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV.

backup_target: A structured data type to represent a target directory.
On Microsoft Windows, we must use directory paths because there is
no variant of CopyFileEx() that would work on file handles.

backup_sink: Wraps a per-thread output stream as well as storage
engine specific context. The member "id" identifies the thread,
between 0 and CONCURRENT-1.

backup_context: Process-wide BACKUP SERVER context.
Handles the directory traversal and copying of
files for built-in storage engines that do not
implement this backup interface.

backup_target_stage: Wrap backup_context.

backup_target_stage::step(),
backup_context::step(): Process a file
from a directory scan, or by invoking
handlerton::backup_step().

handlerton::backup_file(): Check if a file should be
included in the backup. Implemented for ENGINE=Aria
in maria_backup_file().

handlerton::backup_start(), handlerton::backup_end(): Invoked at the
start or end of a backup phase, in the thread that executes a
BACKUP SERVER statement.

handlerton::backup_step(): A backup step that can be invoked from
multiple threads concurrently, between the execution of the corresponding
handlerton::backup_start() and handlerton::backup_end() of the same
phase.

copy_entire_file(): A file copying service for POSIX systems.

clone_file_range(): A wrapper for Linux FICLONERANGE, which
may fail and thus require a fallback to backup::copy().

backup::copy(): A partial or sparse file-copying service.
Note: On Linux we never invoke sendfile(2) for
copying between files, nor copy_file_range(2) for copying
large files, because can be much slower than the alternatives.

backup::copy_or_stream(): Copy or stream a file.

backup::append(): The streaming streaming equivalent of backup::copy()..

backup_stream_zeropad(): Zero-pad the last tar block if needed.

backup_stream_append_async(): A variant of backup::append()
where the source file region is guaranteed to be immutable after the
call returns. Zero-copy Linux sendfile(2) is inherently risky for copying
data files that may be modified in place, because it could introduce
a race condition between a page write that runs concurrently with a
child process that is reading the data from the pipe. This could be
improved with io_uring later.

pread_write(): Allocate a buffer of up to 4 MiB.
This is the "slow path" of copying or streaming files.
For InnoDB data files, BACKUP SERVER TO invokes this on output that
has been opened in unbuffered I/O (O_DIRECT) mode, to avoid polluting
the Linux file system cache (block cache). For the InnoDB log,
regular buffered I/O performed best in our tests.

maria_backup_start(): The entry point for ENGINE=Aria.
Currently, the directory traversal is single-threaded and a memory hog.
This is to be improved in MDEV-39987. The Aria log and data file
copying were implemented by Michael 'Monty' Widenius and
Alessandro Vetere.

maria_backup_log_tail(): Copy the necessary part of the last
ENGINE=Aria log, by making use of block-cloning when possible.
The copy will be trimmed to the logical end of the backup.
The logic was developed by Alessandro Vetere.

InnoDB_backup::init(): Wait for a possible previous BACKUP SERVER
operation to reach the very end of InnoDB_backup::context::cleanup()
so that the context can be safely reused.

InnoDB_backup::queue: Collection of tablespace IDs and payload sizes
at the start of the backup, and the log_sys.first_lsn of log files
that have to be included in the backup. If any data file is created or
extended while the backup is executing, we must have the corresponding
write-ahead-log entries that we are copying since the latest
checkpoint that was completed when the backup started. If any
tablespaces are deleted during the backup, we may or may not copy
them, and the application of a FILE_DELETE record will remove them.
Similarly, applying FILE_RENAME or FILE_CREATE records will rename or
create files during recovery as needed.

InnoDB_backup::context::tracked: Log file queue for the dedicated
log-copying task InnoDB_backup::log_track().

innodb_backup_log_tracking(): Determine if log-tracking backup is
active, which blocks concurrent changes of innodb_log_file_buffering
and innodb_log_file_write_through.

log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this
is to make BACKUP SERVER prevent the concurrent execution of
SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size
when innodb_log_archive=OFF.

log_sys.archived_checkpoint: Keep track of the earliest available
checkpoint, corresponding to log_sys.archived_lsn. This reflects
SET GLOBAL innodb_log_recovery_start (which is settable now), for
incremental backup.

fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed
to be known. To speed up startup, InnoDB does not normally open
all tablespace files.

fil_space_t::create_lsn: Change to Atomic_relaxed and use this to
indicate tablespace creation LSN, in addition to indicate undo
tablespace rebuild LSN.

fil_space_t::backup_end: The first page number that is not being backed up
(by default 0, to indicate that no backup is in progress).

fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be
covered by fil_space_t::backup_end. This is the unit of "page range locking"
during InnoDB backup.

fil_node_t::backup_name: A snapshot of the file name, sampled in
InnoDB_backup::init(). For backing up any files that were created before
the backup start checkpoint LSN, these names will be written to the backup.
On restore, the files will be renamed or deleted as needed, by
applying FILE_RENAME or FILE_DELETE records.

fil_node_t::rename(), fil_node_t::set_backup_name(),
fil_node_t::get_backup_name(), fil_node_t::~fil_node_t(),
fil_node_t::clear_backup_name(): Deal with fil_node_t::backup_name.

buf_page_t::write_fix_try(), buf_page_t::write_unfix_try(): Try to set
or unset a fake "write fix" on a page, to prevent concurrent flush()
during a backup batch. The atomic operations may run concurrently with
set_reinit() and set_freed(). The fake "write fix" does not prevent
any concurrent read or write of the page data in the buffer pool; it
only blocks writes to the underlying data file.

buf_page_t::flush(): Atomically test and set write fix, and
skip the operation if the fake "write fix" was set.

buf_page_t::set_freed(), buf_page_t::set_reinit(): Employ a
compare-and-exchange loop to accommodate for the "write fix".

innodb_backup_batch_wait(): Look up any pages that we are about to
back up. For any dirty pages, invoke buf_page_t::write_fix_try() to
try to set a fake "write fix" lock-free. If the page is currently
write-fixed between buf_page_t::flush() and
buf_page_t::write_complete(), acquire and release a page U-latch to
wait for the conflicting write to complete.

InnoDB_backup::backup_batch_start(),
InnoDB_backup::backup_batch_stop(): Adjust fil_space_t::backup_end and
fake "write fix" of dirty pages to protect the copying of a range of
pages from the underlying file.

InnoDB_backup::commit(): Enqueue the remaining log to be copied.

InnoDB_backup::checkpoint_complete(): If backup is running and
commit() has not been called, add each completed innodb_archive_log=ON
file to the queue. Else, skip or delete, as appropriate.

InnoDB_backup::archive_create(), innodb_backup_archive_create():
Hook for log_t::archive_create().

log_t::backup_start(): If we were running with innodb_log_archive=ON,
ensure that the latest file is a valid recovery starting point.
That is, wait for the latest log checkpoint to be within the file.

buf_flush_list_space(): Check for concurrent backup before writing each
page. This is inefficient, but this function may be invoked from multiple
threads concurrently, and it cannot be changed easily, especially for
fil_crypt_thread().

fil_ibd_create(): Set fil_space_t::create_lsn after the file has been
created.

dict_load_tablespaces(): Determine the size of each file if
upgrade==true. Backup depends on that.

buf_dblwr_t::begin(), buf_dblwr_t::end(), buf_dblwr_t::size():
Accessors to allow BACKUP SERVER to skip the contents of the
doublewrite buffer in the system tablespace. It is only useful for
crash recovery in case a data page had been incompletely written by
the time the server was killed. If the server is killed during a
backup, the backup will be incomplete and unusable anyway.
Furthermore, the page range locking makes page writes and backup
mutually exclusive.

Tested by: Matthias Leich
Tested by: Saahil Alam
Tested by: Rahul Raj
Co-authored by: Michael 'Monty' Widenius
Co-authored by: Alessandro Vetere
Reviewed by: Thirunarayanan Balathandayuthapani
Reviewed by: Marko Mäkelä (the ENGINE=Aria part)
Sergei Petrunia
MDEV-38274: Unexpectedly large pages_read_count in main.analyze_stmt_prefetch_count

The testcase checked the values of pages_prefetch_read_count and
pages_read_count that InnoDB reported for the query.

The numbers depend on how fast the prefetch thread is in comparison with
the query thread. It can also happen that we request to prefetch a page
but then still have to wait for it to be read, which is counted in
pages_read_count.

Replace the checks in the file with ones that will hold in all
circumstances.
Marko Mäkelä
MDEV-40935: Assertion os_file_set_size() failed during shutdown

os_file_set_size(): Consistently handle EINTR also after server
shutdown has been initiated. This will avoid a crash when an
innodb_log_archive=ON log file needs to be created or extended
during shutdown. This only affects the posix_fallocate(2) or
Linux fallocate(2) code path; if the file is being extended by
regular writes, we will retry in case of EINTR.

On Microsoft Windows, there is no clear documentation what would happen
if the process is being terminated while SetFilePointerEx() or
SetEndOfFile() are in progress.
zeyad zahran
MDEV-40294 mariadb-binlog --start/stop-datetime incorrectly accepts over-max timestamps

For a value outside of the TIMESTAMP range, my_system_gmt_sec()
returns 0 and sets ER_WARN_DATA_OUT_OF_RANGE, but
convert_str_to_timestamp() never checked it. So such values silently
became 1970: an over-max --start-datetime printed every event and an
over-max --stop-datetime printed none.

Print a warning for out of range values and clamp the over-max ones
to the max timestamp. Under-min values already became the min.
rusher
Merge branches '3.4-stmt-metadata-copy', '3.4-single-pass-metadata' and '3.4-net-read-fast-path' into 3.4-perf
Oleksandr Byelkin
Merge branch '10.11' into 11.4
Sergei Petrunia
MDEV-38274: Unexpectedly large pages_read_count in main.analyze_stmt_prefetch_count

The testcase checked the values of pages_prefetch_read_count and
pages_read_count that InnoDB reported for the query.

The numbers depend on how fast the prefetch thread is in comparison with
the query thread. It can also happen that we request to prefetch a page
but then still have to wait for it to be read, which is counted in
pages_read_count.

Replace the checks in the file with ones that will hold in all
circumstances.
rusher
CONC-874: Parse result set column definitions in a single pass

Column definitions were processed twice: mthd_my_read_rows() copied each
packet into an intermediate MYSQL_DATA row set, then unpack_fields()
copied it again into MYSQL_FIELD with six allocations per column.

Assisted-by: Claude Fable 5.1
Marko Mäkelä
MDEV-42009: Set GLOBAL innodb_log_file_size=ON is not crash safe

log_t::header_rewrite(): Before setting log_sys.archive=true,
ensure that any garbage entries at the end of the file will be
zeroed out so that they cannot be mistaken for newer records.
On Linux and Microsoft Windows we will attempt a shortcut for
writing zeroes, before falling back to log_file_t::write().

log_t::set_archive(): Block concurrent writes to the log on every
platform, because log_t::header_rewrite() now requires exclusive
access to the log file.

Reviewed by: Alessandro Vetere
Monty
Preparation for MDEV-41866 layout-dependent my.cnf location

- Allow on to change location of /etc/mysql in Debian/Ubunty by setting
  DEFAULT_SYSCONFDIR=/etc/mariadb
- Search for MariaDB configuration files also in /etc/mariadb. In
  14.0 we plan to remove searching in /etc/mysql by default.
- Fixed mariadb.logrotate to use DEFAULT_SYSCONFDIR if used.
- mariadbd_multi now also uses /etc/mariadb
Sergei Golubchik
cleanup: st_mariadb_extension in the server's client library

Like in Connector/C, MYSQL::extension is allocated in mysql_init()
and holds auto_local_infile, tls_self_signed_error and the MariaDB
extended capabilities of the server and the client.
The MYSQL layout does not change.

Assisted-by: Claude:claude-opus-5-5
Marko Mäkelä
MDEV-42009: Set GLOBAL innodb_log_archive=ON is not crash safe

log_t::header_rewrite(): Before setting log_sys.archive=true,
ensure that any garbage entries at the end of the file will be
zeroed out so that they cannot be mistaken for newer records.
On Linux and Microsoft Windows we will attempt a shortcut for
writing zeroes, before falling back to log_file_t::write().

log_t::set_archive(): Block concurrent writes to the log on every
platform, because log_t::header_rewrite() now requires exclusive
access to the log file.

Reviewed by: Alessandro Vetere
Sergei Golubchik
MDEV-40530 Bind the client authentication response to the server certificate

MDEV-31855 introduced ssl_info that let the client verify
that it uses the correct certificate. But MitM could hijack the
connection after the authentication - the client will detect it
and disconnect but the server won't and will continue talking
to the attacker. Let's introduce ssl_info_client packet that
enables the server to verify that the client uses the correct
certificate.

When require_secure_transport=ON and the connection isn't local:
* the server sets MARIADB_CLIENT_SSL_INFO_CLIENT in the initial handshake
* the client sets it in the reply, if it understands it
* when both sides have it, the client sends ssl_info_client right
  after the OK packet, acl_authenticate() reads it before the session
  is used (init_connect, audit, etc)
* a connection is rejected unless ssl_info_client is valid, either
  for the server certificate fingerprint or for "CA" (the client
  verified the certificate itself)
* an account without a password hash (plugin has no hash_password()
  or empty password) is rejected unless the account REQUIRE a client
  certificate

Assisted-by: Claude:claude-opus-5-5