Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Aleksey Midenkov
MDEV-20865 Fix tests and system scripts to foreign key requirements

Storing foreign key metadata in TABLE_SHARE makes opening a foreign table
preopen its referenced tables, so a referenced table must be created
before the foreign one and dropped after it. Rearrange the order of
CREATE/DROP commands in tests and system scripts accordingly.

Affected: the help tables in mariadb_system_tables{,_fix}.sql (with
system_mysql_db_fix* results), fetch_first, union, insert_notembedded,
instant_alter_index_rename and opt_context_store_ddls; some also index a
referenced column or add a missing referenced table, and
opt_context_store_ddls re-records the now-shown MyISAM foreign key.
Aleksey Midenkov
gcov fix

Fix coverage data corruption by enabling atomic profile updates

When compiling with coverage instrumentation, concurrent or forked test executions
can cause corruption of coverage data files (.gcda), resulting in negative hit counts
and invalid coverage reports.

Adding the compiler flag `-fprofile-update=atomic` ensures that updates to
coverage counters are performed atomically, preventing race conditions during
profile data writes.

This change eliminates warnings about unexpected negative hit counts and
improves the accuracy and reliability of coverage measurements in parallel
or multithreaded test environments.
Marko Mäkelä
MDEV-14992 BACKUP SERVER

The following SQL statements will be introduced:

BACKUP SERVER TO '/path/to/directory' [ 1 CONCURRENT ];
BACKUP SERVER WITH [ 1 CONCURRENT ] 'command';

In place of the 1, any positive number of threads may be specified.
For the first variant, '/path/to' must exist and '/path/to/directory'
must be compatible with secure_file_priv and not exist; that is where
the backup will be written to.

For the second variant, 'command' must be the name of a script or
command that will be executed in a child process. The standard input
of that command will be in a format that is compatible with
GNU tar --format=oldgnu (and also BSD tar variants that are also part of
Microsoft Windows and Apple macOS). The command is expected to optionally
compress and encrypt the stream and redirect it to a file on a local or
a remote server. The BACKUP SERVER WITH will append an additional argument,
a positive base-ten number in ASCII, starting with 1, to identify the
current thread. In this way, each concurrent stream can write a separate
file.

The backup or the first stream will contain a file backup.cnf, which
includes parameters needed for restoring the backup. Currently,
these are innodb_log_recovery_start and innodb_log_recovery_target.
If innodb_log_recovery_target>0, InnoDB will be in read-only mode,
not allowing any writes to persistent files other than via the log
application.

To restore a streaming backup made with BACKUP SERVER WITH, an empty
directory needs to be created and all streams be extracted there using
the standard tar utility of the operating system, optionally after
undoing any encryption or compression that had been added by the
backup command. Then, the backup is prepared or MariaDB server started
up on the extracted directory, similar to as if the BACKUP SERVER TO
statement had been used.

Note: The parameter innodb_log_recovery_start in backup.cnf is
STRICTLY NECESSARY TO AVOID CORRUPTION! By default, InnoDB crash recovery
starts from the latest available log checkpoint. However, for restoring
a backup, recovery must start from the checkpoint that was the latest
when the backup was started. Starting recovery from a possible later
checkpoint will result in a corrupted database!

The following will be implemented separately:

MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER
MDEV-40163 Partial backup and restore
MDEV-39091 Back up ENGINE=RocksDB
MDEV-39092 Less blocking backup of ENGINE=Aria

The implementation introduces a basic driver Sql_cmd_backup,
storage engine interfaces, and basic copying of the storage engines
InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV.

aria_backup_end(): A crude prototype that copies non-InnoDB files.
Scans and copies the data directories in a single thread, while
everything is locked. This will be refactored in MDEV-39092.

backup_target: A structured data type to represent a target directory.
On Microsoft Windows, we must use directory paths because there is
no variant of CopyFileEx() that would work on file handles.

backup_sink: Wraps a per-thread output stream as well as storage engine
specific context.

handlerton::backup_start(), handlerton::backup_end(): Invoked at the
start or end of a backup phase, in the thread that executes a
BACKUP SERVER statement.

handlerton::backup_step(): A backup step that can be invoked from
multiple threads concurrently, between the execution of the corresponding
handlerton::backup_start() and handlerton::backup_end() of the same
phase.

copy_entire_file(): A file copying service for POSIX systems.

copy_mmap(): A zero-copy alternative to backup::copy(), to copy from a
memory-mapped buffer.

copy_file_range_try(): A wrapper for Linux copy_file_range(2), which
may fail with EOPNOTSUPP or EXDEV and thus require a fallback to
copy_mmap() or backup::copy().

backup::copy(): A partial or sparse file-copying service.  On other
platforms than FreeBSD or Microsoft Windows, there are shortcut
alternatives to this. Note: On Linux we never invoke sendfile(2) for
copying between files, because can be much slower than the
alternatives.

backup_stream_append_plain(): Equivalent to backup::copy(), but
appending to a stream. On Linux, this uses sendfile(2), which assumes
that the source data will not be changed before the data has been
consumed from the pipe.

backup_stream_append_async(): A variant of backup_stream_append_plain()
where the source file region is guaranteed to be immutable after the
call returns. We must not use zero-copy mmap(2) or Linux sendfile(2)
for copying data files that may be modified in place, because it could
introduce a race condition between a page write that runs concurrently
with a child process that is reading the data from the pipe.

backup_stream_zeropad(): Zero-pad the last tar block if needed.

InnoDB_backup::context: Backup context, attached to backup_sink
so that context can continue to exist between the time a
BACKUP SERVER releases all locks and another BACKUP SERVER starts
executing, with innodb_backup pointing to the new backup, while
the old backup is still being finished.

InnoDB_backup::queue: Collection of tablespace IDs and payload sizes
at the start of the backup, and the log_sys.first_lsn of log files
that have to be included in the backup. If any data file is created or
extended while the backup is executing, we must have the corresponding
write-ahead-log entries that we are copying since the latest
checkpoint that was completed when the backup started. If any
tablespaces are deleted during the backup, we may or may not copy
them, and the application of a FILE_DELETE record will remove them.
Similarly, applying FILE_RENAME or FILE_CREATE records will rename or
create files during recovery as needed.

log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this
is to make BACKUP SERVER prevent the concurrent execution of
SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size
when innodb_log_archive=OFF.

log_sys.archived_checkpoint: Keep track of the earliest available
checkpoint, corresponding to log_sys.archived_lsn. This reflects
SET GLOBAL innodb_log_recovery_start (which is settable now), for
incremental backup.

fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed
to be known. To speed up startup, InnoDB does not normally open
all tablespace files.

fil_space_t::create_lsn: Change to Atomic_relaxed and use this to
indicate tablespace creation LSN, in addition to indicate undo
tablespace rebuild LSN.

fil_space_t::backup_end: The first page number that is not being backed up
(by default 0, to indicate that no backup is in progress).

fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be
covered by fil_space_t::backup_end. This is the unit of "page range locking"
during InnoDB backup.

buf_page_t::write_fix_try(), buf_page_t::write_unfix_try(): Try to set
or unset a fake "write fix" on a page, to prevent concurrent flush()
during a backup batch. The atomic operations may run concurrently with
set_reinit() and set_freed(). The fake "write fix" does not prevent
any concurrent read or write of the page data in the buffer pool; it
only blocks writes to the underlying data file.

buf_page_t::flush(): Atomically test and set write fix, and
skip the operation if the fake "write fix" was set.

buf_page_t::set_freed(), buf_page_t::set_reinit(): Employ a
compare-and-exchange loop to accommodate for the "write fix".

innodb_backup_batch_wait(): Look up any pages that we are about to
back up. For any dirty pages, invoke buf_page_t::write_fix_try() to
try to set a fake "write fix" lock-free. If the page is currently
write-fixed between buf_page_t::flush() and
buf_page_t::write_complete(), acquire and release a page U-latch to
wait for the conflicting write to complete.

InnoDB_backup::backup_batch_start(),
InnoDB_backup::backup_batch_stop(): Adjust fil_space_t::backup_end and
fake "write fix" of dirty pages to protect the copying of a range of
pages from the underlying file.

log_t::backup_start(): If we were running with innodb_log_archive=ON,
ensure that the latest file is a valid recovery starting point.
That is, wait for the latest log checkpoint to be within the file.

buf_flush_list_space(): Check for concurrent backup before writing each
page. This is inefficient, but this function may be invoked from multiple
threads concurrently, and it cannot be changed easily, especially for
fil_crypt_thread().

fil_ibd_create(): Set fil_space_t::create_lsn after the file has been
created.

dict_load_tablespaces(): Determine the size of each file if
upgrade==true. Backup depends on that.

buf_dblwr_t::begin(), buf_dblwr_t::end(), buf_dblwr_t::size():
Accessors to allow BACKUP SERVER to skip the contents of the
doublewrite buffer in the system tablespace. It is only useful for
crash recovery in case a data page had been incompletely written by
the time the server was killed. If the server is killed during a
backup, the backup will be incomplete and unusable anyway.
Furthermore, the page range locking makes page writes and backup
mutually exclusive.
Aleksey Midenkov
Deprecated warning fix

Two compatibility warnings are meant for end users but only add noise to
debug/test builds: mysqld is commonly started through a legacy "mysqld"
symlink, and test .cnf files still reference long-removed options. Emit
them in release builds only.

- mysys/my_init.c: the deprecated-invocation-name warning
  (EE_NAME_DEPRECATED), shown when the program is run via a non-"mariadb"
  symlink to a "mariadb*" binary -- now compiled under #ifdef NDEBUG.

- sql/mysqld.cc: the "'<opt>' was removed ... exists only for
  compatibility" warning for OPT_REMOVED_OPTION -- now under #ifdef DBUG_OFF.
Aleksey Midenkov
Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Aleksey Midenkov
Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Aleksey Midenkov
MDEV-20865 extra2_fields refactored to Extra2_info class

read_extra2() is now Extra2_info::read()
Additional assertions for checking size consistency.

Extra2_info::write() is used by further MDEV-20865 development.
Aleksey Midenkov
Deprecated warning fix

Two compatibility warnings are meant for end users but only add noise to
debug/test builds: mysqld is commonly started through a legacy "mysqld"
symlink, and test .cnf files still reference long-removed options. Emit
them in release builds only.

- mysys/my_init.c: the deprecated-invocation-name warning
  (EE_NAME_DEPRECATED), shown when the program is run via a non-"mariadb"
  symlink to a "mariadb*" binary -- now compiled under #ifdef NDEBUG.

- sql/mysqld.cc: the "'<opt>' was removed ... exists only for
  compatibility" warning for OPT_REMOVED_OPTION -- now under #ifdef DBUG_OFF.
Aleksey Midenkov
MDEV-20865 build_frm_image() readability cleanup

More clear names and constants use.
Aleksey Midenkov
MDEV-20865 extra2_fields refactored to Extra2_info class

read_extra2() is now Extra2_info::read()
Additional assertions for checking size consistency.

Extra2_info::write() is used by further MDEV-20865 development.
Aleksey Midenkov
gcov fix

Fix coverage data corruption by enabling atomic profile updates

When compiling with coverage instrumentation, concurrent or forked test executions
can cause corruption of coverage data files (.gcda), resulting in negative hit counts
and invalid coverage reports.

Adding the compiler flag `-fprofile-update=atomic` ensures that updates to
coverage counters are performed atomically, preventing race conditions during
profile data writes.

This change eliminates warnings about unexpected negative hit counts and
improves the accuracy and reliability of coverage measurements in parallel
or multithreaded test environments.
Marko Mäkelä
MDEV-14992 BACKUP SERVER

The following SQL statements will be introduced:

BACKUP SERVER TO '/path/to/directory' [ 1 CONCURRENT ];
BACKUP SERVER WITH [ 1 CONCURRENT ] 'command';

In place of the 1, any positive number of threads may be specified.
For the first variant, '/path/to' must exist and '/path/to/directory'
must be compatible with secure_file_priv and not exist; that is where
the backup will be written to.

For the second variant, 'command' must be the name of a script or
command that will be executed in a child process. The standard input
of that command will be in a format that is compatible with
GNU tar --format=oldgnu (and also BSD tar variants that are also part of
Microsoft Windows and Apple macOS). The command is expected to optionally
compress and encrypt the stream and redirect it to a file on a local or
a remote server. The BACKUP SERVER WITH will append an additional argument,
a positive base-ten number in ASCII, starting with 1, to identify the
current thread. In this way, each concurrent stream can write a separate
file.

The backup or the first stream will contain a file backup.cnf, which
includes parameters needed for restoring the backup. Currently,
these are innodb_log_recovery_start and innodb_log_recovery_target.
If innodb_log_recovery_target>0, InnoDB will be in read-only mode,
not allowing any writes to persistent files other than via the log
application.

To restore a streaming backup made with BACKUP SERVER WITH, an empty
directory needs to be created and all streams be extracted there using
the standard tar utility of the operating system, optionally after
undoing any encryption or compression that had been added by the
backup command. Then, the backup is prepared or MariaDB server started
up on the extracted directory, similar to as if the BACKUP SERVER TO
statement had been used.

Note: The parameter innodb_log_recovery_start in backup.cnf is
STRICTLY NECESSARY TO AVOID CORRUPTION! By default, InnoDB crash recovery
starts from the latest available log checkpoint. However, for restoring
a backup, recovery must start from the checkpoint that was the latest
when the backup was started. Starting recovery from a possible later
checkpoint will result in a corrupted database!

The following will be implemented separately:

MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER
MDEV-40163 Partial backup and restore
MDEV-39091 Back up ENGINE=RocksDB
MDEV-39092 Less blocking backup of ENGINE=Aria

The implementation introduces a basic driver Sql_cmd_backup,
storage engine interfaces, and basic copying of the storage engines
InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV.

aria_backup_end(): A crude prototype that copies non-InnoDB files.
Scans and copies the data directories in a single thread, while
everything is locked. This will be refactored in MDEV-39092.

backup_target: A structured data type to represent a target directory.
On Microsoft Windows, we must use directory paths because there is
no variant of CopyFileEx() that would work on file handles.

backup_sink: Wraps a per-thread output stream as well as storage engine
specific context.

handlerton::backup_start(), handlerton::backup_end(): Invoked at the
start or end of a backup phase, in the thread that executes a
BACKUP SERVER statement.

handlerton::backup_step(): A backup step that can be invoked from
multiple threads concurrently, between the execution of the corresponding
handlerton::backup_start() and handlerton::backup_end() of the same
phase.

copy_entire_file(): A file copying service for POSIX systems.

copy_mmap(): A zero-copy alternative to backup::copy(), to copy from a
memory-mapped buffer.

copy_file_range_try(): A wrapper for Linux copy_file_range(2), which
may fail with EOPNOTSUPP or EXDEV and thus require a fallback to
copy_mmap() or backup::copy().

backup::copy(): A partial or sparse file-copying service.  On other
platforms than FreeBSD or Microsoft Windows, there are shortcut
alternatives to this. Note: On Linux we never invoke sendfile(2) for
copying between files, because can be much slower than the
alternatives.

backup_stream_append_plain(): Equivalent to backup::copy(), but
appending to a stream. On Linux, this uses sendfile(2), which assumes
that the source data will not be changed before the data has been
consumed from the pipe.

backup_stream_append_async(): A variant of backup_stream_append_plain()
where the source file region is guaranteed to be immutable after the
call returns. We must not use zero-copy mmap(2) or Linux sendfile(2)
for copying data files that may be modified in place, because it could
introduce a race condition between a page write that runs concurrently
with a child process that is reading the data from the pipe.

backup_stream_zeropad(): Zero-pad the last tar block if needed.

InnoDB_backup::context: Backup context, attached to backup_sink
so that context can continue to exist between the time a
BACKUP SERVER releases all locks and another BACKUP SERVER starts
executing, with innodb_backup pointing to the new backup, while
the old backup is still being finished.

InnoDB_backup::queue: Collection of tablespace IDs and payload sizes
at the start of the backup, and the log_sys.first_lsn of log files
that have to be included in the backup. If any data file is created or
extended while the backup is executing, we must have the corresponding
write-ahead-log entries that we are copying since the latest
checkpoint that was completed when the backup started. If any
tablespaces are deleted during the backup, we may or may not copy
them, and the application of a FILE_DELETE record will remove them.
Similarly, applying FILE_RENAME or FILE_CREATE records will rename or
create files during recovery as needed.

log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this
is to make BACKUP SERVER prevent the concurrent execution of
SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size
when innodb_log_archive=OFF.

log_sys.archived_checkpoint: Keep track of the earliest available
checkpoint, corresponding to log_sys.archived_lsn. This reflects
SET GLOBAL innodb_log_recovery_start (which is settable now), for
incremental backup.

fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed
to be known. To speed up startup, InnoDB does not normally open
all tablespace files.

fil_space_t::create_lsn: Change to Atomic_relaxed and use this to
indicate tablespace creation LSN, in addition to indicate undo
tablespace rebuild LSN.

fil_space_t::backup_end: The first page number that is not being backed up
(by default 0, to indicate that no backup is in progress).

fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be
covered by fil_space_t::backup_end. This is the unit of "page range locking"
during InnoDB backup.

buf_page_t::write_fix_try(), buf_page_t::write_unfix_try(): Try to set
or unset a fake "write fix" on a page, to prevent concurrent flush()
during a backup batch. The atomic operations may run concurrently with
set_reinit() and set_freed(). The fake "write fix" does not prevent
any concurrent read or write of the page data in the buffer pool; it
only blocks writes to the underlying data file.

buf_page_t::flush(): Atomically test and set write fix, and
skip the operation if the fake "write fix" was set.

buf_page_t::set_freed(), buf_page_t::set_reinit(): Employ a
compare-and-exchange loop to accommodate for the "write fix".

innodb_backup_batch_wait(): Look up any pages that we are about to
back up. For any dirty pages, invoke buf_page_t::write_fix_try() to
try to set a fake "write fix" lock-free. If the page is currently
write-fixed between buf_page_t::flush() and
buf_page_t::write_complete(), acquire and release a page U-latch to
wait for the conflicting write to complete.

InnoDB_backup::backup_batch_start(),
InnoDB_backup::backup_batch_stop(): Adjust fil_space_t::backup_end and
fake "write fix" of dirty pages to protect the copying of a range of
pages from the underlying file.

log_t::backup_start(): If we were running with innodb_log_archive=ON,
ensure that the latest file is a valid recovery starting point.
That is, wait for the latest log checkpoint to be within the file.

buf_flush_list_space(): Check for concurrent backup before writing each
page. This is inefficient, but this function may be invoked from multiple
threads concurrently, and it cannot be changed easily, especially for
fil_crypt_thread().

fil_ibd_create(): Set fil_space_t::create_lsn after the file has been
created.

dict_load_tablespaces(): Determine the size of each file if
upgrade==true. Backup depends on that.

buf_dblwr_t::begin(), buf_dblwr_t::end(), buf_dblwr_t::size():
Accessors to allow BACKUP SERVER to skip the contents of the
doublewrite buffer in the system tablespace. It is only useful for
crash recovery in case a data page had been incompletely written by
the time the server was killed. If the server is killed during a
backup, the backup will be incomplete and unusable anyway.
Furthermore, the page range locking makes page writes and backup
mutually exclusive.
Aleksey Midenkov
MDEV-20865 Fix tests and system scripts to foreign key requirements

Storing foreign key metadata in TABLE_SHARE makes opening a foreign table
preopen its referenced tables, so a referenced table must be created
before the foreign one and dropped after it. Rearrange the order of
CREATE/DROP commands in tests and system scripts accordingly.

Affected: the help tables in mariadb_system_tables{,_fix}.sql (with
system_mysql_db_fix* results), fetch_first, union, insert_notembedded,
instant_alter_index_rename and opt_context_store_ddls; some also index a
referenced column or add a missing referenced table, and
opt_context_store_ddls re-records the now-shown MyISAM foreign key.
Marko Mäkelä
MDEV-40728 Recovery wrongly fails if FILE_CREATE is followed by FILE_RENAME

deferred_spaces.deferred_dblwr(): Skip newly created tablespaces
to avoid a bogus invocation of fil_space_free().

fil_name_process(): Simplify the logic. If no matching tablespace is
found but file_name_t::create_lsn had been set in response to parsing
a FILE_CREATE record, try to apply FILE_RENAME to deferred_spaces.

log_parse_file(): Parse each FILE_ record only once. In multi-batch
recovery, there may be redundant calls that would break the logic of
fil_name_process().

fil_delete_apply(): Wrappers for fil_space_free(). When recovering
a log in innodb_log_archive=ON format, we must apply FILE_DELETE
records in order to avoid a future clash with FILE_CREATE or FILE_RENAME.
Aleksey Midenkov
Deprecated warning fix

Two compatibility warnings are meant for end users but only add noise to
debug/test builds: mysqld is commonly started through a legacy "mysqld"
symlink, and test .cnf files still reference long-removed options. Emit
them in release builds only.

- mysys/my_init.c: the deprecated-invocation-name warning
  (EE_NAME_DEPRECATED), shown when the program is run via a non-"mariadb"
  symlink to a "mariadb*" binary -- now compiled under #ifdef NDEBUG.

- sql/mysqld.cc: the "'<opt>' was removed ... exists only for
  compatibility" warning for OPT_REMOVED_OPTION -- now under #ifdef DBUG_OFF.
Aleksey Midenkov
MDEV-20865 build_frm_image() readability cleanup

More clear names and constants use.
Aleksey Midenkov
MDEV-20865 extra2 structures moved to datadict.h

datadict.h is appropriate place for such types. These data types are
used by Extra2_info, which is added to datadict.h later.

unireg.cc is for older FRM routines. datadict.cc accepts new and
refactored routines.
Marko Mäkelä
MDEV-40756 Incorrect multi-batch recovery of file size

file_name_t::page0_lsn: Keep track of the last applied
recv_sys_t::parse_page0() so that a multi-batch recovery
will not reset the file to a smaller size.

Reviewed by: Thirunarayanan Balathandayuthapani

(cherry picked from commit 8f00e6caca633c783140db86d3a48a96de67cf38)
Aleksey Midenkov
MDEV-20865 extra2_write_len() fix

Current implementation of extra2_write_len() does not guarantee of
writing correct 2-byte values as it skips writing zero at the
beginning.

Refined DBUG_ASSERT() to more truthful limit.
Aleksey Midenkov
MDEV-20865 Prelocking_strategy refactoring

- Moved has_prelocking_list into Prelocking_strategy (removed it from
  Multiupdate_prelocking_strategy). It is now initialized in
  Prelocking_strategy::reset(), so reset() overrides
  (Multiupdate_prelocking_strategy, Multidelete_prelocking_strategy)
  chain to the base implementation to set it.
- Moved extend_table_list() into Prelocking_strategy;
- Made maybe_need_prelocking() virtual method.

Now maybe_need_prelocking() can be different per prelocking strategy.
Aleksey Midenkov
MDEV-20865 extra2_write_len() fix

Current implementation of extra2_write_len() does not guarantee of
writing correct 2-byte values as it skips writing zero at the
beginning.

Refined DBUG_ASSERT() to more truthful limit.
Thirunarayanan Balathandayuthapani
MDEV-19574: innodb_stats_method is not honored when innodb_stats_persistent=ON

Problem:
=======
When persistent statistics are enabled (innodb_stats_persistent=ON),
the innodb_stats_method setting is not properly utilized during
statistics calculation.

The statistics collection functions always use a hardcoded default
behavior for NULL value comparison instead of respecting the
configured stats method. This affects the accuracy of
n_diff_key_vals (distinct key count), particularly for
indexes with nullable columns containing NULL values.

Moreover, stat_n_non_null_key_vals[] was never computed for
persistent statistics; it stayed at the 0 that
dict_stats_empty_index() assigns.

With innodb_stats_method=nulls_ignored, innodb_rec_per_key()
therefore always found n_diff <= n_null and reported one record
per key for every index. This impacts the query optimizer,
which makes decisions based on inaccurate cardinality estimates.

Solution:
========
Introduced IndexLevelStats to collect statistics at a specific
B-tree level during index analysis.

Introduced PageStats to collect statistics for leaf page analysis.

Refactored the following functions:
dict_stats_analyze_index_level() to IndexLevelStats::analyze_level()
dict_stats_analyze_index_for_n_prefix() to IndexLevelStats::sample_leaf_pages()
dict_stats_analyze_index_below_cur() to PageStats::scan_below()
dict_stats_scan_page() to PageStats::scan()

The innodb_stats_method value is read once per table in
dict_stats_update_persistent() and passed down, so that all
indexes of a table are analyzed with the same method.

Add the stats method name to stat_description when
innodb_stats_method has a non-default value. The suffix is
dropped when the description is already full.

Added the new stat name n_nonnull_fld01, n_nonnull_fld02, etc.
with a stats description, to indicate how many non-null values
exist for the nth field of the index. This value is retrieved
and stored in the index statistics
in dict_stats_fetch_index_stats_step(). The counts are per
column, not per n-column prefix.

rec_get_n_blob_pages(): Calculate the number of
externally stored pages for a record, using ceiling division
by the usable BLOB page payload (blob_part_size), which differs
between ROW_FORMAT=COMPRESSED (zip_size minus FIL_PAGE_DATA) and
the other formats (srv_page_size minus the BLOB header and the
page trailer). For ROW_FORMAT=COMPRESSED the length in
the field reference is the uncompressed length, so the result is an
upper bound.

When the leaf level is scanned in full, the number of leaf pages that
were scanned is reported as n_leaf_pages for a multi level index.
Before, result.n_leaf_pages was overwritten with
index->stat_n_leaf_pages, which dict_stats_empty_index() had
just set to 1, so every index that took the full scan path
reported n_leaf_pages=1. Single page indexes report 1.
This changes cardinality estimates and
therefore leads to multiple changes in existing test cases.

Non-null values are counted only at the leaf level, since only leaf
pages hold actual records. A full scan of the leaf level counts them
exactly. When the level is sampled, the per column count is derived
from the sampled leaves with the same formula as n_diff:

  n_ordinary_leaf_pages * n_non_null_all_analyzed_pages
                        / n_leaf_pages_to_analyze

This is an estimate for NOT NULL columns as well: the sampled leaves
may hold fewer or more records than the average, and a dive that
stops at a boring page contributes nothing to the sum while still
counting in the divisor.

innodb_rec_per_key(): stat_n_non_null_key_vals[i] holds the
number of records in which the i-th indexed column alone is
not NULL, while what has to be excluded here is the number
of records whose first i+1 columns are all not NULL,
because that is the population which the n-column
prefix statistic stat_n_diff_key_vals[i] has to be
corrected against when innodb_stats_method=nulls_ignored:
with NULLs compared as unequal, every record carrying a NULL
anywhere in the prefix adds a distinct value of its own to n_diff.

PageStats::scan(): n_non_null is accumulated and assigned only
for leaf pages, so that a non-leaf scan cannot leave a node
pointer count behind when scan_below() stops at a boring page
without reaching a leaf.

IndexLevelStats::reset_for_level() also clears n_diff[], and
dict_stats_analyze_index() zero initializes the buffer backing it, so
that a level scan which finds no records (a failed
btr_pcur_open_level(), or a non-leaf page whose first record is not
marked as the leftmost one on the level) leaves n_diff[] at 0 instead
of stale values.

IndexLevelStats::sample_leaf_pages() returns early when the group
boundaries for the prefix are empty, which is the same condition.

dict_stats_fetch_index_stats_step() no longer resets
stat_n_non_null_key_vals[] while processing an n_diff_pfxNN row:
dict_stats_empty_table() has already cleared the array before the
fetch, and with n_nonnull_fldNN rows now being read too,
that reset would make the result depend on the order in which the
rows arrive.

dict_stats_save(): now static function in dict0stats.cc that
takes the innodb_stats_method value, and is removed from dict0stats.h.
dict_stats_update_persistent() saves the statistics itself, so its
callers no longer have to.

Replaced btr_rec_get_externally_stored_len() with
rec_get_n_blob_pages() in dict0stats.cc.

btr_rec_get_field_ref_offs() and btr_rec_get_field_ref(),
together with the BTR_BLOB_HDR_* macros, were moved from btr0cur.cc
to btr0cur.h so that rec_get_n_blob_pages() can reuse them;

btr_rec_get_field_ref_offs() is now a noexcept function
returning size_t.

Changed stat_n_diff_key_vals and stat_n_non_null_key_vals from
ib_uint64_t* to uint64_t*

len_is_stored(): simplified to a single comparison, which is
equivalent for the unsigned lengths that it is used with.

Removed the unused UNIV_STATS_DEBUG build macro (univ.i) and
turned the DEBUG_PRINTF() helper in dict0stats.cc into an
unconditional no-op
Aleksey Midenkov
MDEV-20865 FRM image functions cleanups

- build_frm_image(): clearer names and named constants
- build_frm_image(): guard against overflow of the 2-byte key-info offset
- engine_table_options_frm_image(): mark in/out argument directions
sjaakola
MDEV-38869 sequence conflicts with streaming replication

Sequence access conflicts with streaming replication could cause
server hanging, as shown in MDEV-38869
This commit avoids such deadlocks, by skipping fragment replication
for sequence access.
The commit has also new mtr test for testing two sequence/SR conflict
scenarios: galera.galera_sequences_bf_kill_sr
Aleksey Midenkov
MDEV-20865 Prelocking_strategy refactoring

- Moved has_prelocking_list into Prelocking_strategy (removed it from
  Multiupdate_prelocking_strategy). It is now initialized in
  Prelocking_strategy::reset(), so reset() overrides
  (Multiupdate_prelocking_strategy, Multidelete_prelocking_strategy)
  chain to the base implementation to set it.
- Moved extend_table_list() into Prelocking_strategy;
- Made maybe_need_prelocking() virtual method.

Now maybe_need_prelocking() can be different per prelocking strategy.
Aleksey Midenkov
Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Aleksey Midenkov
MDEV-20865 extra2 structures moved to datadict.h

datadict.h is appropriate place for such types. These data types are
used by Extra2_info, which is added to datadict.h later.

unireg.cc is for older FRM routines. datadict.cc accepts new and
refactored routines.
Thirunarayanan Balathandayuthapani
MDEV-19574: innodb_stats_method is not honored when innodb_stats_persistent=ON

Problem:
=======
When persistent statistics are enabled (innodb_stats_persistent=ON),
the innodb_stats_method setting is not properly utilized during
statistics calculation.

The statistics collection functions always use a hardcoded default
behavior for NULL value comparison instead of respecting the
configured stats method. This affects the accuracy of
n_diff_key_vals (distinct key count), particularly for
indexes with nullable columns containing NULL values.

Moreover, stat_n_non_null_key_vals[] was never computed for
persistent statistics; it stayed at the 0 that
dict_stats_empty_index() assigns.

With innodb_stats_method=nulls_ignored, innodb_rec_per_key()
therefore always found n_diff <= n_null and reported one record
per key for every index. This impacts the query optimizer,
which makes decisions based on inaccurate cardinality estimates.

Solution:
========
Introduced IndexLevelStats to collect statistics at a specific
B-tree level during index analysis.

Introduced PageStats to collect statistics for leaf page analysis.

Refactored the following functions:
dict_stats_analyze_index_level() to IndexLevelStats::analyze_level()
dict_stats_analyze_index_for_n_prefix() to IndexLevelStats::sample_leaf_pages()
dict_stats_analyze_index_below_cur() to PageStats::scan_below()
dict_stats_scan_page() to PageStats::scan()

The innodb_stats_method value is read once per table in
dict_stats_update_persistent() and passed down, so that all
indexes of a table are analyzed with the same method.

Add the stats method name to stat_description when
innodb_stats_method has a non-default value. The suffix is
dropped when the description is already full.

Added the new stat name n_nonnull_fld01, n_nonnull_fld02, etc.
with a stats description, to indicate how many non-null values
exist for the nth field of the index. This value is retrieved
and stored in the index statistics
in dict_stats_fetch_index_stats_step(). The counts are per
column, not per n-column prefix.

rec_get_n_blob_pages(): Calculate the number of
externally stored pages for a record, using ceiling division
by the usable BLOB page payload (blob_part_size), which differs
between ROW_FORMAT=COMPRESSED (zip_size minus FIL_PAGE_DATA) and
the other formats (srv_page_size minus the BLOB header and the
page trailer). For ROW_FORMAT=COMPRESSED the length in
the field reference is the uncompressed length, so the result is an
upper bound.

When the leaf level is scanned in full, the number of leaf pages that
were scanned is reported as n_leaf_pages for a multi level index.
Before, result.n_leaf_pages was overwritten with
index->stat_n_leaf_pages, which dict_stats_empty_index() had
just set to 1, so every index that took the full scan path
reported n_leaf_pages=1. Single page indexes report 1.
This changes cardinality estimates and
therefore leads to multiple changes in existing test cases.

Non-null values are counted only at the leaf level, since only leaf
pages hold actual records. A full scan of the leaf level counts them
exactly. When the level is sampled, the per column count is derived
from the sampled leaves with the same formula as n_diff:

  n_ordinary_leaf_pages * n_non_null_all_analyzed_pages
                        / n_leaf_pages_to_analyze

This is an estimate for NOT NULL columns as well: the sampled leaves
may hold fewer or more records than the average, and a dive that
stops at a boring page contributes nothing to the sum while still
counting in the divisor.

innodb_rec_per_key(): stat_n_non_null_key_vals[i] holds the
number of records in which the i-th indexed column alone is
not NULL, while what has to be excluded here is the number
of records whose first i+1 columns are all not NULL,
because that is the population which the n-column
prefix statistic stat_n_diff_key_vals[i] has to be
corrected against when innodb_stats_method=nulls_ignored:
with NULLs compared as unequal, every record carrying a NULL
anywhere in the prefix adds a distinct value of its own to n_diff.

PageStats::scan(): n_non_null is accumulated and assigned only
for leaf pages, so that a non-leaf scan cannot leave a node
pointer count behind when scan_below() stops at a boring page
without reaching a leaf.

IndexLevelStats::reset_for_level() also clears n_diff[], and
dict_stats_analyze_index() zero initializes the buffer backing it, so
that a level scan which finds no records (a failed
btr_pcur_open_level(), or a non-leaf page whose first record is not
marked as the leftmost one on the level) leaves n_diff[] at 0 instead
of stale values.

IndexLevelStats::sample_leaf_pages() returns early when the group
boundaries for the prefix are empty, which is the same condition.

dict_stats_fetch_index_stats_step() no longer resets
stat_n_non_null_key_vals[] while processing an n_diff_pfxNN row:
dict_stats_empty_table() has already cleared the array before the
fetch, and with n_nonnull_fldNN rows now being read too,
that reset would make the result depend on the order in which the
rows arrive.

dict_stats_save(): now static function in dict0stats.cc that
takes the innodb_stats_method value, and is removed from dict0stats.h.
dict_stats_update_persistent() saves the statistics itself, so its
callers no longer have to.

Replaced btr_rec_get_externally_stored_len() with
rec_get_n_blob_pages() in dict0stats.cc.

btr_rec_get_field_ref_offs() and btr_rec_get_field_ref(),
together with the BTR_BLOB_HDR_* macros, were moved from btr0cur.cc
to btr0cur.h so that rec_get_n_blob_pages() can reuse them;

btr_rec_get_field_ref_offs() is now a noexcept function
returning size_t.

Changed stat_n_diff_key_vals and stat_n_non_null_key_vals from
ib_uint64_t* to uint64_t*

len_is_stored(): simplified to a single comparison, which is
equivalent for the unsigned lengths that it is used with.

Removed the unused UNIV_STATS_DEBUG build macro (univ.i) and
turned the DEBUG_PRINTF() helper in dict0stats.cc into an
unconditional no-op
Aleksey Midenkov
MDEV-20865 Prelocking_strategy refactoring

- Moved has_prelocking_list into Prelocking_strategy (removed it from
  Multiupdate_prelocking_strategy). It is now initialized in
  Prelocking_strategy::reset(), so reset() overrides
  (Multiupdate_prelocking_strategy, Multidelete_prelocking_strategy)
  chain to the base implementation to set it.
- Moved extend_table_list() into Prelocking_strategy;
- Made maybe_need_prelocking() virtual method.

Now maybe_need_prelocking() can be different per prelocking strategy.
Aleksey Midenkov
MDEV-20865 Fix tests and system scripts to foreign key requirements

Storing foreign key metadata in TABLE_SHARE makes opening a foreign table
preopen its referenced tables, so a referenced table must be created
before the foreign one and dropped after it. Rearrange the order of
CREATE/DROP commands in tests and system scripts accordingly.

Affected: the help tables in mariadb_system_tables{,_fix}.sql (with
system_mysql_db_fix* results), fetch_first, union, insert_notembedded,
instant_alter_index_rename and opt_context_store_ddls; some also index a
referenced column or add a missing referenced table, and
opt_context_store_ddls re-records the now-shown MyISAM foreign key.
Aleksey Midenkov
MDEV-20865 extra2_fields refactored to Extra2_info class

read_extra2() is now Extra2_info::read()
Additional assertions for checking size consistency.

Extra2_info::write() is used by further MDEV-20865 development.
Aleksey Midenkov
gcov fix

Fix coverage data corruption by enabling atomic profile updates

When compiling with coverage instrumentation, concurrent or forked test executions
can cause corruption of coverage data files (.gcda), resulting in negative hit counts
and invalid coverage reports.

Adding the compiler flag `-fprofile-update=atomic` ensures that updates to
coverage counters are performed atomically, preventing race conditions during
profile data writes.

This change eliminates warnings about unexpected negative hit counts and
improves the accuracy and reliability of coverage measurements in parallel
or multithreaded test environments.
Aleksey Midenkov
Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Alexander Barkov
MDEV-39518 Allow prepared statements in stored functions in assignment right hand

Allowing prepared statements in stored functions when
a stored function is used in an assignment right hand.

Both DEFAULT clause of a variable initialization and
the right side of the SET statement are supported:

  CREATE PROCEDURE p1()
  BEGIN
    -- case 1: DEFAULT clause
    DECLARE spvar1 INT DEFAULT f1_with_ps(); -- OK

    -- case 2: SP variable assignment statement
    DECLARE spvar2 INT;
    SET spvar= f1_with_ps(); -- OK
  END;

- The parser now does not reject PS statements in stored functions.
  PS applicability in stored functions is now detected at run time.
  Note, PS statements in triggers are still prohibited by the parser.

- Functions with PS do not acquire MDL locks on tables.
  They work like procedures in terms of table opening.

- Functions with PS are not replicated as a single `SELECT f1()` call.
  They are replicated per-statement, like procedures.

- Only bare function calls are supported for now. Using a function in
  an expression does not make it PS-safe yet yet:
    SET v= f1()+0;
Aleksey Midenkov
MDEV-20865 extra2_write_len() fix

Current implementation of extra2_write_len() does not guarantee of
writing correct 2-byte values as it skips writing zero at the
beginning.

Refined DBUG_ASSERT() to more truthful limit.
Marko Mäkelä
MDEV-40728 Recovery wrongly fails if FILE_CREATE is followed by FILE_RENAME

deferred_spaces.deferred_dblwr(): Skip newly created tablespaces
to avoid a bogus invocation of fil_space_free().

fil_name_process(): Simplify the logic. If no matching tablespace is
found but file_name_t::create_lsn had been set in response to parsing
a FILE_CREATE record, try to apply FILE_RENAME to deferred_spaces.

log_parse_file(): Parse each FILE_ record only once. In multi-batch
recovery, there may be redundant calls that would break the logic of
fil_name_process().
Vladislav Vaintroub
MDEV-22991 Support SSL over named pipes

Update libmariadb, to remove named pipe SSL check.
Update client.c to remove that check as well.

Add MTR test for named pipe + SSL:
- for Connector/C client, new test named_pipe_ssl
- for in-server clent, run mariabackup with user created as
  "IDENTIFIED WITH named_pipe REQUIRE SSL"
Vladislav Vaintroub
MDEV-22992 Refactor VIO into layered transports and filters

Replace the function-pointer VIO implementation with an abstract C++
interface while retaining the procedural C entry points.

Implement socket and named-pipe transports and composable filters for
client read-ahead, Windows thread-pool prefetch, and TLS. OpenSSL uses a
custom BIO, while wolfSSL uses callbacks that perform I/O through the VIO
below the TLS filter. This keeps waits and timeouts in the transport layer.

Keep sockets nonblocking and implement timed I/O with transport waits.
Named pipes use overlapped I/O for timeout-aware waits and report blocking
waits through the same scheduler callbacks as sockets. Semi-sync
temporarily changes the real VIO read timeout instead of copying VIO state.

Hide transport and TLS implementation state behind accessors. Expose the
TLS handle opaquely and update callers that previously accessed VIO fields
directly. Compile the VIO implementations as C++ and retain PSI memory
accounting for VIO allocations.

Adapt Windows thread-pool pre-read to a Prefetched_vio filter inserted
above the transport so both plain and TLS connections consume prefetched
bytes through the same layered VIO path.

Cleaned header files so that vio header no longer include OpenSSL or
WolfSSL headers.

Removed some legacy functionality

- vio_close() with its double-close guards appeared hard to maintain in
class hierarchy, and had been unnecessary for the last 15 years.
Associated things that are also gone :  preprocessor definition
SIGNAL_WITH_VIO_CLOSE (always defined), VIO_STATE_CLOSED.
VIO_CLOSED type, which was used as sentinel, renamed to VIO_TYPE_INVALID.
- vio_io_wait() used in a single place, replaces by read with timeout.
- vio_reset() to create SSL, replaced by vio_wrap
Aleksey Midenkov
MDEV-20865 extra2 structures moved to datadict.h

datadict.h is appropriate place for such types. These data types are
used by Extra2_info, which is added to datadict.h later.

unireg.cc is for older FRM routines. datadict.cc accepts new and
refactored routines.