Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Aleksey Midenkov
MDEV-20865 mbd:: STL exception-to-error wrappers

The foreign key work needs convenient sequence and associative containers
(vector, set, map) for utility, non-hot-path algorithms, but server code must
not let C++ exceptions propagate: a std::bad_alloc thrown out of an STL
container would unwind through code that is not exception-safe.

Add the mbd:: namespace wrapping std::vector/set/map. The container is inherited
protected, so throwing STL methods (operator[], at, resize, emplace_back,
map::emplace, hint/range inserts, ...) are not callable and cannot escape the
noexcept boundary -- using one is a compile error. Only the non-throwing members
actually needed are re-exposed (begin/end, size, empty, erase, clear, swap,
find, count, back, ...).

The mutating operations are wrapped as noexcept and catch internally:
std::bad_alloc becomes my_error(ER_OUT_OF_RESOURCES), anything else
ER_INTERNAL_ERROR. Failure is reported via a bool status, an end() iterator or
a NULL pointer instead of throwing:

  - vector::push_back, reserve  -> bool (true on failure)
  - set::insert, set::emplace    -> const Key* (NULL on failure)
  - map::insert                  -> T* (NULL on failure)

Benefit: the FK code gets STL ergonomics without weakening the no-exceptions
contract; callers just check the return value. Performance-critical core
structures keep using the native List.
Vladislav Vaintroub
Appveyor - post-fix 1a052f2

After 1a052f2 and 1fb0755, disabling RocksDB build via "git config" alone
no longer works. Add -DPLUGIN_ROCKSDB=NO to cmake config line to workaround
Lawrin Novitsky
CONCPP-159 Adding infer to the Github Actions workflow

Also, some minimal formatting/whitespaces changes made along the way
Alexey Yurchenko
MDEV-38920 MTR tests for Galera-side fixes

MDEV-38920-evs-config-warn checks that there is a warning about bad configuration
values and they are not accepted.
MDEV-38920-install-timer-expired reproduces 'install timer expired' situation.
Both tests require fixed Galera library to pass.
Aleksey Midenkov
Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Georgi (Joro) Kodinov
MDBF-1224: fix the structure of the plugin MD file

This requires moxygen 2.1.16! Added a check

Implemented a structure for the plugin API documentation.
On the top there's a main page to give top level overview and
concepts.
It eventually references the Moxygen documentation.
The moxygen documentation is a series of nested doxygen groups
that define the structure for the whole Plugin API documentation.

There are top levels like "server plugin API" and "client plugin API".
These branch into futher subgroups as needed.
Marko Mäkelä
MDEV-40932 SET GLOBAL innodb_log_archive=OFF may still break recovery

log_t::set_archive(archive=false): Ensure that both the latest checkpoint
and the latest log record (which has possibly not been written out yet)
will carry the log_sys.get_sequence_bit(lsn)==1, to guarantee a successful
recovery after the switch to the innodb_log_archive=OFF format.

Tested by: Matthias Leich
Reviewed by: Thirunarayanan Balathandayuthapani

(cherry picked from commit 16373161744c2ebb62aa0d493b96191c62dd35ec)
Marko Mäkelä
fixup! c909702dfcd2261a73cd0ca6a6ce5703ddda411a
Sergei Petrunia
Continue pwt_manager_base cleanup: introduce locked__process_manager_wakeup().

Note this one:
  //  TODO: the following was done when not holding LOCK_data. Does it
  // matter?
Georgi (Joro) Kodinov
MDBF-1224: fix the structure of the plugin MD file

This requires moxygen 2.1.16! Added a check

Implemented a structure for the plugin API documentation.
On the top there's a main page to give top level overview and
concepts.
It eventually references the Moxygen documentation.
The moxygen documentation is a series of nested doxygen groups
that define the structure for the whole Plugin API documentation.

There are top levels like "server plugin API" and "client plugin API".
These branch into futher subgroups as needed.
Aleksey Midenkov
MDEV-40311 Fix mysqldump-nl leaving slave connection behind

The trailing CHANGE MASTER left master.info in the datadir, so a
later test that restarts the server auto-started the slave and broke
its check-testcase. Use RESET SLAVE ALL to drop the connection.

MTR's internal check of the test case 'sys_vars.default_master_connection_basic' failed.
This means that the test case does not preserve the state that existed
before the test case was executed.  Most likely the test case did not
do a proper clean-up. It could also be caused by the previous test run
by this thread, if the server wasn't restarted.
This is the diff of the states of the servers before and after the
test case was executed:
-Slave_IO_Running No
-Slave_SQL_Running No
+Slave_IO_Running Connecting
+Slave_SQL_Running Yes
...
-Last_IO_Errno 0
-Last_IO_Error
+Last_IO_Errno 1045
+Last_IO_Error error connecting to master '[email protected]:3306' - retry-time: 60  maximum-retries: 100000  message: Access denied for user 'root'@'localhost' (using password: NO)
Aleksey Midenkov
MDEV-40799 Runtime plugin/UDF load errors lost under --silent-startup

Regression from MDEV-32745 (7828fb475b0), which guarded the
plugin-load my_error() calls with opt_silent_startup.  That option is
a lifetime global, set once at startup and never reset, so the guard
suppressed the SQL error for the whole server lifetime, not just
during startup.  Runtime operations (INSTALL PLUGIN, CREATE FUNCTION
... SONAME) then skipped my_error(), never set the diagnostics area
and wrongly succeeded - e.g. main.ps's "call proc_1()" no longer
failed with ER_CANT_OPEN_LIBRARY.

Startup callers pass MYF(ME_ERROR_LOG); runtime callers pass MYF(0).
Gate the silencing on that flag via silent_plugin_startup() so it
applies only to the startup error-log path, and runtime errors always
reach the client.

No new test case: the runtime failure path is already covered by
existing tests (e.g. main.ps's ER_CANT_OPEN_LIBRARY check).  The
regression stayed invisible only because stock MTR does not start
servers with --silent-startup.  A dedicated test would have to restart
the server with --silent-startup solely to assert that a startup-only
option does not affect runtime, which adds little over the restored
invariant.
Sergei Petrunia
Continue pwt_manager_base cleanup: introduce locked__process_manager_wakeup().
Yuchen Pei
MDEV-40168 wip wip
Vladislav Vaintroub
Merge branch '11.4' into 12.3
Vladislav Vaintroub
Appveyor - post-fix 1a052f2

After 1a052f2 and 1fb0755, disabling RocksDB build via "git config" alone
no longer works. Add -DPLUGIN_ROCKSDB=NO to cmake config line to workaround
Aleksey Midenkov
MDEV-34392 Check foreign key column nullability in the server layer

Making a foreign key column NOT NULL must be refused when a referential
action can still write NULL into it -- ON UPDATE/DELETE SET NULL, or ON
UPDATE CASCADE from a NULLable parent. Until now this was enforced through a
per-column nullability bitmap in FK_info (fields_nullable) that the storage
engine allocated and populated, and that the server read back while checking
an ALTER.

Since MDEV-20865 the server keeps the foreign key definitions in TABLE_SHARE,
so this round trip through the engine is redundant. Perform the check
directly in mysql_prepare_alter_table() from the stored referential actions
and the old/new column definitions, and drop the FK_info bitmap along with
its assign_nullable()/set_nullable()/is_nullable() helpers. Incompatible
changes are still rejected with ER_FK_COLUMN_NOT_NULL.

Fixes foreign_null test.
Georgi (Joro) Kodinov
MDBF-1224: fix the structure of the plugin MD file

This requires moxygen 2.1.16! Added a check

Implemented a structure for the plugin API documentation.
On the top there's a main page to give top level overview and
concepts.
It eventually references the Moxygen documentation.
The moxygen documentation is a series of nested doxygen groups
that define the structure for the whole Plugin API documentation.

There are top levels like "server plugin API" and "client plugin API".
These branch into futher subgroups as needed.
Addressed githib copilot review comments.
Aleksey Midenkov
Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
bsrikanth-mariadb
MDEV-40553: unprintable gis ranges in trace and context

When ranges were specified in a query for GIS types, the recorded trace
and context couldn't print the range information. Instead, it only
showed unprintable_geometry_value.

This PR extends the geometric field type Field_geom to print key value
in binary form (same as what is done for Blobs today), along with the
comparison operator, when recorded in the range.
For spatial indexes, the operators like MBRWITHIN, MBRCONTAINS, etc...
are stored appropriately, and for normal indexes, operators like
<, <=, >, >=, etc... are recorded appropriately.

Implementation Details: -
  Add an argument imagetype to Field::print_key_part_value(), to determine
  if an index key part value is to be printed in WKT or binary format.
  For Geometric type, imagetype is set to itMBR. When printing ranges,
  the MBR operators are printed as well, as implemented in
  print_mbr_range_operator().
Georgi (Joro) Kodinov
days work: concluded the group structure for the API.
Marko Mäkelä
fixup! d56de2ab9a81ea95998f1b0dc1e62320f530c581
Sergei Petrunia
Cleanup in pwt_manager_base, pwt_worker_base.
Aleksey Midenkov
MDEV-20865 Store foreign key info in TABLE_SHARE

1. Access foreign keys via TABLE_SHARE::foreign_keys and
  TABLE_SHARE::referenced_keys;

  foreign_keys and referenced_keys are lists in TABLE_SHARE.

2. Remove handler FK interface:

  - get_foreign_key_list()
  - get_parent_foreign_key_list()
  - referenced_by_foreign_key()

3. Invalidate referenced shares on:

  - RENAME TABLE
  - DROP TABLE
  - RENAME COLUMN
  - ADD FOREIGN KEY

  When foreign table is created or altered by the above operations
  all referenced shares are closed. This blocks the operation while
  any referenced shares are used (when at least one its TABLE
  instance is locked).

4. Update referenced shares on:

  - CREATE TABLE

  On CREATE TABLE add items to referenced_keys of referenced
  shares. States of referenced shares are restored in case of errors.

5. Invalidate foreign shares on:

  - RENAME TABLE
  - RENAME COLUMN

  The above-mentioned blocking takes effect.

6. Check foreign/referenced shares consistency on:

  - CHECK TABLE

7. Temporary change until MDEV-21051:

  InnoDB fill foreign key info at handler open().

FOREIGN_KEY_INFO is refactored to FK_info holding Lex_cstring.

On first TABLE open FK_info is loaded from storage engine into
TABLE_SHARE. All referenced shares (if any exists) are closed. This
leads to blocking of first time foreign table open while referenced
tables are used.

MDEV-21311 Converge Foreign_key and supplemental generated Key together

mysql_prepare_create_table() does data validation and such utilities
as automatic name generation. But it does that only for indexes and
ignores Foreign_key objects. Now as Foreign_key data needs to be
stored in FRM files as well this processing must be done for it like
for any other Key objects.

Replace Key::FOREIGN_KEY type with Key::foreign flag of type
Key::MULTIPLE and Key::generated set to true. Construct one object
with Key::foreign == true instead of two objects of type
Key::FOREIGN_KEY and Key::MULTIPLE.

MDEV-21051 datadict refactorings

- Move read_extra2() to datadict.cc
- Refactored extra2_fields to Extra2_info
- build_frm_image() readability

MDEV-21051 build_table_shadow_filename() refactoring

mysql_prepare_alter_table() leaks fixes

MDEV-21051 amend system tables locking restriction

Table mysql.help_relation has foreign key to mysql.help_keyword. On
bootstrap when help_relation is opened, it preopens help_keyword for
READ and fails in lock_tables_check().

If system table is opened for write then fk references are opened for
write.

Related to: Bug#25422, WL#3984
Tests: main.lock

MDEV-21051 Store and read foreign key info into/from FRM files

1. Introduce Foreign_key_io class which creates/parses binary stream
containing foreign key structures. Referenced tables store there only
hints about foreign tables (their db and name), they restore full info
from the corresponding tables.

Foreign_key_io is stored under new EXTRA2_FOREIGN_KEY_INFO field in
extra2 section of FRM file.

2. Modify mysql_prepare_create_table() to generate names for foreign
keys. Until InnoDB storage of foreign keys is removed, FK names must
be unique across the database: the FK name must be based on table
name.

3. Keep stored data in sync on DDL changes. Referenced tables update
their foreign hints after following operations on foreign tables:

  - RENAME TABLE
  - DROP TABLE
  - CREATE TABLE
  - ADD FOREIGN KEY
  - DROP FOREIGN KEY

Foreign tables update their foreign info after following operations on
referenced tables:

  - RENAME TABLE
  - RENAME COLUMN

4. To achieve 3. there must be ability to rewrite extra2 section of
FRM file without full reparse. FRM binary is built from primary
structures like HA_CREATE_INFO and cannot be built from TABLE_SHARE.

Use shadow write and rename like fast_alter_partition_table()
does. Shadow FRM is new FRM file that replaces the old one.

CREATE TABLE workflow:

  1. Foreign_key is constructed in parser, placed into
    alter_info->key_list;

  2. mysql_prepare_create_table() translates them to FK_info, assigns
    foreign_id if needed;

  3. build_frm_image() writes two FK_info lists into FRM's extra2
    section, for referenced keys it stores only table names (hints);

  4. init_from_binary_frm_image() parses extra2 section and fills
    foreign_keys and referenced_keys of TABLE_SHARE.

    It restores referenced_keys by reading hint list of table names,
    opening corresponding shares and restoring FK_info from their
    foreign_keys. Hints resolution is done only when initializing
    non-temporary shares. Usually temporary share has different
    (temporary) name and it is impossible to resolve foreign keys by
    that name (as we identify them by both foreign and referenced
    table names). Another not unimportant reason is performance: this
    saves spare share acquisitions.

ALTER TABLE workflow:

  1. Foreign_key is constructed in parser, placed into
    alter_info->key_list;

  2. mysql_prepare_alter_table() prepares action lists and share list
    of foreigns/references;

  3. mysql_prepare_alter_table() locks list of foreigns/references by
    MDL_INTENTION_EXCLUSIVE, acquires shares;

  4. prepare_create_table() converts key_list into FK_list, assigns
    foreign_id;

  5. shadow FRM of altered table is created;

  6. data is copied;

  7. altered table is locked by MDL_EXCLUSIVE;

  8. fk_handle_alter() processes action lists, creates FK backups,
    modifies shares, writes shadow FRMs;

  9. altered table is closed;

  10. shadow FRMs are installed;

  11. altered table is renamed, FRM backup deleted;

  12. (TBD in MDEV-21053) shadow FRMs installation log closed, backups
      deleted;

On FK backup system:

In case of failed DDL operation all shares that was modified must be
restored into original state. This is done by FK_ddl_backup (CREATE,
DROP), FK_rename_backup (RENAME), FK_alter_backup (ALTER).

On STL usage:

STL is used for utility not performance-critical algorithms, core
structures hold native List. A wrapper was made to convert STL
exception into bool error status or NULL value.

MDEV-20865 fk_check_consistency() in CHECK TABLE

Self-refs fix

Test table_flags fix: "debug" deviation is now gone.

FIXMEs: +16 -1
Thirunarayanan Balathandayuthapani
MDEV-32067 InnoDB linear read-ahead had better be logical

The traditional linear read-ahead, enabled by innodb_read_ahead_threshold,
only helps when consecutively accessed pages have adjacent page numbers.
That is rarely true for B-tree leaf pages: after splits, merges and page
reuse, the logical order of leaves has nothing to do with their
physical page numbers. So for a scan of scattered leaves the
old read-ahead either did nothing or read the wrong pages.

This replaces linear read-ahead of index pages with a logical
read-ahead that prefetches the actual leaf pages a scan is
going to visit, discovered from the B-tree during traversal.
After this change, the exact nonzero value of innodb_read_ahead_threshold
matters only for the read-ahead of undo log pages.

records_in_range() reports the first and last leaf page of a range,
and the optimizer forwards that extent to the engine through
handler::advise_page_range() so read-ahead can be sized and
bounded to the scan.

buf_read_ahead_undo(): Renamed from buf_read_ahead_linear(). Now invoked
only for undo log pages, whose page numbers are sequential. No longer called
on BLOB pages or index pages.

buf_read_ahead_one(): Read ahead a single leaf page. Cannot be disabled.

buf_read_ahead_pages(): Read ahead a set of known B-tree leaf pages.

buf_read_ahead_random(): Split into a decision part (which extent to read,
in buf0buf.cc) and an action part (issue the reads).

The pending-read throttle now compares os_aio_pending_reads_approx() with
buf_pool.curr_size() (a page count) instead of curr_pool_size() (bytes),
which had made the throttle a no-op. buf_read_ahead_pages() releases the
tablespace reference on a failed/corrupted page.

btr_read_ahead_t: A context which records the level-1 page and the
child page number of the last node pointer collected,
so read-ahead can be resumed later.

btr_read_ahead_collect(): scan the node pointers of a PAGE_LEVEL=1
page from a given record in scan order, appending child page numbers.

btr_cur_t::search_leaf(), btr_cur_t::open_leaf(): At PAGE_LEVEL=1, after the
descent has located the child to follow, harvest leaf page numbers starting
at that child in the scan direction. These are exactly the
leaves the cursor will visit, so the prefetch is precise regardless of
physical page numbers.

btr_read_ahead_resume_rec(): Locate the record from which read-ahead
should resume, by re-finding the last harvested child by value. It walks only
genuine records, so a concurrent page reorganization cannot cause an invalid
read (a stored byte offset could).

btr_pcur_move_to_next_page(), btr_pcur_move_backward_from_page(): Invoke
buf_read_ahead_one() on the following/preceding sibling.

btr_copy_blob_prefix(): Simplified; no longer relies on FIL_PAGE_PREV/NEXT.

advise_page_range(): Records the scan's estimated leaf extent.
Used only to size and bound read-ahead; it never
gates the pages the scan actually reads.
mrr_readahead_from_scan_range() derives the read-ahead ceiling from
the advised extent, falling back to a LIMIT-based estimate.
The window ramps up from small number of pages, doubling on each
refill toward the ceiling, so single-row probes such as index_first() for
MIN()/MAX() prefetch only a few leaves while long scans reach full depth.

start_readahead(): For a just-positioned scan, prefetch the collected batch
and set up the rolling cursor.

readahead_refill(): As the scan advances (general_fetch()), resume the
level-1 harvest under an index S-latch, chain across level-1 siblings, and
prefetch the next batch bounded by the advised last leaf, or to the end of
the index when the extent is unknown.

Read-ahead is started from index_read() (range and full index scans),
rnd_init()/general_fetch() (full table scans), and, at one-page depth, from
row_merge_read_clustered_index() (OPTIMIZE/ALTER rebuild). general_fetch()
sets active_handler_stats before issuing read-ahead so prefetched pages are
attributed to the query.

A truncation race is closed with a new STOPPING_READS tablespace flag:
mtr_t::commit_shrink() sets it before shrinking and clears it after, and
buf_read_ahead_undo() re-checks space->is_stopping().

trx_undo_get_prev_rec(), trx_undo_get_prev_rec_from_prev_page():
take the trx_undo_t object instead of a long parameter list
and always latch shared.

handler::multi_range_read_info_const() and DsMrr_impl::dsmrr_info_const()
Passed a page_range parameter; the default MRR implementation aggregates
the per-range leaf extents.

opt_range.cc carries it through the chosen QUICK_RANGE_SELECT, and
QUICK_RANGE_SELECT::reset() calls advise_page_range() before
multi_range_read_init().

main.analyze_stmt_prefetch_count: corrected for double counting now that
pages_read_count includes waits on prefetched pages.
Marko Mäkelä
MDEV-14992/MDEV-38362/MDEV-39092 BACKUP SERVER

The following SQL statements will be introduced:

BACKUP SERVER TO '/path/to/directory' [ 1 CONCURRENT ];
BACKUP SERVER WITH [ 1 CONCURRENT ] 'command';

In place of the 1, any positive number of threads may be specified.
For the first variant, '/path/to' must exist and '/path/to/directory'
must be compatible with secure_file_priv and not exist; that is where
the backup will be written to.

For the second variant, 'command' must be the name of a script or
command that will be executed in a child process. The standard input
of that command will be in a format that is compatible with
GNU tar --format=oldgnu (and also BSD tar variants that are also part of
Microsoft Windows and Apple macOS). The command is expected to optionally
compress and encrypt the stream and redirect it to a file on a local or
a remote server. The BACKUP SERVER WITH will append an additional argument,
a positive base-ten number in ASCII, starting with 1, to identify the
current thread. In this way, each concurrent stream can write a separate
file.

TODO: implement the following:
In multi-threaded backup (2 CONCURRENT or more), one thread will be
responsible for copying the InnoDB write-ahead log from the beginning
of the backup. Other files will be copied by other threads.

Note: In BACKUP SERVER WITH [ 1 CONCURRENT ] ... the InnoDB
write-ahead log will be included near the end of the single backup
stream.

Note: In BACKUP SERVER TO ... [ 1 CONCURRENT ] but not in
multi-threaded backup, the parameter innodb_log_recovery_start in
backup.cnf is STRICTLY NECESSARY TO AVOID CORRUPTION! By default,
InnoDB crash recovery starts from the latest available log
checkpoint. However, for restoring a backup, recovery must start from
the checkpoint that was the latest when the backup was
started. Starting recovery from a possible later checkpoint will
result in a corrupted database!

The backup or the first stream will contain a file backup.cnf, which
includes parameters needed for restoring the backup. Currently,
these are innodb_log_recovery_start and innodb_log_recovery_target.
If innodb_log_recovery_target>0, InnoDB will be in read-only mode,
not allowing any writes to persistent files other than via the log
application.

To restore a streaming backup made with BACKUP SERVER WITH, an empty
directory needs to be created and all streams be extracted there using
the standard tar utility of the operating system, optionally after
undoing any encryption or compression that had been added by the
backup command. Then, the backup is prepared or MariaDB server started
up on the extracted directory, similar to as if the BACKUP SERVER TO
statement had been used.

The following will be implemented separately:

MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER
MDEV-40163 Partial backup and restore
MDEV-39091 Back up ENGINE=RocksDB
MDEV-40333 Concurrent DDL for Aria tables in BACKUP SERVER

The implementation introduces a basic multi-threaded driver
Sql_cmd_backup, storage engine interfaces, and basic copying of the
storage engines InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV.

backup_target: A structured data type to represent a target directory.
On Microsoft Windows, we must use directory paths because there is
no variant of CopyFileEx() that would work on file handles.

backup_sink: Wraps a per-thread output stream as well as storage engine
specific context.

handlerton::backup_start(), handlerton::backup_end(): Invoked at the
start or end of a backup phase, in the thread that executes a
BACKUP SERVER statement.

handlerton::backup_step(): A backup step that can be invoked from
multiple threads concurrently, between the execution of the corresponding
handlerton::backup_start() and handlerton::backup_end() of the same
phase.

copy_entire_file(): A file copying service for POSIX systems.

copy_mmap(): A zero-copy alternative to backup::copy(), to copy from a
memory-mapped buffer.

copy_file_range_try(): A wrapper for Linux copy_file_range(2), which
may fail with EOPNOTSUPP or EXDEV and thus require a fallback to
copy_mmap() or backup::copy().

backup::copy(): A partial or sparse file-copying service.  On other
platforms than FreeBSD or Microsoft Windows, there are shortcut
alternatives to this. Note: On Linux we never invoke sendfile(2) for
copying between files, because can be much slower than the
alternatives.

backup_stream_append_plain(): A wrapper of backup::append(), which is
the streaming equivalent of backup::copy().

backup_stream_zeropad(): Zero-pad the last tar block if needed.

backup_stream_append_async(): A variant of backup_stream_append_plain()
where the source file region is guaranteed to be immutable after the
call returns. Zero-copy mmap(2) or Linux sendfile(2) are inherently
risky for copying data files that may be modified in place, because it
could introduce a race condition between a page write that runs
concurrently with a child process that is reading the data from the
pipe.

backup::append(): On systems where we can determine the size of the
pipe buffer, invoke backup_stream_append_async() for the initial
write, and pread_write() for the last part, to guarantee that the data
written by the zero-copy shortcut will have been consumed before the
call returns and the caller is able to resume writes to the source
region.

pread_write(): On 64-bit systems, allocate a buffer of up to 1 MiB.
This is the "slow path" of copying or streaming files.

struct Aria_backup: Context for multi-threaded backup,
comprising directory handles, a mutex and enum Aria_backup_status.

aria_backup_start(): Prepare the context for aria_backup_step().
Most files are copied in BACKUP_PHASE_NO_DDL after flush_tables(thd,
FLUSH_NON_TRANS_TABLES) has been invoked. All ENGINE=Aria files
(including TRANSACTIONAL=0) are copied in
BACKUP_PHASE_NO_COMMIT. Thanks to Andrzej JarzÄ…bek for writing
test cases and suggesting this logic.

aria_backup_step(): Copy one non-ACID file. Acquires
Aria_backup::mutex, traverses directories to construct one file name,
releases the mutex, and copies the file if one was found.

aria_backup_data(): Copy one data file. On Microsoft Windows, this
assumes that the current directory is the datadir. This assumption
would not hold in the Embedded Server library, which is not supported
on Microsoft Windows.

aria_backup_log(): Copy one ENGINE=Aria log file.

aria_backup_end(): Finish a copying phase and clean up the context.

InnoDB_backup::init(): Wait for a possible previous BACKUP SERVER
operation to reach the very end of InnoDB_backup::context::cleanup()
so that the context can be safely reused.

InnoDB_backup::queue: Collection of tablespace IDs and payload sizes
at the start of the backup, and the log_sys.first_lsn of log files
that have to be included in the backup. If any data file is created or
extended while the backup is executing, we must have the corresponding
write-ahead-log entries that we are copying since the latest
checkpoint that was completed when the backup started. If any
tablespaces are deleted during the backup, we may or may not copy
them, and the application of a FILE_DELETE record will remove them.
Similarly, applying FILE_RENAME or FILE_CREATE records will rename or
create files during recovery as needed.

log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this
is to make BACKUP SERVER prevent the concurrent execution of
SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size
when innodb_log_archive=OFF.

log_sys.archived_checkpoint: Keep track of the earliest available
checkpoint, corresponding to log_sys.archived_lsn. This reflects
SET GLOBAL innodb_log_recovery_start (which is settable now), for
incremental backup.

fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed
to be known. To speed up startup, InnoDB does not normally open
all tablespace files.

fil_space_t::create_lsn: Change to Atomic_relaxed and use this to
indicate tablespace creation LSN, in addition to indicate undo
tablespace rebuild LSN.

fil_space_t::backup_end: The first page number that is not being backed up
(by default 0, to indicate that no backup is in progress).

fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be
covered by fil_space_t::backup_end. This is the unit of "page range locking"
during InnoDB backup.

buf_page_t::write_fix_try(), buf_page_t::write_unfix_try(): Try to set
or unset a fake "write fix" on a page, to prevent concurrent flush()
during a backup batch. The atomic operations may run concurrently with
set_reinit() and set_freed(). The fake "write fix" does not prevent
any concurrent read or write of the page data in the buffer pool; it
only blocks writes to the underlying data file.

buf_page_t::flush(): Atomically test and set write fix, and
skip the operation if the fake "write fix" was set.

buf_page_t::set_freed(), buf_page_t::set_reinit(): Employ a
compare-and-exchange loop to accommodate for the "write fix".

innodb_backup_batch_wait(): Look up any pages that we are about to
back up. For any dirty pages, invoke buf_page_t::write_fix_try() to
try to set a fake "write fix" lock-free. If the page is currently
write-fixed between buf_page_t::flush() and
buf_page_t::write_complete(), acquire and release a page U-latch to
wait for the conflicting write to complete.

InnoDB_backup::backup_batch_start(),
InnoDB_backup::backup_batch_stop(): Adjust fil_space_t::backup_end and
fake "write fix" of dirty pages to protect the copying of a range of
pages from the underlying file.

InnoDB_backup::commit(): Enqueue the remaining log to be copied.

InnoDB_backup::checkpoint_complete(): If backup is running and
commit() has not been called, add each completed innodb_archive_log=ON
file to the queue. Else, skip or delete, as appropriate.

log_t::backup_start(): If we were running with innodb_log_archive=ON,
ensure that the latest file is a valid recovery starting point.
That is, wait for the latest log checkpoint to be within the file.

buf_flush_list_space(): Check for concurrent backup before writing each
page. This is inefficient, but this function may be invoked from multiple
threads concurrently, and it cannot be changed easily, especially for
fil_crypt_thread().

fil_ibd_create(): Set fil_space_t::create_lsn after the file has been
created.

dict_load_tablespaces(): Determine the size of each file if
upgrade==true. Backup depends on that.

buf_dblwr_t::begin(), buf_dblwr_t::end(), buf_dblwr_t::size():
Accessors to allow BACKUP SERVER to skip the contents of the
doublewrite buffer in the system tablespace. It is only useful for
crash recovery in case a data page had been incompletely written by
the time the server was killed. If the server is killed during a
backup, the backup will be incomplete and unusable anyway.
Furthermore, the page range locking makes page writes and backup
mutually exclusive.
Aleksey Midenkov
MDEV-20865 Reuse share in fk_handle_drop()

mysql_rm_table_no_locks() already opens the dropped table's share, so
pass it to fk_handle_drop() instead of re-acquiring it with
GTS_FK_SHALLOW_HINTS.

Skip fk_handle_drop() when the share is unreadable, letting the engine
report the error. This restores ER_GET_ERRNO for
main.partition_not_blackhole, which clear_error() used to mask by
zeroing my_errno.

New Share_acquire::inexistent_t lets DROP keep seeing ER_NO_SUCH_TABLE
independent of foreign_key_checks (INEXISTENT_ALWAYS).

Fixes main.partition_not_blackhole

FIXME: Squash into the main "MDEV-20865 Store foreign key info in
TABLE_SHARE" commit.
sjaakola
MDEV-40903 galera.MDEV-38260 fails with ER_BINLOG_IN_USE on RESET MASTER

The test fails from 12.3 onwards on the final cleanup step:

mysqltest: At line 98: query 'reset master' failed: ER_BINLOG_IN_USE
(4243): Cannot execute RESET MASTER as the binlog is in use by a
connected slave or other RESET MASTER or binlog reader.

The binlog-in-engine work added a use-count guard around RESET MASTER
in MYSQL_BIN_LOG::reset_logs(): every binlog reader, a slave dump thread
included, registers itself through start_use_binlog(), and RESET MASTER
now refuses with ER_BINLOG_IN_USE while the count is non-zero. Earlier
versions reset the binlog even with a slave still attached, which is why
the test passes on 11.8 and older.

The test's cleanup stops the async slave on node_2 and then immediately
runs RESET MASTER on the master node_3. The dump thread on node_3 does
not disappear at STOP SLAVE; it lingers until it next notices the closed
connection, so it is often still registered when RESET MASTER arrives.

Fix here is to use include/reset_master.inc, which terminates dump
threads and retries on ER_BINLOG_IN_USE.
Aleksey Midenkov
Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Aleksey Midenkov
Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Georgi (Joro) Kodinov
MDBF-1224: fix the structure of the plugin MD file

This requires moxygen 2.1.16!

Implemented a structure for the plugin API documentation.
On the top there's a main page to give top level overview and
concepts.
It eventually references the Moxygen documentation.
The moxygen documentation is a series of nested doxygen groups
that define the structure for the whole Plugin API documentation.

There are top levels like "server plugin API" and "client plugin API".
These branch into futher subgroups as needed.
Marko Mäkelä
fixup! 66d16edb97f30e20ae4b7086c73297cb97431d1d
Thirunarayanan Balathandayuthapani
MDEV-32067 InnoDB linear read-ahead had better be logical

The traditional linear read-ahead, enabled by innodb_read_ahead_threshold,
only helps when consecutively accessed pages have adjacent page numbers.
That is rarely true for B-tree leaf pages: after splits, merges and page
reuse, the logical order of leaves has nothing to do with their
physical page numbers. So for a scan of scattered leaves the
old read-ahead either did nothing or read the wrong pages.

This replaces linear read-ahead of index pages with a logical
read-ahead that prefetches the actual leaf pages a scan is
going to visit, discovered from the B-tree during traversal.
After this change, the exact nonzero value of innodb_read_ahead_threshold
matters only for the read-ahead of undo log pages.

records_in_range() reports the first and last leaf page of a range,
and the optimizer forwards that extent to the engine through
handler::advise_page_range() so read-ahead can be sized and
bounded to the scan.

buf_read_ahead_undo(): Renamed from buf_read_ahead_linear(). Now invoked
only for undo log pages, whose page numbers are sequential. No longer called
on BLOB pages or index pages.

buf_read_ahead_one(): Read ahead a single leaf page. Cannot be disabled.

buf_read_ahead_pages(): Read ahead a set of known B-tree leaf pages.

buf_read_ahead_random(): Split into a decision part (which extent to read,
in buf0buf.cc) and an action part (issue the reads).

The pending-read throttle now compares os_aio_pending_reads_approx() with
buf_pool.curr_size() (a page count) instead of curr_pool_size() (bytes),
which had made the throttle a no-op. buf_read_ahead_pages() releases the
tablespace reference on a failed/corrupted page.

btr_read_ahead_t: A context which records the level-1 page and the
child page number of the last node pointer collected,
so read-ahead can be resumed later.

btr_read_ahead_collect(): scan the node pointers of a PAGE_LEVEL=1
page from a given record in scan order, appending child page numbers.

btr_cur_t::search_leaf(), btr_cur_t::open_leaf(): At PAGE_LEVEL=1, after the
descent has located the child to follow, harvest leaf page numbers starting
at that child in the scan direction. These are exactly the
leaves the cursor will visit, so the prefetch is precise regardless of
physical page numbers.

btr_read_ahead_resume_rec(): Locate the record from which read-ahead
should resume, by re-finding the last harvested child by value. It walks only
genuine records, so a concurrent page reorganization cannot cause an invalid
read (a stored byte offset could).

btr_pcur_move_to_next_page(), btr_pcur_move_backward_from_page(): Invoke
buf_read_ahead_one() on the following/preceding sibling.

btr_copy_blob_prefix(): Simplified; no longer relies on FIL_PAGE_PREV/NEXT.

advise_page_range(): Records the scan's estimated leaf extent.
Used only to size and bound read-ahead; it never
gates the pages the scan actually reads.
mrr_readahead_from_scan_range() derives the read-ahead ceiling from
the advised extent, falling back to a LIMIT-based estimate.
The window ramps up from small number of pages, doubling on each
refill toward the ceiling, so single-row probes such as index_first() for
MIN()/MAX() prefetch only a few leaves while long scans reach full depth.

start_readahead(): For a just-positioned scan, prefetch the collected batch
and set up the rolling cursor.

readahead_refill(): As the scan advances (general_fetch()), resume the
level-1 harvest under an index S-latch, chain across level-1 siblings, and
prefetch the next batch bounded by the advised last leaf, or to the end of
the index when the extent is unknown.

Read-ahead is started from index_read() (range and full index scans),
rnd_init()/general_fetch() (full table scans), and, at one-page depth, from
row_merge_read_clustered_index() (OPTIMIZE/ALTER rebuild). general_fetch()
sets active_handler_stats before issuing read-ahead so prefetched pages are
attributed to the query.

A truncation race is closed with a new STOPPING_READS tablespace flag:
mtr_t::commit_shrink() sets it before shrinking and clears it after, and
buf_read_ahead_undo() re-checks space->is_stopping().

trx_undo_get_prev_rec(), trx_undo_get_prev_rec_from_prev_page():
take the trx_undo_t object instead of a long parameter list
and always latch shared.

handler::multi_range_read_info_const() and DsMrr_impl::dsmrr_info_const()
Passed a page_range parameter; the default MRR implementation aggregates
the per-range leaf extents.

opt_range.cc carries it through the chosen QUICK_RANGE_SELECT, and
QUICK_RANGE_SELECT::reset() calls advise_page_range() before
multi_range_read_init().

main.analyze_stmt_prefetch_count: corrected for double counting now that
pages_read_count includes waits on prefetched pages.
Marko Mäkelä
MDEV-40932 SET GLOBAL innodb_log_archive=OFF may still break recovery

log_t::set_archive(archive=false): Ensure that both the latest checkpoint
and the latest log record (which has possibly not been written out yet)
will carry the log_sys.get_sequence_bit(lsn)==1, to guarantee a successful
recovery after the switch to the innodb_log_archive=OFF format.

Tested by: Matthias Leich
Reviewed by: Thirunarayanan Balathandayuthapani
Sergei Petrunia
Cleanup in pwt_manager_base, pwt_worker_base.
Aleksey Midenkov
Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Aleksey Midenkov
MDEV-20865 Refactor Share_acquire::fk_error() into acquire()

Share_acquire::fk_error() inspected thd->is_error() after the acquisition to
decide whether a missing referenced table is tolerable.  That breaks when an
outer Internal_error_handler consumes the error first, e.g. the
Postponed_error_handler installed by mysql_rename_tables() (MDEV-27027).

Move the decision into Share_acquire::acquire(): push a
No_such_table_error_handler for the acquisition when foreign key checks are
not enforced, so ER_NO_SUCH_TABLE is trapped above any outer handler rather
than deferred or leaked into SHOW WARNINGS.  The non-tolerated error outcome
is recorded in Share_acquire::error; the four consumers check it instead of
calling fk_error().

FIXME: Squash into the main "MDEV-20865 Store foreign key info in
TABLE_SHARE" commit.
Rex Johnston
PQ:  allow workers to pre-aggregate SUM, COUNT, MIN, MAX for the manager