Console View

Categories: connectors experimental galera main

Legend:

Passed

Failed

Warnings

Failed Again

Running

Exception

Offline

No data

connectors

experimental

galera

main

fb104af5d5c0...

fb104af5d5c0ccc77b88bb1a8c2097c1bf88e16d

Aleksey Midenkov

midenok@gmail.com

MDEV-20865 mbd:: STL exception-to-error wrappers

The foreign key work needs convenient sequence and associative containers
(vector, set, map) for utility, non-hot-path algorithms, but server code must
not let C++ exceptions propagate: a std::bad_alloc thrown out of an STL
container would unwind through code that is not exception-safe.

Add the mbd:: namespace wrapping std::vector/set/map. The container is inherited
protected, so throwing STL methods (operator[], at, resize, emplace_back,
map::emplace, hint/range inserts, ...) are not callable and cannot escape the
noexcept boundary -- using one is a compile error. Only the non-throwing members
actually needed are re-exposed (begin/end, size, empty, erase, clear, swap,
find, count, back, ...).

The mutating operations are wrapped as noexcept and catch internally:
std::bad_alloc becomes my_error(ER_OUT_OF_RESOURCES), anything else
ER_INTERNAL_ERROR. Failure is reported via a bool status, an end() iterator or
a NULL pointer instead of throwing:

- vector::push_back, reserve -> bool (true on failure)
- set::insert, set::emplace -> const Key* (NULL on failure)
- map::insert -> T* (NULL on failure)

Benefit: the FK code gets STL ergonomics without weakening the no-exceptions
contract; callers just check the return value. Performance-critical core
structures keep using the native List.

f9d9c3a23fc4...

f9d9c3a23fc42ec5b93c3df693c70288c1f43550

Vladislav Vaintroub

vvaintroub@gmail.com

Appveyor - post-fix 1a052f2

After 1a052f2 and 1fb0755, disabling RocksDB build via "git config" alone
no longer works. Add -DPLUGIN_ROCKSDB=NO to cmake config line to workaround

f45be5abcfd7...

f45be5abcfd7b805778a5a10501acf2ae5dcd286

Lawrin Novitsky

lawrin.novitsky@mariadb.com

CONCPP-159 Adding infer to the Github Actions workflow

Also, some minimal formatting/whitespaces changes made along the way

ccpp-sles12-amd64: build linux-connector_cpp failed - stdio warnings (2)

f3fac1855ad9...

f3fac1855ad92d78f4b34f40139a16f3c8897e08

Alexey Yurchenko

alexey.yurchenko@mariadb.com

MDEV-38920 MTR tests for Galera-side fixes

MDEV-38920-evs-config-warn checks that there is a warning about bad configuration
values and they are not accepted.
MDEV-38920-install-timer-expired reproduces 'install timer expired' situation.
Both tests require fixed Galera library to pass.

f2697f4e0647...

f2697f4e0647d9d412cb0c6ddf49daa42c11b2e5

Aleksey Midenkov

midenok@gmail.com

Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

d45c03c22177...

d45c03c2217740cbde2ca2c640de1176a857b4ba

Georgi (Joro) Kodinov

joro@mariadb.org

MDBF-1224: fix the structure of the plugin MD file

This requires moxygen 2.1.16! Added a check

Implemented a structure for the plugin API documentation.
On the top there's a main page to give top level overview and
concepts.
It eventually references the Moxygen documentation.
The moxygen documentation is a series of nested doxygen groups
that define the structure for the whole Plugin API documentation.

There are top levels like "server plugin API" and "client plugin API".
These branch into futher subgroups as needed.

c46764b9b7e5...

c46764b9b7e59e7e11bcf5b93310addfb794beee

Marko Mäkelä

marko.makela@mariadb.com

MDEV-40932 SET GLOBAL innodb_log_archive=OFF may still break recovery

log_t::set_archive(archive=false): Ensure that both the latest checkpoint
and the latest log record (which has possibly not been written out yet)
will carry the log_sys.get_sequence_bit(lsn)==1, to guarantee a successful
recovery after the switch to the innodb_log_archive=OFF format.

Tested by: Matthias Leich
Reviewed by: Thirunarayanan Balathandayuthapani

(cherry picked from commit 16373161744c2ebb62aa0d493b96191c62dd35ec)

bb5ac71a4a5e...

bb5ac71a4a5eadfd2458912902fd5380b724e7b2

Marko Mäkelä

marko.makela@mariadb.com

fixup! c909702dfcd2261a73cd0ca6a6ce5703ddda411a

aa6ea8d422a4...

aa6ea8d422a48ca84b8b1c03898482e1bf5f0d0a

Sergei Petrunia

sergey@mariadb.com

Continue pwt_manager_base cleanup: introduce locked__process_manager_wakeup().

Note this one:
// TODO: the following was done when not holding LOCK_data. Does it
// matter?

aa63464b9e2e...

aa63464b9e2e76f1a81720b1cde77d815f4d57c4

Georgi (Joro) Kodinov

joro@mariadb.org

a0e4f970cd80...

a0e4f970cd8063c07f158b08daac760cb119d5d0

Aleksey Midenkov

midenok@gmail.com

MDEV-40311 Fix mysqldump-nl leaving slave connection behind

The trailing CHANGE MASTER left master.info in the datadir, so a
later test that restarts the server auto-started the slave and broke
its check-testcase. Use RESET SLAVE ALL to drop the connection.

MTR's internal check of the test case 'sys_vars.default_master_connection_basic' failed.
This means that the test case does not preserve the state that existed
before the test case was executed. Most likely the test case did not
do a proper clean-up. It could also be caused by the previous test run
by this thread, if the server wasn't restarted.
This is the diff of the states of the servers before and after the
test case was executed:
-Slave_IO_Running No
-Slave_SQL_Running No
+Slave_IO_Running Connecting
+Slave_SQL_Running Yes
...
-Last_IO_Errno 0
-Last_IO_Error
+Last_IO_Errno 1045
+Last_IO_Error error connecting to master '[email protected]:3306' - retry-time: 60 maximum-retries: 100000 message: Access denied for user 'root'@'localhost' (using password: NO)

a03285c949c5...

a03285c949c5d01d55fd5d1232f9108a8db22a69

Aleksey Midenkov

midenok@gmail.com

MDEV-40799 Runtime plugin/UDF load errors lost under --silent-startup

Regression from MDEV-32745 (7828fb475b0), which guarded the
plugin-load my_error() calls with opt_silent_startup. That option is
a lifetime global, set once at startup and never reset, so the guard
suppressed the SQL error for the whole server lifetime, not just
during startup. Runtime operations (INSTALL PLUGIN, CREATE FUNCTION
... SONAME) then skipped my_error(), never set the diagnostics area
and wrongly succeeded - e.g. main.ps's "call proc_1()" no longer
failed with ER_CANT_OPEN_LIBRARY.

Startup callers pass MYF(ME_ERROR_LOG); runtime callers pass MYF(0).
Gate the silencing on that flag via silent_plugin_startup() so it
applies only to the startup error-log path, and runtime errors always
reach the client.

No new test case: the runtime failure path is already covered by
existing tests (e.g. main.ps's ER_CANT_OPEN_LIBRARY check). The
regression stayed invisible only because stock MTR does not start
servers with --silent-startup. A dedicated test would have to restart
the server with --silent-startup solely to assert that a startup-only
option does not affect runtime, which adds little over the restored
invariant.

943385e571cc...

943385e571cc22351e22a1a25851cd2fa7e6d90b

Sergei Petrunia

sergey@mariadb.com

Continue pwt_manager_base cleanup: introduce locked__process_manager_wakeup().

89885a13d095...

89885a13d095a1a0f1e51439aae0db0958a24a11

Yuchen Pei

ycp@mariadb.com

MDEV-40168 wip wip

80e4e866b93d...

80e4e866b93d569bbcd1c13b84841b375c3808b0

Vladislav Vaintroub

vvaintroub@gmail.com

Merge branch '11.4' into 12.3

79f96bf15d25...

79f96bf15d2538cb78201cc85681571f56548957

Vladislav Vaintroub

vvaintroub@gmail.com

Appveyor - post-fix 1a052f2

After 1a052f2 and 1fb0755, disabling RocksDB build via "git config" alone
no longer works. Add -DPLUGIN_ROCKSDB=NO to cmake config line to workaround

74b237158c7f...

74b237158c7fccb06004492651b833a7746bf261

Aleksey Midenkov

midenok@gmail.com

MDEV-34392 Check foreign key column nullability in the server layer

Making a foreign key column NOT NULL must be refused when a referential
action can still write NULL into it -- ON UPDATE/DELETE SET NULL, or ON
UPDATE CASCADE from a NULLable parent. Until now this was enforced through a
per-column nullability bitmap in FK_info (fields_nullable) that the storage
engine allocated and populated, and that the server read back while checking
an ALTER.

Since MDEV-20865 the server keeps the foreign key definitions in TABLE_SHARE,
so this round trip through the engine is redundant. Perform the check
directly in mysql_prepare_alter_table() from the stored referential actions
and the old/new column definitions, and drop the FK_info bitmap along with
its assign_nullable()/set_nullable()/is_nullable() helpers. Incompatible
changes are still rejected with ER_FK_COLUMN_NOT_NULL.

Fixes foreign_null test.

7464dc2e4120...

7464dc2e4120aa7e66dc0027c5d214ac4118f479

Georgi (Joro) Kodinov

joro@mariadb.org

6e17bdea0b2e...

6e17bdea0b2e771a994ac598beff0a0891b34450

Aleksey Midenkov

midenok@gmail.com

Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

6dae51171bfa...

6dae51171bfacc55288118159d27d09163599f99

bsrikanth-mariadb

srikanth.bondalapati@mariadb.com

MDEV-40553: unprintable gis ranges in trace and context

When ranges were specified in a query for GIS types, the recorded trace
and context couldn't print the range information. Instead, it only
showed unprintable_geometry_value.

This PR extends the geometric field type Field_geom to print key value
in binary form (same as what is done for Blobs today), along with the
comparison operator, when recorded in the range.
For spatial indexes, the operators like MBRWITHIN, MBRCONTAINS, etc...
are stored appropriately, and for normal indexes, operators like
<, <=, >, >=, etc... are recorded appropriately.

Implementation Details: -
Add an argument imagetype to Field::print_key_part_value(), to determine
if an index key part value is to be printed in WKT or binary format.
For Geometric type, imagetype is set to itMBR. When printing ranges,
the MBR operators are printed as well, as implemented in
print_mbr_range_operator().

6c245455fa78...

6c245455fa78bbde00a20b80b167c2b8507f2463

Georgi (Joro) Kodinov

joro@mariadb.org

days work: concluded the group structure for the API.

66d16edb97f3...

66d16edb97f30e20ae4b7086c73297cb97431d1d

Marko Mäkelä

marko.makela@mariadb.com

fixup! d56de2ab9a81ea95998f1b0dc1e62320f530c581

63f0a8ab6c9e...

63f0a8ab6c9e949aacf1df5d12c8b7fb18c6e519

Sergei Petrunia

sergey@mariadb.com

Cleanup in pwt_manager_base, pwt_worker_base.

4b049f122e59...

4b049f122e593603a284872f8f367415943ed47b

Aleksey Midenkov

midenok@gmail.com

MDEV-20865 Store foreign key info in TABLE_SHARE

1. Access foreign keys via TABLE_SHARE::foreign_keys and
TABLE_SHARE::referenced_keys;

foreign_keys and referenced_keys are lists in TABLE_SHARE.

2. Remove handler FK interface:

- get_foreign_key_list()
- get_parent_foreign_key_list()
- referenced_by_foreign_key()

3. Invalidate referenced shares on:

- RENAME TABLE
- DROP TABLE
- RENAME COLUMN
- ADD FOREIGN KEY

When foreign table is created or altered by the above operations
all referenced shares are closed. This blocks the operation while
any referenced shares are used (when at least one its TABLE
instance is locked).

4. Update referenced shares on:

- CREATE TABLE

On CREATE TABLE add items to referenced_keys of referenced
shares. States of referenced shares are restored in case of errors.

5. Invalidate foreign shares on:

- RENAME TABLE
- RENAME COLUMN

The above-mentioned blocking takes effect.

6. Check foreign/referenced shares consistency on:

- CHECK TABLE

7. Temporary change until MDEV-21051:

InnoDB fill foreign key info at handler open().

FOREIGN_KEY_INFO is refactored to FK_info holding Lex_cstring.

On first TABLE open FK_info is loaded from storage engine into
TABLE_SHARE. All referenced shares (if any exists) are closed. This
leads to blocking of first time foreign table open while referenced
tables are used.

MDEV-21311 Converge Foreign_key and supplemental generated Key together

mysql_prepare_create_table() does data validation and such utilities
as automatic name generation. But it does that only for indexes and
ignores Foreign_key objects. Now as Foreign_key data needs to be
stored in FRM files as well this processing must be done for it like
for any other Key objects.

Replace Key::FOREIGN_KEY type with Key::foreign flag of type
Key::MULTIPLE and Key::generated set to true. Construct one object
with Key::foreign == true instead of two objects of type
Key::FOREIGN_KEY and Key::MULTIPLE.

MDEV-21051 datadict refactorings

- Move read_extra2() to datadict.cc
- Refactored extra2_fields to Extra2_info
- build_frm_image() readability

MDEV-21051 build_table_shadow_filename() refactoring

mysql_prepare_alter_table() leaks fixes

MDEV-21051 amend system tables locking restriction

Table mysql.help_relation has foreign key to mysql.help_keyword. On
bootstrap when help_relation is opened, it preopens help_keyword for
READ and fails in lock_tables_check().

If system table is opened for write then fk references are opened for
write.

Related to: Bug#25422, WL#3984
Tests: main.lock

MDEV-21051 Store and read foreign key info into/from FRM files

1. Introduce Foreign_key_io class which creates/parses binary stream
containing foreign key structures. Referenced tables store there only
hints about foreign tables (their db and name), they restore full info
from the corresponding tables.

Foreign_key_io is stored under new EXTRA2_FOREIGN_KEY_INFO field in
extra2 section of FRM file.

2. Modify mysql_prepare_create_table() to generate names for foreign
keys. Until InnoDB storage of foreign keys is removed, FK names must
be unique across the database: the FK name must be based on table
name.

3. Keep stored data in sync on DDL changes. Referenced tables update
their foreign hints after following operations on foreign tables:

- RENAME TABLE
- DROP TABLE
- CREATE TABLE
- ADD FOREIGN KEY
- DROP FOREIGN KEY

Foreign tables update their foreign info after following operations on
referenced tables:

- RENAME TABLE
- RENAME COLUMN

4. To achieve 3. there must be ability to rewrite extra2 section of
FRM file without full reparse. FRM binary is built from primary
structures like HA_CREATE_INFO and cannot be built from TABLE_SHARE.

Use shadow write and rename like fast_alter_partition_table()
does. Shadow FRM is new FRM file that replaces the old one.

CREATE TABLE workflow:

1. Foreign_key is constructed in parser, placed into
alter_info->key_list;

2. mysql_prepare_create_table() translates them to FK_info, assigns
foreign_id if needed;

3. build_frm_image() writes two FK_info lists into FRM's extra2
section, for referenced keys it stores only table names (hints);

4. init_from_binary_frm_image() parses extra2 section and fills
foreign_keys and referenced_keys of TABLE_SHARE.

It restores referenced_keys by reading hint list of table names,
opening corresponding shares and restoring FK_info from their
foreign_keys. Hints resolution is done only when initializing
non-temporary shares. Usually temporary share has different
(temporary) name and it is impossible to resolve foreign keys by
that name (as we identify them by both foreign and referenced
table names). Another not unimportant reason is performance: this
saves spare share acquisitions.

ALTER TABLE workflow:

1. Foreign_key is constructed in parser, placed into
alter_info->key_list;

2. mysql_prepare_alter_table() prepares action lists and share list
of foreigns/references;

3. mysql_prepare_alter_table() locks list of foreigns/references by
MDL_INTENTION_EXCLUSIVE, acquires shares;

4. prepare_create_table() converts key_list into FK_list, assigns
foreign_id;

5. shadow FRM of altered table is created;

6. data is copied;

7. altered table is locked by MDL_EXCLUSIVE;

8. fk_handle_alter() processes action lists, creates FK backups,
modifies shares, writes shadow FRMs;

9. altered table is closed;

10. shadow FRMs are installed;

11. altered table is renamed, FRM backup deleted;

12. (TBD in MDEV-21053) shadow FRMs installation log closed, backups
deleted;

On FK backup system:

In case of failed DDL operation all shares that was modified must be
restored into original state. This is done by FK_ddl_backup (CREATE,
DROP), FK_rename_backup (RENAME), FK_alter_backup (ALTER).

On STL usage:

STL is used for utility not performance-critical algorithms, core
structures hold native List. A wrapper was made to convert STL
exception into bool error status or NULL value.

MDEV-20865 fk_check_consistency() in CHECK TABLE

Self-refs fix

Test table_flags fix: "debug" deviation is now gone.

FIXMEs: +16 -1

48cd3661ac2a...

48cd3661ac2af548e04fb11d55f6584e5b1cad1a

Thirunarayanan Balathandayuthapani

thiru@mariadb.com

MDEV-32067 InnoDB linear read-ahead had better be logical

The traditional linear read-ahead, enabled by innodb_read_ahead_threshold,
only helps when consecutively accessed pages have adjacent page numbers.
That is rarely true for B-tree leaf pages: after splits, merges and page
reuse, the logical order of leaves has nothing to do with their
physical page numbers. So for a scan of scattered leaves the
old read-ahead either did nothing or read the wrong pages.

This replaces linear read-ahead of index pages with a logical
read-ahead that prefetches the actual leaf pages a scan is
going to visit, discovered from the B-tree during traversal.
After this change, the exact nonzero value of innodb_read_ahead_threshold
matters only for the read-ahead of undo log pages.

records_in_range() reports the first and last leaf page of a range,
and the optimizer forwards that extent to the engine through
handler::advise_page_range() so read-ahead can be sized and
bounded to the scan.

buf_read_ahead_undo(): Renamed from buf_read_ahead_linear(). Now invoked
only for undo log pages, whose page numbers are sequential. No longer called
on BLOB pages or index pages.

buf_read_ahead_one(): Read ahead a single leaf page. Cannot be disabled.

buf_read_ahead_pages(): Read ahead a set of known B-tree leaf pages.

buf_read_ahead_random(): Split into a decision part (which extent to read,
in buf0buf.cc) and an action part (issue the reads).

The pending-read throttle now compares os_aio_pending_reads_approx() with
buf_pool.curr_size() (a page count) instead of curr_pool_size() (bytes),
which had made the throttle a no-op. buf_read_ahead_pages() releases the
tablespace reference on a failed/corrupted page.

btr_read_ahead_t: A context which records the level-1 page and the
child page number of the last node pointer collected,
so read-ahead can be resumed later.

btr_read_ahead_collect(): scan the node pointers of a PAGE_LEVEL=1
page from a given record in scan order, appending child page numbers.

btr_cur_t::search_leaf(), btr_cur_t::open_leaf(): At PAGE_LEVEL=1, after the
descent has located the child to follow, harvest leaf page numbers starting
at that child in the scan direction. These are exactly the
leaves the cursor will visit, so the prefetch is precise regardless of
physical page numbers.

btr_read_ahead_resume_rec(): Locate the record from which read-ahead
should resume, by re-finding the last harvested child by value. It walks only
genuine records, so a concurrent page reorganization cannot cause an invalid
read (a stored byte offset could).

btr_pcur_move_to_next_page(), btr_pcur_move_backward_from_page(): Invoke
buf_read_ahead_one() on the following/preceding sibling.

btr_copy_blob_prefix(): Simplified; no longer relies on FIL_PAGE_PREV/NEXT.

advise_page_range(): Records the scan's estimated leaf extent.
Used only to size and bound read-ahead; it never
gates the pages the scan actually reads.
mrr_readahead_from_scan_range() derives the read-ahead ceiling from
the advised extent, falling back to a LIMIT-based estimate.
The window ramps up from small number of pages, doubling on each
refill toward the ceiling, so single-row probes such as index_first() for
MIN()/MAX() prefetch only a few leaves while long scans reach full depth.

start_readahead(): For a just-positioned scan, prefetch the collected batch
and set up the rolling cursor.

readahead_refill(): As the scan advances (general_fetch()), resume the
level-1 harvest under an index S-latch, chain across level-1 siblings, and
prefetch the next batch bounded by the advised last leaf, or to the end of
the index when the extent is unknown.

Read-ahead is started from index_read() (range and full index scans),
rnd_init()/general_fetch() (full table scans), and, at one-page depth, from
row_merge_read_clustered_index() (OPTIMIZE/ALTER rebuild). general_fetch()
sets active_handler_stats before issuing read-ahead so prefetched pages are
attributed to the query.

A truncation race is closed with a new STOPPING_READS tablespace flag:
mtr_t::commit_shrink() sets it before shrinking and clears it after, and
buf_read_ahead_undo() re-checks space->is_stopping().

trx_undo_get_prev_rec(), trx_undo_get_prev_rec_from_prev_page():
take the trx_undo_t object instead of a long parameter list
and always latch shared.

handler::multi_range_read_info_const() and DsMrr_impl::dsmrr_info_const()
Passed a page_range parameter; the default MRR implementation aggregates
the per-range leaf extents.

opt_range.cc carries it through the chosen QUICK_RANGE_SELECT, and
QUICK_RANGE_SELECT::reset() calls advise_page_range() before
multi_range_read_init().

main.analyze_stmt_prefetch_count: corrected for double counting now that
pages_read_count includes waits on prefetched pages.

3f26417495b9...

3f26417495b95b56692da742ffe7e6dfee12c1db

Marko Mäkelä

marko.makela@mariadb.com

MDEV-14992/MDEV-38362/MDEV-39092 BACKUP SERVER

The following SQL statements will be introduced:

BACKUP SERVER TO '/path/to/directory' [ 1 CONCURRENT ];
BACKUP SERVER WITH [ 1 CONCURRENT ] 'command';

In place of the 1, any positive number of threads may be specified.
For the first variant, '/path/to' must exist and '/path/to/directory'
must be compatible with secure_file_priv and not exist; that is where
the backup will be written to.

For the second variant, 'command' must be the name of a script or
command that will be executed in a child process. The standard input
of that command will be in a format that is compatible with
GNU tar --format=oldgnu (and also BSD tar variants that are also part of
Microsoft Windows and Apple macOS). The command is expected to optionally
compress and encrypt the stream and redirect it to a file on a local or
a remote server. The BACKUP SERVER WITH will append an additional argument,
a positive base-ten number in ASCII, starting with 1, to identify the
current thread. In this way, each concurrent stream can write a separate
file.

TODO: implement the following:
In multi-threaded backup (2 CONCURRENT or more), one thread will be
responsible for copying the InnoDB write-ahead log from the beginning
of the backup. Other files will be copied by other threads.

Note: In BACKUP SERVER WITH [ 1 CONCURRENT ] ... the InnoDB
write-ahead log will be included near the end of the single backup
stream.

Note: In BACKUP SERVER TO ... [ 1 CONCURRENT ] but not in
multi-threaded backup, the parameter innodb_log_recovery_start in
backup.cnf is STRICTLY NECESSARY TO AVOID CORRUPTION! By default,
InnoDB crash recovery starts from the latest available log
checkpoint. However, for restoring a backup, recovery must start from
the checkpoint that was the latest when the backup was
started. Starting recovery from a possible later checkpoint will
result in a corrupted database!

The backup or the first stream will contain a file backup.cnf, which
includes parameters needed for restoring the backup. Currently,
these are innodb_log_recovery_start and innodb_log_recovery_target.
If innodb_log_recovery_target>0, InnoDB will be in read-only mode,
not allowing any writes to persistent files other than via the log
application.

To restore a streaming backup made with BACKUP SERVER WITH, an empty
directory needs to be created and all streams be extracted there using
the standard tar utility of the operating system, optionally after
undoing any encryption or compression that had been added by the
backup command. Then, the backup is prepared or MariaDB server started
up on the extracted directory, similar to as if the BACKUP SERVER TO
statement had been used.

The following will be implemented separately:

MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER
MDEV-40163 Partial backup and restore
MDEV-39091 Back up ENGINE=RocksDB
MDEV-40333 Concurrent DDL for Aria tables in BACKUP SERVER

The implementation introduces a basic multi-threaded driver
Sql_cmd_backup, storage engine interfaces, and basic copying of the
storage engines InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV.

backup_target: A structured data type to represent a target directory.
On Microsoft Windows, we must use directory paths because there is
no variant of CopyFileEx() that would work on file handles.

backup_sink: Wraps a per-thread output stream as well as storage engine
specific context.

handlerton::backup_start(), handlerton::backup_end(): Invoked at the
start or end of a backup phase, in the thread that executes a
BACKUP SERVER statement.

handlerton::backup_step(): A backup step that can be invoked from
multiple threads concurrently, between the execution of the corresponding
handlerton::backup_start() and handlerton::backup_end() of the same
phase.

copy_entire_file(): A file copying service for POSIX systems.

copy_mmap(): A zero-copy alternative to backup::copy(), to copy from a
memory-mapped buffer.

copy_file_range_try(): A wrapper for Linux copy_file_range(2), which
may fail with EOPNOTSUPP or EXDEV and thus require a fallback to
copy_mmap() or backup::copy().

backup::copy(): A partial or sparse file-copying service. On other
platforms than FreeBSD or Microsoft Windows, there are shortcut
alternatives to this. Note: On Linux we never invoke sendfile(2) for
copying between files, because can be much slower than the
alternatives.

backup_stream_append_plain(): A wrapper of backup::append(), which is
the streaming equivalent of backup::copy().

backup_stream_zeropad(): Zero-pad the last tar block if needed.

backup_stream_append_async(): A variant of backup_stream_append_plain()
where the source file region is guaranteed to be immutable after the
call returns. Zero-copy mmap(2) or Linux sendfile(2) are inherently
risky for copying data files that may be modified in place, because it
could introduce a race condition between a page write that runs
concurrently with a child process that is reading the data from the
pipe.

backup::append(): On systems where we can determine the size of the
pipe buffer, invoke backup_stream_append_async() for the initial
write, and pread_write() for the last part, to guarantee that the data
written by the zero-copy shortcut will have been consumed before the
call returns and the caller is able to resume writes to the source
region.

pread_write(): On 64-bit systems, allocate a buffer of up to 1 MiB.
This is the "slow path" of copying or streaming files.

struct Aria_backup: Context for multi-threaded backup,
comprising directory handles, a mutex and enum Aria_backup_status.

aria_backup_start(): Prepare the context for aria_backup_step().
Most files are copied in BACKUP_PHASE_NO_DDL after flush_tables(thd,
FLUSH_NON_TRANS_TABLES) has been invoked. All ENGINE=Aria files
(including TRANSACTIONAL=0) are copied in
BACKUP_PHASE_NO_COMMIT. Thanks to Andrzej Jarząbek for writing
test cases and suggesting this logic.

aria_backup_step(): Copy one non-ACID file. Acquires
Aria_backup::mutex, traverses directories to construct one file name,
releases the mutex, and copies the file if one was found.

aria_backup_data(): Copy one data file. On Microsoft Windows, this
assumes that the current directory is the datadir. This assumption
would not hold in the Embedded Server library, which is not supported
on Microsoft Windows.

aria_backup_log(): Copy one ENGINE=Aria log file.

aria_backup_end(): Finish a copying phase and clean up the context.

InnoDB_backup::init(): Wait for a possible previous BACKUP SERVER
operation to reach the very end of InnoDB_backup::context::cleanup()
so that the context can be safely reused.

InnoDB_backup::queue: Collection of tablespace IDs and payload sizes
at the start of the backup, and the log_sys.first_lsn of log files
that have to be included in the backup. If any data file is created or
extended while the backup is executing, we must have the corresponding
write-ahead-log entries that we are copying since the latest
checkpoint that was completed when the backup started. If any
tablespaces are deleted during the backup, we may or may not copy
them, and the application of a FILE_DELETE record will remove them.
Similarly, applying FILE_RENAME or FILE_CREATE records will rename or
create files during recovery as needed.

log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this
is to make BACKUP SERVER prevent the concurrent execution of
SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size
when innodb_log_archive=OFF.

log_sys.archived_checkpoint: Keep track of the earliest available
checkpoint, corresponding to log_sys.archived_lsn. This reflects
SET GLOBAL innodb_log_recovery_start (which is settable now), for
incremental backup.

fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed
to be known. To speed up startup, InnoDB does not normally open
all tablespace files.

fil_space_t::create_lsn: Change to Atomic_relaxed and use this to
indicate tablespace creation LSN, in addition to indicate undo
tablespace rebuild LSN.

fil_space_t::backup_end: The first page number that is not being backed up
(by default 0, to indicate that no backup is in progress).

fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be
covered by fil_space_t::backup_end. This is the unit of "page range locking"
during InnoDB backup.

buf_page_t::write_fix_try(), buf_page_t::write_unfix_try(): Try to set
or unset a fake "write fix" on a page, to prevent concurrent flush()
during a backup batch. The atomic operations may run concurrently with
set_reinit() and set_freed(). The fake "write fix" does not prevent
any concurrent read or write of the page data in the buffer pool; it
only blocks writes to the underlying data file.

buf_page_t::flush(): Atomically test and set write fix, and
skip the operation if the fake "write fix" was set.

buf_page_t::set_freed(), buf_page_t::set_reinit(): Employ a
compare-and-exchange loop to accommodate for the "write fix".

innodb_backup_batch_wait(): Look up any pages that we are about to
back up. For any dirty pages, invoke buf_page_t::write_fix_try() to
try to set a fake "write fix" lock-free. If the page is currently
write-fixed between buf_page_t::flush() and
buf_page_t::write_complete(), acquire and release a page U-latch to
wait for the conflicting write to complete.

InnoDB_backup::backup_batch_start(),
InnoDB_backup::backup_batch_stop(): Adjust fil_space_t::backup_end and
fake "write fix" of dirty pages to protect the copying of a range of
pages from the underlying file.

InnoDB_backup::commit(): Enqueue the remaining log to be copied.

InnoDB_backup::checkpoint_complete(): If backup is running and
commit() has not been called, add each completed innodb_archive_log=ON
file to the queue. Else, skip or delete, as appropriate.

log_t::backup_start(): If we were running with innodb_log_archive=ON,
ensure that the latest file is a valid recovery starting point.
That is, wait for the latest log checkpoint to be within the file.

buf_flush_list_space(): Check for concurrent backup before writing each
page. This is inefficient, but this function may be invoked from multiple
threads concurrently, and it cannot be changed easily, especially for
fil_crypt_thread().

fil_ibd_create(): Set fil_space_t::create_lsn after the file has been
created.

dict_load_tablespaces(): Determine the size of each file if
upgrade==true. Backup depends on that.

buf_dblwr_t::begin(), buf_dblwr_t::end(), buf_dblwr_t::size():
Accessors to allow BACKUP SERVER to skip the contents of the
doublewrite buffer in the system tablespace. It is only useful for
crash recovery in case a data page had been incompletely written by
the time the server was killed. If the server is killed during a
backup, the backup will be incomplete and unusable anyway.
Furthermore, the page range locking makes page writes and backup
mutually exclusive.

374bf88d5a30...

374bf88d5a30d31a9b231f4a40dc218ded158e12

Aleksey Midenkov

midenok@gmail.com

MDEV-20865 Reuse share in fk_handle_drop()

mysql_rm_table_no_locks() already opens the dropped table's share, so
pass it to fk_handle_drop() instead of re-acquiring it with
GTS_FK_SHALLOW_HINTS.

Skip fk_handle_drop() when the share is unreadable, letting the engine
report the error. This restores ER_GET_ERRNO for
main.partition_not_blackhole, which clear_error() used to mask by
zeroing my_errno.

New Share_acquire::inexistent_t lets DROP keep seeing ER_NO_SUCH_TABLE
independent of foreign_key_checks (INEXISTENT_ALWAYS).

Fixes main.partition_not_blackhole

FIXME: Squash into the main "MDEV-20865 Store foreign key info in
TABLE_SHARE" commit.

36885be7d4bb...

36885be7d4bb5d01a39a52ac864ac31d5dd7a000

sjaakola

seppo.jaakola@mariadb.com

MDEV-40903 galera.MDEV-38260 fails with ER_BINLOG_IN_USE on RESET MASTER

The test fails from 12.3 onwards on the final cleanup step:

mysqltest: At line 98: query 'reset master' failed: ER_BINLOG_IN_USE
(4243): Cannot execute RESET MASTER as the binlog is in use by a
connected slave or other RESET MASTER or binlog reader.

The binlog-in-engine work added a use-count guard around RESET MASTER
in MYSQL_BIN_LOG::reset_logs(): every binlog reader, a slave dump thread
included, registers itself through start_use_binlog(), and RESET MASTER
now refuses with ER_BINLOG_IN_USE while the count is non-zero. Earlier
versions reset the binlog even with a slave still attached, which is why
the test passes on 11.8 and older.

The test's cleanup stops the async slave on node_2 and then immediately
runs RESET MASTER on the master node_3. The dump thread on node_3 does
not disappear at STOP SLAVE; it lingers until it next notices the closed
connection, so it is often still registered when RESET MASTER arrives.

Fix here is to use include/reset_master.inc, which terminates dump
threads and retries on ER_BINLOG_IN_USE.

36325556d439...

36325556d4395f888c67dbe028232f8e3d9e673b

Aleksey Midenkov

midenok@gmail.com

Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

2bf3328bfd61...

2bf3328bfd61b5c0d451e6d50bbb51cea735b152

Aleksey Midenkov

midenok@gmail.com

Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

2bbf7cffc4dd...

2bbf7cffc4ddfea5beafc0d1fb63e84c8a80e7cc

Georgi (Joro) Kodinov

joro@mariadb.org

MDBF-1224: fix the structure of the plugin MD file

This requires moxygen 2.1.16!

Implemented a structure for the plugin API documentation.
On the top there's a main page to give top level overview and
concepts.
It eventually references the Moxygen documentation.
The moxygen documentation is a series of nested doxygen groups
that define the structure for the whole Plugin API documentation.

There are top levels like "server plugin API" and "client plugin API".
These branch into futher subgroups as needed.

24e61b2e83ca...

24e61b2e83ca81e00c4573789819bb6787627cad

Marko Mäkelä

marko.makela@mariadb.com

fixup! 66d16edb97f30e20ae4b7086c73297cb97431d1d

1e5f59af0d7e...

1e5f59af0d7ef91b9e2f33041b46eae6209b5592

Thirunarayanan Balathandayuthapani

thiru@mariadb.com

16373161744c...

16373161744c2ebb62aa0d493b96191c62dd35ec

Marko Mäkelä

marko.makela@mariadb.com

12bcf3781f11...

12bcf3781f112fc21c8d653860bcc1f4f1b1f23e

Sergei Petrunia

sergey@mariadb.com

Cleanup in pwt_manager_base, pwt_worker_base.

0f6d688257d3...

0f6d688257d376a9ae6c1e4dbb12bd82ca540b7c

Aleksey Midenkov

midenok@gmail.com

Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

0f11c9d1f5d0...

0f11c9d1f5d0a7e4a51428b9c6ca46d437d9b6c5

Aleksey Midenkov

midenok@gmail.com

MDEV-20865 Refactor Share_acquire::fk_error() into acquire()

Share_acquire::fk_error() inspected thd->is_error() after the acquisition to
decide whether a missing referenced table is tolerable. That breaks when an
outer Internal_error_handler consumes the error first, e.g. the
Postponed_error_handler installed by mysql_rename_tables() (MDEV-27027).

Move the decision into Share_acquire::acquire(): push a
No_such_table_error_handler for the acquisition when foreign key checks are
not enforced, so ER_NO_SUCH_TABLE is trapped above any outer handler rather
than deferred or leaked into SHOW WARNINGS. The non-tolerated error outcome
is recorded in Share_acquire::error; the four consumers check it instead of
calling fk_error().

FIXME: Squash into the main "MDEV-20865 Store foreign key info in
TABLE_SHARE" commit.

0b2d2b017a0d...

0b2d2b017a0d55d56054f93111534838f045f903

Rex Johnston

rex.johnston@mariadb.com

PQ: allow workers to pre-aggregate SUM, COUNT, MIN, MAX for the manager