Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Alessandro Vetere
MDEV-32286 Reuse remembered clustered leaves in secondary-index scans

Row_sel_get_clust_rec_for_mysql::operator() descends the clustered B-tree
from the root for every row whose clustered-index record a secondary-index
scan must read, although consecutive rows land on the same clustered leaf
page wherever the secondary order tracks the clustered one. A non-covering
scan needs one for every row, a locking read needs one whatever the
secondary index holds, because an exclusive select lock type makes
ha_innobase::build_template() build its template against the clustered
index, and a covering scan needs one for every row of a secondary leaf
whose PAGE_MAX_TRX_ID its read view cannot see. ANALYZE FORMAT=JSON
charges each descent its full height, and those descents are nearly the
whole cost: the secondary index is charged its own descent and one page
for each further leaf, and nothing per row, because the position that its
cursor holds between two rows is restored optimistically, which latches
the leaf again without counting an access. So pages_accessed is the row
count times the height of the clustered index, plus a handful: 1000 rows
over a 2-level clustered index cost 2006 and 750 rows over a 3-level one
cost 2291, where a full table scan of the same data costs 23 and 110.

Let a handle remember the clustered leaves that the lookups of one
statement reached, and let the next lookup try them before it descends
again. The reasoning behind each value and each rejection is in the
comments beside it.

row0mysql.h defines clust_leaf_hint_slot, which names one leaf: its page
number, copies of its first and last user record truncated to the key
fields, which bound the key range that the leaf held when it was
remembered, the rec_get_offsets() of both, and the
dict_index_t::n_core_fields that the copies were made under. A slot of a
leaf that had no right sibling names no last key, because every key above
the last record of the rightmost leaf still belongs to it.
CLUST_LEAF_HINT_SLOTS (4) slots hang off the new
row_prebuilt_t::clust_leaf_hint, beside clust_leaf_hint_mru, the most
recently used order held as slot numbers, and clust_leaf_hint_n and
clust_leaf_hint_miss, the used-slot count and the miss counter.

row0sel.cc holds the policy. row_sel_clust_leaf_hint_covers() compares a
key against the remembered ranges, so a lookup that no slot can answer
costs no buffer pool access and no pages_accessed.
row_sel_clust_leaf_hint_search() probes the first slot that covers the key
and moves it to the front of the order.
row_sel_clust_leaf_hint_remember() records the leaf that a descent landed
on, and refreshes the slot of a leaf that is remembered already rather
than spend a second one on the same page. Two descents fill no slot: the
first CLUST_LEAF_HINT_MIN_LOOKUPS (4) lookups of a statement, and a leaf
that is the root. row_sel_clust_leaf_hint_armed() stands a scan down once
the slots stop paying for themselves: a miss adds
CLUST_LEAF_HINT_MISS_WEIGHT (2) to the miss counter and a hit takes one
away, so a scan gives the slots up where it answers too little of its
lookups to pay for them, CLUST_LEAF_HINT_MAX_MISSES (8) misses with no hit
between them still reach the threshold, and one lookup in
CLUST_LEAF_HINT_RETRY (1024) starts the count again, so a scan whose order
becomes correlated only later recovers. Both halves of the cost stop
there, the test of the slots and the copies that refresh them.
Row_sel_get_clust_rec_for_mysql::operator() calls all of this in place of
its btr_pcur_open_with_no_init(), and only where the adaptive hash index
is disabled, whose guess solves the same problem better: it lands on the
record with no page-local search and no page access to charge. That index
is off by default, so the hints are active in a default configuration.

btr0cur.h and btr0cur.cc add btr_cur_t::try_leaf_hint(), a PAGE_CUR_LE,
BTR_SEARCH_LEAF search on one named leaf. It acquires the page with
buf_page_try_get(): a hint is never derived from a latched parent page, so
by the time it is tried it can precede the caller's already-latched
secondary-index leaf in the latching order, where a blocking wait can
deadlock. It then rejects the page unless the checks that it makes on the
latched frame put the match on it. Those checks are the sole authority on
the result, so a stale range costs a wasted probe or a needless descent,
never a wrong result, and the ranges need no invalidation protocol.

ha_innodb.cc: ha_innobase::reset() zeroes the used-slot count and the miss
counter per statement, matching autoinc_last_value. row0mysql.cc:
row_prebuilt_free() frees the key buffers that the slots own.

innodb.clust_leaf_hint measures pages_accessed over key orders that differ
in how closely the secondary order tracks the clustered one, and eight
further tables check query results over the record formats and key shapes
that a clustered-index lookup has to read, down to the metadata
pseudo-record of instant ALTER TABLE, to leaves that split and merge while
a locking read walks them, and to a record that a remembered leaf supplies
for a scan that must then rebuild an older version of it. Two of the
tables scan a covering index, which reads a clustered record under an
exclusive select lock type and under a PAGE_MAX_TRX_ID that the read view
cannot see. clust_leaf_hint_off_debug runs the same body with the hints
turned off, through a debug switch that returns before a lookup tests or
refreshes the slots, so a diff of the two .result files is what the hints
save: 2006 to 1031 (2-level clustered index), 2291 to 1011 (3-level), 4006
to 2015 (two interleaved key ranges), 12016 to 9078 (locality in the
second half alone) and 20020 to 10045 for a covering scan that FOR UPDATE
makes non-covering, where the same scan without FOR UPDATE costs 20 in
both files. Two orders with too little locality to pay for the slots give
them up early and end within a hundred accesses of the unhinted count:
20020 to 19966 (decorrelated) and 20020 to 19999 (shuffled).

innodb.clust_leaf_hint_instant_alter covers the one rejection that no
count reaches, of a slot whose keys were copied under another
dict_index_t::n_core_fields than the index reports.
dict_index_t::clear_instant_alter() is the only writer of that value that
a shared metadata lock allows, and it needs the clustered index to lose
the last user record of its root page, while no leaf that is the root
fills a slot, so the tree has to shrink between the two, which purge does
there. The reader therefore reads uncommitted rows at READ UNCOMMITTED and
waits in a stored function while a rollback and purge take them away, and
one row that arrives above the position it stopped at is the lookup that
tests the slots. The rejection leaves nothing that a query can read, so
that branch writes the two counts to the error log under a debug switch
and the case reads them back with search_pattern_in_file.inc. They are
printed and not named in the pattern, so that a run which reaches the
branch with other counts, or in the direction where the clear lowers them,
is a difference to look at and not a pass.

main.rowid_filter_innodb: 90 to 88, and its ahi combination unchanged.
Oleksandr Byelkin
MDEV-41094 KDF() aliases large iteration/width to weak 32-bit values

Better check parameters of KDF (i.e. iteration count should be
positive 32 bit, key length 16 bit unsigned).
Dave Gosselin
MDEV-33616:  Match the macOS dlopen error in plugins.multiauth

The client reports why it could not load client_ed25519, and macOS names
every path that dlopen() tried.  Two expressions are added, one for the
chunk that holds the start of that message and one for the chunk that
holds the rest of it.

The line runs to 563 bytes, 52 of prefix and the 511 that the client
error buffer holds, while do_exec() reads the output with fgets() into a
512 byte buffer and runs the replacements on each chunk on its own.  A
long enough vardir therefore splits the line, because the path appears
four times in the dlopen text.  The second chunk is the tail of a path
and carries no colon, where the first chunk keeps the colons of the
mysqltest prefix.  That chunk also holds the only line terminator the
error line gets, so the expression captures the newline and the
replacement puts it back.  A replacement is inserted as written, so a \n
spelled there would reach the output as a backslash and an n.

Both expressions stop at a newline.  reg_replace compiles with
REG_DOTALL, so an unrestricted .* runs past the line terminator whenever
the whole message reaches the replacement in one chunk, and the error
line then joins the line after it.
Andrzej Jarzabek
MDEV-37521 Attempt to compare iterators from different sequences in range_set::add_range

fil_space_t::freed_ranges is a range_set guarded by
freed_range_mutex, and every mutator takes that lock except one:
the branch of mtr_t::commit() that frees pages for an unlogged
mini-transaction on the temporary tablespace (the common case,
since temp-space pages never set m_modifications). That path
called fil_space_t::add_free_range() directly, racing with any
other locked mutator of the same std::set - most notably
fil_space_t::flush_freed(), which the page cleaner invokes for
every tablespace while resizing the buffer pool. The race
corrupts the range_set's underlying tree and crashes the server.

Take freed_range_mutex around the loop, matching every other
caller (process_freed_pages(), flush_freed(),
clear_freed_ranges()).

Added innodb.temp_truncate_freed_race, which reproduces the
crash with a configurable probability by racing concurrent
temporary-table churn against a continuously resizing buffer
pool across several rounds against the same server.
Rex Johnston
MDEV-35168 subselects with outer references to derived tables may be incorrectly evaluated as constant

Subselects with outer references to derived tables may be incorrectly
evaluated as having no table references.  This can lead to these
subselects being marked as constant, leading to an incorrect
result.

During the calculation of the tables used in a subselect, we construct a
table map of outer references in our (not necessarily new) "new_parent"
select.  This is currently done purely by finding Item_fields in our tree
and using the attached table to update our bitmap.  It can be that a
reference to a derived table also needs to have it's table added to this
map.  If the derived table can be null, this is the case.

We add a new processor to our item walk system,
enumerate_table_refs_processor which is defined at this stage only
for Item_direct_view_ref items.
This called alongside enumerate_field_refs_processor in
Item_subselect::recalc_used_tables().

Coverage of the merged_into() function is provided at the end of the
tests in subselect4.test.  Be aware that they are not a regression test,
we could not find anything that provided an incorrect output.
Yuchen Pei
Do the same thing as the parent commit to quick_mode_{1,3}
sjaakola
design document update + updates for eligibility checking for
cascaded tables
Thirunarayanan Balathandayuthapani
MDEV-41207 Acquire metadata locks for recovered transaction

Problem:
=======
A transaction being rolled back during recovery holds LOCK_IX on the
table(not the metadata locks), and the rollback thread holds
a reference on it.

An online ALTER TABLE on that table falls back to acquiring LOCK_S,
which conflicts and fails. prepare_inplace_alter_table_dict()
asserted that the reference count is 1 before checking whether
the table lock was acquired, so the reference still held by the
rollback thread makes the assertion fail.

Solution:
========
trx_recovery_thd: A background connection that owns the metadata
locks of the recovered transactions. One connection is shared by
all of recovering transaction.

trx_lists_init_at_db_start(): Create trx_recovery_thd, before any
metadata lock can be acquired.

trx_resurrect_table_locks(): Acquire a shared metadata lock on each
table that the recovered transaction had modified, and remember the
locks in trx_recovery_mdl. Recovered XA PREPARED transactions are
excluded, because they are completed by a user connection.

trx_recovery_mdl: The metadata locks of the recovered transactions,
by transaction. They are kept outside trx_t. Only a recovered
transaction ever has an entry.

trx_recovery_mdl_exists: Whether trx_recovery_mdl is not empty.
It is read for every transaction in trx_t::free(),
so that the map will only be consulted while some recovered
transaction still holds metadata locks. It becomes false as
soon as the rollback of the recovered transactions has been
completed, long before trx_recovery_thd is
destroyed.

trx_t::free(): Release the metadata locks of a recovered transaction,
once its rollback has been completed. This is the only place that
releases them, so that a transaction which was rolled back by
trx_rollback_recovered(false) will not keep its locks until shutdown.

trx_recovery_thd_destroy(): Destroy trx_recovery_thd.
It is invoked by innodb_shutdown() only, after trx_sys.close()
has freed any recovered transaction that was left.

row_undo_mod(): Add the debug injection rollback_wait
Yuchen Pei
MDEV-39525 [wip][to-squash] fix vcol_type_both_bounded

Reverted changes to Field_longstr::is_supertype

Also improve test coverage of the domain check

TODO: check results
Dmitry Shulga
MDEV-30645: CREATE TRIGGER FOR { STARTUP | SHUTDOWN }

Follow-up patch to fix missing call to my_error() in case not all
mandatory columns present in the table mysql.event
Mohammad Tafzeel Shams
MDEV-41242 : Fix resource leaks on InnoDB/mariabackup error paths found by Infer

Several error-handling paths returned without releasing a resource
already acquired earlier in the function, or checked the wrong handle
entirely, risking use of an unopened handle.

Changes:
- SysTablespace::read_lsn_and_check_flags(): close the datafile handle
  on header-validation failure.
- xb_process_datadir(): check the freshly opened `dir` handle instead
  of the stale `dbdir`, fixing a handle leak and a possible use of an
  unopened directory handle.
- wsrep.cc / xb_load_list_file(): close file handles before die(), and
  null-check fopen() results in wsrep.cc.
- datadir_iter_new(): free datadir_path and destroy the mutex on the
  os_file_opendir() failure path.
Dave Gosselin
MDEV-33616:  Normalize the strerror text in innodb_fts.index_table

The injected deadlock reaches the client as ER_GET_ERRNO carrying errno
11, and the text comes from my_strerror().  11 is EAGAIN on Linux and
EDEADLK on macOS, so the message reads "Resource temporarily
unavailable" on one and "Resource deadlock avoided" on the other.
Replace the quoted text so the test does not depend on it.
Dave Gosselin
MDEV-33616:  Skip the redo log upgrade tests without sparse file support

innodb.log_upgrade and innodb.log_upgrade_101_flags build 8GB redo log
files by seeking past the end of an empty file and writing a single
byte.  That needs a filesystem which leaves the skipped range
unallocated.  HFS on macOS allocates every block of it instead, so the
write fails with ENOSPC and the test reports a perl failure.

include/have_sparse_files.inc probes a directory the caller names,
writing one byte 64MB into an empty file there and comparing the
allocated block count against that offset.
bsrikanth-mariadb
MDEV-40837: Sequences used only in a column DEFAULT missing from optimizer context

Problem:
========
A sequence referenced only in a column's DEFAULT expression (e.g.
"a INT DEFAULT NEXTVAL(s1)") is opened only when a statement actually
evaluates DEFAULT values (INSERT, LOAD DATA, etc). A plain SELECT on
the table never opens it, so the sequence never appears in
thd->lex->query_tables and dump_sql_script() had no way to see it.
The dependent table's definition then got captured without the
sequence it needs, making the captured context unusable

Solution:
=========
TABLE::internal_tables already holds the sequence tables
DEFAULT expressions depend on, populated whenever the table is
opened regardless of statement type. For each table bein
walk this list, open any sequence not already open, and dump its
CREATE SEQUENCE and current value (via SETVAL) before th
own CREATE TABLE statement, so replay can recreate both in the
correct order.
Vladislav Vaintroub
MDEV-40967 PROXY protocol host check sent in clear text mid-SSL handshake

Defer the host-privileged/host-blocked check for a PROXY-header-derived
address until after the client's SSL handshake completes, instead of
sending it immediately in clear text.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Dave Gosselin
MDEV-33616:  Make two tests independent of lower_case_table_names

macOS puts the data directory on a case insensitive file system, so
lower_case_table_names is 2 and both tests recorded an answer that only
holds for 0.

period.i_s_notembedded looked up I_S.PERIODS and I_S.KEY_PERIOD_USAGE by
the schema name TEST.  That comparison follows the table name
comparison, so it finds the table under 1 and 2 and finds nothing under
0.  Those four queries move to the new test period.i_s_case_sensitive,
which requires lower_case_table_names=0.  The win rdiff of
period.i_s_notembedded covered the same difference and is no longer
needed.

atomic.drop_db_long_names generated table and view names in upper case
and compared the DROP statements that DDL recovery writes to the binary
log.  Under 2 the names come back from the directory in lower case.
Generating them in lower case to begin with gives the same names on
every setting.  Lower case also changes where the view name sorts
against its table name for the letters after v, which moves one view
between two of the recorded DROP VIEW statements.
Dave Gosselin
MDEV-33616:  Exclude innodb_log_file_mmap from sys_vars.sysvars_innodb

Its default value depends on the operating system, ON where the log can
be memory mapped and OFF elsewhere, so the recorded row only holds on
some platforms.  The other variables whose default depends on the
operating system are already excluded the same way.
Dave Gosselin
MDEV-33616:  Allocate the recovery buffer from the heap

recv_sys.tmp_buf comes from malloc() rather than from the large page
allocator.

main.large_pages fails on macOS with "Warning: Memory not freed: 16375"
at shutdown.  recv_sys_t::find_checkpoint() asks for 1048585 bytes,
my_large_malloc() rounds that up to 1064960 and charges the rounded
figure to the server memory accounting, and recv_sys_t::tmp_free()
credits back the 1048585 that was requested.  ut_malloc_dontdump() takes
the size by value, so it has nowhere to report what my_large_malloc()
wrote back.

The rounding happens whenever my_next_large_page_size() finds a reported
large page size at or below the request.  macOS has no huge page
interface for my_get_large_page_sizes() to consult, so its fallback
branch reports the ordinary page size, 16384 on Apple silicon, and the
request is always rounded.  Linux reads the sizes from
/sys/kernel/mm/hugepages, where the smallest entry is usually 2 MiB, and
a 1 MiB request then gets no large page and no rounding.

The buffer has no alignment requirement.  recv_sys_t::parse() copies a
mini-transaction into it when the record is encrypted in the
FORMAT_ENC_11 log, where it is then decrypted in place, or when the
record wraps around the end of the log file, and reads it back as a byte
sequence.  tmp_free() calls std::free() because the member function
recv_sys_t::free() hides the one from <cstdlib>.

log_sys.buf and log_sys.flush_buf keep the large page allocator.  They
round the same way, so a server started with --large-pages
--innodb-log-buffer-size=2101248 still reports 24576 on macOS.  The core
dump exclusion that recv_sys.tmp_buf gives up applies only where
MADV_DONTDUMP exists, so nothing changes on macOS, while a release build
on Linux would now include the buffer in a dump.  tmp_free() overwrites
the redo log records that innodb_encrypt_log decrypted before releasing
the memory, through a volatile function pointer because GCC removes a
plain memset() that is followed by free().

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Mohammad Tafzeel Shams
MDEV-35154 : dict_sys_t::load_table() is holding exclusive dict_sys.latch
for unnecessarily long time

Issue:

dict_load_table_one() was invoked with the exclusive dict_sys.latch held
and never released it. The whole load ran inside that one critical
section: reading the SYS_TABLES record, opening the .ibd file, and
reading SYS_COLUMNS, SYS_VIRTUAL, SYS_INDEXES, SYS_FIELDS, the clustered
index root page, and SYS_FOREIGN, as well as loading every table related
by FOREIGN KEY constraints.

Almost all of that is I/O. Because dict_sys.latch is the single latch
that guards every dictionary lookup, one cold table open blocked every
other session from opening any table, including tables that were already
cached. A slow enough load could trigger the fatal semaphore wait check.

Fix:

Split the load into three phases. A short latched phase creates the
table object and publishes it as an incomplete "stub"; the I/O runs with
no latch held; a final latched phase links the FOREIGN KEY constraints.

The stub's progress is held in the top two bits of dict_table_t::
n_ref_count (LOADING_DEF, LOADING_FK, LOAD_FAILED), analogous to how
buf_page_t::state() combines a small lifecycle state with the buffer-fix
count in one atomic. A thread that finds a loading table waits by
pinning it (dict_table_t::try_pin_for_wait(), so it cannot be freed while
unlatched) and acquiring dict_table_t::lock_latch in shared mode, the
same latch the loader holds in exclusive mode for the duration of the
load. A failed load can be torn down safely even though other threads
may have already taken such a pin to wait on it: try_pin_for_wait()
refuses once LOAD_FAILED is set, and the teardown path drains any pins
taken earlier before freeing the stub.

lock_latch is otherwise used for record-lock bookkeeping and statistics
on a table; the only place the loader itself needs it for that unrelated
purpose while still holding it for the load is
dict_get_and_save_data_dir_path(), which skips re-acquiring the latch
whenever the table is loading(), since only the loading thread can reach
the table at that point.

A table remains hidden (LOADING_FK) until every table related to it by
FOREIGN KEY constraints has been loaded as well, and dict_sys_t::
load_table() makes them all visible in one step. The intermediate state
is visible to constraint linking via dict_sys_t::find_table_fk(), so that
two threads loading tables that reference each other can still link the
constraint between them.

Tables that InnoDB internal SQL can refer to are loaded without ever
releasing the latch. The internal SQL parser is not reentrant and is
serialized only by the exclusive dict_sys.latch, and it opens tables in
the middle of parsing; releasing the latch there let another thread run
the parser concurrently and corrupt its state.

Changes:

-  dict_table_t : hold the progress of loading in the top bits of
  n_ref_count (std::atomic<uint32_t>): LOADING_DEF, LOADING_FK,
  LOAD_FAILED, loading(). Add the loader-only helpers start_loading(),
  advance_to_loading_fk(), finish_loading(), mark_load_failed(), and
  try_pin_for_wait() for waiters, all built on the existing lock_latch.
  acquire()/release() are unconditional. Add debug load_thread and
  is_loader().

-  dict_load_table_one() : publish the table as a LOADING_DEF stub in
  table_non_LRU and release the latch before loading the tablespace,
  the columns and the indexes; reacquire it, move the table to
  table_LRU and advance it to LOADING_FK before loading the foreign
  key constraints. Failures go through dict_load_table_one_discard().
  Add the hold_latch parameter and the dict_load_table_one_no_latch
  debug sync point.

-  dict_sys_t::load_table() : wait for a concurrent load of the same
  table via wait_for_load(); allocate the foreign key table names on
  a local heap, because the latch is released while draining them;
  in the drain loop, wait for a table whose definition is still being
  loaded by another thread and skip one that is only waiting for its
  own related tables; make all tables loaded by this invocation
  visible in one step, releasing each one's lock_latch.

-  dict_sys_t::wait_for_load() : pins the observed table via
  try_pin_for_wait() (returning immediately if the load already
  failed), releases the exclusive dict_sys.latch, blocks on the
  table's own lock_latch, unpins, and reacquires the latch.

-  dict_load_table_one_discard() : used wherever a stub must be torn
  down (retry after DB_SUCCESS_LOCKED_REC, column or virtual-column
  load failure, a corrupted index or missing FK index). Marks the
  stub LOAD_FAILED, releases lock_latch to wake any already-pinned
  waiters, drains the reference count to zero, then removes the stub.

-  dict_sys_t::add() : take lock_latch in exclusive mode on the
  loader's behalf before a loading stub becomes reachable via
  find_table_any().

-  dict_get_and_save_data_dir_path() : skip re-acquiring lock_latch
  when the table is loading(), because the loading thread already
  holds it in exclusive mode and is the only thread that can reach
  the table at that point.

-  dict_load_hold_latch() : whether a table may be referenced by
  InnoDB internal SQL, and therefore must be loaded without releasing
  the latch: InnoDB system tables, FULLTEXT INDEX auxiliary tables and
  the persistent statistics tables.

-  dict_load_table_on_id() : copy the table name and release the
  SYS_TABLES page latch before calling load_table(), which may now
  block. Restore the cursor position only if the scan has to continue.

-  dict_load_foreign(), dict_load_foreigns() : add the fk_heap
  parameter and allocate the names appended to fk_tables on it.

-  dict_sys_t::find_table_any() : the previous body of find_table(),
  returning tables that are being loaded. Only the name is read.

-  dict_sys_t::find_table() : hide tables that are being loaded, both
  in the by-name and the by-id variant. In the by-id variant, loading
  is checked before any bit-field, to avoid a torn read.

-  dict_sys_t::find_table_fk() : like find_table(), but LOADING_FK
  tables are returned, so that constraints can be linked into them
  while holding the exclusive latch.

-  dict_table_can_be_evicted() : a table that is being loaded may only
  be freed by the thread that is loading it.

-  dict_foreign_add_to_cache() : resolve both sides with
  find_table_fk().

-  btr_search_disable() : skip tables that are being loaded. Walking
  their indexes would race with the loading thread, which appends to
  that list without holding the latch. Such tables cannot have any
  adaptive hash index references.

-  create_table_info_t::create_foreign_keys() : take a temporary
  reference on a referenced table as soon as it is resolved, released
  on every exit path by a scope guard; call dict_sys.prevent_eviction()
  only once the constraint is actually committed to the dictionary
  cache, replacing the temporary reference.

-  create_table_info_t::create_table() : acquire a reference to the
  created table around the foreign key handling, and use a local heap
  for the names of the foreign key related tables.

-  row_rename_table_for_mysql() : use one local heap for both the
  dropped-constraint names and the foreign key table names.

-  assertion fix : the load path may now run without dict_sys.latch,
  but only in the thread that is loading the table, and a cached
  table is no longer necessarily fully loaded. dict_sys.locked() is
  relaxed to "dict_sys.locked() || table->is_loader()" in
  dict_load_columns(), dict_load_virtual_col(), dict_load_fields(),
  dict_load_indexes(), dict_index_add_to_cache(),
  dict_index_find_cols(), dict_index_build_internal_clust(),
  dict_index_build_internal_non_clust() and
  dict_index_build_internal_fts(). Checks of dict_table_t::cached are
  relaxed and reordered after the atomic loading in
  dict_table_add_system_columns(), dict_sys_t::add(),
  dict_table_rename_in_cache() and hash_insert().

-  innodb.dict_load_concurrent
  A load is parked at dict_load_table_one_no_latch while
  holding no latch; another table can be loaded meanwhile, and a
  second opener of the same table waits. A second case checks that a
  table is not made visible while a table related to it by a FOREIGN
  KEY constraint is still being loaded by another thread.
Oleksandr Byelkin
MDEV-19817 Fix crash in Multiupdate_prelocking_strategy::handle_end

Multiupdate tables in case of reopening (in case of deadlock for example
do not clean up views/derived (with trigggers). So SET list stay "fixed"
and do not mark write bits which cause problems.

There is still problem in case of concurent flush table which replace
TABLE object, but it should be covered by MDEV-21630.
Dmitry Shulga
MDEV-40951: System triggers give no visibility into slow/failed execution and misreport server readiness

Added output into a log information about start/finish execution of
startup and shutdown triggers. In case error happens on running any of
startup/shutdown triggers, the error number and message is output
into error log.

Additionally, minor refactoring was done to avoid source code duplication.
Vladislav Vaintroub
MDEV-40967 PROXY protocol host check sent in clear text mid-SSL handshake

Defer the host-privileged/host-blocked check for a PROXY-header-derived
address until after the client's SSL handshake completes, instead of
sending it immediately in clear text.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Marko Mäkelä
fixup! 7e375803bb8a7dcb46c96f4a5727620ff0a6d0bb
Yuchen Pei
Allow a covering index scan for a locking SELECT under a full-scan table lock

Follow-up to MDEV-24813 (innodb_table_lock_on_full_scan) and MDEV-40805.

ha_innobase::build_template() retrieves the whole clustered index record
whenever select_lock_type is LOCK_X, and row_search_with_covering_prefix()
refuses the covering-index optimisation for the same reason. As a result a
covering secondary index scan such as

  SELECT sec, id FROM t1 FOR UPDATE

visits the clustered index once per scanned row, while the same scan with
LOCK IN SHARE MODE stays inside the secondary index. On a one million row
table that is a million extra clustered index lookups.

Only part of that work is inherent to LOCK_X. UPDATE and DELETE do need the
clustered index record, because they are going to write it. A plain locking
SELECT does not: it visits the clustered index only to place the per-row
exclusive lock that gives SELECT ... FOR UPDATE its row level exclusivity.
When innodb_table_lock_on_full_scan made us take a table level LOCK_X for
the whole scan, that per-row lock is redundant, because the table lock is
already mutually exclusive with any other transaction's LOCK_IX, and hence
with any record lock or implicit exclusive lock in the table.

Introduce row_prebuilt_t::full_scan_covering_read, set in
ha_innobase::extra_opt() next to full_table_scan and only when the statement
is a plain SELECT outside the HANDLER interface, and skip the LOCK_X
restriction in both places when it is set. The flag is never set unless
full_table_scan is set, so row level locking behaviour is unchanged when
innodb_table_lock_on_full_scan is off.

Whether a column outside the scanned index is needed is still decided by the
existing logic in build_template(), so SELECT pad ... FOR UPDATE continues to
read the clustered index.

Co-Authored-By: Claude Opus 5 <[email protected]>
Dave Gosselin
MDEV-33616:  Take the read lock many times in perfschema.func_mutex

The wait timer can have a granularity coarser than the time an
uncontended read lock is held, so the recorded duration of one lock can
be zero, which reads back as NULL.  This can cause the test to fail with
a false negative.

Take the lock twenty more times at each measurement point, with the
extra statements silent so the recorded result does not change.  The
mutex part of the test already works this way, since one SELECT
produces ten THR_LOCK::mutex events.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Vladislav Vaintroub
MDEV-39275 XA COMMIT / XA ROLLBACK / XA RECOVER don't require any privileges

Require a new XA RECOVER ADMIN privilege for XA RECOVER, matching
MySQL's XA_RECOVER_ADMIN. XA COMMIT/ROLLBACK are left unchanged.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Vladislav Vaintroub
Fix remaining .result drift from the new privilege bit

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Yuchen Pei
MDEV-39525 [wip][to-squash] fix vcol_type_both_bounded

Also improve test coverage of the domain check

TODO: check results
Dave Gosselin
MDEV-41212:  multi_source.status_vars fails on MacOS platform

Replace the two recorded reads of Slave_received_heartbeats with an
assertion that the counter is nonzero.

The counter advances once per heartbeat period for as long as the
connection is running.  The test waited for it to reach 2 and then
read it again in a separate query, so a heartbeat arriving between
those two queries recorded an unexpected value.

The same wait timed out when the counter was already past 2 at the
first poll, so it now accepts any value at or above the target.
Dave Gosselin
MDEV-33616:  Only one of two routines named in a statement is found

With lower_case_table_names 0 the server can have databases Db1 and db1,
each with a function f1.  A single statement naming both databases, like
SELECT Db1.f1(), db1.f1(), reported that db1.f1 does not exist.

The set of routines a statement uses compared its entries without regard
to case.  Only one routine was loaded but the reference to the other
found nothing.  The set now compares its entries exactly, as the routine
cache and the lock manager already do.
Dave Gosselin
MDEV-33616:  MTR flag to mark tests as incompatible with macOS

Introduces a new MTR include, not_mac.inc, which when included at the
top of a test, prevents that test from running on macOS.

sys_vars.sysvars_readonly_debug is the first user.  It expects the
server to fault when a read only sysvar is written behind the sysvar
interface.  That protection needs the ro_after_init section, which a
linker script places and ld64 has no option to take, so
HAVE_RO_AFTER_INIT stays undefined on macOS.  Without it no variable is
moved into the read only root either, so neither of the two assignments
is refused.
Alexey (Holyfoot) Botchkov
MDEV-36261 XMLTYPE: methods - step 1

Method functions added to the XMLTYPE.
  functions added here are:
  XMLTYPE.isFragment()
  XMLTYPE.getSchemaURL()
  XMLTYPE.getNumberVal()
  XMLTYPE.getStringVal()
  XMLTYPE.getRootElement()
Rex Johnston
MDEV-41228 Test deep clones of Item_cache items

Deep clones of Item items are created in very few places, so most of the
clone code has no test coverage at all, although Parallel Query relies on it
heavily. One of the places where clones are created is the generation of the
key parts for a lookup into a split materialized table, in
TABLE::add_splitting_info_for_key_field().

An Item_cache reaches that place through the IN->EXISTS transformation.
Item_in_optimizer::fix_left() wraps the left expression of the predicate in
an Item_cache, and the equality injected into the subquery refers to it, so
the key field value being cloned is an Item_direct_ref over that cache. Two
more things are needed for the clone to happen: the grouping field the
equality matches has to be the first component of some index of the
underlying table, otherwise it is not among spl_opt_info->spl_fields and the
function returns before cloning, and the subquery must not be converted to a
semi-join. The latter is achieved by the shape of the query, a UNION in the
subquery, rather than by turning optimizer switches off, so that the plan is
the one a user gets with a default optimizer_switch.

Add Item::check_deep_copy(), which validates a clone against its original in
a debug build. It walks both item trees and reports, as notes, whether they
have the same shape with the same Item class at every node, and whether the
clone shares an Item object with the original. Sharing is what distinguishes
a shallow copy from a deep one: an item that is shallow by design, Item_field
for instance, still produces a separate object and only shares a Field, which
is not an Item.

Call it from TABLE::add_splitting_info_for_key_field() under the
"split_materialized_clones" debug flag, which additionally makes the
optimizer use a clone as the value of the generated key part. The clone then
has to work both in the condition pushed into the materialized table and as
the value looked up in the filled table. Note that the clone built for the
pushed condition cannot be reused for this, as it has already been made
dependent on the select that specifies the materialized table.

With the check in place, the clone of the cache turned out not to be a deep
one: every Item_cache_* class implemented deep_copy() as a plain shallow
copy. That is wrong beyond sharing the example item. A cache is filled by the
store()/cache_value() calls of the item that owns it, Item_in_optimizer here,
and nobody does that for a clone, so a clone that inherited the cached value
of the original kept returning that value for the rest of the query.

Implement Item_cache::deep_copy() once for all cache classes instead. The
clone is given an empty cache, so that it computes the value itself out of
the item the value is read from, and a copy of that item. The exception is an
example containing an aggregate or a window function: those are not clonable
yet, as a copy of one shares the per-execution data of the original and both
would free it, so such an example is shared and the check reports the clone
as not fully deep. Item_cache_row is not clonable at all now, as a copy of it
shared the values[] array of element caches with the original.

The test uses a single row in the outer table, because the answer to the same
query with more rows is wrong for an unrelated reason, MDEV-41251: a split
materialized table in a dependently executed subquery is never refilled when
the outer row changes.

This commit was prepared with Claude Code (Opus 5), which located the query
shape that makes the split key part generation clone an Item_cache by
instrumenting TABLE::add_splitting_info_for_key_field() and searching for a
shape that reaches it, wrote Item::check_deep_copy() and the debug hook,
implemented Item_cache::deep_copy(), bisected the crash in Item_sum::cleanup()
that cloning the example item first caused to the aggr and cmp pointers shared
by an Item_sum copy, and separated the remaining wrong result into MDEV-41251
by showing that it survives the clone fix and does not depend on Item_cache.
Yuchen Pei
Do the same thing as the parent commit to quick_mode_{1,3}
Thirunarayanan Balathandayuthapani
MDEV-39061 mariadb-backup compatible wrapper for BACKUP SERVER

This adds a shell script that lets users keep using their existing
mariadb-backup commands while the real work is done by the new
server-side BACKUP SERVER command. The goal is "drop-in": users
should not have to change their backup scripts.

mariadb-backup-server.sh : Understands the usual mariadb-backup
modes and translates each one.

mbstream-server.sh : lets streamed backups be unpacked by
pipelines that expect the mbstream CLI

mariadb-backup-pipe.sh is the stream sink that
BACKUP SERVER itself runs.

All three are documented in README.md.

--backup
========
Connects with the mariadb client and runs
"BACKUP SERVER TO '<dir>'".

Connection options (--user, --host, --port, --socket,
--defaults-file, ssl, ...) are passed through to the client.

--parallel=N becomes "<N+1> CONCURRENT": mariadb-backup
runs a dedicated log_copying_thread() besides its N data-copy
threads, while CONCURRENT counts every BACKUP SERVER worker.
The result is clamped to the 1..256 the parser accepts,
and to a minimum of 2.

After the backup it writes backup-prepare.cnf into the backup
directory, recording what --prepare needs later:
  - where mariadbd lives
  - InnoDB parameters (page size, data file path, undo tablespaces,
                      checksum algorithm, log file size)
  - If the server is encrypted then how to reload the
    encryption key plugin (the file_key_management variables),
    so an encrypted backup can be prepared without extra input.

--backup --stream
=================
Runs "BACKUP SERVER WITH 'pipe'".
The server only accepts a bare command name, which it resolves
n the PATH of the mariadbd process after prepending mariadb-backup-,
so uses the installed mariadb-backup-pipe instead.

mariadb-backup-pipe writes the tar into
.mariadb-backup-pipe-<stream>.tar, a named pipe the wrapper
created in the server @@datadir and is already draining to
its own stdout.

The backup therefore reaches the consumer as it is produced and
never lands on local disk. The wrapper appends backup-prepare.cnf
as a final tar afterwards; the server's tar carries no
end-of-archive marker, so that trailing archive supplies the
only one and the whole stream extracts with a plain "tar -x".
--parallel is ignored here, with a warning.

Four properties follow from how BACKUP SERVER streams,
all differing from mariadb-backup:
- local: the stream command runs inside the server,
so the wrapper must share its filesystem;
- fifo lives in @@datadir, because mariadb-backup-pipe resolves it
relative to the working directory of mariadbd, so --stream needs
write access there;
- tar only: any --stream=<format> yields tar;
- single-threaded: one worker, so no parallel read either.
--target-dir is optional in stream mode;

mbstream-server.sh maps the mbstream CLI onto a plain
"tar -x"/"tar -c", so existing "mbstream -x"/"-c" pipelines
keep working on the wrapper's stream.

mbstream-only flags (-p/--parallel, ...) are accepted and
ignored; any other unknown option is rejected.

Environment overrides:
MARIADB (client),
MARIADBD (the --prepare bootstrap server) and
TAR (the tar implementation, e.g. TAR=bsdtar) can each be overridden.

To run the bootstrap under rr, put it in MARIADBD and
let rr's own _RR_TRACE_DIR choose the trace location, e.g.
  _RR_TRACE_DIR=/dev/shm/rr MARIADBD='rr record mariadbd'

--prepare
=========
Starts "mariadbd --bootstrap" on the backup directory using
backup-prepare.cnf as its defaults file, replays the archived redo
log between the start and target LSN read from backup.cnf,
then builds a fresh ib_logfile0 so a normal server can start
on the directory.

mariadbd is taken from the path recorded in backup-prepare.cnf
if that binary exists, otherwise by searching
/libexec, /sbin, /bin and the configured install directories.

PATH is not searched; set MARIADBD to point elsewhere.
User --defaults-file/-extra-file and encryption options are
layered onto the bootstrap.

--copy-back / --move-back
=========================
Copy or move a prepared backup into the datadir. The datadir
is created if missing, a non-empty datadir is refused unless
--force-non-empty-directories is given, and a chown
reminder is printed.

If --aria-log-dir-path is given, the Aria logs (aria_log_control,
aria_log.*) are relocated into that directory.

Packaging
=========
The wrapper is not installed by default and never replaces the
real mariadb-backup / mbstream binaries.
1. cmake -DWITH_MARIABACKUP_WRAPPER=ON (default OFF) controls it.
2. When ON, the scripts install as /usr/bin/mariadb-backup-server,
/usr/bin/mbstream-server and /usr/bin/mariadb-backup-pipe, tagged
COMPONENT Backup so they ship in the mariadb-backup package.
mariadb-backup-pipe must end up in the PATH of the mariadbd
process, not merely in the PATH of whoever runs the wrapper.
3. RPM: nothing extra to do. the component handles it.
4. DEB: not wired. debian/rules uses --fail-missing and does not
enable the option, so the -server binaries are not listed.
To ship via DEB, make a paired change: add
-DWITH_MARIABACKUP_WRAPPER=ON in debian/rules and list all three of
usr/bin/mariadb-backup-server, usr/bin/mbstream-server and
usr/bin/mariadb-backup-pipe in debian/mariadb-backup.install together.
5. The real mariadb-backup/mbstream binaries and the
mariabackup symlink are left untouched; opt in via an alias or a
symlink early in PATH.

Limitations (not supported yet)
===============================
1) Incremental backup & prepare (--incremental-basedir,
  --incremental-dir, --apply-log-only)
2) --rollback-xa
3) Partial backup (--databases, --tables, --tables-file)
4) Output compression and encryption (--compress, --encrypt)
5) --export is accepted but only warns and runs a plain recovery
6) --extra-lsndir is ignored
7) --parallel is ignored with --stream
8) Windows: POSIX sh only, not installed on Windows

Behaviour differences from native mariadb-backup
================================================
- The wrapper needs the mariadb client on PATH for --backup;
--prepare needs mariadbd recorded in backup-prepare.cnf, in a
standard install directory, or named by MARIADBD
- BACKUP SERVER refuses an already-existing target directory
- BACKUP SERVER does copy the data file as raw pages without
checksum validation, so a corrupted table is not detected
at backup time
- --prepare only works on a wrapper-made backup: it
needs backup-prepare.cnf
- --stream is tar, not xbstream, local-only and single-threaded,
and needs write access to @@datadir for the fifo

Tests
=====
include/have_mariabackup_wrapper.inc redirects $XTRABACKUP to
mariadb-backup-server.sh and $XBSTREAM to mbstream-server.sh,
skipping when a wrapper or the mariadb client is unavailable.

include/have_mariabackup_combination.inc runs a test under both the
[CLIENT] mariadb-backup binary and the [SERVER] wrapper.
Oleksandr Byelkin
MDEV-41094 KDF() aliases large iteration/width to weak 32-bit values

Item_func_kdf::val_str() read the PBKDF2 iteration count as a signed
64-bit longlong but only rejected values <= 0 before narrowing it to
the 32-bit int expected by PKCS5_PBKDF2_HMAC(). Iteration counts that
differ by 2^32 therefore aliased to the same 32-bit value and derived
identical keys: e.g. 4294968296 silently did the work of 1000. Since
the value is reproducible as (iter mod 2^32), any key derived this way
was already only as strong as the aliased low iteration count, so no
previously-derived ciphertext is orphaned by rejecting the alias now;
it simply reports the weak request instead of silently honouring it.

Item_func_kdf::fix_length_and_dec() had the identical bug for the key
width argument: `key_length= (uint)args[4]->val_int()` narrows before
the range check, and because the result is cached as a constant, the
runtime guard in val_str() (which uses a wider type and is otherwise
safe) is never reached for a constant width argument. This let a
width like 4294967552 silently alias to 256, and let negative widths
alias to a plausible positive one, both without warning.

Both call sites now validate the argument's true 64-bit value before
narrowing, reusing the existing invalid_argument_error() and NULL
result already used for other invalid KDF() arguments.
Dave Gosselin
MDEV-33616:  Detect select() on macOS

macOS declares select() in sys/select.h, which the HAVE_SELECT probe did
not include.  clang rejects a call to an undeclared function, so the
probe failed and HAVE_SELECT was left undefined.

my_sleep() then took its last fallback, a busy loop on time() that
rounds the requested interval up to a whole second.  Every sub-second
sleep in the server became a one second spin on a CPU, which is what
made rpl.rpl_perfschema_applier_status_by_worker,
rpl.rpl_shutdown_sighup and rpl.rpl_semi_sync_shutdown_await_ack fail.
Yuchen Pei
MDEV-39525 Add supertype checks to vcol index substitution in WHERE

TODO: fix commit message

- check if vcol field is a supertype to vcol expr
- check if both the vcol expr and vcol field are supertypes to the RHS

If either check returns true, then the substitution is safe.

Skip the checks for IS NULL / IS NOT NULL.

TODO: Does not yet work for the JSON_EXTRACT/JSON_VALUE vcol examples
in vcol_sargable because capacity_limit_is_in_characters returns
different values for varchar and blob/text

MDEV-39525 [to-squash] fix longstr supertype check

The string supertype check is too strict: it bails when src (resp.
dst) is length-limited in chars (i.e. CHAR/VARCHAR) and dst (resp.
src) is length-limited in octets (i.e. TEXT/BLOB).

We change this to conversion of src length limit to that of dst, i.e.

- if src limit is declared in octets and dst limit in chars, convert
  src limit to be in chars, and compare
- if src limit is declared in chars and dst limit in octets, convert
  src limit to be in octets, and compare

This restores the JSON_EXTRACT cases in vcol.vcol_sargable

MDEV-39525 [to-squash] Move collation match check outside of longstr is_supertype

This allows WHERE json_unquote(json_extract(...))=<const_string> to
use compare_collation instead.

MDEV-39525 [to-squash] Check the constant is strictly inside the field domain

The "both bounded" check in vcol_type_both_bounded() used
Field::is_supertype() for the constant as well. That is imprecise in
two ways:

- It compares declared data types, and Item_int always reports INT or
  BIGINT, so for a TINYINT/SMALLINT/MEDIUMINT vcol the check never
  passes, not even for `tinyint_vcol_expr = 2`.

- Being representable in the field's data type is not enough. The check
  should be open interval rather the closed interval For `a
  tinyint, va tinyint as (a+1)` and a row with a=127, the expression is
  128 while va is clamped to 127, so both `a+1 > 127` and `a+1 = 127`
  give different results before and after the substitution.

Also expand eligible Item_bool_rowready_func2 to include "<", "<=",
">" and ">=".

This restores explain results with int in vcol.vcol_sargable cases.

Added testcases accordingly.

Note, Field::is_supertype() is still needed for the vcol field against the
vcol expression: it covers the case when the conversion is not lossy at
all, including a constant sitting exactly on the boundary of the domain.

TODO: check results
Dave Gosselin
MDEV-33616:  Widen the block count filter in the buffer pool resize test

The test replaces the number of buffer pool blocks with a fixed value so
that the message is stable.  The pattern only accepted 5.., and macOS
builds without a futex use SUX_LOCK_GENERIC, which enlarges buf_block_t
enough to bring the count down into 4...
bsrikanth-mariadb
MDEV-40837: Crash while recording context when character_set_results is NULL

The value of character_set_results can be set to null unlike
character_set_client, or collation_connection.
When recording context for a query in opt_context_store_replay.cc,
character_set_results->cs_name was accessed, without checking if
character_set_results value was null or not. Hence the crash.

The solution is to add a null check for the variable character_set_results,
before accessing cs_name, inside Optimizer_context_recorder::dump_sql_script().