Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
sjaakola
MDEV-41385 wsrep_sync_wait has no effect after COM_CHANGE_USER

Also reported as: https://github.com/mariadb-corporation/galera/issues/613.

THD::cleanup() cleared THD::wsrep_client_thread. Besides connection
close, THD::cleanup() is also called from THD::change_user() when
handling COM_CHANGE_USER and COM_RESET_CONNECTION. THD::init() does
not set the flag back, and the only place that sets it is
thd_prepare_connection(). After either command the session was no
longer treated as a wsrep client connection: WSREP_CLIENT() returned
false for the rest of the session.

As a result wsrep_sync_wait was silently ignored, so causal reads
could return stale data. Such connections were also skipped by
wsrep_close_client_connections() and
wsrep_wait_committing_connections_close(). Connection pools issue
COM_RESET_CONNECTION whenever a connection is reused, so in practice
most pooled sessions were affected.

Fix by moving the reset of wsrep_client_thread from THD::cleanup()
to end_connection(), after wsrep_close().

The galera.galera_change_user test is extended to do a causal read
on a fresh connection, after COM_CHANGE_USER and after
COM_RESET_CONNECTION. Each read must increase the provider's
wsrep_causal_reads status counter.
bsrikanth-mariadb
MDEV-40598: Capture sequences used only in column DEFAULT expressions

Problem:
A sequence referenced only in a column's DEFAULT expression (e.g.
"a INT DEFAULT NEXTVAL(s1)") is opened only when a statement
evaluates DEFAULT values (INSERT, LOAD DATA, etc). A plain SELECT
never opens it, so the sequence never appears in
thd->lex->query_tables, and Optimizer_context_recorder::
dump_sql_script() had no way to see it. The dependent table's
definition was then captured without the sequence it depends on,
making the captured context unusable on replay.

TABLE::internal_tables lists these sequences, but its entries'
TABLE_LIST::table pointer is set by open_table() and never reset when
the statement ends, so it can point to a TABLE that has since been
closed, reused or freed. It cannot be trusted as "open now".

Fix:
dump_sql_script() walks TABLE::internal_tables for each dumped table
and calls the new resolve_default_sequence() for each entry. That
function never reads the cached TABLE_LIST::table; it uses only the
entry's db/table_name to resolve the sequence independently:

- Fast path: if the sequence is in thd->open_tables and was used by
  this statement (TABLE::query_id match, which matters under LOCK
  TABLES), dump its CREATE SEQUENCE and current value, as for a
  sequence used directly in the query.
- Slow path: otherwise open it in an isolated LEX, statement arena
  and Open_tables_state (as Table_ident::resolve_table_rowtype_ref()
  and fill_schema_table_by_open() do), so the running statement's
  LEX, sroutines list, locks and a prepared statement's permanent
  arena are not disturbed, and a name that has become a view is
  handled. The MDL request is non-blocking when the thread already
  holds locks. Only the CREATE SEQUENCE is dumped here, since the
  statement never used the sequence. If the open fails (dropped
  sequence, name now a view, MDL conflict), push the new warning
  ER_OPT_CONTEXT_SEQUENCE_CAPTURE_FAILED and carry on instead of
  failing the user's statement. OOM or a killed connection is still
  returned as an error.

dump_sequence_context() now reports through a "newly_dumped"
out-parameter whether it emitted the DDL, so the caller adds SETVAL
exactly once without repeating the dedup lookup.

Sequences are dumped before the dependent table's CREATE TABLE, so
replay can recreate both in order.

Tests (opt_context_store_ddls.test): INSERT-opened sequence; plain
SELECT; SELECT after FLUSH TABLES; LOCK TABLES with the sequence
locked but not used (no SETVAL); two sequences in one table; dropped
sequence (warning, table still captured); stored function in the
query (separate LEX); name replaced by a view; and re-execution as a
prepared statement.
Aleksey Midenkov
MDEV-40462 Move innodb.innodb-virtual-columns-debug into gcol.innodb_virtual_debug

innodb.innodb-virtual-columns-debug contained only the MDEV-17005 test
case. It is about InnoDB virtual column templates, so it belongs with
the other InnoDB virtual column debug tests in gcol.innodb_virtual_debug.

Rationale

Adding a separate test file for every single bugfix tends to grow the
number of MTR tests by tens of thousands (as there are already 40k
MDEV tickets). Each test file carries a fixed overhead that does not
depend on its size: before and after every test MTR runs
check-testcase to compare the server state, checks the error log for
warnings, and may restart the server when the options differ from the
previous test.

For tiny test files this overhead dominates the time of the test
itself, so the total time of MTR runs on buildbot and in local testing
grows considerably.

Besides that, every test is at least two files (.test and .result,
often also .opt or .combinations). Tens of thousands of extra small
files bloat the test directories, which slows down MTR test collection
and any filesystem traversal of the source tree, and they enlarge the
git index and tree objects, which makes git status, checkout, clone
and merges between branches slower.
Alessandro Vetere
Fix the build with cmake -DWITH_INNODB_AHI=OFF

Without BTR_CUR_HASH_ADAPT, buf_block_t::index, btr_search and
innodb_adaptive_hash_index_cells do not exist, so the assertion in
buf_LRU_truncate_temp() and the derivation of btr_search.n_cells in
innodb_init_params() are compiled only with it.

mysql-test/collections/broken-no-innodb-ahi.list names the tests
that fail in such a build, for --skip-test-list. They need the adaptive
hash index variables or counters, or the IS_HASHED column of
INFORMATION_SCHEMA.INNODB_BUFFER_PAGE and INNODB_BUFFER_PAGE_LRU, which
the build omits together with the sys schema views that read it.
mariadb-PranavTiwari
MDEV-39343- Added a system variable Grant_tables_intact that tells if tables are intact or not after dump restore.
Aleksey Midenkov
MDEV-38758 vcol.vcol_partition_innodb fails on buildbot

TABLE_ROWS in INFORMATION_SCHEMA.PARTITIONS for InnoDB is
dict_table_t::stat_n_rows. Each INSERT puts the record into the
clustered index and only afterwards increments stat_n_rows without
any latch. The second INSERT into a fresh partition schedules a
background persistent stats recalc, which is not throttled because
stats_last_recalc is still 0. If the recalc scans the leaf page
after the next INSERT has added its record but before that INSERT
increments stat_n_rows, the recalc stores the exact count and the
increment then adds one more, so p2 shows 4 rows instead of 3.

The fix disables innodb_stats_auto_recalc for the duration of
inc/vcol_partition.inc in the InnoDB variant of the test, so
stat_n_rows is maintained only by the per-row increments and is
exact.
Aleksey Midenkov
do_save_blob(): don't store the previous row's BLOB on OOM

do_save_blob() ignored the result of copy->tmp.copy(). When the
reallocation fails, copy->tmp keeps its old buffer, so the previous
row's BLOB value was stored into the destination field. The error
is already raised by my_malloc(MY_WME), but end_write() does not check
thd->is_error() after copy_fields(), so the stale row was still
written to the temporary table.

The fix resets the destination field and returns when the copy fails.
Pekka Lampio
MDEV-38999  Add results file for the MTR test galera.galera_MDEV-38999
Aleksey Midenkov
MDEV-29483 Heap-use-after-free (Binary_string::copy()) with window functions

JOIN::make_aggr_tables_info(): a query with a window function buffers
join rows into a postjoin-aggr temporary table via copy_fields().
BLOB/TEXT fields were copied by Copy_field::do_field_eq(), a raw
memcpy of the packed record representation (length + pointer),
leaving the tmp table's field pointing at whatever storage backed
the source row. Once the underlying handler (e.g. InnoDB) frees or
reuses that storage on a later row fetch, any later read of the
buffered blob (e.g. Item_field::str_result() via result_field,
referenced from a WHERE/HAVING subquery predicate) dereferences a
dangling pointer.

The fix forces BLOB/TEXT fields to be deep-copied (do_save_blob())
for any query with window functions, the same way simple GROUP BY
already does via save_sum_fields.

save_sum_fields controls an unrelated decision (whether to defer
Item_sum computation instead of wiring an incremental result_field,
see the SUM_FUNC_ITEM handling in Create_tmp_table::add_fields())
and must not be conflated with blob-copy safety. Introduce a
separate save_blobs flag of JOIN::create_postjoin_aggr_table() and
Create_tmp_table that independently controls the Copy_field::set()
deep-copy decision.

Create_tmp_table: add a save_blobs constructor parameter. The old
constructor is kept and sets save_blobs same as save_sum_fields.

create_tmp_table(): split into a variant taking a prepared
Create_tmp_table, used by JOIN::create_postjoin_aggr_table(). The
other create_tmp_table() callers are unchanged and keep save_blobs
same as save_sum_fields.
ParadoxV5
MDEV-4632 multi_source.status_vars test fails sporadically in buildbot

`Slave_received_heartbeats`’s test for independence between
connections relied on timing with seconds-level precision,
which is not consistent on a loaded CI.
To avoid reliance on timing accuracy,
this commit changes the test strategy to instead measure
a live result and use that to show the independence.
Aleksey Midenkov
MDEV-29483 Test for wrong result with window function and IN subquery

Add a test case to main.win that still returns a wrong result with
the previous commit applied. The result file records the current
(wrong) output: an empty set. The expected result is one row 'b'.

  CREATE TABLE t1 (c0 TEXT, c1 INT);
  INSERT INTO t1 VALUES ('a', 1), ('b', 2);
  CREATE TABLE t2 (c0 TEXT);
  INSERT INTO t2 VALUES ('b');
  SELECT COALESCE(c0, LAST_VALUE(c0) OVER (ORDER BY c1))
  FROM (SELECT c0, c1 FROM t1) dt
  WHERE c1 = 0 OR c0 IN (SELECT c0 FROM t2);

Findings so far:

1. The derived table dt is merged. The outer references to c0 in the
  WHERE clause and in the select list become Item_direct_view_ref
  objects pointing to one shared Item_field.

2. The window function forces a post-join aggregation temporary table
  (JOIN::make_aggr_tables_info() -> create_tmp_table()).
  create_tmp_field() with modify_item redirects result_field of that
  shared Item_field to the temporary table field.

3. The WHERE clause is evaluated in evaluate_join_record() for each
  join row, before the row is written to the temporary table. The IN
  predicate caches its left expression: Item_in_optimizer::val_bool()
  -> Item_cache_str::cache_value() -> Item_direct_view_ref::str_result()
  -> Item_field::str_result(), which reads result_field, not field.

4. So the WHERE clause compares the value copied by copy_fields() for
  the last row that passed the WHERE clause, not the current row. In
  the test, row 'a' sees an empty temporary table field and is
  filtered out, so nothing is copied, and row 'b' sees the same empty
  field and is filtered out too.

5. The heap-use-after-free from the original report is the same bug
  with a BLOB column on InnoDB. copy_fields() copied the BLOB with
  do_field_eq(), which keeps a pointer into the InnoDB row blob heap.
  InnoDB frees that heap on the next row fetch
  (row_mysql_prebuilt_free_blob_heap()), and the next WHERE
  evaluation dereferences it in Binary_string::copy(). Recorded with
  rr: the only free of the crashing block happens in the t1 scan on
  the row1 -> row2 transition, and at the crash result_field belongs
  to the temporary table while field holds row2's own value.

6. The previous commit deep-copies BLOBs for window function queries,
  which removes the use-after-free but not the stale read. The wrong
  result is not BLOB-specific and does not depend on InnoDB.

7. With a plain c0 in the select list instead of the COALESCE()
  expression the result is correct, so the shape of the select list
  item affects whether result_field is redirected.

The real fix belongs to the optimizer layer: either the merged
derived table must not share an item between a WHERE condition and a
select list item that gets a temporary table result_field, or the
temporary table setup must not redirect such an item, or the IN
predicate cache must read field when evaluated before the temporary
table is filled. MDEV-39868 (commit c0c4b5573f9) is a similar case of
merged derived table columns sharing one Item_field.
bsrikanth-mariadb
MDEV-40598: Capture sequences used only in column DEFAULT expressions

Problem:
A sequence referenced only in a column's DEFAULT expression (e.g.
"a INT DEFAULT NEXTVAL(s1)") is opened only when a statement
evaluates DEFAULT values (INSERT, LOAD DATA, etc). A plain SELECT
never opens it, so the sequence never appears in
thd->lex->query_tables, and Optimizer_context_recorder::
dump_sql_script() had no way to see it. The dependent table's
definition was then captured without the sequence it depends on,
making the captured context unusable on replay.

TABLE::internal_tables lists these sequences, but its entries'
TABLE_LIST::table pointer is set by open_table() and never reset when
the statement ends, so it can point to a TABLE that has since been
closed, reused or freed. It cannot be trusted as "open now".

Fix:
dump_sql_script() walks TABLE::internal_tables for each dumped table
and calls the new resolve_default_sequence() for each entry. That
function never reads the cached TABLE_LIST::table; it uses only the
entry's db and table name to resolve the sequence independently,
skipping sequences that were already dumped:

- Fast path: if the sequence is in thd->open_tables and was used by
  this statement (TABLE::query_id match, which matters under LOCK
  TABLES), dump its CREATE SEQUENCE and current value, as for a
  sequence used directly in the query.
- Slow path: otherwise open it in an isolated LEX, statement arena
  and Open_tables_state (arena swap as in
  fill_schema_table_by_open(), the rest as in
  Table_ident::resolve_table_rowtype_ref()), so the running
  statement's LEX, sroutines list, locks and a prepared statement's
  permanent arena are not disturbed. The MDL request is non-blocking
  when the thread already holds locks. As the statement never used
  the sequence, the user's SELECT privilege on it is checked first.
  Only the CREATE SEQUENCE is dumped, never SETVAL. If the sequence
  cannot be opened, is not accessible, or the name no longer
  resolves to a sequence (it was dropped, replaced by a view, ...),
  the new warning ER_OPT_CONTEXT_SEQUENCE_CAPTURE_FAILED is pushed,
  in addition to the underlying error turned into a warning, and
  nothing is captured for it. A kill is left as an error, and OOM
  makes the capture fail.

To share code with the existing dump loop, the "register the name in
the dedup hash and emit CREATE DATABASE" step is factored out into
register_table_for_dump(), and the SETVAL logic of the loop into
dump_sequence_current_value(). Sequences are dumped before the
dependent table's CREATE TABLE, so replay can recreate both in order.

Tests (opt_context_store_ddls.test): INSERT-opened sequence; plain
SELECT; LOCK TABLES with the sequence locked but not used (no
SETVAL); a user without privilege on the sequence; two sequences in
one table; dropped sequence (warning, table still captured); stored
function in the query (separate LEX); name replaced by a view; and
re-execution as a prepared statement.
Khaled Riyad
MDEV-38861 heap-use-after-free in Prepared_statement::execute()

DROP PROCEDURE and CREATE OR REPLACE PROCEDURE executed from inside the
routine itself removed it from the SP cache. sp_head::destroy() then freed
the memory root that the running sp_head, its LEX and its instructions
live in, and the caller kept using them.

Skip the removal while the routine is being executed. sp_cache_invalidate()
above has already bumped the cache version, so the stale entry is removed by
the next lookup, after IS_INVOKED has been cleared.
mariadb-PranavTiwari
MDEV-39343- Added a system variable Grant_tables_intact that tells if tables are intact or not after dump restore.
Aleksey Midenkov
MDEV-40462 innodb.innodb-virtual-columns-debug failed in buildbot with wrong result

The test sporadically fails because DELETE does not reach the
ib_open_after_dict_open sync point:

  SET debug_sync= "now WAIT_FOR delete_open";
  +Warnings:
  +Warning 1639 debug sync point wait timed out
  SELECT a FROM t1;
  a
  -NULL
  -NULL

The sync point is in ha_innobase::open(), which is called only when a
new TABLE is created from the share. After ALTER TABLE ... ALGORITHM=COPY
no TABLE for t1 is cached, but t1 has indexed virtual columns, so the
purge thread opens it via open_purge_table() and on close_thread_tables()
leaves the TABLE in the table cache. If that happens before DELETE, the
DELETE reuses the cached TABLE, does not call ha_innobase::open() and
completes without signalling delete_open.

The fix waits for purge with innodb_max_purge_lag_wait=0 and then does
FLUSH TABLES t1 before arming the sync point, so DELETE always opens a
new TABLE.
bsrikanth-mariadb
MDEV-40553 Print GIS ranges in optimizer trace and context

Ranges built over GIS (geometry) columns could not be printed in the
optimizer trace or the recorded optimizer context: Field_geom printed
every key value as the placeholder "unprintable_geometry_value",
regardless of whether the index stored the column's raw value (or a
prefix of it) or, for a SPATIAL index, its MBR (Minimum Bounding
Rectangle).

Field::print_key_part_value() now takes an image_type argument (see
Field::image_type()) that says which of the two the key holds. For a
SPATIAL index (image_type itMBR), Field_geom::print_key_part_value()
decodes the four doubles the key stores and prints them as a WKT
POLYGON. For every other index (image_type itRAW), the key holds the
value or a prefix of it, and it now prints in binary form, the same
way Field_blob does, instead of the placeholder.

print_mbr_range_operator() prints the spatial relation a GEOM range
carries (MBRWITHIN, MBRCONTAINS, MBRINTERSECTS, MBRDISJOINT,
MBREQUALS), inverted where needed so the indexed column reads on the
left. print_range() and print_key_value() thread the new image_type
argument through to Field_geom::print_key_part_value().

Note: this commit deliberately also carries a fix that is not specific
to GIS. Writing a test for this surfaced another bug in the code that
records/replays the optimizer context, fixed here since a test would
otherwise fail for reasons unrelated to GIS: the context literal that
dump_sql_script() writes into the recorded replay script
(opt_context_store_replay.cc) escaped backslashes SQL-style. That does
not round-trip through INFORMATION_SCHEMA.OPTIMIZER_CONTEXT's
regexp-based extraction the same way running the recorded script does,
so a context extracted that way no longer matched the ranges the
optimizer prints. The literal is now written with NO_BACKSLASH_ESCAPES
in effect instead, so its text is identical to the JSON it carries; a
single quote is written as its JSON escape \u0027, since
NO_BACKSLASH_ESCAPES leaves it as the only character that could still
end the literal early.

The tests that list ranges now read num_rows as bigint unsigned,
because MBRDISJOINT records HA_POS_ERROR (18446744073709551615).

Tested with main.opt_trace, main.opt_context_store_stats and
main.opt_context_replay_basic. The first two share the new GIS range
fixture (include/opt_gis_range_print.inc); main.opt_context_replay_basic
has its own GIS record/replay cases.
Pekka Lampio
MDEV-38999: Galera - stale reads in explicit transactions

wsrep_must_sync_wait() exempted every statement from its causal wait
once a transaction was open, regardless of which wsrep_sync_wait bit
it needed - so writes inside BEGIN...COMMIT could run against a stale,
not-yet-applied view. Track per-bit which parts of wsrep_sync_wait
have already been satisfied earlier in the current transaction, and
only skip a statement's wait once its own bit has actually been
honoured.

Assisted-by Claude AI
Khaled Riyad
MDEV-38004 Double free on re-execution of prepared aggregate function

Between two executions of a prepared statement, Item_sp::cleanup() freed
the stored function's memory root but left the arena's free list pointing
into it. That list is filled when the function call returns and the active
arena is restored.

On the next execution Item_sum_sp::clear() walked the stale list and
destroyed already freed items.

Free the items before the memory they live in.  The sequence is now
in Item_sp::free_call_ctx(), so the three places that repeat it
cannot drift apart again.
ParadoxV5
MDEV-4632 multi_source.status_vars test fails sporadically in buildbot

`Slave_received_heartbeats`’s test for independence between
connections relied on timing with second-level precision,
which is inconsistent on a loaded CI.
To avoid relying on timing accuracy, this commit changes the test
strategy to measure a live result and use that to show independence.
mariadb-PranavTiwari
MDEV-39343 mysql_upgrade tool now uses mariadb_info table and grant_tables_intact variable to decide whether upgrade is needed or not.
bsrikanth-mariadb
MDEV-40553 Print GIS ranges in optimizer trace and context

Ranges built over GIS (geometry) columns could not be printed in the
optimizer trace or the recorded optimizer context: Field_geom printed
every key value as the placeholder "unprintable_geometry_value",
regardless of whether the index stored the column's raw value (or a
prefix of it) or, for a SPATIAL index, its MBR (Minimum Bounding
Rectangle).

Field::print_key_part_value() now takes an image_type argument (see
Field::image_type()) that says which of the two the key holds. For a
SPATIAL index (image_type itMBR), Field_geom::print_key_part_value()
decodes the four doubles the key stores and prints them as a WKT
POLYGON. For every other index (image_type itRAW), the key holds the
value or a prefix of it, and it now prints in binary form, the same
way Field_blob does, instead of the placeholder.

print_mbr_range_operator() prints the spatial relation a GEOM range
carries (MBRWITHIN, MBRCONTAINS, MBRINTERSECTS, MBRDISJOINT,
MBREQUALS), inverted where needed so the indexed column reads on the
left. print_range() and print_key_value() thread the new image_type
argument through to Field_geom::print_key_part_value().

Writing a test for this surfaced another bug in the code that
records/replays the optimizer context, fixed here since a test would
otherwise fail for reasons unrelated to GIS: the context literal that
dump_sql_script() writes into the recorded replay script
(opt_context_store_replay.cc) escaped backslashes SQL-style. That does
not round-trip through INFORMATION_SCHEMA.OPTIMIZER_CONTEXT's
regexp-based extraction the same way running the recorded script does,
so a context extracted that way no longer matched the ranges the
optimizer prints. The literal is now written with NO_BACKSLASH_ESCAPES
in effect instead, so its text is identical to the JSON it carries; a
single quote is written as its JSON escape ', since
NO_BACKSLASH_ESCAPES leaves it as the only character that could still
end the literal early.

The tests that list ranges now read num_rows as bigint unsigned,
because MBRDISJOINT records HA_POS_ERROR (18446744073709551615).

Tested with main.opt_trace, main.opt_context_store_stats and
main.opt_context_replay_basic, which share the new GIS range fixture
(include/opt_gis_range_print.inc).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
mariadb-PranavTiwari
MDEV-39343- Added a system variable Grant_tables_intact that tells if tables are intact or not after dump restore.
mariadb-PranavTiwari
MDEV-39343- Added a system variable Grant_tables_intact that tells if tables are intact or not after dump restore.
mariadb-PranavTiwari
MDEV-39343- Added a system variable Grant_tables_intact that tells if tables are intact or not after dump restore.
bsrikanth-mariadb
MDEV-39868 Wrong result with a window fn over merged derived table column

Problem:
========
A query with a window function over a column of a merged derived table
returns an empty set when another table is joined on a condition over the
same column and is accessed with "Range checked for each record":

  SELECT AVG(subq.c2) OVER (), t2.c1
  FROM t1 LEFT JOIN (SELECT * FROM t3) AS subq ON t1.c1 = subq.c1
  STRAIGHT_JOIN t2 ON subq.c2 > t2.c1;

With derived_merge=on all the references to subq.c2 are
Item_direct_view_ref objects sharing one underlying Item_field, because
their ref pointers all point into the derived table's field_translation.
Item::split_sum_func2() calls real_item() and puts that shared Item_field
into the list of the window function's temporary table fields, so
create_tmp_field_from_item_field() sets its result_field to a column of
the temporary table.

Item_field::val_int() reads field, but Item_field::save_in_field() reads
result_field, so the two now return different values. The join condition
is evaluated through the same Item_field, and the runtime range analysis
in Field::get_mm_leaf_int() uses save_in_field_no_warnings(). It reads the
still empty temporary table column instead of the value of t3.c2, treats
the value as NULL, and builds a SEL_TREE::IMPOSSIBLE. Table t2 then
produces no rows.

Solution:
=========
Do not unwrap Item_direct_view_ref in Item::split_sum_func2(). The wrapper
is created per reference and is not shared, so the temporary table field
is attached to the wrapper alone and the conditions that refer to the same
view column keep reading the base table field.

Item_ref::create_tmp_field_ex() already creates the same temporary table
field for a view ref over a column, and change_to_use_tmp_fields() already
handles REF_ITEM, so no other change is needed. Ref access was never
affected: get_store_key() takes real_item()->field explicitly.
Alessandro Vetere
MDEV-32286 Reuse remembered clustered leaves in secondary-index scans

Row_sel_get_clust_rec_for_mysql::operator() descends the clustered B-tree
from the root for every row whose clustered-index record a secondary-index
scan must read, although consecutive rows land on the same clustered leaf
page wherever the secondary order tracks the clustered one. A non-covering
scan needs one for every row, a locking read needs one whatever the
secondary index holds, because an exclusive select lock type makes
ha_innobase::build_template() build its template against the clustered
index, and a covering scan needs one for every row of a secondary leaf
whose PAGE_MAX_TRX_ID its read view cannot see. ANALYZE FORMAT=JSON
charges each descent its full height, and those descents are nearly the
whole cost: the secondary index is charged its own descent and one page
for each further leaf, and nothing per row, because the position that its
cursor holds between two rows is restored optimistically, which latches
the leaf again without counting an access. So pages_accessed is the row
count times the height of the clustered index, plus a handful: 1000 rows
over a 2-level clustered index cost 2006 and 750 rows over a 3-level one
cost 2291, where a full table scan of the same data costs 23 and 110.

Let a handle remember the clustered leaves that the lookups of one
statement reached, and let the next lookup try them before it descends
again. The reasoning behind each value and each rejection is in the
comments beside it.

row0mysql.h defines clust_leaf_hint_slot, which names one leaf: its page
number, copies of its first and last user record truncated to the key
fields, which bound the key range that the leaf held when it was
remembered, the rec_get_offsets() of both, and the
dict_index_t::n_core_fields that the copies were made under. A slot of a
leaf that had no right sibling names no last key, because every key above
the last record of the rightmost leaf still belongs to it.
CLUST_LEAF_HINT_SLOTS (4) slots hang off the new
row_prebuilt_t::clust_leaf_hint, beside clust_leaf_hint_mru, the most
recently used order held as slot numbers, and clust_leaf_hint_n and
clust_leaf_hint_miss, the used-slot count and the miss counter.

row0sel.cc holds the policy. row_sel_clust_leaf_hint_covers() compares a
key against the remembered ranges, so a lookup that no slot can answer
costs no buffer pool access and no pages_accessed.
row_sel_clust_leaf_hint_search() probes the first slot that covers the key
and moves it to the front of the order.
row_sel_clust_leaf_hint_remember() records the leaf that a descent landed
on, and refreshes the slot of a leaf that is remembered already rather
than spend a second one on the same page. Two descents fill no slot: the
lookups of a statement before lookup CLUST_LEAF_HINT_MIN_LOOKUPS (4), and
a leaf that is the root. row_sel_clust_leaf_hint_armed() stands a scan
down once the slots stop paying for themselves: a miss adds
CLUST_LEAF_HINT_MISS_WEIGHT (2) to the miss counter and a hit takes one
away, so a scan gives the slots up where it answers too little of its
lookups to pay for them, CLUST_LEAF_HINT_MAX_MISSES (8) misses with no hit
between them still reach the threshold, and one lookup in
CLUST_LEAF_HINT_RETRY (1024) starts the count again, so a scan whose order
becomes correlated only later recovers. Both halves of the cost stop
there, the test of the slots and the copies that refresh them.
Row_sel_get_clust_rec_for_mysql::operator() calls all of this in place of
its btr_pcur_open_with_no_init(), and only where the adaptive hash index
is disabled. A guess of that index lands on the record with no page-local
search and no page access to charge, but neither of the two is faster for
every shape of index and scan, and a hint that runs first stops that index
from adding entries. The adaptive hash index is off by default, so the
hints are active in a default configuration.

btr0cur.h and btr0cur.cc add btr_cur_t::try_leaf_hint(), a PAGE_CUR_LE,
BTR_SEARCH_LEAF search on one named leaf, which starts from the record
where the last search of that leaf landed while that record is still valid,
and from a binary search of the page otherwise. It acquires the page with
buf_page_try_get(): a hint is never derived from a latched parent page,
so by the time it is tried it can precede the caller's already-latched
secondary-index leaf in the latching order, where a blocking wait can
deadlock. It then rejects the page unless the checks that it makes on the
latched frame put the match on it. Those checks are the sole authority on
the result, so a stale range costs a wasted probe or a needless descent,
never a wrong result, and the ranges need no invalidation protocol.

A slot also holds a btr_leaf_step, where the last search of its leaf
landed: the block, its buf_block_t::modify_clock and the record. A
record pointer stays valid while the block holds the same page and the
clock has not moved, because eviction, deletion and reorganization
advance the clock and an insertion moves no record. While the step is
valid, btr_cur_t::try_leaf_hint() starts from that record instead of a
binary search of the page. page0cur.cc adds page_cur_search_near(),
which compares the key with the record and with one neighbour, and
positions the cursor on the record, on the record after it or on the
record before it. It refuses any other answer and leaves it to
page_cur_search_with_match(). So a scan whose consecutive rows are
neighbours in the clustered index finds each row with two comparisons. A
scan that reads the clustered leaf in falling key order does the same: a
secondary key that falls as the primary key rises, or a secondary index
read backward. Where the secondary order tracks the clustered one with
gaps between the rows, the step fails, and btr_leaf_step::expect (below)
leaves the leaf to the binary search. The step refuses the metadata
pseudo-record, which compares below every key on 0 fields, and a record
before it that is the infimum. A debug build checks every successful
step against the binary search.

A failed step costs comparisons whose outcome no branch predictor can
guess, so btr_leaf_step::expect gates it. The next search tries the step
only where this one landed where a step would have: on the remembered
record, or right before or after it. The successor link of the landing
record tells the record right before, because finding a predecessor takes
a walk of the page directory. So a scan in random order over a few leaves,
which the slots answer in full, keeps the binary search and pays only the
tests that set expect. row_sel_clust_leaf_hint_remember() sets the step on
the record where a descent landed, with expect set.

ha_innodb.cc: ha_innobase::reset() zeroes the used-slot count and the miss
counter per statement, matching autoinc_last_value. row0mysql.cc:
row_prebuilt_free() frees the key buffers that the slots own.

innodb.clust_leaf_hint measures pages_accessed over key orders that differ
in how closely the secondary order tracks the clustered one, and further
tables check query results over the record formats and key shapes
that a clustered-index lookup has to read, down to the metadata
pseudo-record of instant ALTER TABLE, to leaves that split and merge while
a locking read walks them, and to a record that a remembered leaf supplies
for a scan that must then rebuild an older version of it. Two of the
tables scan a covering index, which reads a clustered record under an
exclusive select lock type and under a PAGE_MAX_TRX_ID that the read view
cannot see. clust_leaf_hint_off_debug runs the same body with the hints
turned off, through a debug switch that returns before a lookup tests or
refreshes the slots, so a diff of the two .result files is what the hints
save: 2006 to 1031 (2-level clustered index), 2291 to 1011 (3-level), 4006
to 2015 (two interleaved key ranges), 12016 to 9078 (locality in the
second half alone) and 20020 to 10045 for a covering scan that FOR UPDATE
makes non-covering, where the same scan without FOR UPDATE costs 20 in
both files. Two orders with too little locality to pay for the slots give
them up early and end within a hundred accesses of the unhinted count:
20020 to 19966 (decorrelated) and 20020 to 19999 (shuffled).

Two further tables in innodb.clust_leaf_hint cover the step back. t16
scans a secondary index whose keys fall as the primary key rises, and
reads an ascending secondary index backward: 20020 to 10047 for both. t17
runs the inverse scan on ROW_FORMAT=REDUNDANT with an instantly added
column. innodb.clust_leaf_hint_backward_debug reaches the two refusals of
the step back that no count shows, at the infimum and at the metadata
pseudo-record. A lookup reaches either only for a key that left the
clustered index while the slot still covers it. The rollback of an insert
over a delete-marked record removes the key while purge leaves the
secondary entry, and a PAGE_MAX_TRX_ID that the read view cannot see makes
the reader look that entry up. A debug switch writes each refusal to the
error log, and the case reads both back.

innodb.clust_leaf_hint_instant_alter covers the one rejection that no
count reaches, of a slot whose keys were copied under another
dict_index_t::n_core_fields than the index reports.
dict_index_t::clear_instant_alter() is the only writer of that value that
a shared metadata lock allows, and it needs the clustered index to lose
the last user record of its root page, while no leaf that is the root
fills a slot, so the tree has to shrink between the two, which purge does
there. The reader therefore reads uncommitted rows at READ UNCOMMITTED and
waits in a stored function while a rollback and purge take them away, and
one row that arrives above the position it stopped at is the lookup that
tests the slots. The rejection leaves nothing that a query can read, so
that branch writes the two counts to the error log under a debug switch
and the case reads them back with search_pattern_in_file.inc. They are
printed and not named in the pattern, so that a run which reaches the
branch with other counts, or in the direction where the clear lowers them,
is a difference to look at and not a pass.

main.rowid_filter_innodb: 90 to 88, and its ahi combination unchanged.

A static_assert in Row_sel_get_clust_rec_for_mysql::operator() fails to
compile if btr_search.enabled stops being a Boolean that applies to every
index, because the test !btr_search.enabled that stands the hints down
would then have to become !btr_search.is_enabled(clust_index).
mariadb-PranavTiwari
MDEV-39343- Added a system variable Grant_tables_intact that tells if tables are intact or not after dump restore.
Yuchen Pei
MDEV-41366 Check prefix key match in partition unordered index scan

The idea of Case 2 in can_skip_merging_scans is that when key prefix
is fixed, and the infix that is also the "partition by range" column,
we can scan each partition in order. This relies on accurately
returning EOF when a partition has no (more) matching rows. It is
possible to have no more matching rows at the first index read of a
partition in a LAST_OR_PREV access, in which case we need to check the
prefix match to rule out false positives and return EOF correctly.

TODO: more test coverage?
bsrikanth-mariadb
MDEV-40553 Print GIS ranges in optimizer trace and context

Ranges built over GIS (geometry) columns could not be printed in the
optimizer trace or the recorded optimizer context: Field_geom printed
every key value as the placeholder "unprintable_geometry_value",
regardless of whether the index stored the column's raw value (or a
prefix of it) or, for a SPATIAL index, its MBR (Minimum Bounding
Rectangle).

Field::print_key_part_value() now takes an image_type argument (see
Field::image_type()) that says which of the two the key holds. For a
SPATIAL index (image_type itMBR), Field_geom::print_key_part_value()
decodes the four doubles the key stores and prints them as a WKT
POLYGON. For every other index (image_type itRAW), the key holds the
value or a prefix of it, and it now prints in binary form, the same
way Field_blob does, instead of the placeholder.

print_mbr_range_operator() prints the spatial relation a GEOM range
carries (MBRWITHIN, MBRCONTAINS, MBRINTERSECTS, MBRDISJOINT,
MBREQUALS), inverted where needed so the indexed column reads on the
left. print_range() and print_key_value() thread the new image_type
argument through to Field_geom::print_key_part_value().

Writing a test for this surfaced another bug in the code that
records/replays the optimizer context, fixed here since a test would
otherwise fail for reasons unrelated to GIS: the context literal that
dump_sql_script() writes into the recorded replay script
(opt_context_store_replay.cc) escaped backslashes SQL-style. That does
not round-trip through INFORMATION_SCHEMA.OPTIMIZER_CONTEXT's
regexp-based extraction the same way running the recorded script does,
so a context extracted that way no longer matched the ranges the
optimizer prints. The literal is now written with NO_BACKSLASH_ESCAPES
in effect instead, so its text is identical to the JSON it carries; a
single quote is written as its JSON escape ', since
NO_BACKSLASH_ESCAPES leaves it as the only character that could still
end the literal early.

The tests that list ranges now read num_rows as bigint unsigned,
because MBRDISJOINT records HA_POS_ERROR (18446744073709551615).

Tested with main.opt_trace, main.opt_context_store_stats and
main.opt_context_replay_basic, which share the new GIS range fixture
(include/opt_gis_range_print.inc).
Khaled Riyad
MDEV-38861 heap-use-after-free in Prepared_statement::execute()

DROP PROCEDURE and CREATE OR REPLACE PROCEDURE executed from inside the
routine itself removed it from the SP cache. sp_head::destroy() then freed
the memory root that the running sp_head, its LEX and its instructions
live in, and the caller kept using them.

Skip the removal while the routine is being executed. sp_cache_invalidate()
above has already bumped the cache version, so the stale entry is removed by
the next lookup, after IS_INVOKED has been cleared.
bsrikanth-mariadb
MDEV-41243, MDEV-41273 Don't freeze mem_root after engine pushdown

When a statement is pushed down to an engine, the once-per-statement
part of the optimization (e.g. the first_cond_optimization part of
JOIN::optimize(), or the whole of JOIN::optimize() for a pushed down
UNION) is skipped. Re-executing the same prepared statement or stored
routine without pushdown then allocated from a mem_root already marked
ROOT_FLAG_READ_ONLY, hitting an assertion in alloc_root().

Add LEX::dont_freeze_mem_root, set by the pushdown_handler and
derived_handler constructors so it covers every engine implementing
them. Prepared_statement::execute_loop(), sp_head::execute() (through
sp_head::dont_freeze_mem_root) and the SP instruction reparse path no
longer freeze the mem_root of such statements. The flag is only
declared and used in PROTECT_STATEMENT_MEMROOT builds.

Note this disables the mem_root protection for the statement for good,
even if later executions are not pushed down. For a stored routine one
pushed down instruction disables it for the whole routine.

Add tests for prepared SELECT, UNION, UPDATE and DELETE, prepared EXPLAIN
of a pushed UNION, derived tables, stored procedures, functions,
triggers, cursors and metadata-invalidation reparse, with pushdown
switched on and off in either order. The routine and prepared statement
tests check the statements received by the remote server to verify that
pushdown actually happened.
Khaled Riyad
MDEV-38004 Double free on re-execution of prepared aggregate function

Between two executions of a prepared statement, Item_sp::cleanup() freed
the stored function's memory root but left the arena's free list pointing
into it. That list is filled when the function call returns and the active
arena is restored.

On the next execution Item_sum_sp::clear() walked the stale list and
destroyed already freed items.

Free the items before the memory they live in.  The sequence is now
in Item_sp::free_call_ctx(), so the three places that repeat it
cannot drift apart again.
bsrikanth-mariadb
MDEV-41243, MDEV-41273: fedx pushdown assert for UPDATE/DELETE prepared stmts

When a statement is pushed down to an engine, JOIN::optimize() skips its
once-per-statement work. Re-executing the same prepared statement or
stored routine without pushdown then allocated from a mem_root already
marked ROOT_FLAG_READ_ONLY, hitting an assertion in alloc_root().

Add LEX::dont_freeze_mem_root, set by the pushdown_handler and
derived_handler constructors so it covers every engine implementing
them. Prepared_statement::execute_loop(), sp_head::execute() (through
sp_head::dont_freeze_mem_root, PROTECT_STATEMENT_MEMROOT builds only)
and the SP instruction reparse path no longer freeze the mem_root of
such statements. Note this disables the mem_root protection for them.

Add tests for prepared statements, stored procedures, functions,
triggers, cursors and metadata-invalidation reparse, with pushdown
switched on and off in either order.
Georgi (Joro) Kodinov
more doxygen formatting added.
mariadb-PranavTiwari
MDEV-39343- Added a system variable Grant_tables_intact that tells if tables are intact or not after dump restore.
sjaakola
MDEV-41385 wsrep_sync_wait has no effect after COM_CHANGE_USER

Also reported as: https://github.com/mariadb-corporation/galera/issues/613.

THD::cleanup() cleared THD::wsrep_client_thread. Besides connection
close, THD::cleanup() is also called from THD::change_user() when
handling COM_CHANGE_USER and COM_RESET_CONNECTION. THD::init() does
not set the flag back, and the only place that sets it is
thd_prepare_connection(). After either command the session was no
longer treated as a wsrep client connection: WSREP_CLIENT() returned
false for the rest of the session.

As a result wsrep_sync_wait was silently ignored, so causal reads
could return stale data. Such connections were also skipped by
wsrep_close_client_connections() and
wsrep_wait_committing_connections_close(). Connection pools issue
COM_RESET_CONNECTION whenever a connection is reused, so in practice
most pooled sessions were affected.

Fix by moving the reset of wsrep_client_thread from THD::cleanup()
to end_connection(), after wsrep_close().

The galera.galera_change_user test is extended to do a causal read
on a fresh connection, after COM_CHANGE_USER and after
COM_RESET_CONNECTION. Each read must increase the provider's
wsrep_causal_reads status counter.
Alexander Barkov
MDEV-41246 "Illegal mix of collations" on the mysql.user view

This SQL script failed:

SET NAMES latin1 COLLATE latin1_swedish_ci;
CREATE OR REPLACE VIEW v1 AS SELECT 'Y' AS c1;
SET NAMES big5 COLLATE big5_chinese_ci;
SELECT * FROM v1 WHERE c1='y';

with the following error:

ERROR 1267 (HY000): Illegal mix of collations
(latin1_swedish_ci,COERCIBLE) and (big5_chinese_ci,COERCIBLE)
for operation '='

Note, latin1_swedish_ci and big5_chinese_ci are used here as examples.
The error also happened with different collation combinations.

Fix main idea:

If two collations have equal comparison rules (known as "tailoring")
on a given character repertoire,
like latin1_swedish_ci and big5_chinese_ci on ASCII letters,
then the "Illegal mix of collations" error can be avoided in
a comparison operator. We can choose any of the sides as the operation
effective collation - the result will be equal.

This optimization is not applied when at least one side has an explicit
COLLATE clause. Two explicit COLLATE clauses in one comparison are
already illegal when the character sets are the same, so for
consistency this stays illegal when the character sets differ too.

Most important details:

- Splitting enum_repertoire_t into smaller subsets,
  for better repertoire granularity.
  A variable holding a repertoire value can now have multiple
  MY_REPERTOIRE_XXX flags set.

  This patch implements detecting tailoring equality
  on these repertoires:
  * MY_REPERTOIRE_ASCII_ALNUM - [A..Z,a..z,0..9].
  * MY_REPERTOIRE_ASCII_IDENT - ALNUM + underscore
  * MY_REPERTOIRE_ASCII      - the entire range U+0000..U+007F

- Adding a new virtual function "tailoring" in my_collation_handler_st.

  It returns the tailoring on the given repertoire for the given
  collation.

  If cs1->coll->tailoring(cs1, some_repertoire) returns {0,0},
  it means the illegal mix optimization cannot be used
  for this collation on the given repertoire.

  If these calls:
    tr1= cs1->coll->tailoring(cs1, some_repertoire);
    tr2= cs2->coll->tailoring(cs2, some_repertoire);
  return both non-NULL results and tr1.str==tr2.str,
  then these collations are equal on the given repertoire
  and are mutually replaceable for a comparison operator,
  so "Illegal mix of collations" can be avoided.

- Adding a new method DTCollation::aggregate_by_tailoring().
  Like the other successful branches of DTCollation::aggregate(),
  it makes the resulting repertoire cover both sides.

- Adding a new flag MY_COLL_ALLOW_BY_TAILORING.
  It indicates to DTCollation::aggregate() that the illegal
  mix optimization by repertoire can be used in the given context.

  MY_COLL_CMP_CONV now includes MY_COLL_ALLOW_BY_TAILORING.
  Note, only comparison operators pass this flag.
  Functions returning a string result do not pass this flag,
  because in operations like CONCAT(a,b) we still need to evaluate
  precisely the collation of the result - we cannot just choose
  a collation of one of the sides (even if they are compatible
  on the given repertoire). This also applies to ExtractValue()
  and UpdateXML(), which now aggregate their arguments without
  this flag.

- As in my_repertoire_t the value MY_REPERTOIRE_ASCII is now a set of
  bits rather than a single bit, the way to detect "is only ASCII"
  repertoires has changed in the code.

  For example:
    // repertoire *IS* ascii
    if (repertoire == MY_REPERTOIRE_ASCII)

  has changed in multiple places in the code to

    // repertoire *HAS* only ascii characters
    if (my_repertoire_is_subset_of(repertoire, MY_REPERTOIRE_ASCII))

  The new function my_repertoire_is_subset_of() in m_ctype.h
  is used for this purpose in C code. In C++ code
  DTCollation::repertoire_is_subset_of() is used, e.g.:

    if (collation.repertoire_is_subset_of(MY_REPERTOIRE_ASCII))

- Repertoire of some expressions was adjusted to the new meaning:
  * Item_null now has MY_REPERTOIRE_NONE (was ASCII).
  * MY_LOCALE::repertoire() now returns MY_REPERTOIRE_UNICODE30
    (was EXTENDED).
  * Lex_string_with_metadata_st::repertoire(cs) now scans the string
    contents to detect the actual repertoire.
  * HEX() now has MY_REPERTOIRE_ASCII_ALNUM.
  * QUOTE, MAKE_SET, EXPORT_SET, LPAD, RPAD, GROUP_CONCAT, JSON_ARRAY,
    JSON_OBJECT and JSON_OBJECTAGG add the repertoire of the extra
    characters they put into the result (quotes, separators, padding,
    brackets).
  * LOWER() and UPPER() add the letters of both cases to the repertoire,
    because the result can contain letters of the opposite case.

- The tis620 collations now set CHARSET_INFO::tab_to_uni (was NULL),
  so my_charset_is_ascii_based() is true for tis620.
  This changes the result of subselect_extra_no_semijoin
  from an error to success.

- New flags were added for CHARSET_INFO::state
  * MY_CS_ASCII_BINARY_CI - for simple 8bit case insensitive collations.
    It means that this collation has no irregularities on the ASCII
    repertoire.

  * MY_CS_IDENT_BINARY_CI - for simple 8bit case insensitive collations.
    It means that this collation has no irregularities on the IDENT
    repertoire (but can have irregularities say on punctuation).

  * MY_CS_ASCII_STD_UCA - for UCA collations.
    It means that a UCA collation does not reorder ASCII characters.

- strings/conf_to_src.c was modified to detect and print
  MY_CS_ASCII_BINARY_CI and MY_CS_IDENT_BINARY_CI flags.

- strings/ctype-extra.c was regenerated with new flags.

- Adding a number of MTR tests in
  plugin/func_test/mysql-test/func_test/.
  They display a tailoring by collation name and repertoire as
  returned by:
    cs->coll->tailoring(cs, some_repertoire)
  A dynamically linked plugin function collation_tailoring() was added
  for the purpose of these tests.

- Adding the test plugin/func_test/mysql-test/func_test/ctype_ldml.test.
  It displays the tailorings of collations loaded at runtime from
  Index.xml: simple 8bit collations (flags detected from the weights)
  and Unicode collations (LDML rules which do or do not reorder ASCII).

- Adding a number of MTR tests mysql-test/main/ctype_xxx_tailoring.test
  They display two-dimensional charts showing which collations
  are compatible on which repertoires.

- Adding a number of MTR tests in the form of the originally
  reported script for various collations:

    SELECT Insert_priv FROM mysql.user WHERE Insert_priv='...';

  The repeated blocks of queries are shared through
  mysql-test/include/
  ctype_tailoring_0{1,2,3}_mysql_user_Insert_priv*.inc.

- Adding tests to ctype_cp932.test checking that ExtractValue() and
  UpdateXML() still raise "Illegal mix of collations".
mariadb-PranavTiwari
MDEV-39343- Added a system variable Grant_tables_intact that tells if tables are intact or not after dump restore.
Yuchen Pei
MDEV-41366 Check prefix key match in partition unordered index scan

The idea of Case 2 in can_skip_merging_scans is that when key prefix
is fixed, and the infix that is also the "partition by range" column,
we can scan each partition in order. This relies on accurately
returning EOF when a partition has no (more) matching rows. It is
possible to have no more matching rows at the first index read of a
partition in a non-exct access, in which case we need to check the
prefix match to rule out false positives and return EOF correctly.

Also strengthen the checks in ha_partition::can_skip_merging_scans()
so that AFTER_KEY and BEFORE_KEY reads require both prefix and the
partitioned column in the keypart_map to ensure correctness.

For example, this check disqualifies the index_read_map call in an
existing loose index scan testcase, causing the index scan to use both
unordered (in read_range_first) and ordered.
Oleksandr Byelkin
MDEV-39566 crash on a sorted scan of performance_schema.status_by_*

The bitmap of materialized rows in PFS_table_context was sized by the
live row count of the buffer container, which grows while a scan runs:
a sorted scan sizes it twice and asserted when the two sizes differed,
and set_item() could write past its end.  status_by_account used the
thread local key of status_by_host.  The allocation used the number of
bits in a word as its size in bytes.  m_last_item started at 0, so
set_item(0) did nothing and rnd_pos() dropped the first row.

Size the bitmap by the container maximum, which is fixed at startup;
give status_by_account its own key THR_PFS_SBA; allocate sizeof(ulong)
per word; start m_last_item at a sentinel.

Co-Authored-By: Claude Opus 5 <[email protected]>