Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Marko Mäkelä
fixup! a1c90c41530fdc6558a5299bf9539769bf4e3617
Hemant Dangi
MDEV-40501: Assertion `info->type == READ_CACHE || info->type == WRITE_CACHE' failed in reinit_io_cache upon CHANGE MASTER

Issue:
CHANGE MASTER ... FOR CHANNEL with a channel name within
MAX_CONNECTION_NAME can still overflow the OS file name limit once
escaped into the relay log file name. Relay_log_info::init() then
fails to open the relay log, leaving its index file unopened, but
Master_info_index::remove_master_info() unconditionally calls
reset_logs() on it during CHANGE MASTER's error cleanup, which hits
the assertion in reinit_io_cache().

Solution:
Guard the reset_logs() call in remove_master_info() with is_open(),
so a relay log that was never opened is never passed to it. Also use
MY_SAFE_PATH in open_index_file() so an over-length name fails
deterministically instead of silently falling back to a mangled one.
Vlad Lesin
MDEV-31956 SSD based InnoDB buffer pool extension

In one of the practical cloud MariaDB setups, a server node accesses its
datadir over the network, but also has a fast local SSD storage for
temporary data. The content of such temporary storage is lost when the
server container is destroyed.

The commit uses this ephemeral fast local storage (SSD) as an extension of
the portion of InnoDB buffer pool (DRAM) that caches persistent data
pages. This cache is separated from the persistent storage of data files
and ib_logfile0 and ignored during backup.

The following system variables were introduced:

innodb_extended_buffer_pool_size - the size of external buffer pool
file, if it equals to 0, external buffer pool will not be used;

innodb_extended_buffer_pool_path - the directory in which the external
buffer pool file is created, the data directory is used if the variable
is not set.

If innodb_extended_buffer_pool_size is not equal to 0, external buffer
pool file will be created on startup of a normal server instance. It is
created as a temporary file, so that the operating system removes it
when the server closes it or exits, and no stale file is left behind
after a crash. For this purpose row_merge_file_create_mode() was
generalized into pfs_create_temp_file() and moved to fil0fil.cc, so that
both the merge sort files and the external buffer pool file are created
by the same function.

Only clean pages will be flushed to external buffer pool file. There is
no need to flush dirty pages, as such pages will become clean after
flushing, and then will be evicted when they reach the tail of LRU list.
Freed pages are not written to the external buffer pool file either,
they are just evicted.

The general idea of this commit is to flush clean pages to external
buffer pool file when they are evicted.

A page can be evicted either by transaction thread or by background
thread of page cleaner. In some cases transaction thread is waiting for
page cleaner thread to finish its job. We can't do flushing in external
buffer pool file when transaction threads are waiting for eviction,
that would hurt performance. That's why the only case for flushing is
when page cleaner thread evicts pages in background and there are no
waiters. For this purpose buf_pool_t::done_flush_list_waiters_count
variable was introduced, we flush evicted clean pages only if the
variable is zeroed.

Clean pages are evicted in buf_flush_LRU_list_batch() to keep some
amount of pages in buffer pool's free list. That's why we flush every
second page to external buffer pool file, otherwise there could be not
enough amount of pages in free list to let transaction threads to
allocate buffer pool pages without page cleaner waiting. This might be
not a good solution, but this is enough for prototyping.

External buffer pool page is introduced to store information in buffer
pool page hash about the certain page can be read from external buffer
pool file. The first several members of such page must be the same as the
members of internal page. External page frame must be equal to the
certain value to distinguish external page from internal one. External
buffer pages are preallocated on startup in external pages array. We
could get rid of the frame in external page, and check if the page's
address belongs to the array to distinguish external and internal pages.

There are also external pages free and LRU lists. When some internal page
is decided to be flushed in external buffer pool file, a new external
page is allocated either from the head of external free list, or from
the tail of external LRU list. Both lists are protected with
buf_pool.mutex. It makes sense, because a page is removed from internal
LRU list during eviction under buf_pool.mutex.

Then internal page is locked and the allocated external page is attached
to io request for external buffer pool file, and when write request is
completed, the internal page is replaced with external one in page hash,
external page is pushed to the head of external LRU list and internal
page is unlocked. After internal page was removed from external free list,
it was not placed in external LRU, and placed there only
after write completion, so the page can't be used by the other threads
until write is completed.

Page hash chain get element function has additional template parameter,
which notifies the function if external pages must be ignored or not. We
don't ignore external pages in page hash in two cases, when some page is
initialized for read and when one is reinitialized for new page creating.

When an internal page is initialized for read and external page with the
same page id is found in page hash, the internal page is locked,
the external page in replaced with newly initialized internal page in the
page hash chain, the external page is removed from external LRU list and
attached to io request to external buffer pool file. When the io request
is completed, external page is returned to external free list,
internal page is unlocked. So during read external page is absent in both
external LRU and free lists and can't be reused.

When an internal page is initialized for new page creating and external
pages with the same page id is found in page hash, we just remove external
page from the page hash chain and external LRU list and push it to the
head of external free list. So the external page can be used for future
flushing.

The external buffer pool file is not represented by a fil_space_t. The
requests to it are issued by fil_system_t::ext_bp_io(), and the external
buffer pool page which a request refers to is stored in IORequest in
place of the fil_node_t. The pages are written to and read from the
external buffer pool file in the same form in which they are written to
their tablespaces, i.e. compressed and encrypted pages stay compressed
and encrypted in external buffer pool file.

If a write to the external buffer pool file fails, a warning is written
to the error log and the external buffer pool is disabled for the rest
of the server lifetime.

Currently the commit passed some local smoke tests, mtr and RQG tests
with external buffer pool turned on.

TODO:
1. Add some monitoring, i.e. how much pages are currently in external
  buffer pool, the percent of hits during reading, take a look at the
  current buffer pool monitoring and implement the general monitoring
  tools for external buffer pool.
2. Think about partial initialization of external pages array, as it was
  done for internal pages.
3. Take a look at compressed LRU list, it looks like currently it's not
  covered with eviction to external buffer pool file (I don't currently
  understand if we need it at all).
4. Think about more suitable algorithm for eviction to external buffer
  pool, currently just every second page is flushed.
Alexander Barkov
MDEV-39518 Allow prepared statements in stored functions in assignment right hand

Allowing prepared statements in stored functions when
a stored function is used in an assignment right hand.

Both DEFAULT clause of a variable initialization and
the right side of the SET statement are supported:

  CREATE PROCEDURE p1()
  BEGIN
    -- case 1: DEFAULT clause
    DECLARE spvar1 INT DEFAULT f1_with_ps(); -- OK

    -- case 2: SP variable assignment statement
    DECLARE spvar2 INT;
    SET spvar2= f1_with_ps(); -- OK
  END;

- Only assignments to SP variables works for now:
  * SET spvar= func_with_ps(); -- OK
  * SET @uvar= func_with_ps(); -- Error

- Only bare function calls are supported for now. Using a function in
  an expression does not make it PS-safe yet:
    SET v= f1()+0;

- The parser now does not reject PS statements in stored functions.
  PS applicability in stored functions is now detected at run time.
  Note, PS statements in triggers are still prohibited by the parser.

- Functions with PS do not acquire MDL locks on tables, and no MDL is
  taken on the routines themselves either. They work like procedures in
  terms of table opening and routine locking: a concurrent DROP FUNCTION
  can complete while such a function is executing.

- Functions with PS are not replicated as a single `SELECT f1()` call.
  They are replicated per-statement, like procedures.

Helper changes:
- Changing the return result for LEX::sp_variable_declarations_init()
  from void to bool to catch errors in the caller properly.

Misc:
- This patch incorporates fixes for the following bugs found during debugging:
  MDEV-40224,MDEV-40225,MDEV-40226,MDEV-40227,MDEV-40240,
  MDEV-40285,MDEV-40288,MDEV-40315,MDEV-40318,MDEV-40890,MDEV-40900,
  MDEV-40901,MDEV-40913,MDEV-40914,MDEV-41013,MDEV-41015,MDEV-41019

Assisted-by: Claude - reviews and minor clean-ups
Sergei Golubchik
temporarily remove failing tests
Yuchen Pei
MDEV-40168 nested array handling and json validation
Thirunarayanan Balathandayuthapani
MDEV-38942  i_s_dict_fill_sys_tables() aborts when reading INNODB_SYS_TABLES after innodb_force_recovery

Problem:
========
  A query on INFORMATION_SCHEMA.INNODB_SYS_TABLES crashes when
SYS_TABLES contains a record that was inserted by a transaction
which has not been committed. This can happen after a
crash while a CREATE TABLE was in progress, if the server is
restarted with innodb_force_recovery=4 or greater,
because trx_rollback_recovered() is then skipped and the
recovered transaction remains ACTIVE.

dict_sys_tables_rec_read() returns READ_NOT_FOUND for such a
record, and dict_load_table_low() returns that as success with no
error message and setting *table to nullptr.
i_s_sys_tables_fill_table() checks only the error
message and passes the nullptr table to i_s_dict_fill_sys_tables(),
which dereferences it.

Solution:
========
i_s_sys_tables_fill_table(): Skip the SYS_TABLES record when
dict_load_table_low() reports success but returns no table, because
such a record is not visible.
Sergei Golubchik
ft_json ft parser
Alexander Barkov
MDEV-39518 Allow prepared statements in stored functions in assignment right hand

Allowing prepared statements in stored functions when
a stored function is used in an assignment right hand.

Both DEFAULT clause of a variable initialization and
the right side of the SET statement are supported:

  CREATE PROCEDURE p1()
  BEGIN
    -- case 1: DEFAULT clause
    DECLARE spvar1 INT DEFAULT f1_with_ps(); -- OK

    -- case 2: SP variable assignment statement
    DECLARE spvar2 INT;
    SET spvar2= f1_with_ps(); -- OK
  END;

- Only assignments to SP variables works for now:
  * SET spvar= func_with_ps(); -- OK
  * SET @uvar= func_with_ps(); -- Error

- Only bare function calls are supported for now. Using a function in
  an expression does not make it PS-safe yet:
    SET v= f1()+0;

- The parser now does not reject PS statements in stored functions.
  PS applicability in stored functions is now detected at run time.
  Note, PS statements in triggers are still prohibited by the parser.

- Functions with PS do not acquire MDL locks on tables, and no MDL is
  taken on the routines themselves either. They work like procedures in
  terms of table opening and routine locking: a concurrent DROP FUNCTION
  can complete while such a function is executing.

- Functions with PS are not replicated as a single `SELECT f1()` call.
  They are replicated per-statement, like procedures.

Helper changes:
- Changing the return result for LEX::sp_variable_declarations_init()
  from void to bool to catch errors in the caller properly.

Misc:
- This patch incorporates fixes for the following bugs found during debugging:
  MDEV-40224,MDEV-40225,MDEV-40226,MDEV-40227,MDEV-40240,
  MDEV-40285,MDEV-40288,MDEV-40315,MDEV-40318,MDEV-40890,MDEV-40900,
  MDEV-40901,MDEV-40913,MDEV-40914,MDEV-41013,MDEV-41015,MDEV-41019

Assisted-by: Claude - reviews and minor clean-ups
Marko Mäkelä
WIP: log tracking BACKUP SERVER TO ... CONCURRENT (for HAVE_INNODB_PMEM)

backup_sink::id: The thread identifier (0 to CONCURRENT-1)

innodb_backup_checkpoint_pmem(): Copy the old log file.

InnoDB_backup::log_track(), InnoDB_backup::log_track_pmem():
Keep copying the log until we run out of InnoDB data files to copy.

InnoDB_backup::checkpoint_complete_pmem(): Copy the remaining
part of an old log file right before it is being released.

InnoDB_backup::commit(): In log tracking backup, copy the rest of
the HAVE_INNODB_PMEM log.

FIXME: Implement the non-PMEM code path with minimal blocking.
Khaled Riyad
MDEV-37335: crash on re-parsing an sp_instr_set that does not own its LEX

A DECLARE of several variables with a common DEFAULT expression creates one
sp_instr_set per variable, all sharing a single LEX that only the last of
them owns. On re-parsing after a metadata change, a non-owning instruction
still has its LEX, and both parse_expr() and validate_lex_and_exec_core()
took a non-null LEX to mean a cursor LEX. The cursor-only code then
dereferenced the nullptr returned by get_lex_for_cursor() and the re-parsed
LEX was never adopted.

Branch on whether the LEX is a cursor LEX instead of whether it is non-null.
Sergei Petrunia
This is a combination of 16 commits.

Add comments

Factor out common code into get_mvi_index()

Factor out common code into Item_func_json_contains::get_mvi_access()

Item_func_json_contains::mvi_analyze() and ::create_ft_for_mvi() were
near-identical: both checked the arguments, looked up the matching MVI,
parsed the constant second argument and ran the same scan loop calling
encode_mvi_key(). They differed only in what they did with each encoded
key.

Move all of that into get_mvi_access(), which returns an Mvi_access, and
give Mvi_access two methods:

- add_key(), to collect one encoded element key,
- create_ft_item(), to build the

    MATCH vcol AGAINST ('+encoded_foo +encoded_bar ...' IN BOOLEAN MODE)

  item. It honors Mvi_access::conjunctive, so JSON_OVERLAPS will get the
  OR form for free.

mvi_analyze() and create_ft_for_mvi() are now thin wrappers around
get_mvi_access().

This also fixes a memory leak: the encoded keys were copied with
String::copy(), giving each String in Mvi_access::encoded a heap buffer
that is never freed (the Strings live on the MEM_ROOT, so their
destructors never run). Copy the keys onto the MEM_ROOT instead.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

Make the MVI scan a real access method: QUICK_MVI_SELECT

JSON_CONTAINS() over a multi-valued index used to be optimized by
rewriting the WHERE clause: setup_mvi_for_join() injected a synthetic

  MATCH vcol AGAINST ('+k1 +k2' IN BOOLEAN MODE)

into join->conds and into select_lex->ftfunc_list, and the normal
fulltext machinery then picked it up as JT_FT access. The injected item
showed up in the plan and in the condition even though the user never
wrote it, and because it became ordinary ref access the scan was never
costed against the alternatives - it won by being in the WHERE clause.

Introduce QUICK_MVI_SELECT (QS_TYPE_MVI), a QUICK_SELECT_I that drives
the fulltext index directly through the handler API. Unlike FT_SELECT
there is no Item_func_match to have created the FT_INFO, so the quick
select creates it in reset() with ft_init_ext() and frees it with
close_search() in its destructor. Mvi_access::create_ft_item() is
replaced by build_ft_query(), which builds just the query string.

The analysis in setup_mvi_quick() is now kept: Mvi_context moves to the
header, is allocated on the mem_root and stored as JOIN::mvi_ctx, where
JOIN::get_mvi_access_for_table() looks it up. get_quick_record_count()
builds the quick select before test_quick_select() and keeps whichever
of the two is cheaper; test_quick_select() itself is untouched, so the
MVI quick is held in a local across the call (it deletes select->quick
on entry). The same save/compare is done around the second
test_quick_select() call in make_join_select(), which a LIMIT can reach.

A fulltext key never gets a bit in const_keys or keys, so mark the MVI
key of every table that has an access: the const_keys bit is what lets
the range analysis run for that table at all, the keys bit puts the
index into EXPLAIN's possible_keys.

Collect the accesses from the top-level AND-parts of the WHERE clause
only, instead of walking the whole condition. An MVI scan reads just the
rows the index matches, so it is only valid for a predicate that must
hold for every row of the result: for

  json_contains(j1->'$.tags','"a"') OR json_contains(j2->'$.tags','"a"')

scanning either index would drop the rows that only match the other
branch. The deleted add_ft_for_mvi() refused COND_OR_FUNC for the same
reason; walking the condition tree lost that, which only became visible
once the accesses were actually used.

Costs are placeholders (records=10, read_time=0.001) until the engine
can estimate a fulltext search. Note that while there is no estimate,
an MVI access is also taken when test_quick_select() produced no quick
select at all, without comparing it to the cost of a table scan.

TODO: This doesn't handle UPDATE/DELETE!  Should it be put into
  check_quick() call?
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

Make optimizer trace print "range", not "index_merge" for MVI quick selects.

Add optimizer trace for the multi-valued index access

get_best_mvi_access() picked an Mvi_access and wrapped it in a
QUICK_MVI_SELECT without recording anything, so there was no way to see
which index was chosen or what it would search that index for: the
rows_estimation trace showed a range_analysis that found nothing, and
then a plan using a key the trace never mentioned.

Print a "multi_value_index_use" object:

  {
    "table": "t1",
    "index": "idx",
    "ranges": ["616161"]
  }

Mvi_access::print_json() fills in the index and the element keys,
following TRP_RANGE::trace_basic_info(): same "index" / "ranges" member
names, so an MVI entry reads like a range scan's.

The keys are printed in their encoded form, which is not readable. That
is what is stored in the index and what we search for, so it is still
the useful thing to print; making it readable can come later. It is
plain ASCII (hex plus the xx/xxxx padding from encode_mvi_key()), so it
needs no JSON escaping.

get_best_mvi_access() runs inside the "rows_estimation" array, so the
named object needs an object of its own around it, the same way
make_join_statistics() and the sel_arg_alloc_limit_hit trace do it.
Without it the writer hits an assertion in
Single_line_formatting_helper::on_add_member().

The new test is a separate file because optimizer trace tests need
not_embedded.inc, and putting that in multi_valued_index.test would skip
the whole feature test on embedded builds. It cross-checks the printed
keys against mvi_encode() over the indexed column, which produces the
tokens the index is actually built from.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

Make JSON_OVERLAPS sargable for multi-valued indexes

Both argument orders are handled, since JSON_OVERLAPS is symmetric:

  JSON_OVERLAPS(array_indexed_expr, '[foo, bar, ...]')
  JSON_OVERLAPS('[foo, bar, ...]', array_indexed_expr)

JSON_CONTAINS is true when ALL of the elements have a match, JSON_OVERLAPS
when ANY of them does, so the access it produces has conjunctive=false and
build_ft_query() leaves the keys optional instead of prefixing them with
'+'.

The two get_mvi_access() implementations share collect_mvi_keys(), which
is the scan of the JSON literal that used to sit inside
Item_func_json_contains::get_mvi_access().

The two differ in one way beyond the flag. An element that cannot be
encoded for the index (a number against a CHAR array, say) is skipped for
JSON_CONTAINS: dropping a key from an AND makes the index scan less
selective, so it still returns a superset of the rows the predicate
matches and the predicate does the exact filtering afterwards.

That reasoning does not hold for an OR. A row can satisfy the predicate
through the very element we failed to encode, and MVI_ENCODE skips such
elements as well, so that row has no key in the index for the scan to find
it by - dropping the key would lose it. So for a disjunctive access we
give up instead of skipping. With a row {"tags": [123]} in the table,

  select * from t1 where json_overlaps(j->'$.tags','[123,"aaa"]')

must not use the index, and the test checks its result against the same
query with IGNORE INDEX.

Also print "match": "all"/"any" in the optimizer trace. Now that an access
can be either, the printed ranges alone did not say whether a row has to
have all of the keys or just one.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

Move the JSON function MVI code into opt_mvi_jsonfuncs.cc

opt_multi_valued_index.cc held two separate concerns: the index side
(how a value is encoded into the index, the access descriptor, the quick
select) and the predicate side (which JSON functions can be computed
from an MVI, which of their arguments holds the indexed expression, and
what to search the index for). Split the second one out.

Moved verbatim:

  get_mvi_index()
  collect_mvi_keys()
  Item_func_json_contains::get_mvi_access() and ::mvi_analyze()
  Item_func_json_overlaps::get_mvi_access() and ::mvi_analyze()
  add_mvi_access()

The only code change is that encode_mvi_key() is no longer static: it is
used both by Item_func_mvi_encode::val_str_ascii(), which stays, and by
collect_mvi_keys(), which moves. It is declared in
opt_multi_valued_index.h now. The other three moved helpers had no
callers outside the moved code and stay static.

Item_func_mvi_encode is not a JSON function and stays put: it is how the
values get into the index in the first place.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

MDEV-40168: JSON-over-fulltext: add estimates.

Add records_in_range-like estimates for fulltext index

Estimate the number of records an MVI access will read

QUICK_MVI_SELECT carried records=10 and read_time=0.001, numbers picked
low enough that the access always won over a table scan. Ask the engine
instead, through the fulltext_estimate() added by the previous commit.

Mvi_access::estimate_records() estimates one element key at a time and
combines the answers the way the query combines the keys:

- A disjunctive access (JSON_OVERLAPS) reads the rows of every key, so
  the estimates add up.

- A conjunctive access (JSON_CONTAINS) reads the rows that have all of
  the keys, so the rarest key alone bounds the result. We use its
  estimate and drop the other keys from the query: reading the rarest
  key and letting the WHERE clause discard the rest is not worse than
  having the engine intersect the terms. This is the trade-off
  collect_mvi_keys() already makes for the keys it cannot encode - a
  shorter AND matches a superset of the rows, and the JSON predicate
  does the exact filtering.

The engine may be unable to estimate a key: ha_innobase only looks at
the fulltext auxiliary tables, so until the words are flushed out of the
FTS cache the answer is "unknown" for everything. Such a key takes no
part in the choice of the rarest one, and if not a single key could be
estimated the old guess stands and the query is left as it is.

For an OR we cannot do that: we have to read that key and have no idea
what it costs. Give the access a DBL_MAX read_time and do not use it.
That has to be acted on in get_best_mvi_access() rather than left to the
cost comparison, because best_access_path() takes a quick select to be
cheaper than a table scan without checking - true of anything the range
optimizer proposes, but this access does not come from there.

read_time is now the cost of reading the estimated rows plus evaluating
the WHERE clause on them, which is what the join optimizer expects of a
quick select's read_time. It does not account for the fulltext search
that produces the rowids in the first place.

The trace prints the estimate, or says the access is unusable and why.

The two existing tests ran on tables whose words were still in the FTS
cache, so the JSON_OVERLAPS sections would have stopped using the index
entirely. They now flush the cache with OPTIMIZE TABLE under
innodb_optimize_fulltext_only, the way innodb_fts.estimate does. The
trace test gets a table with a skewed distribution ("bbb" in every row,
"aaa" in one) to show the conjunctive access keeping only the rarest key
and still producing the same rows as the same query with IGNORE INDEX.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

MDEV-40168: JSON-over-fulltext: let the estimate consult the FTS cache.

fts_estimate_word_docs() probed only the on-disk auxiliary INDEX_[1..6]
table.  Documents inserted but not SYNCed yet are only in the in-memory
FTS cache, so they were missed entirely, and on a table that has never
been SYNCed the auxiliary table is empty and the estimate was simply
"unknown".

Look in the cache as well.  fts_index_cache_t::words is an rb tree of
fts_tokenizer_word_t, and fts_node_t::doc_count already holds the number
of documents in the node's ilist, so this is one rbt_search plus a walk
over a short vector: no ilist decoding and no I/O.

Three details:

  - the cache mutex is taken with trylock.  This runs during
    optimization, where a SYNC holding cache->lock across SQL execution
    would stall the optimizer; dropping the cache contribution is the
    better trade.

  - nodes flagged fts_node_t::synced are skipped.  An in-flight SYNC has
    already written them out, and the estimator reads the B-tree without
    a read view, so it sees those records; counting the node too would
    count its documents twice.

  - index_cache->words is NULL between fts_cache_clear() and
    fts_cache_init().  The query path never observes that because it
    runs after fts_init_index(); the estimate does not run it.

DB_RECORD_NOT_FOUND now means both sources are empty.  The cache on its
own cannot prove a word absent, because it is only complete once
fts_init_index() has run, and an estimate must have no side effects.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

MDEV-40168: JSON-over-fulltext: move the estimator into fts0est.cc.

Pure code motion, no functional change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

Adjust the MVI tests to the estimate that reads the FTS cache

fulltext_estimate() used to see only the on-disk auxiliary table, so the
multi-valued index tests, which insert and immediately EXPLAIN, got
"unknown" for every element key: the conjunctive accesses fell back to a
guess of 10 rows and the disjunctive ones were dropped for want of a
cost. Both tests worked around that with OPTIMIZE TABLE under
innodb_optimize_fulltext_only. The estimate consults the cache now, so
the workaround is gone and the row counts in the plans are real.

Two sections of the trace test were written before the estimate existed
and no longer showed what they said they did:

- "Several element keys" printed one range, not several, because a
  conjunctive access now keeps only the rarest key. It runs on a table
  with a skewed distribution instead ("bbb" in every row, "aaa" in one),
  which makes the choice of key visible rather than a tie, and checks
  the rows against the same query with IGNORE INDEX.

- The JSON_OVERLAPS section is where several ranges are printed now, so
  it says so, along with the estimates adding up.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

Trivial cleanups and comments

Move QUICK_MVI_SELECT into opt_multi_valued_index.cc

Keep the MVI access of a table in its JOIN_TAB

The access was looked up through JOIN::get_mvi_access_for_table(), which
indexed a per-JOIN array by table->tablenr. Nothing else about how a
table is going to be read lives there, so put it where the rest does:
JOIN_TAB::mvi_access. Mvi_context is then just the result of the WHERE
analysis - the indexes and the accesses it found - and the choice of
which access a table uses is made per table, where it belongs.

setup_mvi_access_for_table() makes that choice and marks the index in
const_keys and keys, which make_join_statistics() used to do in a loop of
its own right after update_ref_and_keys(). It runs next to
add_group_and_distinct_keys(), the other place that adds to const_keys
for something the range optimizer would not find by itself, and just
before the range analysis those bits exist for.

get_quick_record_count() takes the JOIN_TAB rather than the TABLE now,
which is all it needed the JOIN for.

Also fix the header comment of Mvi_access::estimate_records(), which
still described the old fallback for a conjunctive access that could not
be estimated at all.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Rex Johnston
PQ: tidy up some of the mess, remove pointless code.
Vladislav Vaintroub
MDEV-40608 build mysqlservices without an embedded CRT requirement

mysqlservices only exposes a thin C API, no CRT state crosses it, so
don't force whatever CRT/config built the server onto a plugin linking
it. Without /Zl, a plugin built in a config with no matching installed
mysqlservices variant (CMake silently substitutes one - verified with
a toy project) gets an ignorable but noisy LNK4098 warning.

Assisted-by: Claude:claude-5-sonnet
Alessandro Vetere
MDEV-40408 btr_page_reorganize_low() uses the buffer pool just to obtain a scratch block

Add buf_pool.scratch_buf, a pool of page frames that the page
reorganization operations use instead of taking a block from the global
buffer pool.

buf_pool_t::scratch_buffer: A singly linked list of chunks of
buf_tmp_buffer_t slots. A thread holds at most one slot at a time, and
it holds a page latch while doing so, which is why reserve() must not
wait for the buffer pool or for I/O. When every slot is in use,
reserve() appends a chunk that holds twice the slots of the last one, up
to a limit on the size of one chunk. Only grow() waits, on the mutex
that serializes it and on the allocator. A chunk is never moved or freed
before close(), so a thread can keep using the slot that it reserved
while another thread appends a chunk. Debug builds start with a single
slot, so that a second concurrent page reorganization exercises grow().

buf_pool_t::scratch_buffer::shrink(): Free the page frames of the slots
that are not in use. The slots and the chunks are kept, because a slot
costs 32 bytes while a page frame costs srv_page_size. A frame survives
the first pass, because that pass only clears the used flag. The master
thread calls this often while the server is idle and rarely while it is
active, and buf_pool_t::garbage_collect() calls it under memory
pressure, where it ignores the used flag, because releasing these frames
is much cheaper than shrinking the buffer pool.

buf_pool_t::io_buf_t::acquire(): Factor out the scan for an unreserved
slot, which io_buf_t::reserve() ran twice and the scratch buffer reuses.

btr_page_reorganize_low(), page_zip_reorganize(): Obtain the scratch
page frame from buf_pool.scratch_buf. This removes the buf_pool.mutex
acquisition and the free block wait that buf_block_alloc() could
perform while at least a page X-latch was being held.
btr_page_reorganize_low() also used to leak the block on its error
paths; a single exit now releases the slot.

page_copy_rec_list_end_no_locks(), lock_move_reorganize_page(): Take
the source page frame instead of a source buf_block_t, because the
source is no longer a buffer pool block. In
page_copy_rec_list_end_no_locks() the source page is page_align(rec) at
every call site, so only the record is passed.

buf_tmp_buffer_t::acquire(), buf_tmp_buffer_t::release(): Use acquire
and release memory ordering, so that the page frame pointer of a slot
is published to the next thread that reserves the slot. acquire() also
reads the flag before the exchange, so that a scan across an array of
slots does not write to the slots that it finds reserved. release()
also marks the page frame undefined for Valgrind and MSan, because the
frame stays allocated for the next reserver and no deallocation marks
it.
Sergei Golubchik
w
Rex Johnston
MDEV-40005 Parallel Query: a worker scanned the wrong index

The SQL layer opens a private TABLE, and so a private handler, per
worker. parallel_init_coordinator() records the scan parameters on the
master's handler, which a worker's handler has never seen: its
m_pscan_keynr is still MAX_KEY, the clustered index, while the chunk
boundaries it gets handed were computed on the secondary index the plan
chose.

Pass the coordinator's handler to parallel_init_worker() and take the
parameters from it in pscan_adopt_scan_params(), before anything reads
m_pscan_keynr. The ranges are borrowed rather than copied: they live in
the master's m_pscan_range_heap, freed by parallel_end_coordinator()
only once every worker has been joined, so parallel_end_worker() drops
the pointers again on a handler that does not own the heap.
Hemant Dangi
MDEV-40501: Assertion `info->type == READ_CACHE || info->type == WRITE_CACHE' failed in reinit_io_cache upon CHANGE MASTER

Issue:
CHANGE MASTER ... FOR CHANNEL with a channel name within
MAX_CONNECTION_NAME can still overflow the OS file name limit once
escaped into the relay log file name. Relay_log_info::init() then
fails to open the relay log, leaving its index file unopened, but
Master_info_index::remove_master_info() unconditionally calls
reset_logs() on it during CHANGE MASTER's error cleanup, which hits
the assertion in reinit_io_cache().

Solution:
Guard the reset_logs() call in remove_master_info() with is_open(),
so a relay log that was never opened is never passed to it. Also use
MY_SAFE_PATH in open_index_file() so an over-length name fails
deterministically instead of silently falling back to a mangled one.
Vladislav Vaintroub
MDEV-41008: Fix X509 issuer/subject comparison for OpenSSL 3

OpenSSL 3 escapes '/' and '+' in X509_NAME_oneline() output;
OpenSSL 1.1 and WolfSSL don't. A REQUIRE ISSUER/SUBJECT grant from
one library can stop matching after switching to another.

Default comparison stays strcmp(). old_mode=X509_LENIENT_COMPARE
opts into ignoring the escaping backslash, at the cost of reopening
the single-RDN-vs-multi-RDN ambiguity a crafted certificate could
exploit to impersonate another identity.

Also fixes sysvars_server_embedded/notembedded for the new old_mode
value, and guards my_x509_oneline_cmp() against X509_NAME_oneline()
returning NULL.
Oleksandr Byelkin
Fix the version
Khaled Riyad
MDEV-37335: crash on re-parsing an sp_instr_set that does not own its LEX

A DECLARE of several variables with a common DEFAULT expression creates one
sp_instr_set per variable, all sharing a single LEX that only the last of
them owns. On re-parsing after a metadata change, a non-owning instruction
still has its LEX, and both parse_expr() and validate_lex_and_exec_core()
took a non-null LEX to mean a cursor LEX. The cursor-only code then
dereferenced the nullptr returned by get_lex_for_cursor() and the re-parsed
LEX was never adopted.

Branch on whether the LEX is a cursor LEX instead of whether it is non-null.
Thirunarayanan Balathandayuthapani
MDEV-19574: innodb_stats_method is not honored when innodb_stats_persistent=ON

Problem:
=======
When persistent statistics are enabled (innodb_stats_persistent=ON),
the innodb_stats_method setting is not properly utilized during
statistics calculation.

The statistics collection functions always use a hardcoded default
behavior for NULL value comparison instead of respecting the
configured stats method. This affects the accuracy of
n_diff_key_vals (distinct key count), particularly for
indexes with nullable columns containing NULL values.

Moreover, stat_n_non_null_key_vals[] was never computed for
persistent statistics; it stayed at the 0 that
dict_stats_empty_index() assigns.

With innodb_stats_method=nulls_ignored, innodb_rec_per_key()
therefore always found n_diff <= n_null and reported one record
per key for every index. This impacts the query optimizer,
which makes decisions based on inaccurate cardinality estimates.

Solution:
========
Introduced IndexLevelStats to collect statistics at a specific
B-tree level during index analysis.

Introduced PageStats to collect statistics for leaf page analysis.

Refactored the following functions:
dict_stats_analyze_index_level() to IndexLevelStats::analyze_level()
dict_stats_analyze_index_for_n_prefix() to IndexLevelStats::sample_leaf_pages()
dict_stats_analyze_index_below_cur() to PageStats::scan_below()
dict_stats_scan_page() to PageStats::scan()

The innodb_stats_method value is read once per table in
dict_stats_update_persistent() and passed down, so that all
indexes of a table are analyzed with the same method.

Add the stats method name to stat_description when
innodb_stats_method has a non-default value. The suffix is
dropped when the description is already full.

Added the new stat name n_nonnull_fld01, n_nonnull_fld02, etc.
with a stats description, to indicate how many non-null values
exist for the nth field of the index. This value is retrieved
and stored in the index statistics
in dict_stats_fetch_index_stats_step(). The counts are per
column, not per n-column prefix.

rec_get_n_blob_pages(): Calculate the number of
externally stored pages for a record, using ceiling division
by the usable BLOB page payload (blob_part_size), which differs
between ROW_FORMAT=COMPRESSED (zip_size minus FIL_PAGE_DATA) and
the other formats (srv_page_size minus the BLOB header and the
page trailer). For ROW_FORMAT=COMPRESSED the length in
the field reference is the uncompressed length, so the result is an
upper bound.

When the leaf level is scanned in full, the number of leaf pages that
were scanned is reported as n_leaf_pages for a multi level index.
Before, result.n_leaf_pages was overwritten with
index->stat_n_leaf_pages, which dict_stats_empty_index() had
just set to 1, so every index that took the full scan path
reported n_leaf_pages=1. Single page indexes report 1.
This changes cardinality estimates and
therefore leads to multiple changes in existing test cases.

Non-null values are counted only at the leaf level, since only leaf
pages hold actual records. A full scan of the leaf level counts them
exactly. When the level is sampled, the per column count is derived
from the sampled leaves with the same formula as n_diff:

  n_ordinary_leaf_pages * n_non_null_all_analyzed_pages
                        / n_leaf_pages_to_analyze

This is an estimate for NOT NULL columns as well: the sampled leaves
may hold fewer or more records than the average, and a dive that
stops at a boring page contributes nothing to the sum while still
counting in the divisor.

innodb_rec_per_key(): stat_n_non_null_key_vals[i] holds the
number of records in which the i-th indexed column alone is
not NULL, while what has to be excluded here is the number
of records whose first i+1 columns are all not NULL,
because that is the population which the n-column
prefix statistic stat_n_diff_key_vals[i] has to be
corrected against when innodb_stats_method=nulls_ignored:
with NULLs compared as unequal, every record carrying a NULL
anywhere in the prefix adds a distinct value of its own to n_diff.

PageStats::scan(): n_non_null is accumulated and assigned only
for leaf pages, so that a non-leaf scan cannot leave a node
pointer count behind when scan_below() stops at a boring page
without reaching a leaf.

IndexLevelStats::reset_for_level() also clears n_diff[], and
dict_stats_analyze_index() zero initializes the buffer backing it, so
that a level scan which finds no records (a failed
btr_pcur_open_level(), or a non-leaf page whose first record is not
marked as the leftmost one on the level) leaves n_diff[] at 0 instead
of stale values.

IndexLevelStats::sample_leaf_pages() returns early when the group
boundaries for the prefix are empty, which is the same condition.

IndexLevelStats::analyze_level(): Instead of copying the last record
of the page, retain the latch on the page until the record has been
compared with the first record of the next page

dict_stats_fetch_index_stats_step() no longer resets
stat_n_non_null_key_vals[] while processing an n_diff_pfxNN row:
dict_stats_empty_table() has already cleared the array before the
fetch, and with n_nonnull_fldNN rows now being read too,
that reset would make the result depend on the order in which the
rows arrive.

dict_stats_save(): now static function in dict0stats.cc that
takes the innodb_stats_method value, and is removed from dict0stats.h.
dict_stats_update_persistent() saves the statistics itself, so its
callers no longer have to.

Replaced btr_rec_get_externally_stored_len() with
rec_get_n_blob_pages() in dict0stats.cc.

btr_rec_get_field_ref_offs() and btr_rec_get_field_ref(),
together with the BTR_BLOB_HDR_* macros, were moved from btr0cur.cc
to btr0cur.h so that rec_get_n_blob_pages() can reuse them;

btr_rec_get_field_ref_offs() is now a noexcept function
returning size_t.

Changed stat_n_diff_key_vals and stat_n_non_null_key_vals from
ib_uint64_t* to uint64_t*

len_is_stored(): simplified to a single comparison, which is
equivalent for the unsigned lengths that it is used with.

Removed the unused UNIV_STATS_DEBUG build macro (univ.i) and
turned the DEBUG_PRINTF() helper in dict0stats.cc into an
unconditional no-op
Vladislav Vaintroub
MDEV-41008: Fix X509 issuer/subject comparison for OpenSSL 3

OpenSSL 3 escapes '/' and '+' in X509_NAME_oneline() output;
OpenSSL 1.1 and WolfSSL don't. A REQUIRE ISSUER/SUBJECT grant from
one library can stop matching after switching to another.

Default comparison stays strcmp(). old_mode=X509_LENIENT_COMPARE
opts into ignoring the escaping backslash, at the cost of reopening
the single-RDN-vs-multi-RDN ambiguity a crafted certificate could
exploit to impersonate another identity.
Sergei Golubchik
fixup: do not remove WITH_WSREP

this creates ABI incompatiility. install wsrep headers instead
Thirunarayanan Balathandayuthapani
MDEV-26057 Assertion `!vcol->v_indexes.empty() in trx_undo_log_v_idx

Problem:
========
-  Rollback of an INPLACE ALTER TABLE is executed while holding only a
shared metadata lock on the table, so DML can run concurrently.
rollback_inplace_alter_table() resets dict_col_t::ord_part in a
critical section of its own, after row_merge_drop_indexes() already
removed the aborted indexes from the dictionary cache and emptied
dict_v_col_t::v_indexes. During this time, DML statement can see a
virtual column with ord_part set and an empty v_indexes, which
makes assert failure in trx_undo_report_insert_virtual().

Solution:
========
row_merge_reset_ord_part(): Added a function to reset
dict_col_t::ord_part for the columns that are no longer a field of
any index remaining in the dictionary cache.
For virtual columns the decision is based on dict_v_col_t::v_indexes
being empty, and no element is ever removed from that list.

row_merge_drop_indexes(): Added a call to row_merge_reset_ord_part()
in the branch that removes the indexes from the cache, in the same
dict_sys.latch critical section. That branch is taken only when
MDL_EXCLUSIVE is held or when this is the only handle to the table,
so no concurrent DML can observe the intermediate state.
In the lazy drop branch the indexes and their v_indexes entries
stay in the cache and nothing is reset; that is done later,
when the indexes are dropped while holding MDL_EXCLUSIVE.

check_col_exists_in_indexes(): Removed the only_committed parameter,
which no longer has any caller.

row_quiesce_col_ord_part(): Added a function to get
dict_col_t::ord_part and dict_col_t::max_prefix of a column
from the committed indexes that are
present in the dictionary cache.

row_quiesce_write_table(): Write the row_quiesce_col_ord_part() return
values to the .cfg file instead of the cached dict_col_t fields,
because a rolled back ADD INDEX leaves ord_part set until the
aborted index is removed by a later DDL, and
max_prefix is never reset when an index is dropped, which makes
IMPORT TABLESPACE reject the tablespace with a bogus schema mismatch.
Sergei Petrunia
A plain KEY is the only index type allowed over an ARRAY

    create table t1 (j json, unique key idx ((CAST(j->'$.a' AS CHAR(6) ARRAY))));
    create table t1 (j json, primary key ((CAST(j->'$.a' AS CHAR(6) ARRAY))));
    create table t1 (j json, fulltext key idx ((CAST(j->'$.a' AS CHAR(6) ARRAY))));

were all accepted without a word, and all produced the same thing: a plain
index. The key type the user wrote was simply overwritten with
Key::FULLTEXT, so the table ended up with no unique constraint, or no
primary key, or with a FULLTEXT index that MATCH() finds nothing in - the
index holds encoded element keys, not the text.

What the server builds for an ARRAY is a fulltext index over those encoded
elements, and it can only mean what a plain KEY means. Say so: reject any
other type, in the grammar, before that overwrite loses what was asked for.
CONSTRAINT ... UNIQUE and the ALTER TABLE forms go the same way. SPATIAL
and VECTOR are already syntax errors for an ARRAY key part; they are in the
switch anyway so it stays exhaustive.

FOREIGN KEY is unaffected: it builds its key with Key::MULTIPLE and cannot
be told apart here. It is rejected, further down, by the engine - "Foreign
key constraint is incorrectly formed".

The count of key parts is now also checked in the grammar, and not only in
init_key_part_spec(). Otherwise the first ARRAY part of a two-part key sets
the type to FULLTEXT, and the second part reports "Incorrect usage of
FULLTEXT and ARRAY" for a key nobody declared FULLTEXT. As a side effect
KEY idx (c,(CAST(... ARRAY))) now gives the same "max 1 parts" error as the
other orders, instead of ER_BAD_FT_COLUMN for `c'.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Sergei Petrunia
Only allow one key part in an index over an ARRAY

    create table t1 (j json, key idx ((CAST(j->'$.a' AS CHAR(6) ARRAY)),
                                      (CAST(j->'$.b' AS CHAR(6) ARRAY))));

was accepted without a word. Each ARRAY key part gets an internal column of
its own, and they all became key parts of one fulltext key: the tokens of
both arrays end up mixed in a single index, and the optimizer would then
search that index for the keys of one array and get the rows of the other as
well. There is also no way to show such a key, or to read one back.

init_key_part_spec() now rejects a key that has an ARRAY key part and more
than one key part, on both the CREATE TABLE and the ALTER TABLE path.

The other order, KEY idx (c,(CAST(... ARRAY))), was already rejected: the
ARRAY part makes the key FULLTEXT, and `c' cannot be part of one.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Aleksey Midenkov
MDEV-25529 Auto-create: Pre-existing historical data is not partitioned as specified by ALTER

Adds logic into prep_alter_part_table() for AUTO to check the history
range (vers_get_history_range()) and based on (max_ts - min_ts)
difference compute the number of created partitions and set STARTS
value to round down min_ts value (vers_set_starts()) if it was not
specified by user or if the user specified it incorrectly. In the
latter case it will print warning about wrongly specified user value.

In case of fast ALTER TABLE, f.ex. when partitioning already exists,
the above logic is ignored unless FORCE clause is specified. When user
specifies partition list explicitly the above logic is ignored even
with FORCE clause.

vers_get_history_range() detects if the index can be used for row_end
min/max stats and if so it gets it with ha_index_first() and
HA_READ_BEFORE_KEY (as it must ignore current data). Otherwise it does
table scan to read the stats. There is test_mdev-25529 debug keyword
to check the both and compare results. A warning is printed if the
algorithm uses slow scan.

Field_vers_trx_id::get_timestamp() is implemented for TRX_ID based
versioning to get epoch value. It works in vers_get_history_range()
but since partitioning is not enabled for TRX_ID versioning create
temporary table fails with error, requiring timestamp-based system
fields. This method will be useful when partitioning will be enabled
for TRX_ID which is mostly performance problems to solve.

Static key_cmp was renamed to key_eq to resolve compilation after
key.h was included as key_cmp was already declared there.
Sergei Petrunia
This is a combination of 7 commits.

Do the MVI analysis one table at a time

setup_mvi_quick() ran once per JOIN, from JOIN::optimize_inner(): it
collected the MV indexes of every leaf table into one list, walked the
WHERE clause once, and left the accesses it found in JOIN::mvi_ctx for
setup_mvi_access_for_table() to dig through, filtering by
access->index->vcol->table == tab->table.

Nothing about it needed to be JOIN-wide. Each JOIN_TAB now has its own
Mvi_context describing the access to its own table, and everything
setup_mvi_quick() did happens in setup_mvi_access_for_table(): collect
that table's MV indexes, analyze the condition, pick the access. The
context is only kept when there is an access to use, so the chosen one is
Mvi_context::best and JOIN_TAB has a single MVI member. The table filter
becomes a DBUG_ASSERT: ctx->indexes holds only this table's indexes and
get_mvi_index() matches the predicate against those with Item::eq(),
which compares Field pointers, so an access can only ever be on the table
whose column the predicate names - even when two tables carry identical
MVI definitions.

The condition to analyze comes from get_sargable_cond(), the same one the
range analysis of that table uses twenty lines later. For a table on the
inner side of an outer join that is the ON expression rather than the
WHERE clause, so MVI access now works there too. It is sound for the same
reason the range optimizer may do it: the scan is a necessary, not a
sufficient condition, the JSON predicate stays in the ON expression and
does the exact filtering, and outer rows that find no match are
NULL-complemented as usual. The new test checks both, against the same
queries with IGNORE INDEX.

Two consequences of the analysis no longer being a pre-pass:

- It runs on the condition the range optimizer will see, after
  simplify_joins(), substitute_indexed_vcols_for_join() and
  optimize_cond(), rather than on the freshly parsed WHERE.

- Const tables are not analyzed at all, the per-table loop having
  skipped them before this point. They never get range analysis.

JOIN_TAB::mvi_ctx also needs no explicit reset between re-optimizations:
make_join_statistics() bzeroes the JOIN_TAB array.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

Make innodb_fts.estimate's clamped count deterministic

The "deleted rows are still counted" case failed about one full-suite run
in three:

  -Note 1105 fulltext_estimate('gamma')= 4
  +Note 1105 fulltext_estimate('gamma')= 5

The number it checks is the clamp in ha_innobase::fulltext_estimate(),
which is dict_table_get_n_rows() - the table statistics. The DELETE just
before it changes half the rows, which queues a background statistics
recalculation, and whether that has run by the time of the SELECT depends
on how loaded the machine is.

ANALYZE TABLE after the DELETE recalculates them on the spot and clears
the counter that would have triggered the background one, so the clamp
has one value to report.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

Rename collect_mvi_vcols_for_table to collect_mvi_indexes_for_table

It collects Mv_index objects, not vcols.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

Keep only the chosen Mvi_access in JOIN_TAB

JOIN_TAB held the whole Mvi_context the analysis produced, but outside
setup_mvi_access_for_table() the only thing ever read out of it was
mvi_ctx->best. The rest is scratch: indexes feeds get_mvi_index(),
accesses is what the last-wins loop picks best out of, and thd is there
for the mvi_analyze() callbacks.

So JOIN_TAB keeps the access itself, and the context becomes a local of
setup_mvi_access_for_table() - which is what the TODO there asked for:
nothing is allocated for a table that has no MVI key, or whose condition
yields no access. The access outliving the context is safe because
neither it nor the Mv_index it refers to belongs to the context: both are
allocated on the MEM_ROOT, and the lists only hold link nodes.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

Show multi-valued indexes in SHOW CREATE TABLE

    create table t1 (c int, j json,
                    key idx ((CAST(j->'$.tags' AS CHAR(6) ARRAY))));

printed the two columns and no index at all. Nobody had decided to hide it:
it inherited invisibility from the internal column that backs it. The
grammar makes DB_MVI_<n> INVISIBLE_FULL, init_from_binary_frm_image() turns
a hidden key part into a hidden key, and store_create_info() skips keys with
HA_INVISIBLE_KEY. Long unique hash keys - the other kind of key built over a
column the user cannot name - are already exempted from that; exempt the
multi-valued index the same way, and print it as

  KEY `idx` ((cast(json_extract(`j`,'$.tags') as char(6) array)))

which is the expression it was declared with and all it takes to re-create
it. The internal column stays out of the output: it cannot be printed as a
column, since there is no syntax that would recreate the pairing.

Item_func_mvi_encode::print() cannot produce that form. Its output is what
pack_expression() puts in the FRM, and that is read back as a call of
mvi_encode(), the only form the parser accepts outside an index definition.
So the CAST spelling gets a printer of its own, sharing the type printing.

SHOW INDEX and I_S.STATISTICS list the key now too - the key part is let
through the invisibility filter - so they no longer need
debug_dbug=test_invisible_index, and the tests stop setting it (it also
injected a stray invisible1 column and key into their output).

The same HA_INVISIBLE_KEY drove mysql_prepare_alter_table(), which drops
such keys from the list of keys carried into the rebuilt table - and
INVISIBLE_FULL columns from the list of columns. So

    ALTER TABLE t1 ADD COLUMN x INT;

silently dropped the index. The key survives now, and its column is carried
over with it, for exactly as long as the key lives: DROP KEY takes the
column with it, so the name is free again afterwards.

While at it, make_internal_field_name() looped forever when create_list is
empty: dup_found started at true and the loop that clears it does not run.
The MVI path is the only caller that can hit that, and it does - with
ALTER TABLE ... ADD KEY ((CAST(... ARRAY))), which used to hang the server
and now works.

A fulltext key over several arrays,

    KEY idx ((CAST(j->'$.a' AS CHAR(6) ARRAY)),
            (CAST(j->'$.b' AS CHAR(6) ARRAY)))

has no single expression to print and no syntax of its own to be read back,
so it stays hidden, exactly as before. The optimizer still uses its parts.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

Only allow one key part in an index over an ARRAY

    create table t1 (j json, key idx ((CAST(j->'$.a' AS CHAR(6) ARRAY)),
                                      (CAST(j->'$.b' AS CHAR(6) ARRAY))));

was accepted without a word. Each ARRAY key part gets an internal column of
its own, and they all became key parts of one fulltext key: the tokens of
both arrays end up mixed in a single index, and the optimizer would then
search that index for the keys of one array and get the rows of the other as
well. There is also no way to show such a key, or to read one back.

init_key_part_spec() now rejects a key that has an ARRAY key part and more
than one key part, on both the CREATE TABLE and the ALTER TABLE path.

The other order, KEY idx (c,(CAST(... ARRAY))), was already rejected: the
ARRAY part makes the key FULLTEXT, and `c' cannot be part of one.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

A plain KEY is the only index type allowed over an ARRAY

    create table t1 (j json, unique key idx ((CAST(j->'$.a' AS CHAR(6) ARRAY))));
    create table t1 (j json, primary key ((CAST(j->'$.a' AS CHAR(6) ARRAY))));
    create table t1 (j json, fulltext key idx ((CAST(j->'$.a' AS CHAR(6) ARRAY))));

were all accepted without a word, and all produced the same thing: a plain
index. The key type the user wrote was simply overwritten with
Key::FULLTEXT, so the table ended up with no unique constraint, or no
primary key, or with a FULLTEXT index that MATCH() finds nothing in - the
index holds encoded element keys, not the text.

What the server builds for an ARRAY is a fulltext index over those encoded
elements, and it can only mean what a plain KEY means. Say so: reject any
other type, in the grammar, before that overwrite loses what was asked for.
CONSTRAINT ... UNIQUE and the ALTER TABLE forms go the same way. SPATIAL
and VECTOR are already syntax errors for an ARRAY key part; they are in the
switch anyway so it stays exhaustive.

FOREIGN KEY is unaffected: it builds its key with Key::MULTIPLE and cannot
be told apart here. It is rejected, further down, by the engine - "Foreign
key constraint is incorrectly formed".

The count of key parts is now also checked in the grammar, and not only in
init_key_part_spec(). Otherwise the first ARRAY part of a two-part key sets
the type to FULLTEXT, and the second part reports "Incorrect usage of
FULLTEXT and ARRAY" for a key nobody declared FULLTEXT. As a side effect
KEY idx (c,(CAST(... ARRAY))) now gives the same "max 1 parts" error as the
other orders, instead of ER_BAD_FT_COLUMN for `c'.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Alexander Barkov
MDEV-39518 Allow prepared statements in stored functions in assignment right hand

Allowing prepared statements in stored functions when
a stored function is used in an assignment right hand.

Both DEFAULT clause of a variable initialization and
the right side of the SET statement are supported:

  CREATE PROCEDURE p1()
  BEGIN
    -- case 1: DEFAULT clause
    DECLARE spvar1 INT DEFAULT f1_with_ps(); -- OK

    -- case 2: SP variable assignment statement
    DECLARE spvar2 INT;
    SET spvar2= f1_with_ps(); -- OK
  END;

- Only assignments to SP variables works for now:
  * SET spvar= func_with_ps(); -- OK
  * SET @uvar= func_with_ps(); -- Error

- Only bare function calls are supported for now. Using a function in
  an expression does not make it PS-safe yet:
    SET v= f1()+0;

- The parser now does not reject PS statements in stored functions.
  PS applicability in stored functions is now detected at run time.
  Note, PS statements in triggers are still prohibited by the parser.

- Functions with PS do not acquire MDL locks on tables, and no MDL is
  taken on the routines themselves either. They work like procedures in
  terms of table opening and routine locking: a concurrent DROP FUNCTION
  can complete while such a function is executing.

- Functions with PS are not replicated as a single `SELECT f1()` call.
  They are replicated per-statement, like procedures.

Helper changes:
- Changing the return result for LEX::sp_variable_declarations_init()
  from void to bool to catch errors in the caller properly.

Misc:
- This patch incorporates fixes for the following bugs found during debugging:
  MDEV-40224,MDEV-40225,MDEV-40226,MDEV-40227,MDEV-40240,
  MDEV-40285,MDEV-40288,MDEV-40315,MDEV-40318,MDEV-40890,MDEV-40900,
  MDEV-40901,MDEV-40913,MDEV-40914,MDEV-41013,MDEV-41015,MDEV-41019

Assisted-by: Claude - reviews and minor clean-ups
Alexander Barkov
MDEV-39518 Allow prepared statements in stored functions in assignment right hand

Allowing prepared statements in stored functions when
a stored function is used in an assignment right hand.

Both DEFAULT clause of a variable initialization and
the right side of the SET statement are supported:

  CREATE PROCEDURE p1()
  BEGIN
    -- case 1: DEFAULT clause
    DECLARE spvar1 INT DEFAULT f1_with_ps(); -- OK

    -- case 2: SP variable assignment statement
    DECLARE spvar2 INT;
    SET spvar2= f1_with_ps(); -- OK
  END;

- Only assignments to SP variables works for now:
  * SET spvar= func_with_ps(); -- OK
  * SET @uvar= func_with_ps(); -- Error

- Only bare function calls are supported for now. Using a function in
  an expression does not make it PS-safe yet:
    SET v= f1()+0;

- The parser now does not reject PS statements in stored functions.
  PS applicability in stored functions is now detected at run time.
  Note, PS statements in triggers are still prohibited by the parser.

- Functions with PS do not acquire MDL locks on tables, and no MDL is
  taken on the routines themselves either. They work like procedures in
  terms of table opening and routine locking: a concurrent DROP FUNCTION
  can complete while such a function is executing.

- Functions with PS are not replicated as a single `SELECT f1()` call.
  They are replicated per-statement, like procedures.

Helper changes:
- Changing the return result for LEX::sp_variable_declarations_init()
  from void to bool to catch errors in the caller properly.

Misc:
- This patch incorporates fixes for the following bugs found during debugging:
  MDEV-40224,MDEV-40225,MDEV-40226,MDEV-40227,MDEV-40240,
  MDEV-40285,MDEV-40288,MDEV-40315,MDEV-40318,MDEV-40890,MDEV-40900,
  MDEV-40901,MDEV-40913,MDEV-40914,MDEV-41013,MDEV-41015,MDEV-41019

Assisted-by: Claude - reviews and minor clean-ups
Vladislav Vaintroub
MDEV-40608 propagate DBUG_OFF, ENABLED_DEBUG_SYNC and SAFE_MUTEX to external plugins

they affect ABI, but aren't in headers, so must be passed separately
Sergei Petrunia
Show multi-valued indexes in SHOW CREATE TABLE

    create table t1 (c int, j json,
                    key idx ((CAST(j->'$.tags' AS CHAR(6) ARRAY))));

printed the two columns and no index at all. Nobody had decided to hide it:
it inherited invisibility from the internal column that backs it. The
grammar makes DB_MVI_<n> INVISIBLE_FULL, init_from_binary_frm_image() turns
a hidden key part into a hidden key, and store_create_info() skips keys with
HA_INVISIBLE_KEY. Long unique hash keys - the other kind of key built over a
column the user cannot name - are already exempted from that; exempt the
multi-valued index the same way, and print it as

  KEY `idx` ((cast(json_extract(`j`,'$.tags') as char(6) array)))

which is the expression it was declared with and all it takes to re-create
it. The internal column stays out of the output: it cannot be printed as a
column, since there is no syntax that would recreate the pairing.

Item_func_mvi_encode::print() cannot produce that form. Its output is what
pack_expression() puts in the FRM, and that is read back as a call of
mvi_encode(), the only form the parser accepts outside an index definition.
So the CAST spelling gets a printer of its own, sharing the type printing.

SHOW INDEX and I_S.STATISTICS list the key now too - the key part is let
through the invisibility filter - so they no longer need
debug_dbug=test_invisible_index, and the tests stop setting it (it also
injected a stray invisible1 column and key into their output).

The same HA_INVISIBLE_KEY drove mysql_prepare_alter_table(), which drops
such keys from the list of keys carried into the rebuilt table - and
INVISIBLE_FULL columns from the list of columns. So

    ALTER TABLE t1 ADD COLUMN x INT;

silently dropped the index. The key survives now, and its column is carried
over with it, for exactly as long as the key lives: DROP KEY takes the
column with it, so the name is free again afterwards.

While at it, make_internal_field_name() looped forever when create_list is
empty: dup_found started at true and the loop that clears it does not run.
The MVI path is the only caller that can hit that, and it does - with
ALTER TABLE ... ADD KEY ((CAST(... ARRAY))), which used to hang the server
and now works.

A fulltext key over several arrays,

    KEY idx ((CAST(j->'$.a' AS CHAR(6) ARRAY)),
            (CAST(j->'$.b' AS CHAR(6) ARRAY)))

has no single expression to print and no syntax of its own to be read back,
so it stays hidden, exactly as before. The optimizer still uses its parts.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Rex Johnston
PQ: tidy up some of the mess, remove pointless code.

reworked the code in sql_select.cc to make it clearer and shift more
out.
Huge tidy up.
Sergei Golubchik
don't install config.h

high chance of name conflict. not used by any other headers
identical to my_config.h (which is used by other headers), so redundant.
Sergei Golubchik
a.test: fix encoding
Yuchen Pei
MDEV-40168 Fix cast to int arrays