Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Jan Lindström
Merge remote-tracking branch 'origin/MDEV-40281' into 4.ee-MDEV-40281
Rex Johnston
Parallel Query: count a worker in under the lock that counts it out

A query could hang for good, with the manager waiting on COND_data_avail for a
worker that had already finished and gone.

active_workers is decremented in report_worker_final_state() under LOCK_data,
and read in locked__process_manager_wakeup() with LOCK_data held, but
register_worker() was incrementing it under no lock at all. The two do run at
the same time: init_parallel_workers() builds the team one worker at a time and
starts each thread as it creates it, so worker 1 can be counting itself out
while worker 2 is being counted in.

  manager thread                      worker 1
  ------------------------------      ---------------------------------
  new pwt_worker  -> reads 2
                                      thread_func_end()
                                        LOCK_data, reads 2, writes 1
  writes 3

The decrement is lost and the count stands one too high. From then on
claim_next_result() waits for a worker that will never publish a result and
never signal again, because every worker there was has already exited.

Taking LOCK_data around the increment is the whole fix.

It wants a small table and enough load: a table the engine divides into one
chunk leaves the other workers with nothing to scan, so they reach
thread_func_end() almost at once, in the window while the rest of the team is
still being built. main.function_defaults_innodb is such a test, and this
reproduced it within six attempts:

  ./mtr --parallel=16 --mem --force --ps \
        --mysqld=--parallel-worker-threads=4 \
        main.function_defaults_innodb main.function_defaults_innodb \
        main.function_defaults_innodb main.function_defaults_innodb \
        main.function_defaults_innodb main.function_defaults_innodb

It now runs clean twelve times over. No test is added: a lost update needs a
race to be lost, and a test that only sometimes fails is worse than none.

This commit was prepared with Claude Code: it reproduced the hang, read the
manager's bookkeeping out of the stopped server with gdb -- active_workers 1
against four workers, all four with a destroyed THD and the statistics that
only thread_func_end() writes, so all four had decremented -- and reasoned back
from four decrements and a count of one to a lost increment.
Dmitry Shulga
MDEV-40076: ASAN global-buffer-overflow in reconstruct_create_trigger_stmt

Follow-up patch to address review comments found by reviewer
Alexander Barkov
MDEV-39518 Allow prepared statements in stored functions in assignment right hand

Allowing prepared statements in stored functions when
a stored function is used in an assignment right hand.

Both DEFAULT clause of a variable initialization and
the right side of the SET statement are supported:

  CREATE PROCEDURE p1()
  BEGIN
    -- case 1: DEFAULT clause
    DECLARE spvar1 INT DEFAULT f1_with_ps(); -- OK

    -- case 2: SP variable assignment statement
    DECLARE spvar2 INT;
    SET spvar2= f1_with_ps(); -- OK
  END;

- Only assignments to SP variables works for now:
  * SET spvar= func_with_ps(); -- OK
  * SET @uvar= func_with_ps(); -- Error

- Only bare function calls are supported for now. Using a function in
  an expression does not make it PS-safe yet:
    SET v= f1()+0;

- The parser now does not reject PS statements in stored functions.
  PS applicability in stored functions is now detected at run time.
  Note, PS statements in triggers are still prohibited by the parser.

- Functions with PS do not acquire MDL locks on tables, and no MDL is
  taken on the routines themselves either. They work like procedures in
  terms of table opening and routine locking: a concurrent DROP FUNCTION
  can complete while such a function is executing.

- Functions with PS are not replicated as a single `SELECT f1()` call.
  They are replicated per-statement, like procedures.

Helper changes:
- Changing the return result for LEX::sp_variable_declarations_init()
  from void to bool to catch errors in the caller properly.

Misc:
- This patch incorporates fixes for the following bugs found during debugging:
  MDEV-40224,MDEV-40225,MDEV-40226,MDEV-40227,MDEV-40240,
  MDEV-40285,MDEV-40288,MDEV-40315,MDEV-40318,MDEV-40890,MDEV-40900,
  MDEV-40901,MDEV-40913,MDEV-40914,MDEV-41013,MDEV-41015,MDEV-41019

Assisted-by: Claude - reviews and minor clean-ups
Alexander Barkov
MDEV-39518 Allow prepared statements in stored functions in assignment right hand

Allowing prepared statements in stored functions when
a stored function is used in an assignment right hand.

Both DEFAULT clause of a variable initialization and
the right side of the SET statement are supported:

  CREATE PROCEDURE p1()
  BEGIN
    -- case 1: DEFAULT clause
    DECLARE spvar1 INT DEFAULT f1_with_ps(); -- OK

    -- case 2: SP variable assignment statement
    DECLARE spvar2 INT;
    SET spvar2= f1_with_ps(); -- OK
  END;

- Only assignments to SP variables works for now:
  * SET spvar= func_with_ps(); -- OK
  * SET @uvar= func_with_ps(); -- Error

- Only bare function calls are supported for now. Using a function in
  an expression does not make it PS-safe yet:
    SET v= f1()+0;

- The parser now does not reject PS statements in stored functions.
  PS applicability in stored functions is now detected at run time.
  Note, PS statements in triggers are still prohibited by the parser.

- Functions with PS do not acquire MDL locks on tables, and no MDL is
  taken on the routines themselves either. They work like procedures in
  terms of table opening and routine locking: a concurrent DROP FUNCTION
  can complete while such a function is executing.

- Functions with PS are not replicated as a single `SELECT f1()` call.
  They are replicated per-statement, like procedures.

Helper changes:
- Changing the return result for LEX::sp_variable_declarations_init()
  from void to bool to catch errors in the caller properly.

Misc:
- This patch incorporates fixes for the following bugs found during debugging:
  MDEV-39518,MDEV-40224,MDEV-40225,MDEV-40226,MDEV-40227,MDEV-40240,
  MDEV-40285,MDEV-40288,MDEV-40315,MDEV-40318,MDEV-40890,MDEV-40900,
  MDEV-40901,MDEV-40913,MDEV-40914,MDEV-41013,MDEV-41015,MDEV-41019

Assisted-by: Claude - reviews and minor clean-ups
Sergei Petrunia
MDEV-40168: JSON-over-fulltext: let the estimate consult the FTS cache.

fts_estimate_word_docs() probed only the on-disk auxiliary INDEX_[1..6]
table.  Documents inserted but not SYNCed yet are only in the in-memory
FTS cache, so they were missed entirely, and on a table that has never
been SYNCed the auxiliary table is empty and the estimate was simply
"unknown".

Look in the cache as well.  fts_index_cache_t::words is an rb tree of
fts_tokenizer_word_t, and fts_node_t::doc_count already holds the number
of documents in the node's ilist, so this is one rbt_search plus a walk
over a short vector: no ilist decoding and no I/O.

Three details:

  - the cache mutex is taken with trylock.  This runs during
    optimization, where a SYNC holding cache->lock across SQL execution
    would stall the optimizer; dropping the cache contribution is the
    better trade.

  - nodes flagged fts_node_t::synced are skipped.  An in-flight SYNC has
    already written them out, and the estimator reads the B-tree without
    a read view, so it sees those records; counting the node too would
    count its documents twice.

  - index_cache->words is NULL between fts_cache_clear() and
    fts_cache_init().  The query path never observes that because it
    runs after fts_init_index(); the estimate does not run it.

DB_RECORD_NOT_FOUND now means both sources are empty.  The cache on its
own cannot prove a word absent, because it is only complete once
fts_init_index() has run, and an estimate must have no side effects.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Teemu Ollakka
Fix fragile branch name parsing
Sergei Petrunia
Keep the MVI access of a table in its JOIN_TAB

The access was looked up through JOIN::get_mvi_access_for_table(), which
indexed a per-JOIN array by table->tablenr. Nothing else about how a
table is going to be read lives there, so put it where the rest does:
JOIN_TAB::mvi_access. Mvi_context is then just the result of the WHERE
analysis - the indexes and the accesses it found - and the choice of
which access a table uses is made per table, where it belongs.

setup_mvi_access_for_table() makes that choice and marks the index in
const_keys and keys, which make_join_statistics() used to do in a loop of
its own right after update_ref_and_keys(). It runs next to
add_group_and_distinct_keys(), the other place that adds to const_keys
for something the range optimizer would not find by itself, and just
before the range analysis those bits exist for.

get_quick_record_count() takes the JOIN_TAB rather than the TABLE now,
which is all it needed the JOIN for.

Also fix the header comment of Mvi_access::estimate_records(), which
still described the old fallback for a conjunctive access that could not
be estimated at all.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Teemu Ollakka
Merge remote-tracking branch 'origin/4.x-MDEV-36926' into 4.ee-MDEV-36926
Jan Lindström
Merge remote-tracking branch 'origin/4.x-26.4.27' into 4.ee
Alexander Barkov
MDEV-39518 Allow prepared statements in stored functions in assignment right hand

Allowing prepared statements in stored functions when
a stored function is used in an assignment right hand.

Both DEFAULT clause of a variable initialization and
the right side of the SET statement are supported:

  CREATE PROCEDURE p1()
  BEGIN
    -- case 1: DEFAULT clause
    DECLARE spvar1 INT DEFAULT f1_with_ps(); -- OK

    -- case 2: SP variable assignment statement
    DECLARE spvar2 INT;
    SET spvar2= f1_with_ps(); -- OK
  END;

- Only assignments to SP variables works for now:
  * SET spvar= func_with_ps(); -- OK
  * SET @uvar= func_with_ps(); -- Error

- Only bare function calls are supported for now. Using a function in
  an expression does not make it PS-safe yet:
    SET v= f1()+0;

- The parser now does not reject PS statements in stored functions.
  PS applicability in stored functions is now detected at run time.
  Note, PS statements in triggers are still prohibited by the parser.

- Functions with PS do not acquire MDL locks on tables, and no MDL is
  taken on the routines themselves either. They work like procedures in
  terms of table opening and routine locking: a concurrent DROP FUNCTION
  can complete while such a function is executing.

- Functions with PS are not replicated as a single `SELECT f1()` call.
  They are replicated per-statement, like procedures.

Helper changes:
- Changing the return result for LEX::sp_variable_declarations_init()
  from void to bool to catch errors in the caller properly.

Misc:
- This patch incorporates fixes for the following bugs found during debugging:
  MDEV-39518,MDEV-40224,MDEV-40225,MDEV-40226,MDEV-40227,MDEV-40240,
  MDEV-40285,MDEV-40288,MDEV-40315,MDEV-40318,MDEV-40890,MDEV-40900,
  MDEV-40901,MDEV-40913,MDEV-40914,MDEV-41013,MDEV-41015,MDEV-41019

Assisted-by: Claude - reviews and minor clean-ups
Alexander Barkov
MDEV-39518 Allow prepared statements in stored functions in assignment right hand

Allowing prepared statements in stored functions when
a stored function is used in an assignment right hand.

Both DEFAULT clause of a variable initialization and
the right side of the SET statement are supported:

  CREATE PROCEDURE p1()
  BEGIN
    -- case 1: DEFAULT clause
    DECLARE spvar1 INT DEFAULT f1_with_ps(); -- OK

    -- case 2: SP variable assignment statement
    DECLARE spvar2 INT;
    SET spvar2= f1_with_ps(); -- OK
  END;

- Only assignments to SP variables works for now:
  * SET spvar= func_with_ps(); -- OK
  * SET @uvar= func_with_ps(); -- Error

- Only bare function calls are supported for now. Using a function in
  an expression does not make it PS-safe yet:
    SET v= f1()+0;

- The parser now does not reject PS statements in stored functions.
  PS applicability in stored functions is now detected at run time.
  Note, PS statements in triggers are still prohibited by the parser.

- Functions with PS do not acquire MDL locks on tables, and no MDL is
  taken on the routines themselves either. They work like procedures in
  terms of table opening and routine locking: a concurrent DROP FUNCTION
  can complete while such a function is executing.

- Functions with PS are not replicated as a single `SELECT f1()` call.
  They are replicated per-statement, like procedures.

Helper changes:
- Changing the return result for LEX::sp_variable_declarations_init()
  from void to bool to catch errors in the caller properly.

Misc:
- This patch incorporates fixes for the following bugs found during debugging:
  MDEV-40224,MDEV-40225,MDEV-40226,MDEV-40227,MDEV-40240,
  MDEV-40285,MDEV-40288,MDEV-40315,MDEV-40318,MDEV-40890,MDEV-40900,
  MDEV-40901,MDEV-40913,MDEV-40914,MDEV-41013,MDEV-41015,MDEV-41019

Assisted-by: Claude - reviews and minor clean-ups
Sergei Petrunia
Estimate the number of records an MVI access will read

QUICK_MVI_SELECT carried records=10 and read_time=0.001, numbers picked
low enough that the access always won over a table scan. Ask the engine
instead, through the fulltext_estimate() added by the previous commit.

Mvi_access::estimate_records() estimates one element key at a time and
combines the answers the way the query combines the keys:

- A disjunctive access (JSON_OVERLAPS) reads the rows of every key, so
  the estimates add up.

- A conjunctive access (JSON_CONTAINS) reads the rows that have all of
  the keys, so the rarest key alone bounds the result. We use its
  estimate and drop the other keys from the query: reading the rarest
  key and letting the WHERE clause discard the rest is not worse than
  having the engine intersect the terms. This is the trade-off
  collect_mvi_keys() already makes for the keys it cannot encode - a
  shorter AND matches a superset of the rows, and the JSON predicate
  does the exact filtering.

The engine may be unable to estimate a key: ha_innobase only looks at
the fulltext auxiliary tables, so until the words are flushed out of the
FTS cache the answer is "unknown" for everything. Such a key takes no
part in the choice of the rarest one, and if not a single key could be
estimated the old guess stands and the query is left as it is.

For an OR we cannot do that: we have to read that key and have no idea
what it costs. Give the access a DBL_MAX read_time and do not use it.
That has to be acted on in get_best_mvi_access() rather than left to the
cost comparison, because best_access_path() takes a quick select to be
cheaper than a table scan without checking - true of anything the range
optimizer proposes, but this access does not come from there.

read_time is now the cost of reading the estimated rows plus evaluating
the WHERE clause on them, which is what the join optimizer expects of a
quick select's read_time. It does not account for the fulltext search
that produces the rowids in the first place.

The trace prints the estimate, or says the access is unusable and why.

The two existing tests ran on tables whose words were still in the FTS
cache, so the JSON_OVERLAPS sections would have stopped using the index
entirely. They now flush the cache with OPTIMIZE TABLE under
innodb_optimize_fulltext_only, the way innodb_fts.estimate does. The
trace test gets a table with a skewed distribution ("bbb" in every row,
"aaa" in one) to show the conjunctive access keeping only the rarest key
and still producing the same rows as the same query with IGNORE INDEX.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Thirunarayanan Balathandayuthapani
MDEV-26057 Assertion `!vcol->v_indexes.empty() in trx_undo_log_v_idx

Problem:
========
-  Rollback of an INPLACE ALTER TABLE is executed while holding only a
shared metadata lock on the table, so DML can run concurrently.
rollback_inplace_alter_table() resets dict_col_t::ord_part in a
critical section of its own, after row_merge_drop_indexes() already
removed the aborted indexes from the dictionary cache and emptied
dict_v_col_t::v_indexes. During this time, DML statement can see a
virtual column with ord_part set and an empty v_indexes, which
makes assert failure in trx_undo_report_insert_virtual().

Solution:
========
row_merge_reset_ord_part(): Added a function to reset
dict_col_t::ord_part for the columns that are no longer a field of
any index remaining in the dictionary cache.
For virtual columns the decision is based on dict_v_col_t::v_indexes
being empty, and no element is ever removed from that list.

row_merge_drop_indexes(): Added a call to row_merge_reset_ord_part()
in the branch that removes the indexes from the cache, in the same
dict_sys.latch critical section. That branch is taken only when
MDL_EXCLUSIVE is held or when this is the only handle to the table,
so no concurrent DML can observe the intermediate state.
In the lazy drop branch the indexes and their v_indexes entries
stay in the cache and nothing is reset; that is done later,
when the indexes are dropped while holding MDL_EXCLUSIVE.

check_col_exists_in_indexes(): Removed the only_committed parameter,
which no longer has any caller.

row_quiesce_col_ord_part(): Added a function to get
dict_col_t::ord_part and dict_col_t::max_prefix of a column
from the committed indexes that are
present in the dictionary cache.

row_quiesce_write_table(): Write the row_quiesce_col_ord_part() return
values to the .cfg file instead of the cached dict_col_t fields,
because a rolled back ADD INDEX leaves ord_part set until the
aborted index is removed by a later DDL, and
max_prefix is never reset when an index is dropped, which makes
IMPORT TABLESPACE reject the tablespace with a bogus schema mismatch.
Alexey Yurchenko
MDEV-38920 MTR tests for Galera-side fixes

MDEV-38920-evs-config-warn checks that there is a warning about bad configuration
values and they are not accepted.
MDEV-38920-install-timer-expired reproduces 'install timer expired' situation.
Both tests require fixed Galera library to pass.
Jan Lindström
MDEV-40281 Tolerate a concurrently closed socket in set_fd_options()

set_fd_options() threw when the socket had already been closed before the
connect or accept completion handler ran. The gu::Exception escaped the
handler and left the gcomm event loop, taking the whole backend down.
Return early instead and let the operation which follows report the error.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Teemu Ollakka
Merge branch '4.x-fix-ghprb-mysql-branch-parsing' into 4.ee-fix-ghprb-mysql-branch-parsing
Sergei Petrunia
Trivial cleanups and comments
Jan Lindström
MDEV-40281 : galera.galera_wsrep_new_cluster test failure

Rejoining with an emptied datadir requires a full SST, which on a loaded
machine does not finish within the 60 seconds galera_wait_ready.inc allowed,
aborting the test while the SST was still running. Let the readiness wait
take an optional $galera_wait_ready_timeout and give that restart 300
seconds.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Sergei Petrunia
MDEV-40168: JSON-over-fulltext: add estimates.

Add records_in_range-like estimates for fulltext index
Jan Lindström
Merge remote-tracking branch 'origin/4.x-26.4.28' into 4.ee-26.4-28
Sergei Petrunia
Move the JSON function MVI code into opt_mvi_jsonfuncs.cc

opt_multi_valued_index.cc held two separate concerns: the index side
(how a value is encoded into the index, the access descriptor, the quick
select) and the predicate side (which JSON functions can be computed
from an MVI, which of their arguments holds the indexed expression, and
what to search the index for). Split the second one out.

Moved verbatim:

  get_mvi_index()
  collect_mvi_keys()
  Item_func_json_contains::get_mvi_access() and ::mvi_analyze()
  Item_func_json_overlaps::get_mvi_access() and ::mvi_analyze()
  add_mvi_access()

The only code change is that encode_mvi_key() is no longer static: it is
used both by Item_func_mvi_encode::val_str_ascii(), which stays, and by
collect_mvi_keys(), which moves. It is declared in
opt_multi_valued_index.h now. The other three moved helpers had no
callers outside the moved code and stay static.

Item_func_mvi_encode is not a JSON function and stays put: it is how the
values get into the index in the first place.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Dmitry Shulga
MDEV-40076: ASAN global-buffer-overflow in reconstruct_create_trigger_stmt

Follow-up patch to address review comments found by reviewer
Dmitry Shulga
MDEV-40952: CREATE TRIGGER fails with ER_EVENT_STORE_FAILED after unrelated failed ALTER

Attempt to create a system trigger right after the previous statement
finished with warning, resulted in execution failure with diagnostics:
  'ER_EVENT_STORE_FAILED (1538): Failed to store event on_completion.'

The reason of failure is that on storing metadata of a system trigger
the value ON_COMPLETION_DEFAULT was used for storing data in the column
`on_completion` of the table mysql.event. The enumerator
ON_COMPLETION_DEFAULT has numeric value 0 that is considered as invalid.
In case previous statement finished with warning, it would resulted
in raising error since THD::count_cuted_fields would greater than
the value CHECK_FIELD_EXPRESSION and the following pice of code
be executed in the method Field_enum::store()
    if (nr != 0 || get_thd()->count_cuted_fields > CHECK_FIELD_EXPRESSION)
    {
      nr= 0;
      error= 1;
    }

To fix the issue use the value ON_COMPLETION_PRESERVE for the column
`on_completion` when metadata for the new system trigger has to be stored.
Alexey Yurchenko
Merge branch 'MDEV-38920-ensure-timeouts' into MDEV-38920-ensure-timeouts-4.ee
Dmitry Shulga
MDEV-40947: ASAN global-buffer-overflow in events_to_string/reconstruct_create_trigger_stmt

Manual set of mysql.event.kind to the value 'DDL' resulted in crash
on running the statement SHOW CREATE TRIGGER.

The crash was caused on accessing the array base_event_names
by index for DDL trigger event, but the array missing this
entry.

To fix the issue, add an entry for the DDL event in
the array base_event_names.
Sergei Petrunia
MDEV-40168: JSON-over-fulltext: move the estimator into fts0est.cc.

Pure code motion, no functional change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Dmitry Shulga
MDEV-40105: Information_schema.triggers doesn't show schema for system triggers

On querying data from information schema about ON STARTUP/ON SHUTDOWN triggers
original implementation of the task MDEV-30645 erroneously considered
such triggers as not associated with any schema that is incorrect assumption.
System triggers like any other objects are stored in some schemas so information
about their schema should be output in query from information_schema.triggers
Jan Lindström
Merge remote-tracking branch 'upstream/4.ee' into mariadb-4.x
Rex Johnston
Parallel Query: the gate says why it turned a query away

"Why did this not run in parallel" was a question the reader had to answer
themselves, from a plan, in a debugger, some way from the query that provoked
it. It is now a question the server answers:

  SELECT JSON_EXTRACT(trace, '$**.parallel_scan_declined_because')
    FROM information_schema.optimizer_trace;

  ["grouped: the aggregation table is keyed by a unique constraint rather than
    by the group -- create_tmp_table() does that for a GROUP BY element too
    wide for a key part, and a hash of the row is not something a group can be
    looked up in",
  "the plan needs a temporary table and the workers cannot pre-aggregate into
    it"]

The first entry is the check that objected; the rest is that refusal working
its way back up. Every one of the gate's refusals now goes through
pwt_decline(), which keeps the DBUG_PRINT and adds the reason to the trace.
Sixteen of them had neither before -- a bare "return false" -- and those were
in the two functions a grouped query spends most of its time in.

Three things this needed beyond saying the words.

The gate is asked twice. Once from make_join_readinfo() while the plan is still
being built, and again from parallel_join_check() when it is finished. The
early call runs before make_aggr_tables_info() has built the aggregation table,
so it reported "the terminal is not end_update()" for every grouped query --
an answer about a plan that no longer exists by the time the query runs, and a
worse place to start than no answer at all. A trace flag threaded through the
gate means only the decision that stands speaks. Threaded rather than kept in a
static, because several connections optimize at once.

Which check fires first decides what you are told. Two reasons were being lost
to a coarser test reached earlier: ordered_index_usage now precedes the
access-method test, since "an index supplies the ORDER BY order" says more than
"this is not a scan the engine divides", and both refuse the same plans. And
the duplicated access-method test in table_can_be_parallel_scanned() is gone,
so is_parallel_scan_applicable() owns that question alone and can tell its
several answers apart.

A reason has to name the cause, not the symptom. A GROUP BY on a column too
wide for a key part is refused because create_tmp_table() keys the container by
a unique constraint, which makes the terminal end_unique_update(), which is why
pwt_plan_group_key() finds nothing -- and "the terminal is not end_update()" is
a true statement that sends the reader nowhere. The check that would have named
the width is never even reached. So when there is no group key, this asks
make_aggr_tables_info()'s own condition of the same table it asked it of, and
says which of the two it is.

main.parallel_query_why pins one refusal per shape that can reach it, so a
reason cannot quietly go missing or attach itself to the wrong query.

This commit was prepared with Claude Code, after a report that finding why a
wide GROUP BY column would not run in parallel took an afternoon in gdb. It
counted the refusal points first -- 22 of 38 traced, 16 silent, the one being
hunted among the silent -- and worked from there.
Sergei Golubchik
workaround for https://bugzilla.redhat.com/show_bug.cgi?id=2390105
Sergei Petrunia
Adjust the MVI tests to the estimate that reads the FTS cache

fulltext_estimate() used to see only the on-disk auxiliary table, so the
multi-valued index tests, which insert and immediately EXPLAIN, got
"unknown" for every element key: the conjunctive accesses fell back to a
guess of 10 rows and the disjunctive ones were dropped for want of a
cost. Both tests worked around that with OPTIMIZE TABLE under
innodb_optimize_fulltext_only. The estimate consults the cache now, so
the workaround is gone and the row counts in the plans are real.

Two sections of the trace test were written before the estimate existed
and no longer showed what they said they did:

- "Several element keys" printed one range, not several, because a
  conjunctive access now keeps only the rarest key. It runs on a table
  with a skewed distribution instead ("bbb" in every row, "aaa" in one),
  which makes the choice of key visible rather than a tie, and checks
  the rows against the same query with IGNORE INDEX.

- The JSON_OVERLAPS section is where several ranges are printed now, so
  it says so, along with the estimates adding up.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Sergei Petrunia
Move QUICK_MVI_SELECT into opt_multi_valued_index.cc
Sergei Petrunia
Make optimizer trace print "range", not "index_merge" for MVI quick selects.
Hemant Dangi
Merge branch 'MDEV-36621-4.x' into MDEV-36621-4.ee_1

Adjust page_size in top_level_seqno_lock_protects_ist_buffers to
account for 4.ee per-page meta overhead (gcache::Page::meta_size()).
Alexey Yurchenko
Merge branch 'MDEV-38920-ensure-timeouts' into MDEV-38920-ensure-timeouts-4.ee
Sergei Petrunia
Add optimizer trace for the multi-valued index access

get_best_mvi_access() picked an Mvi_access and wrapped it in a
QUICK_MVI_SELECT without recording anything, so there was no way to see
which index was chosen or what it would search that index for: the
rows_estimation trace showed a range_analysis that found nothing, and
then a plan using a key the trace never mentioned.

Print a "multi_value_index_use" object:

  {
    "table": "t1",
    "index": "idx",
    "ranges": ["616161"]
  }

Mvi_access::print_json() fills in the index and the element keys,
following TRP_RANGE::trace_basic_info(): same "index" / "ranges" member
names, so an MVI entry reads like a range scan's.

The keys are printed in their encoded form, which is not readable. That
is what is stored in the index and what we search for, so it is still
the useful thing to print; making it readable can come later. It is
plain ASCII (hex plus the xx/xxxx padding from encode_mvi_key()), so it
needs no JSON escaping.

get_best_mvi_access() runs inside the "rows_estimation" array, so the
named object needs an object of its own around it, the same way
make_join_statistics() and the sel_arg_alloc_limit_hit trace do it.
Without it the writer hits an assertion in
Single_line_formatting_helper::on_add_member().

The new test is a separate file because optimizer trace tests need
not_embedded.inc, and putting that in multi_valued_index.test would skip
the whole feature test on embedded builds. It cross-checks the printed
keys against mvi_encode() over the indexed column, which produces the
tokens the index is actually built from.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Sergei Petrunia
Make JSON_OVERLAPS sargable for multi-valued indexes

Both argument orders are handled, since JSON_OVERLAPS is symmetric:

  JSON_OVERLAPS(array_indexed_expr, '[foo, bar, ...]')
  JSON_OVERLAPS('[foo, bar, ...]', array_indexed_expr)

JSON_CONTAINS is true when ALL of the elements have a match, JSON_OVERLAPS
when ANY of them does, so the access it produces has conjunctive=false and
build_ft_query() leaves the keys optional instead of prefixing them with
'+'.

The two get_mvi_access() implementations share collect_mvi_keys(), which
is the scan of the JSON literal that used to sit inside
Item_func_json_contains::get_mvi_access().

The two differ in one way beyond the flag. An element that cannot be
encoded for the index (a number against a CHAR array, say) is skipped for
JSON_CONTAINS: dropping a key from an AND makes the index scan less
selective, so it still returns a superset of the rows the predicate
matches and the predicate does the exact filtering afterwards.

That reasoning does not hold for an OR. A row can satisfy the predicate
through the very element we failed to encode, and MVI_ENCODE skips such
elements as well, so that row has no key in the index for the scan to find
it by - dropping the key would lose it. So for a disjunctive access we
give up instead of skipping. With a row {"tags": [123]} in the table,

  select * from t1 where json_overlaps(j->'$.tags','[123,"aaa"]')

must not use the index, and the test checks its result against the same
query with IGNORE INDEX.

Also print "match": "all"/"any" in the optimizer trace. Now that an access
can be either, the printed ranges alone did not say whether a row has to
have all of the keys or just one.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Rex Johnston
Parallel Query: AVG, and an aggregate inside an expression

TPC-H Q1 now runs in the workers. Two things were stopping it, and both are
here.

AVG
---
An average does not compose out of averages, so what a worker has for a group
is two numbers: the sum of its rows and how many there were. Nothing had to be
invented to carry them. Item_sum_avg::create_tmp_field() already packs a sum
and a count into one field, because that is how the serial grouped plan
accumulates an average, and a worker's grouped container was already
maintaining both.

What was missing was the merge. The manager primes its own aggregate with a
worker's partial through direct_add(), and Item_sum_sum::direct_add() carries
only the sum -- Item_sum_avg::count was left untouched, so every partial
counted as one row. direct_add() now takes the count alongside the sum, and
reset_field()/update_field() add it to the field's count instead of the 1 an
ordinary row contributes, in the same shape Item_sum_sum's already had.

Nothing changes for a serial plan: direct_added is false there, and the new
branches are not taken.

An aggregate inside an expression
---------------------------------
SUM(x)+1 was refused, and so was SUM(x)/COUNT(x) -- the obvious way to write an
average by hand. A select list item that is not itself an aggregate is
evaluated once per group from whatever row the terminal is holding, so a field
in it that GROUP BY does not name takes an arbitrary value; that is worth
refusing. But the check walked into the aggregates as well, found x under
SUM(x), and refused a query that was never in doubt.

It now steps over Item_sum subtrees, using with_sum_func() to answer an
aggregate-free subtree wholesale with the walk that was there before, and
decomposing only the node kinds that can hold an aggregate. Anything else
carrying one is refused rather than guessed at: being wrong in the permissive
direction here is wrong results. SUM(x)+x is still correctly refused, because
the bare x is walked and is not a group column.

The shipped row's shape
-----------------------
With more than one AVG the second onward came back wrong, which was not
arithmetic. Item_sum_avg is the one aggregate whose create_tmp_field() differs
between a keyed and an unkeyed temporary table -- it packs the count beside the
sum only for the keyed form -- and the two ends of the transport were built
differently:

  recv, group_container  make_container(..., group)  16-byte {sum, count}
  exec.result (shipped)  make_container(...)          plain 8-byte double

flush_groups() copies a group container record into the shipping container by
reclength, so it wrote past a narrower record and the manager read back a wider
one, drifting by one AVG's worth of bytes at a time. For SUM, COUNT, MIN and
MAX the two widths are identical, which is why this sat there unnoticed. The
shipping container is now built with the same key; a worker ships one row per
group, so there is nothing for the key to collapse.

main.parallel_query_aggregate gains an AVG section -- serial against parallel
over a DECIMAL average, and three AVGs at once, which is the case that exposed
the shape bug -- and AVG leaves its "what is not accepted" list. STDDEV,
BIT_XOR and COUNT(DISTINCT) stay there.

This commit was prepared with Claude Code: it bisected TPC-H Q1 down to the two
refusals, found the container shape mismatch from the pattern that only the
last AVG was wrong, and fixed a null Item passed to VDec that the aggregate
test caught as a crash.
Teemu Ollakka
Merge remote-tracking branch 'origin/4.x-MDEV-36926' into 4.ee-MDEV-36926