Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Khaled Riyad
MDEV-37563 Crash on a FOR loop with a row comparison in the upper bound

Both bounds of a FOR loop, and all parameters of a cursor FOR loop,
are parsed before their instructions are added. add_instr() then gave
all their items to the first instruction, and in a trigger also all
NEW and OLD fields. So a row comparison used by a later instruction
was cleaned up by the wrong one, on the next CALL, after its memory
had already been freed. This crashed in Arg_comparator::cleanup().

Give each instruction only the items of its own expression, and its
own trigger fields. To find them, cut THD::free_list at the newest
item of each expression, starting with the oldest expression.
Dave Gosselin
MDEV-40620:  Drop the rowid filter for the FULL JOIN null complement scan

The pass that emits the right side rows of a FULL JOIN which never
matched reads the right table with a plain sequential scan rather than
the key lookup the plan chose.  A rowid filter belongs to that lookup
because it holds the primary keys the lookup was allowed to return, and
an engine is entitled to assume it never sees one outside an index read.
InnoDB asserts on !prebuilt->index->is_primary() when it does.

The filter is set aside for the length of the scan and put back once the
scan is closed, which is what the executor already does where it swaps a
key lookup for a table scan while running a subquery.  Section 24 covers
a plan that reads the right table by key with a filter on a second index.
Khaled Riyad
MDEV-31567 SIGSEGV in reinit_stmt_before_use | Prepared_statement::execute

If the first execution of a prepared statement was killed (for
example by max_statement_time) after the top-level JOIN had done the
once-per-statement semi-join rewrite of the nested subqueries, but
before the nested selects saved their rewritten WHERE into prep_where,
the statement was left half-transformed. A later execution then used
a freed Item from prep_where or an unfixed Item_field and crashed.

Set LEX::needs_reprepare when JOIN::optimize() fails during the
select's first optimization, so the statement is re-prepared before
it runs again.

Also make the handling of needs_reprepare safe when the re-prepare
fails: keep the flag so it is tried again, do not skip a pending
long data error, and invalidate the SP instruction so that a failed
re-parse is retried.
DerZc
MDEV-40762: Wrong MIN()/MAX() in loose index scan

Problem: GROUP BY with MIN()/MAX() can return the opposite extremum
when loose index scan chooses among indexes with different ordering.
MDEV-32732 (ec6f3208837d3e3625761d26555e7899dd2e7005) added support for
descending indexes in loose index scan. It exposed a previously harmless
incorrect MIN/MAX key-part pointer: index enumeration overwrites
min_max_arg_part even when the candidate is rejected or not selected.
The pointer can then refer to a different index from the chosen plan.
Using its index-specific key_part_flag to determine scan direction
causes the regression.

Fix: Keep the MIN/MAX key-part pointer local to each candidate and
use it for the candidate's gap, infix and predicate checks. Save it
with the chosen index only when that candidate becomes the best plan.

(Cosmetic edits to testcase and commit comment by Sergei Petrunia)
Dave Gosselin
MDEV-40620:  Record every FULL JOIN match on the right side

A FULL JOIN runs as a LEFT JOIN of its left side over its right side,
then a pass emits the right side rows that matched no left row.  The pass
reads a record of which right side rows matched, and a row missing from
that record is emitted as a row that matched nothing.

The WHERE predicates on the right side are held back by a guard that
stays closed until the first match of the current left side row, which is
what let the first match be recorded whatever the WHERE then made of the
row.  For a second or later matching row of the same left row the guard
is already open, so a WHERE predicate could reject the row before its
match was recorded, and the pass brought it back with the left side all
NULL.

The match is now recorded for every right side row whose match condition
holds.  Closing the guard again gives that condition on its own, since
the guard withholds exactly what is no part of the match.  Section 23
covers a left row with two matching right rows, one of them rejected by
the WHERE.  Before this change that row came back a second time with the
left side NULL, which the UNION of a LEFT JOIN and a RIGHT JOIN does not
produce.
Andrzej Jarzabek
MDEV-40239, MDEV-41616: Fix undo log page add failure injection

The debug injection ib_err_ins_undo_page_add_failure in
trx_undo_report_row_operation() cleared undo_block to simulate a
failure of trx_undo_add_page(), but left err as DB_SUCCESS.
The function then returned DB_SUCCESS without writing an undo log
record or setting *roll_ptr, so callers stored an invalid DB_ROLL_PTR
in clustered index records. This caused assertion failures in
trx_undo_page_report_modify() and trx_undo_prev_version_build().

The injection now also sets err=DB_OUT_OF_FILE_SPACE, simulating
running out of tablespace capacity.

MDEV-41616: when that failure was reported, ib_errf() was given the
dberr_t value DB_OUT_OF_FILE_SPACE where an error number was expected,
so the client received an unrelated message. Use
ER_INNODB_UNDO_LOG_FULL.
Daniel Bartholomew
bump the VERSION
Rex Johnston
MDEV-41631 Check the class of the copy made by Item::shallow_copy()

Item::shallow_copy() was meant to create a copy of the same class as the
item it is called on, but nothing enforced that. A class that does not
declare its own override inherits the one of its nearest concrete parent,
and get_item_copy<Parent>() then silently creates an object of the parent
class, losing the child's virtual behaviour (see MDEV-38801 for an earlier
instance with Item_cache_year).

Add the ITEM_SHALLOW_COPY(CLASS) macro, which declares the override, and
Item::dbug_check_shallow_copy_type(), a debug-only check called from
Query_arena::free_items() for every item freed. Once per dynamic type it
creates a shallow copy, unlinks it from thd->free_list again (the copy shares
pointers with the original and must not be destroyed) and asserts that the
copy has the same typeid as the original. The copy is never destroyed: its
destructor is not run, and its memory is released when the MEM_ROOT it was
allocated on, the current thd->mem_root, is freed. A comment in the check
says so.

The check found these classes copied as their parent class, now fixed with
ITEM_SHALLOW_COPY():
  Item_join_operator_plus, Item_null_result, Item_old_field, Item_datetime,
  Item_empty_string, Item_return_int, Item_trigger_type_of_statement,
  in_string::Item_string_for_in_vector, Item_is_not_null_test,
  Item_interval_DDhhmmssff_typecast, Item_string_xml_non_const,
  Item_func_trt_trx_sees_eq, Item_field_refcursor (type_cursor plugin),
  Item_field_packable_scalar (type_assoc_array plugin).

Item_field_packable_row::shallow_copy() asserted to mark the class as never
copied, which the check trips over since it copies every type it meets. It
now returns nullptr, the existing way for an item to refuse a copy.

No test is added: the check runs under every existing test in a debug build.
main and the SQL-layer suites (compat/*, json, funcs_*, vcol, gcol,
versioning, period, sys_vars, type_*) pass with it in both normal and
--ps-protocol mode.

This commit was prepared with Claude Code (Opus 5), which wrote the runtime
type check, ran the main and SQL-layer suites in normal and --ps-protocol
mode to find the classes that inherited a parent's copy method, added the
ITEM_SHALLOW_COPY overrides for them, changed Item_field_packable_row to
return nullptr, and documented where the copy made by the check is
released.

Sergei Petrunia:-
Add comments, do "delete item_copy" to destroy the copy.

This has exposed items that do not support shallow_copy():
- Item_cache_wrapper
- Item_sum_sp
Thirunarayanan Balathandayuthapani
MDEV-41666 Corruption on statement rollback when ON UPDATE SET NULL affects the base column

Problem:
=======
  - When a foreign key column is the base column of a
virtual column that is indexed together with other virtual
columns, a cascaded update (ON UPDATE/DELETE SET NULL)
computed the old value of only the dependent virtual column.
The other virtual key parts of the index stayed NULL in
update->old_vrow. If the statement was rolled back,
row_upd_store_v_row() used those NULLs, so
row_upd_sec_index_entry() searched the secondary
index for a record  that did not exist and hit
the "Record in index was not found on update" assertion.

Solution:
=========
dict_mem_fill_vcol_set_for_base_col():  Add every virtual key part
of each index that contains the dependent virtual column to
dict_foreign_t::v_cols, as dict_mem_fill_vcol_from_v_indexes()
already does. row_ins_foreign_fill_virtual() then computes the
old values of all of them.

Based on mysql/mysql-server@c48fa1e57a8e810f89b43d3e94bcbefa1fd69806
Dave Gosselin
MDEV-XXXXX:  Keep the last top level base table search in bounds

Attaching a condition to the last table of a join walks back from the end
of the join order looking for the last table of the top level plan.  The
search decremented an unsigned index without a lower bound, so a plan
whose last entries all stand for runs of other tables ran the index past
zero and the walk read memory before the array.

The search now stops at the start of the array.

The shape that reaches this is a FULL JOIN with a nested join on both
sides, which is rejected until the phase 2 gates come off, so the
covering test arrives with the queries that shape allows.  The defect
itself is older than the FULL JOIN work and does not depend on it.
Oleg Smirnov
MDEV-38819 Wrong result of 'Range checked for each record' after pushdown from HAVING

When a condition was pushed from HAVING into WHERE, its fields were
unwrapped in place. The pushed condition then shared field items with
the SELECT list. Creating the GROUP BY temporary table later bound
those items to the temporary table. With a 'Range checked for each
record' plan, range analysis took the value of the outer table's field
from the not yet filled temporary table. It built an impossible range,
and the query returned no rows.

The pushed condition is now cloned before its fields are unwrapped, so
it gets its own field items. Items that cannot be cloned (e.g.
subqueries) are still unwrapped in place, to keep the pushdown. Such
conditions are unlikely to get a 'Range checked for each record' plan.

Cloning can also fail with an error, which the pushdown did not handle
properly. It modifies HAVING before all of its steps have succeeded,
yet on a failure it emptied the list of pushed conditions and let
the query continue, so the conditions already removed from HAVING were
not checked at all.
pushdown_from_having_into_where() now returns an error status, and
the query errors out if the pushdown fails after it may have modified
HAVING.

Based on a contribution by Omar Gamal (PR #4815).

Co-authored-by: OmarGamal10 <[email protected]>
Kristian Nielsen
Merge branch 'knielsen_log_event_read_validation_base_11.4' into 11.8

This merges the patch series that fixes a lot of missing validation
checks for read_log_event(). The merge includes a few added fixes that
are needed in new code introduced in 11.8.

Signed-off-by: Kristian Nielsen <[email protected]>
Thirunarayanan Balathandayuthapani
MDEV-41667 InnoDB: Failing assertion: buf <= (byte*) original_upd_buff + buff_len

Problem:
=======
An UPDATE on an InnoDB table with several indexed virtual columns
could corrupt the heap in calc_row_difference() because the
write pointer went past the end of the update buffer.

ha_innobase::update_row() sized the buffer from the row length, the
maximum key length and fixed padding. That did not allow for the
extra packed values calc_row_difference() stores for indexed
virtual columns for both the new value and the old value.

Solution:
========
  When allocating the update buffer in ha_innobase::update_row(),
add pack_length() of every virtual column. The row length
already counts each virtual column once, so the buffer can now
hold both the new and old packed values. All virtual columns
are counted, not only the indexed ones, because the buffer
is reused and a column may become indexed after it is allocated.

Based on mysql/mysql-server@12655140610674db615547cdf86af1f7cbf563c1
Khaled Riyad
MDEV-37563 Crash on a FOR loop with a row comparison in the upper bound

Both bounds of a FOR loop, and all parameters of a cursor FOR loop,
are parsed before their instructions are added. add_instr() then gave
all their items to the first instruction, and in a trigger also all
NEW and OLD fields. So a row comparison used by a later instruction
was cleaned up by the wrong one, on the next CALL, after its memory
had already been freed. This crashed in Arg_comparator::cleanup().

Give each instruction only the items of its own expression, and its
own trigger fields. To find them, cut THD::free_list at the newest
item of each expression, starting with the oldest expression.
Sergei Petrunia
MDEV-35318 Assertion `tl->jtbm_subselect' failed in JOIN::calc_allowed_top_level_tables

Alternative, more general fix, Variant 2.

The problem was as follows: Suppose we are running a PS/SP statement and
we get an error while doing optimization that is done once per statement
life. This may leave the statement data structures in an undefined state,
where it is not safe to execute it again.

The fix: introduce LEX::needs_reprepare and set it in such cases.
Make PS and SP runtime check it and re-prepare the statement before
executing it again.

We do not use Reprepare_observer, because it turns out it is tightly tied
to watching versions of statement's objects. For example, it must not be
used when running the statement for the first time, exactly when the
once-per-statement-lifetime optimizations are done.

(cherry picked from commit 0e21ff8ca47ab637564b4c120322215caa70835d)
Vladislav Vaintroub
MDEV-41691 PROXY protocol v1 header parser is too lenient

Follow the specification: the line is at most 107 bytes including CRLF,
ends with CRLF (not a bare LF), and fields are separated by exactly one
space, with nothing trailing (except after UNKNOWN). Ports are decimal
numbers up to 65535, without signs or leading zeroes. Also reject NUL
bytes and unknown address families (these were accepted before).

test_proxy_header_limits() now checks that malformed headers are
rejected, and no longer makes the server resolve 1.2.3.4.

Assisted-by: Claude:claude-sonnet-5-5
Khaled Riyad
MDEV-31567 SIGSEGV in reinit_stmt_before_use | Prepared_statement::execute

If the first execution of a prepared statement was killed (for
example by max_statement_time) after the top-level JOIN had done the
once-per-statement semi-join rewrite of the nested subqueries, but
before the nested selects saved their rewritten WHERE into prep_where,
the statement was left half-transformed. A later execution then used
a freed Item from prep_where or an unfixed Item_field and crashed.

Set LEX::needs_reprepare when JOIN::optimize() fails during the
select's first optimization, so the statement is re-prepared before
it runs again.

Also make the handling of needs_reprepare safe when the re-prepare
fails: keep the flag so it is tried again, do not skip a pending
long data error, and invalidate the SP instruction so that a failed
re-parse is retried.
Thirunarayanan Balathandayuthapani
MDEV-41281 Unbounded Recursion in InnoDB FTS Boolean-Mode causes Stack-Overflow fts_ast_visit

expr_lst: expr_lst sub_expr wrapped the expression list parsed so
far in a newly created FTS_AST_LIST for every sub-expression,
so a query such as '(word)' repeated N times built N nested lists.
fts_ast_visit() recurses once per FTS_AST_LIST and overflowed the
thread stack; the default thread_stack was exhausted at about
2300 repetitions.

Append the sub-expression to the list that is already being built,
and create a list only when there is none yet, the way
the neighbouring expr_lst: expr_lst expr rule has always done.
The sub-expressions now become siblings of a single list and
the depth no longer grows.

This also corrects '+term (subexpr)'. The FTS_EXIST pass used to
run to completion in the inner list before the sub-expression was
unioned at the outer level, so rows that matched only the
sub-expression were returned.

Based on mysql/mysql-server@e46e9db484a051db6ffb0abd2b06d3b38ef71a35
Marko Mäkelä
Replace backup_phase with extended backup_stages

This is based on a commit 281a0c99986824784a66659f2332db3ec362846f
by Monty.

According to Monty, this:
- Galera should now be supported (needs testing)
- Enables ddl logging (so we can use it in the future)
- Flushes binary logs (we still must add code to copy them)
- mdl locks are consistent between mariadb-backup and backup command
- Give errors if backup command is done under a transaction, global
  read lock or lock tables.
- Retry for MDL_BACKUP_WAIT_DDL (needed for
  backup.backup_ddl_concurrent_verify)

Things to do (in addition to the things in Monty's earlier review):
- At backup_stage stage start, force rotate of aria log files. This
  allows us to copy all old logs without any locks
- Copy all transactional tables and old aria logs under BACKUP_START (as
- maria-backup does)
- Copy the active aria log file under block commit (Only one file as aria
  log rotation is disabled while backup is running.
- Improve speed of copying aria tables by copy files in up to 1M
  blocks and run checksum on the blocks and only re-read blocks with
-  wrong checksum.
- Copy non transactional files under BACKUP_START.
Thirunarayanan Balathandayuthapani
MDEV-41550 MTR doesn't produce core dump or stack trace when a server hangs during startup

Problem:
========
If a server hangs in startup, mtr either waits for its pid file for
--start-timeout seconds and gives up with "Failed to start" (or)
when the test restarts the server, leaves the waiting to mysqltest,
which ends after about 500 seconds with "Server failed to restart".
In both cases the hung server is killed with SIGKILL, so
no core dump and no stack trace is produced, and it is
impossible to find out where the server got stuck.

Solution:
=========
dump_core_for_hung_server(): Sends SIGABRT to a server that is
still alive and waits for it to finish dumping core.
The server's fatal signal handler writes the stack trace
to the error log.

sleep_until_file_created(): call it when the wait for the pid file
times out and the process is still alive. This covers the servers
that mtr starts and waits for itself.

check_expected_crash_and_restart(): when mtr restarts a server on
behalf of the test, remember the time by which the
server must have created its pid file ($opt_start_timeout)

run_testcase() wakes up at that time, and if the server is
still alive without a pid file, it dumps core, fails the test with
"Failed to restart mysqldN, no pid file after N seconds" and kills
mysqltest, instead of waiting for it to give up.

mysql_server_wait(): append the server log of the failed server
to the failure report, as the other failure paths do.

Behavior change:
================
A restart done by the test now has to finish within --start-timeout
(180 seconds by default, MTR_START_TIMEOUT), the same as an initial
start. Before, only the test's own wait applied, about 500 seconds
with restart_mysqld.inc. A slow but healthy restart that needs more
than --start-timeout now fails with a core dump. Raise
MTR_START_TIMEOUT for such tests. Under a debugger the timeouts are
much longer, so nothing changes there.

Servers that are only slow, or that mtr has not waited for yet
are not aborted before the timeout.
Kristian Nielsen
Merge branch 'knielsen_log_event_read_validation_base_10.11' into 11.4

This merges the patch series that fixes a lot of missing validation
checks for read_log_event(). The merge includes a few added fixes that
are needed in new code introduced in 11.4.

Signed-off-by: Kristian Nielsen <[email protected]>
Dave Gosselin
MDEV-40620:  Add optimizer switches for the four join transformations

Four switches, all on by default, so behavior does not change.  Each one
turns off a transformation that simplify_joins performs, which helps
narrow down where a wrong result comes from.

  full_join_rewrite    rewriting a FULL JOIN as a one sided outer join
                      when the WHERE clause rejects NULLs on one of its
                      operands.  Off keeps the FULL JOIN and forces the
                      null complement pass to run.
  outer_join_to_inner  converting an outer join to an inner join when a
                      conjunctive predicate rejects NULLs for one of
                      its inner tables.
  flatten_join_nests  replacing a join nest that carries no ON
                      expression with its children.  Off keeps the
                      nesting the parser produced, which costs plan
                      quality without changing results.
  simplify_joins      an umbrella over the other three.

Each decision moved out of the code that carries it out, so a switch has
one place to read.  classify_full_join already held the whole FULL JOIN
decision.  The outer join conversion shared a block with work that runs
for a plain inner join as well, so that decision moved to
classify_outer_join, and taking the dependencies of the enclosing nest,
moving the ON expression into the WHERE clause, and dropping the outer
join marks now happen both for a converted join and for one that was
already inner.  Registering a semi-join nest moved to
register_semijoin_nest, which always runs since it is the only producer
of the semi-join nest list, leaving the flattening test on its own.

Gating the call to simplify_joins rather than its three decisions would
give wrong results, since that pass is the only producer of the nest
attributes and table dependencies the rest of the optimizer reads.
join_transform_enabled holds the umbrella relationship in one place.

subselect_exists2in turns every switch off by rewriting the whole
optimizer_switch string, so it now turns flatten_join_nests off as well,
and the nest it leaves in place shows up in one line of that test's
expected output.  One switch list result file also gained
reorder_outer_joins, which had been left out when that switch was added.
Thirunarayanan Balathandayuthapani
MDEV-41281 Unbounded Recursion in InnoDB FTS Boolean-Mode causes Stack-Overflow fts_ast_visit

expr_lst: expr_lst sub_expr wrapped the expression list parsed so
far in a newly created FTS_AST_LIST for every sub-expression,
so a query such as '(word)' repeated N times built N nested lists.
fts_ast_visit() recurses once per FTS_AST_LIST and overflowed the
thread stack; the default thread_stack was exhausted at about
2300 repetitions.

Append the sub-expression to the list that is already being built,
and create a list only when there is none yet, the way
the neighbouring expr_lst: expr_lst expr rule has always done.
The sub-expressions now become siblings of a single list and
the depth no longer grows.

This also corrects '+term (subexpr)'. The FTS_EXIST pass used to
run to completion in the inner list before the sub-expression was
unioned at the outer level, so rows that matched only the
sub-expression were returned.

Based on mysql/mysql-server@e46e9db484a051db6ffb0abd2b06d3b38ef71a35
Khaled Riyad
MDEV-37563 Crash on a FOR loop with a row comparison in the upper bound

Both bounds of a FOR loop, and all parameters of a cursor FOR loop,
are parsed before their instructions are added. add_instr() then gave
all their items to the first instruction, and in a trigger also all
NEW and OLD fields. So a row comparison used by a later instruction
was cleaned up by the wrong one, on the next CALL, after its memory
had already been freed. This crashed in Arg_comparator::cleanup().

Give each instruction only the items of its own expression, and its
own trigger fields. To find them, cut THD::free_list at the newest
item of each expression, starting with the oldest expression.
Dave Gosselin
MDEV-40620:  Allow a nested join as a FULL JOIN operand

Phase 2 rejected two shapes.  A FULL JOIN whose right operand was a
nested join raised ER_FULL_JOIN_BASE_TABLES_ONLY, and a FULL JOIN on the
inner side of an enclosing LEFT or RIGHT JOIN raised
ER_FULL_JOIN_NOT_ALLOWED_IN_OUTER_JOIN.  Both shapes are allowed now and
neither error is raised any more.

A FULL JOIN emits the rows of one side that never matched with the other
side null complemented.  An enclosing join's condition has to see those
rows complete, which it cannot do while the operand is only a set of
tables spread through the join order.  An operand that is a nested join
is therefore computed into a temporary table before the enclosing join
runs, so one JOIN_TAB stands for the whole operand and carries the FULL
JOIN's own marks.

The chosen plan has to keep such an operand on an unbroken span of the
join order, and that span becomes a run of JOIN_TABs of its own, the way
a materialized semi join already is.  Nothing outside the operand is
positioned while the operand is computed, so a table inside it can carry
no condition that names an outside table, an equality class that spans
the boundary is split at it, and the outer join scope of a join outside
it stops there.  A nest goes on naming a table that table elimination
removed, and an eliminated table is a const table, so the test for
entering the run leaves the eliminated tables out.  The test for
entering a materialized semi join run reads the same way.

The operand keeps a copy of its own ON expression, since simplifying the
join tree merges an inner join's ON into the enclosing outer join's ON,
which is not equivalent for a FULL JOIN because it changes which rows of
the right side never match.  An operand can itself be built from more
than one join, so an ON expression can sit a level or two below the top
of it, and every such expression is recorded on the operand as a whole,
which is the nest that reads them back.  A predicate over the operand is
checked again at the entry standing for it, once the null complementing
has happened.  A FULL JOIN's left side WHERE predicates are deferred to
the table that completes its right side even when that side is a nest.
The enclosing ON contributes no access path for the operand.

Reading a row back restores what a nest table cannot hold on its own,
whether the row was null complemented for a column declared NOT NULL,
and the status that says whether the table holds a usable row at all.

The JOIN_TAB standing for a run covers every table of the nest, the
eliminated ones included.  The plan output therefore skips only a
JOIN_TAB that is a table of the query, since a JOIN_TAB standing for a
run is never eliminated.

The FULL JOIN code moves out of sql_select.cc, sql_base.cc and table.cc
into opt_full_join.cc and opt_full_join.h.  JOIN::full_joins holds a
Full_join_info, declared in opt_full_join_info.h, for each FULL JOIN
that simplify_joins did not rewrite.  It holds the two sides, the WHERE
conjuncts held back for the FULL JOIN, the tab that reads its right
side, the duplicate filter of the right rows that matched, and the next
FULL JOIN whose null complement pass runs from the same tab.  It
replaces fj_dups, fj_first_target and fj_next_target on JOIN_TAB and
fj_left_cond on TABLE_LIST.  The list is built for each JOIN, so a
prepared statement starts every execution with nothing held.

opt_join_runs.cc and opt_join_runs.h hold the runs.
get_best_combination records every run of the chosen join order in a
Join_run_layout before it writes any JOIN_TAB, and the layout gives the
exact number of top level JOIN_TABs.  make_join_select walks the plan
with Join_tab_walk, which keeps the tables a run sets aside on entry
until the walk reaches the run's placeholder.  A FULL JOIN run can be
inside a materialized semi join, so the walk keeps the runs on a stack.

make_join_select decides again which conditions wait for a FULL JOIN's
right partner, on each condition about to be attached to a table.
Distributing the WHERE builds conditions for one table out of
disjunctions that span both sides, and a HAVING predicate moved into
the WHERE can name the left side alone, so the choice made while the
WHERE is rewritten misses both.  That earlier choice stays, since it
keeps the left side out of ref and range analysis.  Whether a condition
reads a table after the partner is decided from the join order
positions of the tables it reads.

An enclosing join's ON supplies no key value for a table on a FULL
JOIN's left side, because the null complement pass is complete only
once every left row has been read.  A key value that reads only tables
of the same left side is still allowed.  An ORDER BY over a FULL JOIN
that simplify_joins kept sorts through a temporary table, because the
null complement pass emits right rows after the scan of the first
table.  EXPLAIN gathers the tables of each run at its own depth, and an
operand nest inside a materialized semi join reports under that semi
join's id.

Sections 22 to 28 cover the new shapes.  The section covering more than
one matching right side row becomes Section 28, which adds a nest operand
to the same case, and the two sections that came with the fixes before
this one are renumbered to follow it.  Sections 35 to 42 cover an ON
expression below the top of an operand, an enclosing ON that keeps the
left side NULLs, operand nests that table elimination shrinks, a
condition offered to a table inside a run, a prepared WHERE that folds
away on a later execution, a FULL JOIN run inside a materialized semi
join, and a left side condition over a table before the partner.
Sections 8, 18 and 27 gain left side predicates that appear only once
the WHERE is rewritten.  The helper that gives every column an index of
its own comes with them, ahead of the axis that turns it on.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Dave Gosselin
MDEV-40620:  Run the FULL JOIN tests over more engines and switches

full_join.test runs once for every point of three crossed axes, the
optimizer switch axis in full_join.combinations, the storage engine axis
in full_join_engine.combinations and the index axis in
full_join_index.combinations, thirty runs in all.  The result file in the
tree is the run with the default engine, no added indexes and the
switches at their defaults.  Every other run's expected output is a diff
against it, so a run whose output matches it has no file of its own, and
a query added to the test shows up in all thirty runs.

The index axis gives every column of every base table a non-unique index
on itself.  The work is decided by what the database holds at the point
full_join_add_indexes.inc is sourced, so a section sources it after the
tables it creates.  A non-unique index is the only kind that can be added
without knowing the data, since a primary key or a unique index would
fail on the duplicates and the NULLs that many sections put in a join
column on purpose.

The switch axis holds the four switches at their defaults and each of
them off in turn, which is what makes a wrong result attributable to one
transformation.
Sergei Petrunia
MDEV-36057: Assertion failure on 2nd execution of parameterized PS

Followup to fix for MDEV-35318: In Prepared_statement::execute_loop(),
if we enter this function with lex->needs_reprepare=true, we need to
re-prepare the statement.
We also need to call set_parameters() to get PS parameter values. We
failed to do this and that's the bug.

Note that we need to also handle the other reprepare scenario: when
we get into Prepared_statement::execute_loop(), call set_parameters()
and then hit a "DDL was changed" error with reprepare_observer,
we should NOT call set_parameters() again.
The parameters have already been transferred to the reprepared statement
in Prepared_statement::reprepare()|swap_parameter_array().

(cherry picked from commit 80067a69feaeb5df30abb1bfaf7d4e713ccbf027)
Dave Gosselin
MDEV-40620:  Null complement a merged derived table inside a FULL JOIN

A FULL JOIN's null complement pass marks the whole opposite operand as a
null row before it emits a row of the side it rescans.  The walk that
marks the operand treated a TABLE_LIST as either a plain table or a nest.
A merged derived table or view is both, its own placeholder and the
tables it was merged into, so the walk marked the placeholder and
stopped.  The query reads the fields of the merged tables, which still
held the values of an earlier row, and those values reached the client
where NULL belonged.

The walk now descends into the nested tables as well as marking the
placeholder.  Section 22 covers the shape with the flattening of join
nests turned off, which leaves the derived table in a nest of its own so
the walk reaches it.  Before this change the FULL JOIN returned the left
side value 3 against the right side rows 3 and 4, where the UNION of a
LEFT JOIN and a RIGHT JOIN returns NULL.
Dave Gosselin
MDEV-XXXXX:  Keep an ON equality that a FULL JOIN left side reads by key

A ref access guarantees an equality only for the rows it read, so the
optimizer normally drops that equality from the outer join's ON
condition.  A table on the left side of a surviving FULL JOIN also
reaches the ON condition as a null row, produced by the pass that emits
the right side rows which never matched on the left, and the ON condition
has to reject those rows.  With the equality gone, nothing did, and a row
the outer join owed as a null complement was lost.

The equality now stays in the condition for such a table.  The existing
deferral machinery moves it onto the FULL JOIN's right side under the
found match guard, so it is checked once the match has been recorded.

The shape that reaches this is a FULL JOIN whose right side is a nested
join, which is rejected until the phase 2 gates come off, so the covering
test arrives with the queries that shape allows.
Dave Gosselin
MDEV-XXXXX:  Save and restore the status of tables inside a run

Walking back over the tables a join buffer scan feeds from takes a copy
of each table's status and clears it, then puts the copy back when the
scan closes.  For a table standing for a run of other tables, the tables
in that run were left out of the copy but were still written to when the
copy was put back, so they received a value the copy had never taken.
They now take part in both directions.

The work for one table moved into a function that reaches the tables of
any run it stands for, however deeply those runs nest, which is what the
rest of the join order walking already does.

This is a defect in the handling of a materialized semi join, which is
the only kind of run a released version produces, so it is not specific
to FULL JOIN.  No query is known to reach it today.  The FULL JOIN work
that follows produces runs that nest, which reaches it.
Sergei Petrunia
MDEV-36080: Assertion on 2nd PS execution with error and Array Binding

The fix for MDEV-35318 has introduced LEX::needs_reprepare and logic in
Prepared_statement::execute_loop() and sp_lex_keeper::
validate_lex_and_exec_core() to re-prepare the statement if it has hit
an error when doing once-per-statement-life optimizations.

But there is also third code path: PS with Array Binding is handled in
Prepared_statement::execute_bulk_loop(). Add handling there as well.

(cherry picked from commit 33e0796e7a154e02a5e53c55cefc5d6feb4f5e6d)
Sergei Petrunia
MDEV-35318 Assertion `tl->jtbm_subselect' failed in JOIN::calc_allowed_top_level_tables

Alternative, more general fix, Variant 2.

The problem was as follows: Suppose we are running a PS/SP statement and
we get an error while doing optimization that is done once per statement
life. This may leave the statement data structures in an undefined state,
where it is not safe to execute it again.

The fix: introduce LEX::needs_reprepare and set it in such cases.
Make PS and SP runtime check it and re-prepare the statement before
executing it again.

We do not use Reprepare_observer, because it turns out it is tightly tied
to watching versions of statement's objects. For example, it must not be
used when running the statement for the first time, exactly when the
once-per-statement-lifetime optimizations are done.

(cherry picked from commit 0e21ff8ca47ab637564b4c120322215caa70835d)
Sergei Petrunia
MDEV-36057: Assertion failure on 2nd execution of parameterized PS

Followup to fix for MDEV-35318: In Prepared_statement::execute_loop(),
if we enter this function with lex->needs_reprepare=true, we need to
re-prepare the statement.
We also need to call set_parameters() to get PS parameter values. We
failed to do this and that's the bug.

Note that we need to also handle the other reprepare scenario: when
we get into Prepared_statement::execute_loop(), call set_parameters()
and then hit a "DDL was changed" error with reprepare_observer,
we should NOT call set_parameters() again.
The parameters have already been transferred to the reprepared statement
in Prepared_statement::reprepare()|swap_parameter_array().

(cherry picked from commit 80067a69feaeb5df30abb1bfaf7d4e713ccbf027)
Alessandro Vetere
MDEV-41694 Read view after DDL returns wrong data when the table definition was reloaded

dict_index_t::trx_id exists only in memory. After a table was evicted
from the InnoDB data dictionary cache and loaded again, it was 0, so
an old read view could use indexes whose data was replaced by DDL.
The old read view then got wrong data after ALTER TABLE ...
ALGORITHM=COPY, a table rebuild, ADD INDEX, TRUNCATE TABLE and
IMPORT TABLESPACE, instead of ER_TABLE_DEF_CHANGED.

dict_load_indexes(): Restore dict_index_t::trx_id from the DB_TRX_ID
of the SYS_INDEXES record, except for a discarded tablespace, which
cannot be read anyway. RENAME INDEX also updates the record, so after
a reload an old read view cannot use a renamed index either.

innodb.mvcc_ddl_evict runs a matrix of DDL statements against an old
read view, first without and then with eviction of the table from the
InnoDB data dictionary cache.

Assisted-by: Claude:claude-opus-5-5
Marko Mäkelä
fixup! 92d9175305639ea3e0e9d817540fa3d1d1ebde82
Kristian Nielsen
MDEV-40745: Validate field parsing of optional_metadata

Protect accesses into the m_optional_metadata buffer to not access outside
of the buffer in case of corrupt/malicious event data.

Protect against buffer overflow of the m_column_name array in case of
excessive column names in the event data.

Signed-off-by: Kristian Nielsen <[email protected]>
Sergei Petrunia
MDEV-36080: Assertion on 2nd PS execution with error and Array Binding

The fix for MDEV-35318 has introduced LEX::needs_reprepare and logic in
Prepared_statement::execute_loop() and sp_lex_keeper::
validate_lex_and_exec_core() to re-prepare the statement if it has hit
an error when doing once-per-statement-life optimizations.

But there is also third code path: PS with Array Binding is handled in
Prepared_statement::execute_bulk_loop(). Add handling there as well.

(cherry picked from commit 33e0796e7a154e02a5e53c55cefc5d6feb4f5e6d)