Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Sergei Golubchik
MDEV-40336 SET AUTHORIZATION un-expired passwords

don't allow SET SESSION AUTHORIZATION if the password expired
Georgi (Joro) Kodinov
MDEV-39307: Remove the verbose part of a comment.
Dave Gosselin
MDEV-41211:  FederatedX multi-table DELETE keeps a const table row

A multi-table DELETE on a FederatedX table left a row in place when a
primary key lookup made the target a const table.  The optimizer reads
a const table's row through index_read_idx_map(), whose default
implementation ends the index scan and frees the result set.  The
server asks for the row's position later, during execution, so the
saved position was empty and the row was skipped.

FederatedX now overrides index_read_idx_map() so that the lookup leaves
its result set open, as index_read() does.  position() then records a
valid position, and the result set is freed at the end of the
statement.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Rex Johnston
MDEV-41225 Parse the ORDER BY tail of a wrapped parenthesized query in the wrapping select

A parenthesized query expression that already has its own ORDER BY or
LIMIT and is followed by another ORDER BY, e.g.

  (SELECT c1 FROM t1 ORDER BY c1 LIMIT 2) ORDER BY <expr> LIMIT 1

is wrapped into a derived table, and the outer ORDER BY/LIMIT is attached
to the wrapping select.  The grammar pushed the inner select before
parsing the tail, and add_tail_to_query_expression_body_ext_parens()
created the wrapper only after the tail had been parsed.  Anything that
is attached to the current select while the tail is being parsed was
therefore attached to the inner select, while the ORDER BY list itself
was moved to the wrapper:

- Window functions and their window specs were added to the inner
  select's window_funcs/window_specs.  The same Item_window_func was then
  in the wrapper's ORDER BY and in the inner select's window function
  list, which triggers the original assertion reported in MDEV-41225.

  (SELECT 1 FROM t1 LIMIT 1)
    ORDER BY PERCENTILE_DISC(1) WITHIN GROUP(ORDER BY TIME'0') OVER();

  The function is computed by the inner select over the rows before its
  LIMIT, so the outer ORDER BY sorts on wrong values
  (e.g. COUNT(*) OVER () returned 3 for a 2-row result).

- Subqueries were registered as units of the inner select.  An
  uncorrelated subquery in the tail crashed the server with SIGSEGV when
  it was executed by the outer filesort.  A correlated one resolved its
  outer references in the inner select (t1.c1) instead of the derived
  table.

The decision to wrap depends only on whether the inner select already
has a tail and on whether the outer tail has ORDER BY.  LIMIT and locking
clauses cannot contain window functions or subqueries.
We replace query_expression_tail with
query_expression_tail_with_order and query_expression_tail_no_order
so the mid-rule action knows whether an ORDER BY follows.
The new LEX::push_select_for_ext_parens_tail() creates the wrapper before
an ORDER BY tail is parsed when wrapping is required, and pushes it instead
of the inner select.  The tail's items are then created in the wrapper's
context, and window functions and subqueries are attached to the wrapper.
add_tail_to_query_expression_body_ext_parens() takes the pushed select and
skips wrapping when it has already happened; the other cases keep the
previous logic.  The grammar has the same number of conflicts as before.

A subquery in such a tail can no longer refer to the tables of the wrapped
query expression (e.g. t1.c1 above) and gets ER_BAD_FIELD_ERROR,
as those tables are not visible outside the derived table.

This commit was prepared with Claude Code (Opus 5.5). It traced the
wrong-result and crash cases to the parse-time registration of window
functions and subqueries in the inner select, tested the grammar split
with bison, wrote the code change and the brackets test.
Marko Mäkelä
MDEV-41310: Assertion current_lsn < archive_header_was_reset failed

log_t::write_checkpoint(): Correct an off-by-one error in the assertion
expression. We may reach the very end of the current log file.
A subsequent log_t::write_buf() will invoke archive_new_write(),
which will create a new log file if needed.

Tested by: Saahil Alam
Thirunarayanan Balathandayuthapani
MDEV-40274 Server aborts while reloading the table after failed DISCARD TABLESPACE

Problem:
========
row_discard_tablespace() updates SYS_TABLES, SYS_INDEXES and
reassigns the table identifier in SYSTEM TABLES.
These changes cannot be rolled back, and
row_discard_tablespace_for_mysql() commits the transaction even when
row_discard_tablespace() returned an error.
If the reassignment in row_mysql_table_id_reassign() fails
in the middle then the committed data dictionary is inconsistent
and the table can no longer be loaded from it, while the
in-memory table definition stays in the dictionary cache and
remains usable.

Solution:
=========
row_discard_tablespace_for_mysql(): If row_discard_tablespace()
failed, flag the table as corrupted and report the data dictionary
inconsistency in the error log.

dict_table_open_on_name(): Check dict_table_t::space before reading
fil_space_t::get_compression_algo() from it.
Aleksey Midenkov
MDEV-41344 Timestamp conversion does not work for TRADITIONAL mode

Field_temporal::get_copy_func() returns do_field_datetime whenever the
session's sql_mode has NO_ZERO_DATE or NO_ZERO_IN_DATE (e.g. under
sql_mode=TRADITIONAL), or when the two fields are not eq_def(),
regardless of whether the field is a system-versioned
row_end.

get_copy_func() checked for that value and returned early, before ever
reaching the VERS_ROW_END check that installs
do_field_versioned_timestamp. So an ALTER TABLE .. FORCE meant to
convert an old-format row_end silently copied the value unchanged
instead: the conversion check correctly demanded a copy, but the copy
step never applied it, with no error or warning.

The fix checks the row_end conversion need first, independently of
what Field_temporal::get_copy_func() picked. row_end is
server-maintained and never zero, so NO_ZERO_DATE does not apply to
it.

Tested by "traditional" combination in old_timestamp.test, the test
restores real pre-11.5 row_end fixtures and runs mariadb-upgrade
--force.
Kristian Nielsen
MDEV-41264: binlog-in-engine dump thread not stopping at server shutdown

The binlog-in-engine code for binlog dump thread didn't check
should_stop(info, true) to stop on KILL_SERVER when it reaches the end of
the current binlog. This caused the dump threads to hang indefinitely during
SHUTDOWN WAIT FOR ALL SLAVES, and thus the shutdown to hang as well.

Signed-off-by: Kristian Nielsen <[email protected]>
Marko Mäkelä
fixup! 92d9175305639ea3e0e9d817540fa3d1d1ebde82
Khaled Riyad
MDEV-40551 Copy/Paste friendly output format for MariaDB Command Line Client

Copy/paste friendly output was only reachable by starting the client with
--silent --skip-column-names, which cannot be done from a running
interactive session.

Add \S, a statement terminator which prints the result of one statement in
the tab separated format without column names.

com_silent() sets output_plain, opt_silent and column_names around
com_go(), then restores them, the same way com_ego() handles vertical.
output_plain selects print_tab_data() ahead of the vertical and table
branches, so \S gives the same output whether the session was started
plainly or with --table, --vertical or --silent. --html and --xml still
win, matching \G.
Lawrin Novitsky
Updtated the Connector/C submodule to v3.4.11
Prathamesh Hukkeri
MDEV-39307: Fix %f in audit plugin timestamp rendering zero microseconds

The server_audit_timestamp_format %f specifier always rendered zero
microseconds. The server downcast the precise time to seconds before
passing it to audit plugins, and the plugin then re-fetched the time
itself at write time.

Pass the server's high-resolution time through the audit API instead:
- extend mysql_event_general with general_time_microseconds (added in
  MYSQL_AUDIT_INTERFACE_VERSION 0x0304), keeping general_time in
  seconds for backward compatibility
- the server_audit plugin uses event->general_time_microseconds for
  query log entries instead of re-fetching the time at write time

Connection and table events carry no timestamp in the audit API, so the
plugin keeps taking the time at event time for those entries.
bsrikanth-mariadb
MDEV-41340: Push a merged view down like the table it stands for

A multi-table UPDATE/DELETE through a mergeable view wasn't pushed down
into FederatedX at all, even when the view was a transparent stand-in for
one of the engine's own tables. The eligibility check (find_multi_upddel_
handler -> get_fed_table_for_pushdown) ran before JOIN::optimize()'s
DT_MERGE step, so it only ever saw the view's own not-yet-merged
TABLE_LIST placeholder, whose table (if any) isn't FederatedX-backed, and
rejected the whole statement.

Naively fixing that by treating an unmerged view as equivalent to a plain
derived table (defer to its inner SELECT, as already done for anonymous
FROM-list subqueries) is unsafe for a genuinely materialized view: the
statement is reproduced as text via TABLE_LIST::print(), which always
prints a named view by its name, merged or not; a view that needs
materialization (e.g. because of LIMIT) has no such name on the remote
server, so the pushed-down statement referenced a table that only exists
locally, and failed with ER_NO_SUCH_TABLE.

A second, related problem: a merged view's underlying table keeps the
table name/alias it has inside the view's own definition, not the alias
the statement used for the view. If that collides with another reference
to the same table elsewhere in the statement (a self-join through the
view), the printed statement can't tell the two occurrences apart, and
either the remote server rejects it as an ambiguous "not unique
table/alias" statement, or worse, a column silently binds to the wrong
occurrence.

- sql_select.cc: Sql_cmd_dml::execute_inner() now runs the DT_MERGE step
  before asking engines whether they can take over the statement, so a
  merged view's real underlying table(s) are visible to the check instead
  of just the view's own placeholder. Idempotent, so it does not repeat
  the same step inside optimize_inner() further down.
- federatedx_pushdown.cc: get_fed_table_for_pushdown()'s per-table check is
  now the recursive check_fed_table_for_pushdown(), which:
  - skips a merged view's own inert placeholder but recurses into what it
    actually stands for, however DT_MERGE represented it (a spliced
    sibling TABLE_LIST, or a NESTED_JOIN wrapping the placeholder when the
    view is itself one of the statement's targets);
  - rejects a named view that still needs materialization instead of
    deferring to its inner SELECT, closing the ER_NO_SUCH_TABLE gap above;
  - rejects pushdown outright when the same remote table would be printed
    under colliding names, closing the self-join gap above.

Test: federated.federatedx_pushdown_upd_del.
Vladislav Vaintroub
MDEV-41343 REVOKE DENY FROM PUBLIC allowed with column-level UPDATE

REVOKE DENY .. FROM PUBLIC needs table-level UPDATE privilege for
mysql.global_priv. Tighten the privilege check, which also accepted
column-level UPDATE, because check_grant() defers the column check.

Assisted-by: Claude:claude-sonnet-5-5
Rex Johnston
MDEV-41225 Parse the ORDER BY tail of a wrapped parenthesized query in the wrapping select

A parenthesized query expression that already has its own ORDER BY or
LIMIT and is followed by another ORDER BY, e.g.

  (SELECT c1 FROM t1 ORDER BY c1 LIMIT 2) ORDER BY <expr> LIMIT 1

is wrapped into a derived table, and the outer ORDER BY/LIMIT is attached
to the wrapping select.  The grammar pushed the inner select before
parsing the tail, and add_tail_to_query_expression_body_ext_parens()
created the wrapper only after the tail had been parsed.  Anything that
is attached to the current select while the tail is being parsed was
therefore attached to the inner select, while the ORDER BY list itself
was moved to the wrapper:

- Window functions and their window specs were added to the inner
  select's window_funcs/window_specs.  The same Item_window_func was then
  in the wrapper's ORDER BY and in the inner select's window function
  list, which triggers the original assertion reported in MDEV-41225.

  (SELECT 1 FROM t1 LIMIT 1)
    ORDER BY PERCENTILE_DISC(1) WITHIN GROUP(ORDER BY TIME'0') OVER();

  The function is computed by the inner select over the rows before its
  LIMIT, so the outer ORDER BY sorts on wrong values
  (e.g. COUNT(*) OVER () returned 3 for a 2-row result).

- Subqueries were registered as units of the inner select.  An
  uncorrelated subquery in the tail crashed the server with SIGSEGV when
  it was executed by the outer filesort.  A correlated one resolved its
  outer references in the inner select (t1.c1) instead of the derived
  table.

The decision to wrap depends only on whether the inner select already
has a tail and on whether the outer tail has ORDER BY.  LIMIT and locking
clauses cannot contain window functions or subqueries.
We replace query_expression_tail with
query_expression_tail_with_order and query_expression_tail_no_order
so the mid-rule action knows whether an ORDER BY follows.
The new LEX::push_select_for_ext_parens_tail() creates the wrapper before
an ORDER BY tail is parsed when wrapping is required, and pushes it instead
of the inner select.  The tail's items are then created in the wrapper's
context, and window functions and subqueries are attached to the wrapper.
add_tail_to_query_expression_body_ext_parens() takes the pushed select and
skips wrapping when it has already happened; the other cases keep the
previous logic.  The grammar has the same number of conflicts as before.

A subquery in such a tail can no longer refer to the tables of the wrapped
query expression (e.g. t1.c1 above) and gets ER_BAD_FIELD_ERROR,
as those tables are not visible outside the derived table.

This commit was prepared with Claude Code (Opus 5.5). It traced the
wrong-result and crash cases to the parse-time registration of window
functions and subqueries in the inner select, tested the grammar split
with bison, wrote the code change and the brackets test.
Daniel Bartholomew
bump the VERSION
Vladislav Vaintroub
MDEV-41343 REVOKE DENY FROM PUBLIC allowed with column-level UPDATE

REVOKE DENY .. FROM PUBLIC needs table-level UPDATE privilege for
mysql.global_priv. Tighten the privilege check, which also accepted
column-level UPDATE, because check_grant() defers the column check.

Assisted-by: Claude:claude-sonnet-5-5
mariadb-PranavTiwari
Ad
Jan Lindström
MDEV-41075 : Assertion .thd->in_active_multi_stmt in TOI

Clear OPTION_NOT_AUTOCOMMIT/OPTION_BEGIN before opening the table below,
not after. A storage engine may register itself into the "all"
transaction as part of the table open/lock (e.g. InnoDB's
external_lock()), depending on those bits. Clearing them only after the
table is open is too late: the engine has already registered into "all"
using the still-set bits, and since record_gtid is meant to be a standalone
autocommit-style write, nothing will later issue the matching "all"-level
commit to clear that registration and the performance-schema transaction
handle, and it leaks into whatever runs on this THD next.
mariadb-PranavTiwari
MDEV-39343: Restoring from a mysqldump from older version makes
mysql_upgrade version test fail.
The mariadb-upgrade tool no longer uses the mysql_upgrade_info file to
track the server version during an upgrade.
Changes:

1- Create the mysql.mysql_upgrade_info table and seed it with the server
binary version when MariaDB is installed.

2- Make mariadb-dump add a DROP TABLE mysql.mysql_upgrade_info statement
to the dump. This ensures that the table is removed on the target server
when restoring a dump created from an older server. The table will then be
recreated if it exists on the source server.

3- Update mariadb-upgrade to perform the upgrade if
mysql.mysql_upgrade_info is missing or contains a version older than the
current server binary version. Otherwise, reject the upgrade.
Rex Johnston
MDEV-41225 Parse the ORDER BY tail of a wrapped parenthesized query in the wrapping select

A parenthesized query expression that already has its own ORDER BY or
LIMIT and is followed by another ORDER BY, e.g.

  (SELECT c1 FROM t1 ORDER BY c1 LIMIT 2) ORDER BY <expr> LIMIT 1

is wrapped into a derived table, and the outer ORDER BY/LIMIT is attached
to the wrapping select.  The grammar pushed the inner select before
parsing the tail, and add_tail_to_query_expression_body_ext_parens()
created the wrapper only after the tail had been parsed.  Anything that
is attached to the current select while the tail is being parsed was
therefore attached to the inner select, while the ORDER BY list itself
was moved to the wrapper:

- Window functions and their window specs were added to the inner
  select's window_funcs/window_specs.  The same Item_window_func was then
  in the wrapper's ORDER BY and in the inner select's window function
  list, which triggers the original assertion reported in MDEV-41225.

  (SELECT 1 FROM t1 LIMIT 1)
    ORDER BY PERCENTILE_DISC(1) WITHIN GROUP(ORDER BY TIME'0') OVER();

  The function is computed by the inner select over the rows before its
  LIMIT, so the outer ORDER BY sorts on wrong values
  (e.g. COUNT(*) OVER () returned 3 for a 2-row result).

- Subqueries were registered as units of the inner select.  An
  uncorrelated subquery in the tail crashed the server with SIGSEGV when
  it was executed by the outer filesort.  A correlated one resolved its
  outer references in the inner select (t1.c1) instead of the derived
  table.

The decision to wrap depends only on whether the inner select already
has a tail and on whether the outer tail has ORDER BY.  LIMIT and locking
clauses cannot contain window functions or subqueries.
We replace query_expression_tail with
query_expression_tail_with_order and query_expression_tail_no_order
so the mid-rule action knows whether an ORDER BY follows.
The new LEX::push_select_for_ext_parens_tail() creates the wrapper before
an ORDER BY tail is parsed when wrapping is required, and pushes it instead
of the inner select.  The tail's items are then created in the wrapper's
context, and window functions and subqueries are attached to the wrapper.
add_tail_to_query_expression_body_ext_parens() takes the pushed select and
skips wrapping when it has already happened; the other cases keep the
previous logic.  The grammar has the same number of conflicts as before.

A subquery in such a tail can no longer refer to the tables of the wrapped
query expression (e.g. t1.c1 above) and gets ER_BAD_FIELD_ERROR,
as those tables are not visible outside the derived table.

This commit was prepared with Claude Code (Opus 5.5). It traced the
wrong-result and crash cases to the parse-time registration of window
functions and subqueries in the inner select, tested the grammar split
with bison, wrote the code change and the brackets test.
Aleksey Midenkov
MDEV-39074 trans_rollback_stmt(THD *): Assertion `! thd->in_sub_stmt' failed.

slave_close_thread_tables() unconditionally called
trans_commit_stmt()/trans_rollback_stmt(), which assert
!thd->in_sub_stmt. A BINLOG statement with malformed base64 payload
executed from an AFTER INSERT trigger hits this: on decode failure,
mysql_client_binlog_statement() sets thd->is_error() and calls
slave_close_thread_tables() from within the trigger's sub-statement.

Guard it with spcont/in_sub_stmt, deferring cleanup to the
enclosing top-level statement. Other callers run only from the
top-level SQL slave applier thread, so this doesn't change their
behavior.

mysql_client_binlog_statement() itself refuses spcont/in_sub_stmt
with ER_SP_BADSTATEMENT before decoding anything. The guard above
is defense-in-depth for its other, top-level-only callers.

1. BINLOG statement executed from a trigger, SF or SP is disabled
by the patch:

Row events (Rows_log_event) open their target table via a
one-shot check that only fires at the top of a fresh statement.
Inside a trigger, the table is never opened this way. It may be
fixed by reusing query_tables, but:

  - find_locked_table() matched only by table name -- could return
    the TABLE instance the enclosing statement was actively writing
    through, not an idle one. Reusing it would require pre-saving its
    state: record[0]/bitmaps/handler/etc. (will deprecate
    ER_CANT_UPDATE_USED_TABLE_IN_SF_OR_TRG)

  - set_stmt_row_injection()/set_time() calls mutated thd->lex, which
    at that point is main_lex -- shared with the enclosing statement,
    not something safe to touch.

Statement events (Query_log_event) run the embedded query via
mysql_parse(): it bundles lex_start(), reset_for_next_command() and
parse_sql() as one unit meant for a genuinely new top-level statement,
not a one-off nested parse.

Calling parse_sql() directly instead avoids that, but then we own
everything mysql_parse() was doing for us: a private LEX and a
private Query_arena (or allocations land on main_lex/whatever arena
is currently active, shared with the enclosing statement), plus
calling mysql_execute_command() ourselves afterwards.

In any case, DML for query_tables cannot be done due to
ER_CANT_UPDATE_USED_TABLE_IN_SF_OR_TRG reasons explained above.

2. PS for BINLOG statement still works and needs a leak fix for
statement events:

A BINLOG statement decoding to a Query_log_event, executed via
PREPARE/EXECUTE, leaked its nested query's allocations onto the PS's
own persistent arena: thd->stmt_arena pointed at it while mysql_parse()
ran the decoded query. Second EXECUTE asserted on ROOT_FLAG_READ_ONLY,
since PROTECT_STATEMENT_MEMROOT marks that arena read-only after a
successful execution.

The fix redirects thd->stmt_arena to thd itself for the duration of
the nested mysql_parse() call, so stmt_arena->is_conventional() reads
true and activate_stmt_arena_if_needed() (called e.g. from
save_leaf_tables()) never redirects allocations to the PS's arena in
the first place.

Harmless for what it's protecting: leaf_tables_exec is normally cached
on the persistent arena so a repeatedly-executed statement's
SELECT_LEX doesn't rebuild it every time, but our SELECT_LEX is torn
down and reparsed fresh (due to mysql_parse() semantics) on every
EXECUTE, so there's nothing to cache here regardless of which arena is
used.
Lawrin Novitsky
Fixed the test for the case of SSPS
Aleksey Midenkov
MDEV-41050 versioned DELETE via row_end index leaves a row undeleted

1. Aria/MyISAM engines

A system-versioned DELETE on a MyISAM or Aria table indexed by row_end
could leave the last current row undeleted. The delete scans the current
rows via an equality search on row_end = MAX, which is driven by
mi_rnext_same/maria_rnext_same. That function keeps the search's
reference key in lastkey2 and uses the HA_STATE_RNEXT_SAME flag to
remember it has already stored it. This reference-key mechanism is
exactly what lets the scan modify rows it is walking without skipping
them.

Deleting a versioned row is an in-place update of row_end, and the update
reuses lastkey2 as scratch space for the changed key, so it clears
HA_STATE_RNEXT_SAME to request that rnext_same re-store its reference on
the next call. However, TABLE::delete_row wraps the update in
HA_EXTRA_REMEMBER_POS/HA_EXTRA_RESTORE_POS, and RESTORE_POS restored the
whole saved info->update word, resurrecting the HA_STATE_RNEXT_SAME bit
that the update had just cleared.

As a result rnext_same skipped rebuilding its reference key and compared
subsequent keys against the now-overwritten lastkey2, hitting a spurious
end-of-file and terminating the scan one row early, defeating the engine's
own protection against a Halloween-style skip.

Fixed by preserving the current HA_STATE_RNEXT_SAME bit across
RESTORE_POS instead of restoring the stale saved value. See also the HEAP
fix below: same root cause, different per-engine mechanism.

2. HEAP engine

The row loss also reproduces on the MEMORY (HEAP) engine. A system-
versioned DELETE scans the current rows on the row_end index and turns
each delete into an in-place update of row_end, so it modifies the very
index it is walking.

When the changed key is the scanned one (info->lastinx), hp_delete_key()
repositions the cursor but heap_update() leaves info->update untouched, so
HA_STATE_NEXT_FOUND from the preceding heap_rnext() stays set. The next
heap_rnext() then sees current_ptr == 0 with that bit and takes the
"!current_ptr && HA_STATE_NEXT_FOUND" guard as a false end-of-file,
stopping one row early.

Fixed by clearing HA_STATE_NEXT_FOUND when the scanned index key changed.
HA_STATE_AKTIV is kept (unlike heap_delete): the row is updated, not
removed, so a following op must not fail test_active(). The bit is only
set after a heap_rnext(), so a plain single-row UPDATE never reaches this.
See also the Aria/MyISAM fix above: same root cause, different per-engine
mechanism.

3. Why the fix differs per engine, and InnoDB

Aria and HEAP both trace their handler design back to MyISAM, hence the
same root cause (stale scan bookkeeping after an in-place key change) in
all three, fixed at each engine's own bookkeeping spot. InnoDB needs no
fix: its persistent cursor survives concurrent index modification by
design, already covered by this same test under the timestamp
combination (default-storage-engine=innodb), which passes unmodified.
Marko Mäkelä
MDEV-41040 BACKUP SERVER TO is slower than mariadb-backup

mariadb-backup --backup always uses a dedicated log_copying_thread()
that eagerly copies the log from the server. Let us do the same
in multi-threaded BACKUP SERVER TO, unless innodb_log_file_buffering=OFF
(which prevents arbitrary-size reads from the log file).

backup_sink::id: The thread identifier (0 to CONCURRENT-1)

InnoDB_backup::context::tracked: Log file queue.

innodb_backup_checkpoint_pmem(), innodb_backup_checkpoint():
Enqueue or detach the old log file.

innodb_backup_log_tracking(): Determine if log-tracking backup
is active, preventing SET GLOBAL innodb_log_file_buffering and
SET GLOBAL innodb_log_file_write_through.

InnoDB_backup::log_track(), Keep copying the log until we
run out of InnoDB data files to copy.
Marko Mäkelä
MDEV-33966: buf_page_make_young() is a contention point

The buf_pool.LRU list needs to reasonably accurately reflect recently
accessed blocks, so that they will not be evicted prematurely.
Because the list is protected by buf_pool.mutex, it is not a good idea
to maintain the position on every page access.

Instead of maintaining the LRU position on each access, we will
decide on each access whether the block qualifies for promotion, and
record the decision in a "promote" flag in the block descriptor. The
buf_flush_page_cleaner() thread as well as some traversal of the
buf_pool.LRU list, all of which hold buf_pool.mutex anyway, will move
the flagged blocks to the "recently used" end of buf_pool.LRU.

A block qualifies only if it is accessed again at least
innodb_old_blocks_time after the reference point, which is its first
access and, after that, its most recent promotion. The age is measured
in whole seconds and checked on the access, not when a sweep reaches
the block, so that a block that a table scan accessed in one burst
will not be promoted, however long it stays in buf_pool.LRU_old. After
a promotion, the age is measured from the time the sweep moved the
block, which can be later than the access that qualified it. The rule
applies at any position in buf_pool.LRU, and the flag stays set until
a sweep moves the block. Thus, a block that qualified outside
buf_pool.LRU_old keeps the flag until a sweep reaches it, which is
usually after it has moved into buf_pool.LRU_old; before, such a
block was moved on any access when freed_page_clock showed that it was
no longer close to the "recently used" end.

buf_page_make_young_if_needed(), buf_page_make_young(),
buf_page_peek_if_too_old(), buf_page_peek_if_young(),
btr_cur_nonleaf_make_young(), buf_page_t::set_accessed(): Replaced by
buf_page_t::touch(), buf_page_t::touch_no_stamp() and
buf_page_t::make_young_if_needed().

buf_pool_t::freed_page_clock, buf_page_t::freed_page_clock: Remove.
This is no longer meaningful in the revised design.
INFORMATION_SCHEMA.INNODB_BUFFER_PAGE(_LRU).FREE_PAGE_CLOCK now
always reports 0. Before the first eviction, buf_LRU_stat_update() now
records statistics intervals, and buf_LRU_evict_from_unzip_LRU() uses
its formula instead of assuming a disk-bound workload.

page_zip_des_t::state: An atomic 16-bit field that will include
the PROMOTE and OLD flags that would more logically belong
to buf_page_t. We maintain them here (along with some
ROW_FORMAT=COMPRESSED specific state that is protected by page
latches) in order to avoid race conditions and unnecessary
memory overhead. m_end moves out to its own field, shrinking the
packed word to 16 bits; n_blobs narrows from 12 bits to 10 to make
room for PROMOTE and OLD, still comfortably above the 744-column
maximum.

The helpers that set or clear one bit of page_zip_des_t::state always
perform the atomic read-modify-write, which is a full barrier on x86.
A caller that can find the bit already in the target state checks it
first: touch_no_stamp() for PROMOTE, buf_LRU_add_block() for OLD, and
page_zip_write_rec() for NONEMPTY. page_zip_des_t::copy_from() and
set_n_blobs_and_empty() skip the fetch_add() when the delta is 0.
page_zip_decompress_clust() counts the BLOB pointers locally and
invokes add_n_blobs() once.

page_zip_des_t::copy_from(): Replaces the copy constructor that
page_zip_copy_recs() invoked. It copies m_end, NONEMPTY and n_blobs
from the source page, but preserves the PROMOTE and OLD flags of the
destination block, because they describe the position of that block
in buf_pool.LRU. Copying OLD from the source would make
buf_pool.LRU_old_len inconsistent with the list. Like
set_n_blobs_and_empty(), it updates the state with a single
fetch_add(), which preserves any concurrent change of PROMOTE or OLD.

buf_page_t::init(): Takes the ROW_FORMAT=COMPRESSED shift size
(ssize) directly and clears the zip descriptor state itself, so
callers (buf_block_t::initialise(), buf_page_init_for_read()) no
longer need a separate page_zip_des_init()/page_zip_set_size() call.

buf_page_t::invalidate(): Replaces buf_block_modify_clock_inc().
Instead of maintaining a 64-bit counter, we will maintain one
comprising 32+16=48 bits, in modify_clock_low,modify_clock_high.
Worst case there will be exactly n<<48 calls to
buf_page_t::invalidate() before some operation such as
btr_pcur_t::restore_position() is executed. Such a count should
be extremely unlikely but not completely impossible. It is worth
noting that the DB_TRX_ID is only 48 bits, and each transaction
start and commit/rollback will consume an identifier.

buf_page_t::modify_clock(): Replaces the read access of
buf_page_t::modify_clock. Assert that the caller is holding
a page latch. Note: because invalidate() and modify_clock() are
protected with buf_pool.mutex or the buf_page_t::lock, there can
be no issue with regard to the atomicity of accessing the 48-bit field.

buf_page_t::access_time: Store the 16-bit buf_pool.access_clock
rather than a 32-bit millisecond ut_time_ms(). It wraps around every
18.2 hours; ages are computed as uint16_t(now - access_time), which
stays correct across that wrap. This avoids any alignment loss: the
adjacent fields modify_clock_low, modify_clock_high, access_time of
32+16+16 bits nicely add up to 64 bits. access_time is stamped on the
first access after the block was initialized, and on each promotion
by make_young_if_needed(), so that the age of a frequently promoted
block stays exact across the wrap.

buf_pool_t::access_clock: uint16_t(my_interval_timer() / 1000000000),
never 0 (see buf_pool_t::now()), refreshed about once per second by
buf_pool_t::refresh_clock() so that page accesses need not read the
system clock.
srv_master_callback() refreshes it. From buf_pool_t::create() until
srv_master_timer is started, and for good when srv_master_timer is
not started (innodb_read_only, innodb_force_recovery>=2,
mariadb-backup), a separate buf_pool_clock_timer refreshes it.

buf_pool_t::access_clock, buf_pool_t::LRU_old_threshold: Located in a
cache line of their own, because they are read on page accesses and
the adjacent buf_pool fields are frequently written.

buf_page_t::touch(): Stamp access_time on the first access, then
invoke touch_no_stamp(). Return whether this was not the first access,
as the result of buf_page_make_young_if_needed() used to be. A
buffer-fix is sufficient, as in MVCC undo page lookups.

buf_page_t::touch_no_stamp(): Set the PROMOTE flag if
innodb_old_blocks_time is 0, or if accessed_at() is at least that old,
at any position of the block in buf_pool.LRU. Because
buf_pool.access_clock advances once per refresh_clock(), about once per
second, a difference of threshold ticks can span less than
innodb_old_blocks_time; the difference must exceed the threshold, so
that a burst of accesses that crosses a refresh is not promoted. Like
btr_cur_nonleaf_make_young(), do not stamp access_time. Once PROMOTE is
set, later accesses only load the state and return.

buf_page_t::make_young_if_needed(): If PROMOTE is set, invoke
buf_page_t::make_young(). This part is inline, so that a sweep pays no
function call for a block that stays in place. It reads the state of
the block once: buf_pool.mutex, which the caller holds, protects OLD,
and only make_young() clears PROMOTE. The block need not be in
buf_pool.LRU_old: while buf_pool.LRU is shorter than
BUF_LRU_OLD_MIN_LEN, buf_pool.LRU_old does not exist, and blocks are
evicted without ever becoming old. The template parameter only_old
leaves a block that is not old in place; buf_do_flush_list_batch()
uses it, because it visits blocks in the order of oldest_modification,
and moving a block that is not old would not save it from eviction.
The template parameter count_not_young selects whether an old block
that was accessed but not flagged is counted in
buf_pool.stat.n_pages_not_made_young; only the eviction sweeps
buf_LRU_free_from_common_LRU_list() and buf_flush_LRU_list_batch() do
this, and a block that a sweep leaves in the list can be counted again
by a later sweep. As before, buf_pool.stat.n_pages_made_young counts
only the moves of old blocks. Because a sweep moves the block, not the
access, Innodb_buffer_pool_pages_made_young can differ from before for
the same workload.

buf_page_t::make_young(): Clear PROMOTE, stamp access_time, and move
the block to the "recently used" end of buf_pool.LRU. The block can be
read-fixed, because buf_pool_t::unzip() copies PROMOTE and OLD from the
compressed-only descriptor and releases buf_pool.mutex during
buf_zip_decompress(). Unlike buf_page_make_young(), we do not skip such
a block: a page access no longer acquires buf_pool.mutex, and the
sweeps hold buf_pool.mutex but no buffer-fix on the block.

buf_LRU_remove_block(): When buf_pool.LRU becomes too short for
buf_pool.LRU_old, clear the OLD flags from buf_pool.LRU_old to the end
of the list, which holds all the old blocks, not the flags of every
block. While buf_pool.LRU_old does not exist, no block in buf_pool.LRU
is old, so on later removals the loop does nothing.

buf_LRU_add_block(): Write the OLD flag only if it changes. In a list
that is too short for buf_pool.LRU_old, assert that buf_pool.LRU_old
is not defined and that the block is not old, instead of writing the
flag.

innodb_old_blocks_time_update(): New sysvar update callback,
replacing a NULL one, that calls buf_pool_t::set_old_threshold_ms(),
so that SET GLOBAL innodb_old_blocks_time also updates the
LRU_old_threshold in seconds that page accesses read. The threshold
is clamped to 65535 seconds, matching the access_time wrap period;
the sysvar itself still accepts up to UINT_MAX32 milliseconds.

buf_pool_t::set_old_threshold_ms(): Round the millisecond threshold
up, not down, to the nearest second. innodb_old_blocks_time is
documented and accepted in milliseconds; flooring instead of ceiling
would make any configured value from 1 to 999 silently behave as 0
(disabled).

buf_page_t::is_accessed(): Renamed accessed_at(), to stop reading as
a boolean. It returns the access_time stamp of the first access or
of the last promotion, in seconds;
INFORMATION_SCHEMA.INNODB_BUFFER_PAGE(_LRU).ACCESS_TIME now reflects
that.

buf_read_ahead_random(): A page now qualifies once accessed_at()
holds, together with either zip.is_promote() or !zip.old().

buf_read_ahead_linear(): Compare access_time stamps as a signed
16-bit difference, not raw unsigned, so the monotonic-access check
stays correct across the access_time wrap. The resolution of the
stamps is 1 second instead of 1 millisecond.

buf_flush_page_cleaner(): Refresh abstime before proceeding to LRU
eviction after an idle period, so that the next my_cond_timedwait()
will not return immediately on a stale deadline.

buf_LRU_scan_and_free_block(): Declare static.

buf_flush_LRU_list_batch(): In a run of blocks that
make_young_if_needed() moves, release and reacquire buf_pool.mutex
after every 512 scanned blocks, except on the first scanned block.
A move costs much less than an eviction, so the stride is longer than
the one of the eviction path.

buf_pool_invalidate(): Define in the same compilation unit with
buf_LRU_scan_and_free_block().

buf_pool.LRU_old_time_threshold: Replaces buf_LRU_old_threshold_ms.

PageConverter::run(): Renamed from fil_iterate(). In debug builds,
initialize and acquire a dummy exclusive latch on the block, so the
assertion in buf_page_t::invalidate() is satisfied; free the latch on
every path, not only on success.

AbstractCallback::m_zip_ssize: Replaces m_zip_size.

page_zip_des_t: Add calc_ssize()/zip_size() helpers, replacing the
zip_size<->ssize conversion duplicated across buf0buf.cc, buf0rea.cc
and row0import.cc.

innodb.buf_lru_scan_resistance: A new big test that checks that pages
of a table that is accessed in one burst are evicted, even if a sweep
reaches them only after innodb_old_blocks_time, that pages accessed
again after that time are promoted, and that pages read while the
buffer pool is being filled are not promoted later.

innodb_zip.n_blobs_700: A new test that stores 700 BLOB pointers on
one ROW_FORMAT=COMPRESSED page, within the 10-bit n_blobs field. The
index_page_splits counter confirms that the clustered index consists
of its root page alone, without depending on which pages remain in the
buffer pool.

Co-Authored-By: Alessandro Vetere <[email protected]>
Dave Gosselin
MDEV-24907:  Empty the source list after merging an Item_equal

With save_merged false, Item_equal::merge_into_list() merges this
object's Item_equal instances into another, linking *this object's
items into the target.  However this allowed callers to free an
item via one list but later read it through the other; a case of
use-after-free.

If save_merged is false then delete the items from the source.
Daniel Bartholomew
bump the VERSION
  • cc-x-codbc-windows: 'dojob pwd if '3.4' == '3.4' ls win32/test SET TEST_DSN=master SET TEST_DRIVER=master SET TEST_PORT=3306 SET TEST_SCHEMA=odbcmaster if '3.4' == '3.4' cd win32/test if '3.4' == '3.4' ctest --output-on-failure' failed -  stdio
Marko Mäkelä
MDEV-41040 BACKUP SERVER TO is slower than mariadb-backup

mariadb-backup --backup always uses a dedicated log_copying_thread()
that eagerly copies the log from the server. Let us do the same
in multi-threaded BACKUP SERVER TO, unless innodb_log_file_buffering=OFF
(which prevents arbitrary-size reads from the log file).

backup_sink::id: The thread identifier (0 to CONCURRENT-1)

InnoDB_backup::context::tracked: Log file queue.

innodb_backup_checkpoint_pmem(), innodb_backup_checkpoint():
Enqueue or detach the old log file.

innodb_backup_log_tracking(): Determine if log-tracking backup
is active, preventing SET GLOBAL innodb_log_file_buffering and
SET GLOBAL innodb_log_file_write_through.

InnoDB_backup::log_track(), Keep copying the log until we
run out of InnoDB data files to copy.
Aleksey Midenkov
MDEV-40854 Use of uninitialized table_will_be_deleted in federated engine

MemorySanitizer report:

==851408==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 ha_federated::end_bulk_insert() storage/federated/ha_federated.cc:2035:30
    #1 mysql_insert(THD*, ...) sql/sql_insert.cc:1258:11
    ...
  Memory was marked as uninitialized
    #0 __msan_allocated_memory
    #1 my_malloc mysys/my_malloc.c:116:7
SUMMARY: MemorySanitizer: use-of-uninitialized-value ... end_bulk_insert()

table_will_be_deleted is a handler member with no in-class
initializer, and the handler object itself is heap-allocated via
my_malloc(), so it starts out as garbage. It was only ever set in
extra(HA_EXTRA_PREPARE_FOR_DROP) and in external_lock().  For a
TEMPORARY table, statement execution can reach
end_bulk_insert()/write_row(), which reads the flag, without
external_lock() having run first, so the read sees uninitialized
memory.

The fix initializes table_will_be_deleted in reset(), which runs
after each statement, so the flag is cleared before the next
statement reaches end_bulk_insert() regardless of locking path.
Kristian Nielsen
MDEV-41264: binlog-in-engine dump thread not stopping at server shutdown

The binlog-in-engine code for binlog dump thread didn't check
should_stop(info, true) to stop on KILL_SERVER when it reaches the end of
the current binlog. This caused the dump threads to hang indefinitely during
SHUTDOWN WAIT FOR ALL SLAVES, and thus the shutdown to hang as well.

Signed-off-by: Kristian Nielsen <[email protected]>
Lawrin Novitsky
The C/C has been updated to the v3.4.11
Marko Mäkelä
MDEV-41010 BACKUP SERVER TO is slower than mariadb-backup

mariadb-backup --backup always uses a dedicated log_copying_thread()
that eagerly copies the log from the server. Let us do the same
in multi-threaded BACKUP SERVER TO, unless innodb_log_file_buffering=OFF
(which prevents arbitrary-size reads from the log file).

backup_sink::id: The thread identifier (0 to CONCURRENT-1)

InnoDB_backup::context::tracked: Log file queue.

innodb_backup_checkpoint_pmem(), innodb_backup_checkpoint():
Enqueue or detach the old log file.

InnoDB_backup::log_track(), Keep copying the log until we
run out of InnoDB data files to copy.
Oleksandr Byelkin
MDEV-41258 Crash in Item_func_nextval on prepared re-execution

A table's DEFAULT NEXTVAL(seq) column keeps a per-open TABLE_LIST
describing the sequence table, allocated on that TABLE's own mem_root.
add_internal_tables() cached a matching prelocking entry across
statement re-executions by comparing db.str/table_name.str pointer
identity. If the owning table was closed and reopened between two
executions (FLUSH TABLE, or ordinary table-cache eviction under
concurrent load), those pointers went stale, and pointer-identity
matching also failed to recognize the reopened table's fresh entry as
the same one, leaving a dangling duplicate in the prelocking list that
still got opened and crashed.

Record the owning TABLE_LIST on each internal-table entry and match an
owner's entries by position instead of by name pointer, which survives
a reopen. Deep-copy db/table_name onto the statement's own arena once,
at entry creation, instead of aliasing TABLE-owned memory, so they
never need refreshing.

Separately, the relink step was gated the same way as trigger/routine
prelocking discovery (has_prelocking_list), which stays true forever
once a statement's prelocking set includes a trigger -- silently
skipping the relink on every execution after the first for any such
statement. Moved it into its own function, called unconditionally per
table from open_and_process_table(), guarded so it still only applies
where DML_prelocking_strategy used to run it (not ALTER or INSERT
DELAYED, whose statements can have for_insert_data set without meaning
"open my DEFAULT-sequence dependencies").

Also:
- clear a stale internal-table entry's linked_table in
  TABLE_LIST::reinit_before_use() when its owning table's own open gets
  skipped in a given execution, so a later open_table() backlink write
  can't land on memory the owner may have already freed;
- keep track of whether the prepared statement text itself uses DEFAULT
  (as opposed to a parameter bound "USING DEFAULT"): set_parameters()
  overwrote lex->default_used every execution from this execution's own
  parameters only, so "UPDATE t SET a = DEFAULT" stopped relinking on
  the second EXECUTE.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Mohammad Tafzeel Shams
MDEV-41242 : Fix resource leaks on InnoDB/mariabackup error paths found by Infer

Several error-handling paths returned without releasing a resource
already acquired earlier in the function, or checked the wrong handle
entirely, risking use of an unopened handle.

Changes:
- SysTablespace::read_lsn_and_check_flags(): close the datafile handle
  on header-validation failure.
- xb_process_datadir(): check the freshly opened `dir` handle instead
  of the stale `dbdir`, fixing a handle leak and a possible use of an
  unopened directory handle.
- wsrep.cc / xb_load_list_file(): close file handles before die(), and
  null-check fopen() results in wsrep.cc.
- datadir_iter_new(): free datadir_path and destroy the mutex on the
  os_file_opendir() failure path.
Kristian Nielsen
mysqltest: Implement --enable_sync_gtid option

The --enable_sync_gtid option switches to use GTID-based
--sync_slave_with_master (eg. using MASTER_GTID_WAIT() instead of
MASTER_POS_WAIT()).

This is useful to run adapt existing test cases for use with
--binlog-storage-engine. But it is also useful in general for replication
using GTID (which is the default).

The option is off by default to not randomly break existing tests.

Signed-off-by: Kristian Nielsen <[email protected]>