Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Sergei Golubchik
MDEV-39596 Unexpected "No database selected" error for a CALL inside an anonymous block

For name1.name2() the PATH loop searches CURRENT_SCHEMA for a
package name1. In an anonymous block without a current database
resolve_current_schema() called copy_db_normalized(), which raised
ER_NO_DB_ERROR. But without a current schema there is no package
to find, name1.name2 can only be schema.routine.

Skip the unresolvable CURRENT_SCHEMA in an anonymous block, as it is
done outside of stored programs. Unqualified names never get here
without a current database, make_sp_name() and
Create_qfunc::create_func() reject them first.

Assisted-by: Claude:claude-opus-5-5
Marko Mäkelä
fixup! 2f9d9c14cb2550c95163e0a11fdb4a60732783d1

InnoDB_backup::archive_create(): Be PMEM agnostic
Sergei Golubchik
MDEV-40543 Assertion `opt->value.str' failed in append_create_options on ALTER TABLE ... REORGANIZE PARTITION

partition syntax is printed from the parsed option list, which
contains =DEFAULT (NULL value) entries. Skip them, like frm writer does.

Assisted-by: Claude:claude-opus-5-5
Jan Lindström
MDEV-41171 I_S.TABLES fast path ignores ROWNUM and ROWS EXAMINED

The fast path of MDEV-31342 streams rows of a simple SELECT on
INFORMATION_SCHEMA.TABLES. ROWNUM() in the WHERE clause and LIMIT
ROWS EXAMINED are handled by the join. The fast path bypassed both
and returned every row without a warning.

Do not use the fast path for a query with ROWNUM() or with LIMIT
ROWS EXAMINED.

The test also covers @@sql_select_limit and FOUND_ROWS().

Assisted-by: Claude:claude-4.6-sonnet
Vladislav Vaintroub
MDEV-38741 MYSQL_METHODS: the same layout in all builds

The members of MYSQL_METHODS after on_close_free existed only if
!MYSQL_SERVER or EMBEDDED_LIBRARY, so the client in the server
(sql-common/client.c, built with MYSQL_SERVER) had a shorter table than
libmysqld/libmysql.c, which mariabackup builds as a client.
mysql_next_result() read methods->next_result beyond the end of that
table, and crashed.

Declare all the members always, and set all of them in client_methods
and in local_methods, next_result of a client is read_query_result.
Revert the check of the embedded server in mysql_next_result(), it is
not needed now.

Assisted-By: Claude:claude-sonnet-5-5
Vladislav Vaintroub
MDEV-38741 perfschema.nesting: one more THD::LOCK_thd_data wait

dispatch_command() now takes THD::LOCK_thd_data around set_command() and
the check of the disconnect state, which makes the disconnect watcher
and the start of a statement agree. The test lists every instrumented
mutex wait of a statement, so each of its five statements has one more
wait/synch/mutex/sql/THD::LOCK_thd_data in the "starting" stage, and
the event ids after it are shifted by one.

Assisted-By: Claude:claude-sonnet-5-5
Jan Lindström
MDEV-40212 galera_duplicate_primary_value stalls on single applier

Failure:
- The test needs two write sets applied concurrently by two
  appliers on node 1, but never sets wsrep-slave-threads, so it
  defaults to 1.
- lock_wait_wsrep_kill's debug sync parks the applier thread
  itself on WAIT_FOR wsrep_after_kill_continue; with only one
  applier, the second write set has no free thread, and WAIT_FOR
  wsrep_after_kill_reached_2 blocks until --debug-sync-timeout
  (300s).
- The timeout does not fail the test, so it reports pass without
  ever reaching the concurrent brute-force state MDEV-31658 added
  it to cover.

Fix:
- Set wsrep-slave-threads=2 on node 1, so the two independent
  write sets apply concurrently, matching what the test is meant
  to exercise.

Assisted-by: Teemu Ollakka <[email protected]>
Assisted-by: https://mariadb.org/governance/governance-ai-policy/
bsrikanth-mariadb
MDEV-41748: Syntax error with BETWEEN in FederatedX pushdown

A multi-table UPDATE/DELETE pushed down to FederatedX failed on the
remote server with a syntax error when the WHERE clause had a BETWEEN
whose bounds are constants, e.g.

  UPDATE t1 AS a, t1 AS b SET a.x=NULL WHERE a.x BETWEEN NULL AND NULL;

The bounds are wrapped in Item_cache objects. The cache was not filled
yet when the query was printed, so Item_cache::print() emitted
"<cache>(NULL)", which is not valid SQL.

Fix: when printing with QT_PARSABLE, an Item_cache that holds no value
now prints the item it caches. A cache that already holds a value still
prints it as a literal. Output for callers that do not pass
QT_PARSABLE is unchanged. Only FederatedX pushdown uses QT_PARSABLE.

Add a test that pushes down multi-table SELECT, UPDATE and DELETE with
NULL and DATE BETWEEN bounds, and checks both the statements the remote
server received and the resulting data.
Vladislav Vaintroub
MDEV-38741 mariabackup: wait for the statements of the DDL thread

The debug thread of mariabackup that emulates a DDL sends several
statements in one query, and closed its connection as soon as the first
of them returned. The server used to run the rest, now the disconnect
watcher aborts the ALTER of a client that is gone, and
mariabackup.innodb_ddl_on_intermediate_table hangs, as the debug sync
signals the main thread waits for do not come. The thread reads the
results of all the statements before it closes the connection.

mysql_next_result() of libmysqld/libmysql.c, which mariabackup uses,
called methods->next_result. Only the methods of the embedded server
have it, the client in the server (sql-common/client.c, built with
MYSQL_SERVER) has a shorter table, and the call crashed. Outside of the
embedded server it calls read_query_result, which is next_result of the
client library.

Assisted-By: Claude:claude-sonnet-5-5
bsrikanth-mariadb
MDEV-41749: FederatedX pushdown sends "<nop>(" in pushed-down SQL

A pushed-down statement containing "x < SOME (subquery)" failed on the
remote server with a syntax error, as the query text contained the
internal "<nop>(...)" wrapper. "x < ALL (subquery)" is wrapped in
"<not>(...)" in the same way.

Item_func_not_all::print() prints the wrapper whenever `show` is set.
Omit it when QT_PARSABLE is requested and print the wrapped predicate in
parentheses, as precedence() reports the wrapper, so a parent expression
would not add parentheses itself.

While the ALL/ANY subquery item is intact, args[0] prints the original
"x op ALL|ANY (subquery)". When the subquery is reduced to a constant
("x < ALL (SELECT 2)"), it is replaced by a comparison with the inverted
comparator, and the negation is held by "<not>". Print it explicitly as
"NOT (x >= 2)" for ALL, otherwise the remote server returns the
complementary rows. ANY/SOME needs no negation.

Item_allany_subselect::print() printed its left operand without
parentheses, so "(a XOR b) < ALL (subquery)" was pushed as
"a xor b < all (...)" and parsed by the remote server as
"a XOR (b < ALL ...)". Use print_parenthesised(), as
Item_in_subselect::print() does.

Add tests to federatedx_pushdown_upd_del covering SOME and ALL in
UPDATE, DELETE and SELECT, with results compared with pushdown OFF and
the pushed-down text checked in the remote general log.
Dave Gosselin
MDEV-40779:  Wrong result for DISTINCT with HAVING, ROLLUP

A query with DISTINCT over GROUP BY now reads AVG, STD and VARIANCE
values from the DISTINCT temp table, and its ROLLUP rows pass HAVING
while keeping NULL from the columns that they roll-up.

The following scenario now returns correct results of -1.0000:

  CREATE TABLE t1 (a INT);
  INSERT INTO t1 VALUES (1),(2),(3);
  SELECT DISTINCT -AVG(a) FROM t1 GROUP BY a HAVING AVG(a) < 2;

Only the group with average 1 passes HAVING and goes into the DISTINCT
temp table.  Duplicate removal checks HAVING again on that row, after
all three groups have been read.  The row now stores its own average,
and the second check reads 1 from it.  Before, the check read 3, the
average of the last group, and deleted the row.

This scenario now returns 2, 2:

  SELECT DISTINCT a, a+0 FROM t1 GROUP BY a WITH ROLLUP HAVING a=2;

The ROLLUP total row has a=NULL and fails HAVING.  For this query,
HAVING is on the JOIN_TAB of the grouping table while JOIN::having is
NULL.  rollup_write_data considers JOIN_TAB::having if available,
otherwise it falls back to JOIN::having which was set during
JOIN::prepare.  Without HAVING, SELECT DISTINCT a+0 over the same
groups returns 1, 2, 3 and NULL.  The ROLLUP row stores NULL for a
even though a is not in the select list, and a+0 is computed from that
stored a.

Assisted-By: Claude Opus 5.5 <[email protected]>
bsrikanth-mariadb
MDEV-41862: Syntax error with DELETE and IN in FederatedX pushdown

A single-table DELETE with an IN subquery on the same table, pushed
down to FederatedX, failed on the remote server with a syntax error:

  DELETE FROM t3 WHERE a IN (SELECT a FROM t3 WHERE a < 3);

The target table of a single-table DELETE is not in top_join_list, so
the list is empty. st_select_lex::print() printed it as
"delete  from  using dual where ...".

Fix: in st_select_lex::print() use top_join_list only when it is not
empty, and otherwise print the tables from leaf_tables. For DELETE that
branch emits the "from " keyword. UPDATE is unchanged.

As a side effect, EXPLAIN EXTENDED of a single-table DELETE no longer
shows the dummy "using dual" and prints "delete from `t` where ...".
A DELETE through a view now prints the view instead of its base table.
Update the affected results (explain_non_select,
myisam_explain_non_select_all, opt_hints, opt_hints_index, opt_trace,
versioning.delete_history).

Add a test to federatedx_pushdown_upd_del that pushes down DELETE (also
with ORDER BY ... LIMIT and as a prepared statement) and UPDATE with an
IN subquery on the same table, and checks both the resulting data and
the statements received by the remote server in its general log.
Jan Lindström
MDEV-41171 I_S.TABLES fast path does not count examined rows

The fast path of MDEV-31342 sent rows but did not update the
examined row count. The slow query log showed rows_examined=0 for a
query that read three rows. Count each row that the fast path reads,
as the regular path does for the rows it reads from the temporary
table.

The test also covers Rows_sent, Empty_queries and the slow log.
It adds the ANALYZE SELECT case of MDEV-41165.

Assisted-by: Claude:claude-4.6-sonnet
Vladislav Vaintroub
MDEV-38741 Abort the running statement when its client disconnects

The server notices that a client is gone only when it reads from or
writes to the socket. While a statement runs, for example waiting for
an InnoDB row lock, a disconnect goes unnoticed. The statement keeps
running, and its transaction keeps owning its locks, so other
connections wait for locks of a client that is not there.

Watch the client socket from the end of login until the connection
ends, or the client sends COM_QUIT, using only the events the OS
reports on its own. When the client has gone, abort the running
statement with the new KILL_DISCONNECT state, found by thread id like
KILL does, through a kill_thd_locked() that KILL shares. A quitting
connection ignores it.

KILL_DISCONNECT is statement-level, like KILL QUERY. When it does abort
a statement, the connection ends after the error of that statement is
sent, so the commands that the client pipelined behind it do not run.
A disconnect noticed while the connection is idle, before the statement
starts, is remembered and applied when the next statement starts.
The reason, "Client disconnected", is logged once, and a failed write
of the error is not reported. An orderly COM_QUIT, or a close without
it, is handled as before.

The global variable disconnect_watcher, ON by default, can be set OFF
to switch the watcher off. It affects the connections made after the
change, and exists only where there is a watcher: Linux, Windows and
the BSD systems, not the embedded server. If the watcher cannot start,
the variable is OFF and cannot be set ON.

Linux: an epoll set, polled by the connection accepting loop.
EPOLLRDHUP is requested, as a close and a crashed client can be only a
FIN.
Windows sockets: WSAEventSelect(FD_CLOSE), with a wait per socket on a
thread pool of our own that has a single thread, so the callbacks run
one by one.
Windows named pipes: the wait is for the exit of the local client
process. A client that only closes the pipe is not noticed.
BSD and macOS: kqueue, EV_EOF.
Replication dump threads are watched too.

mysqltest: dirty_close takes an optional "linger" (the default, a FIN)
or "nolinger" (SO_LINGER 0, so the server gets a reset) argument.

main.disconnect_lock_wait, disconnect_lock_owner (the client's
transaction is rolled back and its locks are released),
disconnect_wait_kinds (a CPU-bound statement and a table level lock
wait, which do not poll the connection themselves),
disconnect_lock_wait_idle (debug), disconnect_lock_wait_pipe (Windows)
and sys_vars.disconnect_watcher_basic are new. mysql_client_test gets
test_disconnect_pipelined: a client sends two commands in one write and
half-closes while the first waits for a lock. The connection ends, and
the second command is not run.
disconnect_lock_wait and disconnect_lock_wait_pipe run with the watcher
ON and OFF, the others need it ON. innodb.xa_recovery no longer needs
the shortened lock wait timeout with the watcher ON. rpl_ssl1 and
rpl_password_boundaries wait for the session of the slave to end
before DROP USER, max_binlog_total_size and rpl_fail_register kill the
dump thread only if it is still there, and rpl_get_lock suppresses the
read error of a reset connection on Windows (errno 10054).

Assisted-By: Claude:claude-sonnet-5-5
bsrikanth-mariadb
MDEV-41862: Syntax error with DELETE and IN in FederatedX pushdown

A single-table DELETE with an IN subquery on the same table, pushed
down to FederatedX, failed on the remote server with a syntax error:

  DELETE FROM t3 WHERE a IN (SELECT a FROM t3 WHERE a < 3);

The target table of a single-table DELETE is not in top_join_list, so
the list is empty. st_select_lex::print() printed it as
"delete  from  using dual where ...".

Fix: in st_select_lex::print() use top_join_list only when it is not
empty, and otherwise print the tables from leaf_tables. For DELETE that
branch emits the "from " keyword. UPDATE is unchanged.

As a side effect, EXPLAIN EXTENDED of a single-table DELETE no longer
shows the dummy "using dual" and prints "delete from `t` where ...".
A DELETE through a view now prints the view instead of its base table.
Update the affected results (explain_non_select,
myisam_explain_non_select_all, opt_hints, opt_hints_index, opt_trace,
versioning.delete_history).

Add a test to federatedx_pushdown_upd_del that pushes down DELETE (also
with ORDER BY ... LIMIT and as a prepared statement) and UPDATE with an
IN subquery on the same table, and checks both the resulting data and
the statements received by the remote server in its general log.
Vladislav Vaintroub
MDEV-38741 perfschema.nesting: one more THD::LOCK_thd_data wait

dispatch_command() now takes THD::LOCK_thd_data around set_command() and
the check of the disconnect state, which makes the disconnect watcher
and the start of a statement agree. The test lists every instrumented
mutex wait of a statement, so each of its five statements has one more
wait/synch/mutex/sql/THD::LOCK_thd_data in the "starting" stage, and
the event ids after it are shifted by one.

Assisted-By: Claude:claude-sonnet-5-5
Sergei Golubchik
MDEV-40543 Assertion `opt->value.str' failed in append_create_options on ALTER TABLE ... REORGANIZE PARTITION

partition syntax is printed from the parsed option list, which
contains =DEFAULT (NULL value) entries. Skip them, like frm writer does.

Assisted-by: Claude:claude-5.5-opus
Jan Lindström
MDEV-31300 Suppress RESET SLAVE's Using_Gtid note across galera tests

RESET SLAVE/RESET SLAVE ALL emits a Note when it changes Using_Gtid,
and whether it does depends on the prior value, which can drift with
unrelated replication changes - galera_query_cache_invalidate,
galera_2_cluster and galera_gtid_2_cluster depend on it appearing.
Wrap every RESET SLAVE/RESET SLAVE ALL across the galera and
galera_3nodes suites in --disable_warnings/--enable_warnings, so no
test depends on that value.

Assisted-by: Claude:claude-4.6-sonnet
Khaled Riyad
MDEV-39001 Server crashes in find_type2 / Field_enum::store upon returning ENUM from function

Item_sp keeps sp_result_field, built from m_sp, across executions. A
trigger or prepared statement can outlive that sp_head, when CREATE
TEMPORARY TABLE flushes the routine cache or the function is replaced.
The next execution reused a field with a dangling ENUM TYPELIB or the
old return type. Regression of MDEV-26115.

Delete the field in Item_sp::cleanup() and build it from the current
m_sp on each execution. The field memory now belongs to the item, in
result_field_mem_root, so cleanup() can run at any time, which is what
MDEV-26115 needed.
Jan Lindström
MDEV-40211 galera_connect.inc sleeps one second on each invocation

Failure:
- galera_connect.inc sleeps 1 second, unconditionally, before every
  connect, as a 2015 workaround for a node racing its own join/IST
  against another node's concurrent join (MDEV-7178).
- ~700 tests in the galera/wsrep suites go through this include, so
  the blind sleep costs roughly that many seconds in a full run,
  whether or not any node is still joining.

Fix:
- Replace the sleep with include/galera_wait_ready.inc after
  connecting, polling the actual wsrep_ready state instead of
  guessing a fixed delay. galera_wait_ready.inc is the existing,
  already widely used helper for this, so this reuses it rather
  than adding a new check.
- Add a $galera_no_wait_ready opt-out for a node that is expected
  to join a non-Primary component and so never becomes ready;
  galera_pc_weight.test isolates node_1 before connecting node_3,
  so node_3 needs it.
- The ready check itself must not wait on causality: a test can
  legitimately connect a second session to a node whose applier is
  stuck behind a local lock (galera_lock_table.test does, via LOCK
  TABLES), and wsrep_sync_wait would otherwise block the check's own
  first query on that same stuck applier. Set it to 0 for the check
  and restore it from the global value afterward, without reading it
  back first, since even a bare SELECT of the session variable hits
  the same wait.

Assisted-by: https://mariadb.org/governance/governance-ai-policy/
Sergei Golubchik
MDEV-41630 ST_IsValid crashes on polygons with many rings
bsrikanth-mariadb
MDEV-41293: FederatedX: connection state after a net_read_timeout

Problem:
When reading the reply of the remote server times out, the client
library compiled into the server returns without closing the connection
and without setting a client error. FederatedX took the raw
ER_NET_READ_INTERRUPTED for an error of the remote server and went on
using a connection that still had the reply of the failed statement to
arrive. As a result:
* the next statement on that connection failed in various ways, either
  with ER_QUERY_ON_FOREIGN_DATA_SOURCE ("Server has gone away") or with
  an error packet that mysqltest took for a malformed packet;
* inside a transaction, the following statements continued on a new
  remote session in autocommit mode, so they were committed on the
  remote server one by one even though the local transaction was later
  rolled back. The savepoints of the lost session were also still
  remembered by FederatedX.
The same happened when the connection was lost while reading the reply.

Fix:
* After a timeout FederatedX closes the connection, so that the next
  statement opens a new one. The client still gets the error of the
  timeout, ER_NET_READ_INTERRUPTED.
* Whenever the connection is gone after a failed statement, timeout or
  not, the state of the lost session (autocommit mode, savepoints) is
  forgotten and the local transaction is marked to be rolled back, so a
  new connection starts the transaction again. This includes the case
  where the connection is lost while SET AUTOCOMMIT=0 is sent for the
  first statement of the transaction. The closed connection is also
  freed, which was leaked before.
* Errors of the client library are reported as
  ER_QUERY_ON_FOREIGN_DATA_SOURCE by ha_federatedx::info() as well, the
  same way as by the pushed down multi-table UPDATE/DELETE.
* The error state of ha_federatedx, read by info(), is initialized.
* A non-debug build checks the socket for EOF before it sends a command
  (net_clear()). If the remote server has already closed the connection, the
  write fails without the network layer raising an error, and the client
  would get ER_GET_ERRMSG ("Got error 10000 ...") intermittently. When the
  connection is lost and no error has been raised, raise
  ER_QUERY_ON_FOREIGN_DATA_SOURCE with the error of the client library. The
  COMMIT and ROLLBACK of the transaction still return their error to the
  caller.

Test: federatedx_pushdown_upd_del covers the timeout of a pushed down
multi-table UPDATE and DELETE and of reading a result, the statements
after it, the same inside a transaction, a connection killed in the
middle of a statement, a connection lost while SET AUTOCOMMIT=0 is sent
(also followed by a disconnect and a transaction in another session),
client library errors in info(), and that an error of the remote server
which merely has the code of ER_NET_READ_INTERRUPTED is not taken for a
timeout.
The two statements after the idle connection was killed accept
ER_NET_READ_ERROR (reported by the network layer in a debug build) or
ER_QUERY_ON_FOREIGN_DATA_SOURCE (non-debug build).
Marko Mäkelä
MDEV-40935: Assertion os_file_set_size() failed during shutdown

os_file_set_size(): Consistently handle EINTR also after server
shutdown has been initiated. This will avoid a crash when an
innodb_log_archive=ON log file needs to be created or extended
during shutdown. This only affects the posix_fallocate(2) or
Linux fallocate(2) code path; if the file is being extended by
regular writes, we will retry in case of EINTR.

On Microsoft Windows, there is no clear documentation what would happen
if the process is being terminated while SetFilePointerEx() or
SetEndOfFile() are in progress.
Kristian Nielsen
Fix corrupt query string in mysqltest embedded

The mysqltest --send command re-used the same buffer for the query string to
send to each connection thread. This caused a race where a following --send
could overwrite the query before the first connection thread had time to
copy out the string, resulting in executing a corrupt query.

Instead, allocate a new string for each --send and free it in the connection
thread.

This was seen as a sporadic failure in --embeded-server in test
innodb.temp_truncate_freed_race (and probably others).

Signed-off-by: Kristian Nielsen <[email protected]>
Sergei Petrunia
Merge: MDEV-39226: Push whole multi-table update/delete down into engines
Marko Mäkelä
fixup! 2f9d9c14cb2550c95163e0a11fdb4a60732783d1

Fix InnoDB_backup::de_hardlink()
Vladislav Vaintroub
MDEV-38741 mariabackup: wait for the statements of the DDL thread

The debug thread of mariabackup that emulates a DDL sends several
statements in one query, and closed its connection as soon as the first
of them returned. The server used to run the rest, now the disconnect
watcher aborts the ALTER of a client that is gone, and
mariabackup.innodb_ddl_on_intermediate_table hangs, as the debug sync
signals the main thread waits for do not come. The thread reads the
results of all the statements before it closes the connection.

mysql_next_result() of libmysqld/libmysql.c, which mariabackup uses,
called methods->next_result. Only the methods of the embedded server
have it, the client in the server (sql-common/client.c, built with
MYSQL_SERVER) has a shorter table, and the call crashed. Outside of the
embedded server it calls read_query_result, which is next_result of the
client library.

Assisted-By: Claude:claude-sonnet-5-5
Marko Mäkelä
MDEV-14992/MDEV-38362/MDEV-39092 BACKUP SERVER

The following SQL statements will be introduced:

BACKUP SERVER TO '/path/to/directory' [ 1 CONCURRENT ];
BACKUP SERVER WITH [ 1 CONCURRENT ] 'command';

In place of the 1, any positive number of threads may be specified.
For the first variant, '/path/to' must exist and '/path/to/directory'
must not exist; that is where the backup will be written to.

For BACKUP SERVER WITH, the string mariadb-backup-
will be prepended to the specified command and quoted.
Path separators or quotes are not allowed in command.
We expect the mariadb-backup-command to be in the
search PATH of the mariadbd process.
For now, we distribute no such script or program;
the user has to write one, something like this:

zstdmt|exec ssh [email protected] "exec cat > $1.tar.zstd"

The standard input of that command will be in a format compatible with
GNU tar --format=oldgnu (and also BSD tar variants that are also part of
Microsoft Windows and Apple macOS). The command is expected to optionally
compress and encrypt the stream and redirect it to a file on a local or
a remote server. The BACKUP SERVER WITH will append an additional argument,
a positive base-ten number in ASCII, starting with 1, to identify the
current thread. In this way, there can be multiple concurrent streams.
The InnoDB write-ahead log will be streamed near the end, because
the stream format requires the file sizes to be declared upfront.

In BACKUP SERVER TO ... 2 CONCURRENT or more, one thread will be
responsible for copying the InnoDB write-ahead log from the beginning
of the backup, running InnoDB_backup::log_track(). Other files will be
copied by other threads.

Note: In single-threaded BACKUP SERVER TO, the parameter
innodb_log_recovery_start that is written out to backup.cnf is
STRICTLY NECESSARY TO AVOID CORRUPTION during recovery! By default,
InnoDB crash recovery starts from the latest available log
checkpoint. However, for restoring a backup, recovery must start from
the checkpoint that was the latest when the backup was
started. Starting recovery from a possible later checkpoint will
result in a corrupted database!

The backup or the first stream will contain a file backup.cnf, which
includes parameters needed for restoring the backup. Currently,
these are innodb_log_recovery_start and innodb_log_recovery_target.
If innodb_log_recovery_target>0, InnoDB will be in read-only mode,
not allowing any writes to persistent files other than via the log
application.

To restore a streaming backup made with BACKUP SERVER WITH, an empty
directory needs to be created and all streams be extracted there using
the standard tar utility of the operating system, optionally after
undoing any encryption or compression that had been added by the
backup command. Then, the backup is prepared or MariaDB server started
up on the extracted directory, similar to as if the BACKUP SERVER TO
statement had been used.

The following will be implemented separately:

MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER
MDEV-40163 Partial backup and restore
MDEV-39091 Back up ENGINE=RocksDB
MDEV-40333 Concurrent DDL for Aria tables in BACKUP SERVER
MDEV-39987 Improve BACKUP SERVER performance for ENGINE=Aria

The implementation introduces a basic multi-threaded driver
Sql_cmd_backup, storage engine interfaces, and basic copying of the
storage engines InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV.

backup_target: A structured data type to represent a target directory.
On Microsoft Windows, we must use directory paths because there is
no variant of CopyFileEx() that would work on file handles.

backup_sink: Wraps a per-thread output stream as well as storage
engine specific context. The member "id" identifies the thread,
between 0 and CONCURRENT-1.

backup_context: Process-wide BACKUP SERVER context.
Handles the directory traversal and copying of
files for built-in storage engines that do not
implement this backup interface.

backup_target_stage: Wrap backup_context.

backup_target_stage::step(),
backup_context::step(): Process a file
from a directory scan, or by invoking
handlerton::backup_step().

handlerton::backup_file(): Check if a file should be
included in the backup. Implemented for ENGINE=Aria
in maria_backup_file().

handlerton::backup_start(), handlerton::backup_end(): Invoked at the
start or end of a backup phase, in the thread that executes a
BACKUP SERVER statement.

handlerton::backup_step(): A backup step that can be invoked from
multiple threads concurrently, between the execution of the corresponding
handlerton::backup_start() and handlerton::backup_end() of the same
phase.

copy_entire_file(): A file copying service for POSIX systems.

clone_file_range(): A wrapper for Linux FICLONERANGE, which
may fail and thus require a fallback to backup::copy().

backup::copy(): A partial or sparse file-copying service.
Note: On Linux we never invoke sendfile(2) for
copying between files, nor copy_file_range(2) for copying
large files, because can be much slower than the alternatives.

backup::copy_or_stream(): Copy or stream a file.

backup::append(): The streaming streaming equivalent of backup::copy()..

backup_stream_zeropad(): Zero-pad the last tar block if needed.

backup_stream_append_async(): A variant of backup::append()
where the source file region is guaranteed to be immutable after the
call returns. Zero-copy Linux sendfile(2) is inherently risky for copying
data files that may be modified in place, because it could introduce
a race condition between a page write that runs concurrently with a
child process that is reading the data from the pipe. This could be
improved with io_uring later.

pread_write(): Allocate a buffer of up to 4 MiB.
This is the "slow path" of copying or streaming files.
For InnoDB data files, BACKUP SERVER TO invokes this on output that
has been opened in unbuffered I/O (O_DIRECT) mode, to avoid polluting
the Linux file system cache (block cache). For the InnoDB log,
regular buffered I/O performed best in our tests.

maria_backup_start(): The entry point for ENGINE=Aria.
Currently, the directory traversal is single-threaded and a memory hog.
This is to be improved in MDEV-39987. The Aria log and data file
copying were implemented by Michael 'Monty' Widenius and
Alessandro Vetere.

maria_backup_log_tail(): Copy the necessary part of the last
ENGINE=Aria log, by making use of block-cloning when possible.
The copy will be trimmed to the logical end of the backup.
The logic was developed by Alessandro Vetere.

InnoDB_backup::init(): Wait for a possible previous BACKUP SERVER
operation to reach the very end of InnoDB_backup::context::cleanup()
so that the context can be safely reused.

InnoDB_backup::queue: Collection of tablespace IDs and payload sizes
at the start of the backup, and the log_sys.first_lsn of log files
that have to be included in the backup. If any data file is created or
extended while the backup is executing, we must have the corresponding
write-ahead-log entries that we are copying since the latest
checkpoint that was completed when the backup started. If any
tablespaces are deleted during the backup, we may or may not copy
them, and the application of a FILE_DELETE record will remove them.
Similarly, applying FILE_RENAME or FILE_CREATE records will rename or
create files during recovery as needed.

InnoDB_backup::context::tracked: Log file queue for the dedicated
log-copying task InnoDB_backup::log_track().

innodb_backup_log_tracking(): Determine if log-tracking backup is
active, which blocks concurrent changes of innodb_log_file_buffering
and innodb_log_file_write_through.

log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this
is to make BACKUP SERVER prevent the concurrent execution of
SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size
when innodb_log_archive=OFF.

log_sys.archived_checkpoint: Keep track of the earliest available
checkpoint, corresponding to log_sys.archived_lsn. This reflects
SET GLOBAL innodb_log_recovery_start (which is settable now), for
incremental backup.

fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed
to be known. To speed up startup, InnoDB does not normally open
all tablespace files.

fil_space_t::create_lsn: Change to Atomic_relaxed and use this to
indicate tablespace creation LSN, in addition to indicate undo
tablespace rebuild LSN.

fil_space_t::backup_end: The first page number that is not being backed up
(by default 0, to indicate that no backup is in progress).

fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be
covered by fil_space_t::backup_end. This is the unit of "page range locking"
during InnoDB backup.

fil_node_t::backup_name: A snapshot of the file name, sampled in
InnoDB_backup::init(). For backing up any files that were created before
the backup start checkpoint LSN, these names will be written to the backup.
On restore, the files will be renamed or deleted as needed, by
applying FILE_RENAME or FILE_DELETE records.

fil_node_t::rename(), fil_node_t::set_backup_name(),
fil_node_t::get_backup_name(), fil_node_t::~fil_node_t(),
fil_node_t::clear_backup_name(): Deal with fil_node_t::backup_name.

buf_page_t::write_fix_try(), buf_page_t::write_unfix_try(): Try to set
or unset a fake "write fix" on a page, to prevent concurrent flush()
during a backup batch. The atomic operations may run concurrently with
set_reinit() and set_freed(). The fake "write fix" does not prevent
any concurrent read or write of the page data in the buffer pool; it
only blocks writes to the underlying data file.

buf_page_t::flush(): Atomically test and set write fix, and
skip the operation if the fake "write fix" was set.

buf_page_t::set_freed(), buf_page_t::set_reinit(): Employ a
compare-and-exchange loop to accommodate for the "write fix".

innodb_backup_batch_wait(): Look up any pages that we are about to
back up. For any dirty pages, invoke buf_page_t::write_fix_try() to
try to set a fake "write fix" lock-free. If the page is currently
write-fixed between buf_page_t::flush() and
buf_page_t::write_complete(), acquire and release a page U-latch to
wait for the conflicting write to complete.

InnoDB_backup::backup_batch_start(),
InnoDB_backup::backup_batch_stop(): Adjust fil_space_t::backup_end and
fake "write fix" of dirty pages to protect the copying of a range of
pages from the underlying file.

InnoDB_backup::commit(): Enqueue the remaining log to be copied.

InnoDB_backup::checkpoint_complete(): If backup is running and
commit() has not been called, add each completed innodb_archive_log=ON
file to the queue. Else, skip or delete, as appropriate.

InnoDB_backup::archive_create(), innodb_backup_archive_create():
Hook for log_t::archive_create().

log_t::backup_start(): If we were running with innodb_log_archive=ON,
ensure that the latest file is a valid recovery starting point.
That is, wait for the latest log checkpoint to be within the file.

buf_flush_list_space(): Check for concurrent backup before writing each
page. This is inefficient, but this function may be invoked from multiple
threads concurrently, and it cannot be changed easily, especially for
fil_crypt_thread().

fil_ibd_create(): Set fil_space_t::create_lsn after the file has been
created.

dict_load_tablespaces(): Determine the size of each file if
upgrade==true. Backup depends on that.

buf_dblwr_t::begin(), buf_dblwr_t::end(), buf_dblwr_t::size():
Accessors to allow BACKUP SERVER to skip the contents of the
doublewrite buffer in the system tablespace. It is only useful for
crash recovery in case a data page had been incompletely written by
the time the server was killed. If the server is killed during a
backup, the backup will be incomplete and unusable anyway.
Furthermore, the page range locking makes page writes and backup
mutually exclusive.

Tested by: Matthias Leich
Tested by: Saahil Alam
Co-authored by: Michael 'Monty' Widenius
Co-authored by: Alessandro Vetere
Reviewed by: Thirunarayanan Balathandayuthapani
Reviewed by: Marko Mäkelä (the ENGINE=Aria part)
Oleg Smirnov
MDEV-40439 Evaluate cheap subconditions before expensive ones in WHERE and ON conditions

The arguments of AND and OR are evaluated left to right with
short-circuiting, in the order written in the query. An expensive
subcondition written first is therefore evaluated even for rows that
a cheap one would reject (AND) or accept (OR).

Reorder the arguments of each AND/OR so that the expensive ones come
after the cheap ones, keeping the relative order within each group.
The walk goes down into nested AND/OR and into trigger conditions.
A subcondition is expensive if it contains a subquery, a stored
function call, or an item that marks itself as expensive through
is_expensive_processor() (a UDF, MATCH()).

The reordering is applied to:
- the conditions attached to tables (select_cond, select->cond,
  on_precond), at the end of make_join_select() and before the
  attached_conditions_summary trace, so the trace shows the final
  order;
- BNL/BNLH scan filters, in make_scan_filter(). They are built from
  the original WHERE/ON, and check_join_cache_usage() may build them
  after make_join_select();
- outer_ref_cond and pseudo_bits_cond, the conditions a subquery
  checks once per execution before running its join.
The steps that later build new conditions from these (index condition
pushdown, removal of duplicate BNL conditions) take subsets and keep
the order.

Not covered yet: HAVING, exec_const_cond, terms added after
make_join_select() (the index condition remainder rejected by an
engine, HAVING moved to a table's condition), and SLEEP()/GET_LOCK(),
which do not mark themselves as expensive through
is_expensive_processor().
bsrikanth-mariadb
MDEV-41293: FederatedX: connection state after a net_read_timeout

Problem:
When reading the reply of the remote server times out, the client
library compiled into the server returns without closing the connection
and without setting a client error. FederatedX took the raw
ER_NET_READ_INTERRUPTED for an error of the remote server and went on
using a connection that still had the reply of the failed statement to
arrive. As a result:
* the next statement on that connection failed in various ways, either
  with ER_QUERY_ON_FOREIGN_DATA_SOURCE ("Server has gone away") or with
  an error packet that mysqltest took for a malformed packet;
* inside a transaction, the following statements continued on a new
  remote session in autocommit mode, so they were committed on the
  remote server one by one even though the local transaction was later
  rolled back. The savepoints of the lost session were also still
  remembered by FederatedX.
The same happened when the connection was lost while reading the reply.

Fix:
* After a timeout FederatedX closes the connection, so that the next
  statement opens a new one. The client still gets the error of the
  timeout, ER_NET_READ_INTERRUPTED.
* Whenever the connection is gone after a failed statement, timeout or
  not, the state of the lost session (autocommit mode, savepoints) is
  forgotten and the local transaction is marked to be rolled back, so a
  new connection starts the transaction again. This includes the case
  where the connection is lost while SET AUTOCOMMIT=0 is sent for the
  first statement of the transaction. The closed connection is also
  freed, which was leaked before.
* Errors of the client library are reported as
  ER_QUERY_ON_FOREIGN_DATA_SOURCE by ha_federatedx::info() as well, the
  same way as by the pushed down multi-table UPDATE/DELETE.
* The error state of ha_federatedx, read by info(), is initialized.
* A non-debug build checks the socket for EOF before it sends a command
  (net_clear()). If the remote server has already closed the connection, the
  write fails without the network layer raising an error, and the client
  would get ER_GET_ERRMSG ("Got error 10000 ...") intermittently. When the
  connection is lost and no error has been raised, raise
  ER_QUERY_ON_FOREIGN_DATA_SOURCE with the error of the client library. The
  COMMIT and ROLLBACK of the transaction still return their error to the
  caller.

Test: federatedx_pushdown_upd_del covers the timeout of a pushed down
multi-table UPDATE and DELETE and of reading a result, the statements
after it, the same inside a transaction, a connection killed in the
middle of a statement, a connection lost while SET AUTOCOMMIT=0 is sent
(also followed by a disconnect and a transaction in another session),
client library errors in info(), and that an error of the remote server
which merely has the code of ER_NET_READ_INTERRUPTED is not taken for a
timeout.
The two statements after the idle connection was killed accept
ER_NET_READ_ERROR (reported by the network layer in a debug build) or
ER_QUERY_ON_FOREIGN_DATA_SOURCE (non-debug build).
Marko Mäkelä
MDEV-41395 fixup: GCC 16 -Wstringop-overflow

fil_system_t::default_encrypt_next(): Dereference the iterator only once
to avoid a confusing and bogus -Wstringop-overflow by GCC 16.2.0.
Sergei Golubchik
MDEV-39596 Unexpected "No database selected" error for a CALL inside an anonymous block

For name1.name2() the PATH loop searches CURRENT_SCHEMA for a
package name1. In an anonymous block without a current database
resolve_current_schema() called copy_db_normalized(), which raised
ER_NO_DB_ERROR. But without a current schema there is no package
to find, name1.name2 can only be schema.routine.

Skip the unresolvable CURRENT_SCHEMA in an anonymous block, as it is
done outside of stored programs. Unqualified names never get here
without a current database, make_sp_name() and
Create_qfunc::create_func() reject them first.

Assisted-by: Claude:claude-opus-5-5
Sergei Petrunia
Merge: MDEV-39226: Push whole multi-table update/delete down into engines
Jan Lindström
MDEV-41165 Assertion in set_eof_status in I_S.TABLES fast path

The fast path of MDEV-31342 streams rows for a simple SELECT on
INFORMATION_SCHEMA.TABLES. Evaluating the WHERE clause can fail
after the statement is killed. SLEEP() sets ER_QUERY_INTERRUPTED
or ER_STATEMENT_TIMEOUT. get_schema_tables_result() then called
my_eof() with the error already set. This failed the assertion in
Diagnostics_area::set_eof_status().

Do not call my_eof() when the statement has an error. The existing
error handling after the loop reports the error once.

Do not use the fast path when max_statement_time is set.

Tests cover SET STATEMENT max_statement_time, KILL QUERY and KILL
CONNECTION.

Assisted-by: Claude:claude-4.6-sonnet
Oleg Smirnov
MDEV-40439 Evaluate cheap subconditions before expensive ones in WHERE and ON conditions

The arguments of AND and OR are evaluated left to right with
short-circuiting, in the order written in the query. An expensive
subcondition written first is therefore evaluated even for rows that
a cheap one would reject (AND) or accept (OR).

Reorder the arguments of each AND/OR so that the expensive ones come
after the cheap ones, keeping the relative order within each group.
The walk goes down into nested AND/OR and into trigger conditions.
A subcondition is expensive if it contains a subquery, a stored
function call, or an item that marks itself as expensive through
is_expensive_processor() (a UDF, MATCH()).

The reordering is applied to:
- the conditions attached to tables (select_cond, select->cond,
  on_precond), at the end of make_join_select() and before the
  attached_conditions_summary trace, so the trace shows the final
  order;
- BNL/BNLH scan filters, in make_scan_filter(). They are built from
  the original WHERE/ON, and check_join_cache_usage() may build them
  after make_join_select();
- outer_ref_cond and pseudo_bits_cond, the conditions a subquery
  checks once per execution before running its join.
The steps that later build new conditions from these (index condition
pushdown, removal of duplicate BNL conditions) take subsets and keep
the order.

Not covered yet: HAVING, exec_const_cond, terms added after
make_join_select() (the index condition remainder rejected by an
engine, HAVING moved to a table's condition), and SLEEP()/GET_LOCK(),
which do not mark themselves as expensive through
is_expensive_processor().
Susil
MDEV-41995 galera.MDEV-40727 result file has latin1 instead of utf8mb4

The four SHOW CREATE TABLE outputs in galera/r/MDEV-40727.result record
DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci. That is the server
default on 10.11 and 11.4, but not on 11.8, where the tables that the
test creates get utf8mb4 / utf8mb4_uca1400_ai_ci. The test therefore
fails on 11.8.

The merge of 11.4 into 11.8 (62b26c2f7cb8) brought in the 10.11/11.4
version of the result file. Use the utf8mb4 version, which 12.3 and
later already have.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>