Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Monty
Update backup code to use new my_dir and my_copy interfaces
Oleksandr Byelkin
MDEV-39993 Force InnoDB in the sys routine grants upgrade test

vaintroub (PR #5698): without an explicit engine requirement, this
test could run in a config where InnoDB isn't the checked table's
engine, making mariadb-upgrade's "Checking and upgrading" phase
print misleading "Unknown storage engine 'InnoDB'" / "error :
Corrupt" lines for mysql.innodb_index_stats, mysql.innodb_table_stats
and mysql.transaction_registry, and a spurious "Repairing tables"
section. Add --source include/have_innodb.inc so the test only runs
with InnoDB available, and re-record the now-clean result.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Monty
Implement copying of Aria files with reading of big blocks

The new interface for copy transactional Aria files:

Copying of an aria tables starts and end with these calls:
int aria_open_files_for_backup(THD *thd,
                              const char *path, my_bool trans_type,
                              ARIA_BACKUP_CONTEXT *context);
void aria_close_files_for_backup(ARIA_BACKUP_CONTEXT *context);

Reading through an aria file is done wih these calls:
longlong aria_read_index_file(ARIA_BACKUP_CONTEXT *context,
                              uchar *buffer, size_t buff_length);
longlong aria_read_data_file(ARIA_BACKUP_CONTEXT *context,
                            uchar *buffer, size_t buff_length);
One should call both in a loop until the return is <= 0
0 means end of file, a negative value means error.

storage/maria/test_ma_backup.c tests the code and can be used as an example
of how to use the functions.
Dave Gosselin
MDEV-33616:  Allocate the recovery buffer from the heap

recv_sys.tmp_buf comes from aligned_malloc() rather than from the large
page allocator.

main.large_pages fails on macOS with "Warning: Memory not freed: 16375"
at shutdown.  recv_sys_t::find_checkpoint() asks for 1048585 bytes,
my_large_malloc() rounds that up to 1064960 and charges the rounded
figure to the server memory accounting, and recv_sys_t::tmp_free()
credits back the 1048585 that was requested.  ut_malloc_dontdump() takes
the size by value, so it has nowhere to report what my_large_malloc()
wrote back.

Only macOS rounds.  my_get_large_page_sizes() has no huge page interface
to consult there, so its fallback branch reports the ordinary page size
of 16384 as the only large page size.  Linux offers 2 MiB and larger,
and my_next_large_page_size() returns only sizes at or below the
request, so this buffer is never rounded there.

The alignment is srv_page_size because recv_dblwr_t::validate_page()
uses the start of the buffer as two page frames.

log_sys.buf and log_sys.flush_buf keep the large page allocator and stay
excluded from core dumps.  They round the same way, so a server started
with --large-pages --innodb-log-buffer-size=2101248 still reports 24576
on macOS.  recv_sys.tmp_buf can now appear in a core dump, holding redo
log records that innodb_encrypt_log has decrypted.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Thirunarayanan Balathandayuthapani
MDEV-40324 use-of-uninitialized-value after creation of FULLTEXT table failure

Problem:
=======
For fulltext index, row_create_index_for_mysql() calls
fts_create_index_tables(). If creating FTS auxiliary table fails,
error handling performs trx->rollback() of the dictionary
transaction. Rollback removes the parent table from
dictionary cache and frees it. After that,
convert_error_code_to_mysql() reads table->flags after table->heap.
This leads to read of freed memory.

Solution:
========
create_index(): Read table->flags into a local variable before
calling row_create_index_for_mysql()
Sergei Golubchik
don't install config.h

high chance of name conflict. not used by any other headers
identical to my_config.h (which is used by other headers), so redundant.
Monty
Fixed that translog_walk_filenames() in Aria properly recognized aria
log filenames.
Vladislav Vaintroub
MDEV-40608 propagate DBUG_OFF, ENABLED_DEBUG_SYNC and SAFE_MUTEX to external plugins

they affect ABI, but aren't in headers, so must be passed separately
Thirunarayanan Balathandayuthapani
MDEV-41207 Assertion `user_table->get_ref_count() == 1' failed during ALTER TABLE conflicting with rollback of recovered transaction

Problem:
=======
A transaction being rolled back during recovery holds LOCK_IX on the
table, and the rollback thread holds a reference on it.
An online ALTER TABLE on that table falls back to acquiring LOCK_S,
which conflicts and fails.
prepare_inplace_alter_table_dict() asserted that the reference
count is 1 before checking whether the table lock was acquired,
so the reference still held by the rollback thread makes the
assertion fail.

Solution:
========
prepare_inplace_alter_table_dict(): Move the assertion after the
table lock error handling, so that it is evaluated only when
the table locks were acquired successfully.
Val Doroshchuk
MDEV-41154: Report an error when the engine fails in ha_rnd_pos

DELETE skips the row without reporting anything when error is returned by handler::ha_rnd_pos()

Report every error except the two that genuinely mean "the row is not there
any more".
Sergei Golubchik
fix mariadb-plugin-defines.cmake for multi-config
Monty
Update backup code to use new my_dir and my_copy interfaces
Oleksandr Byelkin
MDEV-41205 Report CREATE OR REPLACE routine failures, don't crash/hide them

CREATE OR REPLACE FUNCTION/PROCEDURE/PACKAGE replaces an existing
routine by first deleting its mysql.proc row
(Sp_handler::sp_drop_routine_internal(), or
Sp_handler_package_spec::sp_find_and_drop_routine() for a PACKAGE,
which drops the PACKAGE BODY row then the spec row) and then
writing the new one. If either step failed, no my_error() was ever
called: sp_drop_routine_internal() returned SP_DELETE_ROW_FAILED
silently, its two callers in Sp_handler::sp_create_routine() just
did "goto done;", and a failing ha_write_row() was always reported
as ER_SP_ALREADY_EXISTS regardless of the real cause. The statement
finished with an empty Diagnostics_area, which
Protocol::end_statement() turns into DBUG_ASSERT(0) on a debug
build (crashing the server) and a silent client-visible OK on a
release build, for a routine that may have been left half-dropped.

Fix:
- sp_drop_routine_internal() now captures ha_delete_row()'s error
  code and reports it via print_error() before returning
  SP_DELETE_ROW_FAILED.
- The SP_TYPE_PACKAGE and SP_TYPE_PACKAGE_BODY/FUNCTION/PROCEDURE
  arms in sp_create_routine()'s CREATE OR REPLACE switch now call
  my_error(ER_SP_DROP_FAILED, ...) before "goto done;", mirroring
  the identical safety net the explicit DROP PROCEDURE/FUNCTION
  path already has in sql_parse.cc. Because the first error raised
  wins, this only fires when the delete didn't already report one
  itself (e.g. SP_KEY_NOT_FOUND from a concurrent drop) -- a
  comment explains this so a future reader doesn't "clean up" it
  as dead code.
- ha_write_row()'s error code is now captured too: only a genuine
  HA_ERR_FOUND_DUPP_KEY still gets ER_SP_ALREADY_EXISTS; any other
  code goes through print_error(), with a my_error(ER_SP_STORE_FAILED)
  fallback in case print_error() itself doesn't raise anything
  (a couple of handler error codes are legitimately silent).

Two DBUG_EXECUTE_IF fault-injection points let this be tested without
a real storage-engine failure: "sp_drop_routine_internal_fail" (any
delete), "sp_drop_routine_internal_fail_package_spec_only" (only the
PACKAGE spec delete, letting a test target the harder "body already
gone, spec delete now fails" ordering), and
"sp_create_routine_write_row_fail" for the write path.

New test mysql-test/main/sp-error-debug.test covers FUNCTION,
PROCEDURE, PACKAGE BODY alone, and both PACKAGE delete orderings,
plus the write-row failure path.

Known limitations, deliberately not addressed here:
- Sp_handler_package_spec::sp_find_and_drop_routine() drops the
  PACKAGE BODY row before the spec row; if the spec delete then
  fails, the body is durably gone (already flushed) while the spec
  survives, and neither the drop nor a subsequent DROP PACKAGE
  binlogs that sub-operation. This is a genuine pre-existing gap
  (mysql.proc is Aria, non-transactional, so a true atomic two-row
  drop isn't achievable without a larger change) that this patch
  does not make worse -- it only makes the failure visible via a
  proper error instead of a crash or silent OK. Left for a
  follow-up MDEV.
- sql_parse.cc's explicit DROP PROCEDURE/FUNCTION path calls
  sp_revoke_privileges() whenever sp_result != SP_KEY_NOT_FOUND,
  which now includes a reachable SP_DELETE_ROW_FAILED -- so a
  failed DROP can still strip the routine's grants. Pre-existing,
  unrelated to CREATE OR REPLACE; left for a follow-up MDEV.
- This same code is unchanged in 10.6 and 10.11; the reproducer
  applies there too. Kept on 11.4 per the reporter's request.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Monty
Fixed that translog_walk_filenames() in Aria properly recognized aria
log filenames.
Oleksandr Byelkin
Fix cursor protocol
Sergei Golubchik
fix the build for -G "Ninja Multi-Config"
Vladislav Vaintroub
MDEV-40608 build mysqlservices without an embedded CRT requirement

mysqlservices only exposes a thin C API, no CRT state crosses it, so
don't force whatever CRT/config built the server onto a plugin linking
it. Without /Zl, a plugin built in a config with no matching installed
mysqlservices variant (CMake silently substitutes one - verified with
a toy project) gets an ignorable but noisy LNK4098 warning.

Assisted-by: Claude:claude-5-sonnet
Dave Gosselin
MDEV-33616:  Skip the redo log upgrade tests without sparse file support

innodb.log_upgrade and innodb.log_upgrade_101_flags build 8GB redo log
files by seeking past the end of an empty file and writing a single
byte.  That needs a filesystem which leaves the skipped range
unallocated.  HFS on macOS allocates every block of it instead, so the
write fails with ENOSPC and the test reports a perl failure.

include/have_sparse_files.inc probes a directory the caller names,
writing one byte 64MB into an empty file there and comparing the
allocated block count against that offset.
Sergei Golubchik
don't INCLUDE(CPack) automatically
Marko Mäkelä
fixup! ae8056d2279c5a96ce73206a4f70b0d69e5f3817
Sergei Golubchik
misc
Monty
Implement copying of Aria files with reading of big blocks

The new interface for copy transactional Aria files:

Copying of an aria tables starts and end with these calls:
int aria_open_files_for_backup(THD *thd,
                              const char *path, my_bool trans_type,
                              ARIA_BACKUP_CONTEXT *context);
void aria_close_files_for_backup(ARIA_BACKUP_CONTEXT *context);

Reading through an aria file is done wih these calls:
longlong aria_read_index_file(ARIA_BACKUP_CONTEXT *context,
                              uchar *buffer, size_t buff_length);
longlong aria_read_data_file(ARIA_BACKUP_CONTEXT *context,
                            uchar *buffer, size_t buff_length);
One should call both in a loop until the return is <= 0
0 means end of file, a negative value means error.

storage/maria/test_ma_backup.c tests the code and can be used as an example
of how to use the functions.
Monty
fixup!
Thirunarayanan Balathandayuthapani
MDEV-40324 use-of-uninitialized-value after creation of FULLTEXT table failure

Problem:
=======
For fulltext index, row_create_index_for_mysql() calls
fts_create_index_tables(). If creating FTS auxiliary table fails,
error handling performs trx->rollback() of the dictionary
transaction. Rollback removes the parent table from
dictionary cache and frees it. After that,
convert_error_code_to_mysql() reads table->flags after table->heap.
This leads to read of freed memory.

Solution:
========
create_index(): Read table->flags into a local variable before
calling row_create_index_for_mysql()
Sergei Golubchik
MDEV-40608 MariaDB-devel is incomplete for plugins

This works on Linux and on Windows, with rpm/deb/tar.gz/zip
installations.

For rpm/deb it just works, for tar.gz/zip there is no
standard location, so one needs to configure plugin with

  -DCMAKE_PREFIX_PATH=/pah/to/mariadb/basedir

after that, `cmake --install .` works too, installing in the same
basedir.

`cmake --build . --target package` works, creating rpm/deb/targz/zip
depending on whether it's Linux or Windows and whether -DRPM or -DDEB
was specified.

* create and install mariadb-plugin-config.cmake
* deb: move all headers that plugins need to libmariadb-dev,
  together with libmysqlservices.a. At least until we'll
  create mariadb-plugin-dev. Nobody should need huge
  libmariadbd-dev to develop a plugin
* rpm: all in MariaDB-devel already, no changes here
* adjust plugin.cmake to work for external plugins
* move server-internal part to top-level CMakeLists.txt
* remove WITH_WSREP from my_config.h (it upsets external plugins)
* disable DBUG in plugins, can be enabled with -UDBUG_OFF, if needed
* remove double-defined macros from unireg.h (the guard doesn't help
  if unireg.h is included first)

ColumnStore, until fixed, needs a backward-compatibility workaround
Kristian Nielsen
MDEV-40729: Add functionality to verify if a transaction is committed or not after failure

Basic proof-of-concept patch, only implements the basics to put trx_id into
the binlog file and search for it with trx_status().

A test case rpl.rpl_client_trx_id demonstrates the functionality.

Some limitations/considerations from this patch:

There is no support for MARIADB_TRX_IN_PROGRESS (do we want/need such
functionality?).

Based on 10.11, however pushing to stable 10.11 will be controversial as it
changes GTID event format.

Since 10.11 has no GTID indexes, linear scan of at least one entire binlog
file will be required. An in-memory cache may be needed to handle client
reconnect-storm after a crash or network outage that caused many ongoing
commits to fail.

When starting gtid is not specified for trx_status(), it is not possible to
distinguish between MARIADB_TRX_ABORTED or MARIADB_TRX_UNKNOWN. In this
case, we return MARIADB_TRX_ABORTED.

Knowing where to start scanning binlogs is important, for performance (to
avoid scanning _entire_ binlog history), and to distinguish
MARIADB_TRX_ABORTED from MARIADB_TRX_UNKNOWN. It is somewhat tricky
though:

- A GTID position is multi-dimensional. The starting GTID _must_ be with
  the same domain_id as the transaction being searched for, otherwise
  searching on a slave may start too late in the slave's binlog and wrongly
  return MARIADB_TRX_ABORTED for a committed transaction.

- The client will need to obtain a starting GTID for the very first
  transaction done on the connection. One possible way could be to
  SELECT @@GLOBAL.gtid_binlog_pos and pick out the one with the domain_id
  which will be used for subsequent transactions.

Signed-off-by: Kristian Nielsen <[email protected]>
Dave Gosselin
MDEV-33616:  Match the macOS dlopen error in plugins.multiauth

The client reports why it could not load client_ed25519, and macOS names
every path that dlopen() tried.  Two expressions are added, one for the
chunk that holds the start of that message and one for the chunk that
holds the rest of it.

The line runs to 563 bytes, 52 of prefix and the 511 that the client
error buffer holds, while do_exec() reads the output with fgets() into a
512 byte buffer and runs the replacements on each chunk on its own.  A
long enough vardir therefore splits the line, because the path appears
four times in the dlopen text.  The second chunk is the tail of a path
and carries no colon, where the first chunk keeps the colons of the
mysqltest prefix.  That chunk also holds the only line terminator the
error line gets, so the expression captures the newline and the
replacement puts it back.  A replacement is inserted as written, so a \n
spelled there would reach the output as a backslash and an n.

Both expressions stop at a newline.  reg_replace compiles with
REG_DOTALL, so an unrestricted .* runs past the line terminator whenever
the whole message reaches the replacement in one chunk, and the error
line then joins the line after it.
Sergei Golubchik
fixup: do not remove WITH_WSREP

this creates ABI incompatiility. install wsrep headers instead
Sergei Golubchik
misc fixes

1. set PLUGIN_HEX_VERSION correctly for bundled plugins
2. don't add GenError dependency for external plugins
3. don't change the policy globally
4. only do EXTERNAL_PLUGIN_POST() if EXTERNAL_PLUGIN_PRE() was done
Rucha Deodhar
MDEV-41181: ASAN heap-buffer-overflow after SELECT JSON_SCHEMA_VALID
Dave Gosselin
MDEV-33616:  Take the read lock many times in perfschema.func_mutex

The wait timer can have a granularity coarser than the time an
uncontended read lock is held, so the recorded duration of one lock can
be zero, which reads back as NULL.  This can cause the test to fail with
a false negative.

Take the lock twenty more times at each measurement point, with the
extra statements silent so the recorded result does not change.  The
mutex part of the test already works this way, since one SELECT
produces ten THR_LOCK::mutex events.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Monty
Added new mysys functions my_open_dir and improved my_copy

- Improved my_copy() using copy_file_range and memmap

- New mysys functions:
int my_copy_file(File from, File to, myf MyFlags);
int my_copy_file_range(File from, File to, my_off_t start,
                        my_off_t end, myf MyFlags);

- New functions for looping over files in a directory:
MY_NO_CACHE_DIR *my_dir_open();
int my_dir_read_next()
int my_dir_rewind();
int my_dir_close();

Other things
- Fixed #ifdef's in sql_backup.cc to use the new define
  HAVE_COPY_FILE_RANGE
Monty
Integrate the old and new backup code

Fixes a lot of issues in current backup code:
- Galera should now be supported (needs testing)
- Enables ddl logging (so we can use it in the future)
- Flushes binary logs (we still must add code to copy them)
- mdl locks are consistent between maria-backup and backup command
- startup backup code for InnoDB moved to innodb_prepare_for_backup()
  called by prepare_for_backup hton handler.
- Give errors if backup command is done under a transaction, global
  read lock or lock tables.
- Retry for MDL_BACKUP_WAIT_DDL (needed for
  backup.backup_ddl_concurrent_verify)
- Removed wrong log locks in Aria

Things to do (in addition to the things in my earlier review) :
- At backup_stage stage start, force rotate of aria log files. This
  allows us to copy all old logs without any locks
- Copy all transactional tables and old aria logs under BACKUP_START (as
- maria-backup does)
- Copy the active aria log file under block commit (Only one file as aria
  log rotation is disabled while backup is running.
- Improve speed of copying aria tables by copy files in up to 1M
  blocks and run checksum on the blocks and only re-read blocks with
-  wrong checksum.
- Copy non transactional files under BACKUP_PHASE_NO_BEGIN_NON_TRANS.
  Note that Aria does not support the documented
  BACKUP_PHASE_NO_DML_NON_TRANS . The BACKUP_PHASE_NO_BEGIN_NON_TRANS
  state is already blocking changes to non transactional tables
Sergei Golubchik
fix errmsg-utf8.txt dependencies for Ninja generator

GenError's custom command must specify headers as OUTPUT,
otherwise ninja cannot deduce that mysqld.cc depends on errmsg-utf8.txt

As a bonus, BYPRODUCTS lists generated files for `ninja clean`
Dave Gosselin
MDEV-41212:  multi_source.status_vars fails on MacOS platform

Replace the two recorded reads of Slave_received_heartbeats with an
assertion that the counter is nonzero.

The counter advances once per heartbeat period for as long as the
connection is running.  The test waited for it to reach 2 and then
read it again in a separate query, so a heartbeat arriving between
those two queries recorded an unexpected value.
Sergei Golubchik
debian, again
Pekka Lampio
MDEV-39143 Replicate MySQL binary JSON to a MariaDB slave

Let a MariaDB slave apply ROW-format binlog events from a MySQL
master that contain JSON columns, both as full documents and as
MySQL's partial (diff) JSON updates (PARTIAL_UPDATE_ROWS_EVENT).
Monty
Added new mysys functions my_open_dir, my_win_open and improved my_copy

- Improved my_copy() using copy_file_range and memmap
- Improve performance of my_win_open() by taking free entries from a list
  instead of searching after a free space in an array.

New mysys copy functions:

int my_copy_file(File from, File to, myf MyFlags);
int my_copy_file_range(File from, File to, my_off_t start,
                        my_off_t end, myf MyFlags);

Added compatibility functions for easy converting a windows HANDLE to
a File and back:

File my_convert_handle_to_file(my_native_file handle, int oflag);
void my_detach_file(File fd);
which complements the existing my_native_file_handle(fd)

New functions for looping over files in a directory:

MY_NO_CACHE_DIR *my_dir_open();
int my_dir_read_next()
int my_dir_rewind();
int my_dir_close();

Other things
- Fixed #ifdef's in sql_backup.cc to use the new define
  HAVE_COPY_FILE_RANGE
Oleksandr Byelkin
fix MDEV-31342 view protocol