Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Marko Mäkelä
fixup! b387a4a6f9f9b3194a29c1a80c39c983d5dc4fd5
Marko Mäkelä
WIP: log tracking BACKUP SERVER TO ... CONCURRENT (for HAVE_INNODB_PMEM)

backup_sink::id: The thread identifier (0 to CONCURRENT-1)

innodb_backup_checkpoint_pmem(): Copy the old log file.

InnoDB_backup::log_track(), InnoDB_backup::log_track_pmem():
Keep copying the log until we run out of InnoDB data files to copy.

InnoDB_backup::checkpoint_complete_pmem(): Copy the remaining
part of an old log file right before it is being released.

InnoDB_backup::commit(): In log tracking backup, copy the rest of
the HAVE_INNODB_PMEM log.

FIXME: Implement the non-PMEM code path with minimal blocking.
Oleksandr Byelkin
new version of main (3.2) part 2
Marko Mäkelä
fixup! b387a4a6f9f9b3194a29c1a80c39c983d5dc4fd5
Dave Gosselin
MDEV-33616:  Charge and credit the same size for the recovery buffer

main.large_pages fails on macOS with "Warning: Memory not freed: 16375"
at shutdown and no accompanying safemalloc report.  The residual stays
at 16375 whether innodb_buffer_pool_size is 8M or 128M, and dropping
--large-pages makes it go away.

recv_sys_t::find_checkpoint() asks for tmp_buf_size, which is
MTR_SIZE_MAX + 9, or 1048585 bytes.  my_large_malloc() rounds that up to
a multiple of the large page size and charges the rounded figure to
global_memory_used, while recv_sys_t::tmp_free() credits back the
1048585 that was requested.  The page size here is 16384, 1048585 rounds
up to 1064960, and the difference is the 16375 reported.  The caller
cannot see the rounded figure because ut_malloc_dontdump() takes the
size by value and, with a null ut_new_pfx_t, has nowhere to report what
it allocated.  ut_malloc_dontdump_size() writes the size back, and
recv_sys_t keeps it in tmp_buf_alloc_size for the free.  tmp_buf_size
remains the capacity that parse() asserts against.

Only macOS rounds up.  my_get_large_page_sizes() has no huge page
interface to consult there, so its fallback branch reports the ordinary
page size as the only large page size and the plain mmap() always
succeeds.  On Linux the candidate is 2 MiB, the MAP_HUGETLB mapping
fails with ENOMEM when no huge pages are reserved, and the retry loop
settles on large_page_size == 0, which records the request unrounded.
No memory was lost either way, since munmap() rounds its length up to a
whole page.  The counter was wrong, and the counter is what MTR checks.

Co-Authored-By: Claude Opus 5 <[email protected]>
Monty
Fixed that translog_walk_filenames() in Aria properly recognized aria
log filenames.
Khaled Riyad
MDEV-40551 Copy/Paste friendly output format for MariaDB Command Line Client

Copy/paste friendly output was only reachable by starting the client with
--silent --skip-column-names, which cannot be done from a running
interactive session.

Add \S, a statement terminator which prints the result of one statement in
the tab separated format without column names.

com_silent() sets output_plain, opt_silent and column_names around
com_go(), then restores them, the same way com_ego() handles vertical.
output_plain selects print_tab_data() ahead of the vertical and table
branches, so \S gives the same output whether the session was started
plainly or with --table, --vertical or --silent. --html and --xml still
win, matching \G.
Marko Mäkelä
MDEV-40410: Tight innodb_buffer_pool_size_max on ThreadSanitizer

bur_pool_t::size_in_bytes_max_default: Define as 0 also on
ThreadSanitizer. The symbol __SANITIZE_THREAD__ is predefined
starting with Clang 22 or GCC 7 when building with -fsanitize=thread.
Georgi (Joro) Kodinov
more doxygen formatting added.
Vladislav Vaintroub
MDEV-41072 add SBOM author/tool metadata

- set SBOM author to ${CPACK_PACKAGE_VENDOR}
- extract email address from ${CPACK_PACKAGE_CONTACT}
- add metadata.tools.components describing the generator
Oleksandr Byelkin
Merge branch '13.0' into 13.1
Vladislav Vaintroub
MDEV-37996 CMake: MariaDB:: targets for bundled-or-system libraries

Stop overwriting the standard find_package() result variables (ZLIB_FOUND,
ZLIB_LIBRARIES, ZLIB_INCLUDE_DIR(S), ...), this breaks vcpkg.

Provide namespaced INTERFACE targets that point at either the bundled or
the system library and carry their include directories (and, for SSL, the
compile definitions)

  MariaDB::zlib, MariaDB::OpenSSL, MariaDB::pcre2-8, MariaDB::pcre2-posix,
  MariaDB::fmt, MariaDB::readline

Link these consistently instead of the scattered ${*_LIBRARIES} and
${*_INCLUDE_DIR(S)} variables sprinkled across the tree.

Bundled pcre2-posix depends on pcre2-8 so the static link order is correct
for consumers that link only posix.

wolfssl carries its own usage requirements (HAVE_WOLFSSL,
WOLFSSL_USER_SETTINGS); MariaDB::OpenSSL keeps only HAVE_OPENSSL.

Update libmariadb to 3.3 (47a31a98): handles MariaDB::zlib as a submodule.
Dave Gosselin
MDEV-33616:  Normalize the strerror text in innodb_fts.index_table

The injected deadlock reaches the client as ER_GET_ERRNO carrying errno
11, and the text comes from my_strerror().  11 is EAGAIN on Linux and
EDEADLK on macOS, so the message reads "Resource temporarily
unavailable" on one and "Resource deadlock avoided" on the other.
Replace the quoted text so the test does not depend on it.
Georgi (Joro) Kodinov
MDEV-38144: Re-recorded the test
Oleksandr Byelkin
Merge branch '13.1' into mariadb-13.1.1
Dave Gosselin
MDEV-33616:  Match the macOS dlopen error in plugins.multiauth

The client reports why it could not load client_ed25519, and macOS names
every path that dlopen() tried.  Two expressions are added, one for the
chunk that holds the start of that message and one for the chunks that
continue it.

Whether the message arrives in one chunk or several depends on the
vardir, because the path appears four times in the dlopen text.  With
--vardir /Volumes/<repo>/var the line is 417 bytes and fits the 512 byte
buffer that --exec output is read in.  With the default vardir it does
not.

Both expressions stop at a newline.  reg_replace compiles with
REG_DOTALL, so an unrestricted .* runs past the line terminator whenever
the whole message reaches the replacement in one chunk, and the error
line then joins the line after it.
Oleksandr Byelkin
new version of main (3.2) part 2
Dave Gosselin
MDEV-33616:  Skip the redo log upgrade tests without sparse file support

innodb.log_upgrade and innodb.log_upgrade_101_flags build 8GB redo log
files by seeking past the end of an empty file and writing a single
byte.  That needs a filesystem which leaves the skipped range
unallocated.  HFS on macOS allocates every block of it instead, so the
write fails with ENOSPC and the test reports a perl failure.

include/have_sparse_files.inc probes a directory the caller names,
writing one byte 64MB into an empty file there and comparing the
allocated block count against that offset.  Both tests name the vardir
tmp, where they build their redo log files.  The offset stays above 16MB
since APFS allocates the whole range for a file smaller than that rather
than recording a hole.
Marko Mäkelä
Address some review comments
Monty
Fixed that translog_walk_filenames() in Aria properly recognized aria
log filenames.
Daniel Bartholomew
bump the VERSION
Eric
MDEV-38144: update Optional_metadata_fields to use MariaDB types

Currently Optional_metadata_fields has many members that use classes
from the C++ standard library, most notably the use of std::vector and
std::string. This is inconsistent with the coding standards, as MariaDB
defines its own types for arrays/lists/strings. This patch updates
these variables to use existing MariaDB types.

Additionally, the structure of the fields is changed. Instead of having
a separate list to track each metadata field; the structure now exists
at the column level (Column_metadata), and it has individual member
variables to describe its metadata. This allows for more
straightforward memory management, as everything will now be allocated
at once.

There is also a bug fix in this patch. Prior to this refactor, YEAR
columns would not consume their "unsigned" bit flag, resulting in
numeric columns after a YEAR type to have incorrect signage. This is
now fixed.

Reviewed-by: Brandon Nesterenko <[email protected]>.
Alexander Barkov
Cleanup for MDEV-39518 Allow PS in SF in assignment right hand

Fixing the "End of 13.1 tests" to "End of 13.2 tests",
as the task got into 13.2 release only.
Oleksandr Byelkin
new columnstore (df4f261f1f622adb2c266527e3fd3c8c4ddaa636 25.10.7)
drrtuy
MDEV-41001: cpack now skips data path that brings symbolic links to fix Windows build.
Georgi (Joro) Kodinov
MDBF-1243: Make moxygen doc be a good GitBook citizen.

Removed --anchors from the moxygen command line since this is not
useful to GitBook and renders as garbage in it.
Kristian Nielsen
MDEV-40729: Add functionality to verify if a transaction is committed or not after failure

Basic proof-of-concept patch, only implements the basics to put trx_id into
the binlog file and search for it with trx_status().

A test case rpl.rpl_client_trx_id demonstrates the functionality.

Some limitations/considerations from this patch:

There is no support for MARIADB_TRX_IN_PROGRESS (do we want/need such
functionality?).

Based on 10.11, however pushing to stable 10.11 will be controversial as it
changes GTID event format.

Since 10.11 has no GTID indexes, linear scan of at least one entire binlog
file will be required. An in-memory cache may be needed to handle client
reconnect-storm after a crash or network outage that caused many ongoing
commits to fail.

When starting gtid is not specified for trx_status(), it is not possible to
distinguish between MARIADB_TRX_ABORTED or MARIADB_TRX_UNKNOWN. In this
case, we return MARIADB_TRX_ABORTED.

Knowing where to start scanning binlogs is important, for performance (to
avoid scanning _entire_ binlog history), and to distinguish
MARIADB_TRX_ABORTED from MARIADB_TRX_UNKNOWN. It is somewhat tricky
though:

- A GTID position is multi-dimensional. The starting GTID _must_ be with
  the same domain_id as the transaction being searched for, otherwise
  searching on a slave may start too late in the slave's binlog and wrongly
  return MARIADB_TRX_ABORTED for a committed transaction.

- The client will need to obtain a starting GTID for the very first
  transaction done on the connection. One possible way could be to
  SELECT @@GLOBAL.gtid_binlog_pos and pick out the one with the domain_id
  which will be used for subsequent transactions.

Signed-off-by: Kristian Nielsen <[email protected]>
Dave Gosselin
MDEV-33616:  Widen the block count filter in the buffer pool resize test

The test replaces the number of buffer pool blocks with a fixed value so
that the message is stable.  The pattern only accepted 5.., and macOS
builds without a futex use SUX_LOCK_GENERIC, which enlarges buf_block_t
enough to bring the count down into 4...
Monty
MDEV-25292 Atomic CREATE OR REPLACE TABLE

Atomic CREATE OR REPLACE allows to keep an old table intact if the
command fails or during the crash. That is done by renaming the
original table to temporary name, as a backup and restoring it if the
CREATE fails. When the command is complete and logged the backup
table is deleted.

Atomic replace algorithm

Two DDL chains are used for CREATE OR REPLACE:
ddl_log_state_create (C) and ddl_log_state_rm (D).

  1. (C) Log rename of ORIG to TMP table (Rename TMP to original).
  2. Rename orignal to TMP.
  3. (C) Log CREATE_TABLE_ACTION of ORIG (drops ORIG);
  4. Do everything with ORIG (like insert data)
  5. (D) Log drop of TMP
  6. Write query to binlog (this marks (C) to be closed in
    case of failure)
  7. Execute drop of TMP through (D)
  8. Close (C) and (D)

  If there is a failure before 6) we revert the changes in (C)
  Chain (D) is only executed if 6) succeded (C is closed on
  crash recovery).

Foreign key errors will be found at the 1) stage.

Additional notes

- CREATE TABLE without REPLACE and temporary tables is not affected
  by this commit.
  set @@drop_before_create_or_replace=1 can be used to
  get old behaviour where existing tables are dropped
  in CREATE OR REPLACE.

- CREATE TABLE is reverted if binlogging the query fails.

- Engines having HTON_EXPENSIVE_RENAME flag set are not affected by
  this commit. Conflicting tables marked with this flag will be
  deleted with CREATE OR REPLACE.

- Replication execution is not affected by this commit.
  - Replication will first drop the conflicting table and then
    creating the new one.

- CREATE TABLE .. SELECT XID usage is fixed and now there is no need
  to log DROP TABLE via DDL_CREATE_TABLE_PHASE_LOG (see comments in
  do_postlock()). XID is now correctly updated so it disables
  DDL_LOG_DROP_TABLE_ACTION. Note that binary log is flushed at the
  final stage when the table is ready. So if we have XID in the
  binary log we don't need to drop the table.

- Three variations of CREATE OR REPLACE handled:

  1. CREATE OR REPLACE TABLE t1 (..);
  2. CREATE OR REPLACE TABLE t1 LIKE t2;
  3. CREATE OR REPLACE TABLE t1 SELECT ..;

- Test case uses 6 combinations for engines (aria, aria_notrans,
  myisam, ib, lock_tables, expensive_rename) and 2 combinations for
  binlog types (row, stmt). Combinations help to check differences
  between the results. Error failures are tested for the above three
  variations.

- expensive_rename tests CREATE OR REPLACE without atomic
  replace. The effect should be the same as with the old behaviour
  before this commit.

- Triggers mechanism is unaffected by this change. This is tested in
  create_replace.test.

- LOCK TABLES is affected. Lock restoration must be done after new
  table is created or TMP is renamed back to ORIG

- Moved ddl_log_complete() from send_eof() to finalize_ddl(). This
  checkpoint was not executed before for normal CREATE TABLE but is
  executed now.

- CREATE TABLE will now rollback also if writing to the binary
  logging failed. See rpl_gtid_strict.test

backup ddl log changes

- In case of a successfull CREATE OR REPLACE we only log
  the CREATE event, not the DROP TABLE event of the old table.

ddl_log.cc changes

  ddl_log_execute_action() now properly return error conditions.
  ddl_log_disable_entry() added to allow one to disable one entry.
  The entry on disk is still reserved until ddl_log_complete() is
  executed.

On XID usage

  Like with all other atomic DDL operations XID is used to avoid
  inconsistency between master and slave in the case of a crash after
  binary log is written and before ddl_log_state_create is closed. On
  recovery XIDs are taken from binary log and corresponding DDL log
  events get disabled.  That is done by
  ddl_log_close_binlogged_events().

On linking two chains together

  Chains are executed in the ascending order of entry_pos of execute
  entries. But entry_pos assignment order is undefined: it may assign
  bigger number for the first chain and then smaller number for the
  second chain. So the execution order in that case will be reverse:
  second chain will be executed first.

  To avoid that we link one chain to another. While the base chain
  (ddl_log_state_create) is active the secondary chain
  (ddl_log_state_rm) is not executed. That is: only one chain can be
  executed in two linked chains.

  The interface ddl_log_link_chains() was defined in "MDEV-22166
  ddl_log_write_execute_entry() extension".

Atomic info parameters in HA_CREATE_INFO

  Many functions in CREATE TABLE pass the same parameters. These
  parameters are part of table creation info and should be in
  HA_CREATE_INFO (or whatever). Passing parameters via single
  structure is much easier for adding new data and
  refactoring.

Aria changes:
- Fixed issue in Aria engine with CREATE + locked tables
  that data was not properly commited in some cases in
  case of crashes.

InnoDB related changes (by Marko Mäkelä):
- table_name_t::is_create_or_replace(): A new predicate to check for
  CREATE OR REPLACE TABLE will rename an old table to
  and eventually drop after creating the replacement.
- dict_table_t::parse_name(): Do acquire MDL on #sql-create- names
  for partitioned tables.
- dict_table_rename_in_cache(): On CREATE OR REPLACE TABLE ... SELECT,
  forget the original dict_table_t::mdl_name so that purge will
  acquire MDL on the #sql-create- name instead. In this way, the
  MDL_EXCLUSIVE that the CREATE OR REPLACE TABLE holds on the
  user-visible name will not unnecessarily block any purge of old history
  until the very end when the #sql-create- table will be dropped.
- ha_innobase::delete_table(): Do not check FOREIGN KEY consistency
  when dropping an #sql-create- table.
- row_rename_table_for_mysql(): Update SYS_FOREIGN.ID also
  when renaming to #sql-create- in order to avoid any
  duplicate key error when CREATE OR REPLACE TABLE is
  creating some FOREIGN KEY constraints by names
  that existed in the old table.

Other changes:
- Removed some auto variables in log.cc for better code readability.
- Fixed old bug that CREATE ... SELECT would not be able to auto repair
  a table that is part of the SELECT.
- Marked MyISAM that it does not support ROLLBACK (not required but
  done for better consistency with other engines).
- maria_create_trn_for_mysql() does not register a new transaction
  handler for commits. This was needed to ensure create or replace
  will not end with an active transaction.
- We do not get anymore warnings about "Engine not supporting atomic
  create" when doing a legal CREATE OR REPLACE on a table with
  foreign key constraints.
- Updated VIDEX engine flags to disable CREATE SEQUENCE.
- Removed mysql_mutex_unlock(&LOCK_gdl) / mysql_mutex_lock(&LOCK_gdl)
  around calls to binlog as these are unsafe. The binlog code uses
  global variables that needs protection from other caller.
- EITS data is preserved if create or replace fails if
  drop_before_create_or_replace=OFF. If ON, then create or replace
  will drop EITS before the drop of the original table (as before).
- Using CREATE OR REPLACE on a encrypted table that the user cannot
  decrypt will fail instead of replacing the encrypted table.
  The encrypted table will unchanged.

Known issues:
- One cannot use create or replace on an InnoDB tables that has foreign
  key references point to it
- CREATE OR REPLACE TEMPORARY table is not full atomic. Any conflicting
  table will always be dropped before creating a new one. (Old behaviour).

Bug fixes related to this MDEV:
MDEV-36435 Assertion failure in finalize_locked_tables()
MDEV-36439 Assertion `thd_arg->lex->sql_command != SQLCOM_CREATE_SEQUENCE...
MDEV-36498 Failed CoR in non-atomic mode no longer generates DROP in RBR...
MDEV-36508 Temporary files #sql-create-....frm occasionally stay after
          crash recovery
MDEV-38479 Crash in CREATE OR REPLACE SEQUENCE when new sequence cannot
          be created
MDEV-36497 Assertion failure after atomic CoR with Aria under lock in
          transactional context
MDEV-36501 EITS data is lost after failed attempt to CREATE OR REPLACE
          table
MDEV-36493 Atomic CREATE OR REPLACE ... SELECT blocks InnoDB purge
MDEV-39367 MSAN/valgrind errors in temp_file_size_cb_func,
          main.tmp_space_usage fails
MDEV-39446 Atomic CREATE OR REPLACE fails if a table cannot be decrypted
MDEV-40776 Atomic CREATE OR REPLACE silently breaks the foreign key
MDEV-40765 Assertion `"unexpected references" == 0' failed upon failing
          CREATE OR REPLACE

Reverted commits:
- MDEV-36685 "CREATE-SELECT may lose in binlog side-effects of
  stored-routine" as it did not take into account that it safe to
  clear binlogs if the created table is non transactional and there
  are no other non transactional tables used.
  This was done because it caused extra logging when it is not
  needed (not using any non transactional tables) and it also did
  not solve side effects when using statement based loggging.
Marko Mäkelä
fixup! 73e050520f5d14f04bd3d80ac49907804571c6f6
Dave Gosselin
MDEV-33616:  Take the read lock many times in perfschema.func_mutex

The wait timer can have a granularity coarser than the time an
uncontended read lock is held, so the recorded duration of one lock can
be zero, which reads back as NULL.  This can cause the test to fail with
a false negative.

Take the lock twenty more times at each measurement point, with the
extra statements silent so the recorded result does not change.  The
mutex part of the test already works this way, since one SELECT
produces ten THR_LOCK::mutex events.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Georgi (Joro) Kodinov
Keep the generated docs as an github action artifact for inspection.
Vladislav Vaintroub
MDEV-41072 add SBOM author/tool metadata

- set SBOM author to ${CPACK_PACKAGE_VENDOR}
- extract email address from ${CPACK_PACKAGE_CONTACT}
- add metadata.tools.components describing the generator
Oleksandr Byelkin
Merge branch 'bb-13.0-release' into main