Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Georgi (Joro) Kodinov
MDEV-40815: resolveip is not built for the minbuild cmake target

Added the resolveip target to the minbuild target.
Alessandro Vetere
MDEV-40408 btr_page_reorganize_low() uses the buffer pool just to obtain a scratch block

Add buf_pool.scratch_buf, a pool of page frames that the page
reorganization operations use instead of taking a block from the global
buffer pool.

buf_pool_t::scratch_buffer: A singly linked list of chunks of
buf_tmp_buffer_t slots. A thread holds at most one slot at a time, and
it holds a page latch while doing so, which is why reserve() must not
wait for the buffer pool or for I/O. When every slot is in use,
reserve() appends a chunk that holds twice the slots of the last one, up
to a limit on the size of one chunk. Only grow() waits, on the mutex
that serializes it and on the allocator. A chunk is never moved or freed
before close(), so a thread can keep using the slot that it reserved
while another thread appends a chunk. Debug builds start with a single
slot, so that a second concurrent page reorganization exercises grow().

buf_pool_t::scratch_buffer::shrink(): Free the page frames of the slots
that are not in use. The slots and the chunks are kept, because a slot
costs 32 bytes while a page frame costs srv_page_size. A frame survives
the first pass, because that pass only clears the used flag. The master
thread calls this often while the server is idle and rarely while it is
active, and buf_pool_t::garbage_collect() calls it under memory
pressure, where it ignores the used flag, because releasing these frames
is much cheaper than shrinking the buffer pool.

buf_pool_t::io_buf_t::acquire(): Factor out the scan for an unreserved
slot, which io_buf_t::reserve() ran twice and the scratch buffer reuses.

btr_page_reorganize_low(), page_zip_reorganize(): Obtain the scratch
page frame from buf_pool.scratch_buf. This removes the buf_pool.mutex
acquisition and the free block wait that buf_block_alloc() could
perform while at least a page X-latch was being held.
btr_page_reorganize_low() also used to leak the block on its error
paths; a single exit now releases the slot.

page_copy_rec_list_end_no_locks(), lock_move_reorganize_page(): Take
the source page frame instead of a source buf_block_t, because the
source is no longer a buffer pool block. In
page_copy_rec_list_end_no_locks() the source page is page_align(rec) at
every call site, so only the record is passed.

buf_tmp_buffer_t::acquire(), buf_tmp_buffer_t::release(): Use acquire
and release memory ordering, so that the page frame pointer of a slot
is published to the next thread that reserves the slot. acquire() also
reads the flag before the exchange, so that a scan across an array of
slots does not write to the slots that it finds reserved. release()
also marks the page frame undefined for Valgrind and MSan, because the
frame stays allocated for the next reserver and no deallocation marks
it.
forkfun
MDEV-39566 fix status_by_thread crash on live thread-count change

PFS_table_context snapshots the live thread/user/host/account
count at scan start and again on restore (filesort's second
rnd_init). If the count changed between the two, m_map_size
mismatched and the server aborted.

Skip the wasted re-sample on restore, bound each table's scan by
the frozen snapshot instead of the container's live count.
Monty
Removed some not needed checks and add a DBUG_ASSERT() for not covered code

- In ha_partition.cc:check_parallel_search(), remove check if
  item_field->field is null. This is not needed as the function is run
  after fix_field() which guarnatees that the field is always set.
- Added DBUG_ASSERT(new_field) to Item_field::fix_fields() to check if a
  select-list item, found by name or alias when resolving ORDER BY/GROUP
  BY/HAVING, can have field == 0. This error path is not covered by any
  mtr test.
Monty
MDEV-25292 Atomic CREATE OR REPLACE TABLE

Atomic CREATE OR REPLACE allows to keep an old table intact if the
command fails or during the crash. That is done by renaming the
original table to temporary name, as a backup and restoring it if the
CREATE fails. When the command is complete and logged the backup
table is deleted.

Atomic replace algorithm

  Two DDL chains are used for CREATE OR REPLACE:
  ddl_log_state_create (C) and ddl_log_state_rm (D).

  1. (C) Log rename of ORIG to TMP table (Rename TMP to original).
  2. Rename orignal to TMP.
  3. (C) Log CREATE_TABLE_ACTION of ORIG (drops ORIG);
  4. Do everything with ORIG (like insert data)
  5. (D) Log drop of TMP
  6. Write query to binlog (this marks (C) to be closed in
    case of failure)
  7. Execute drop of TMP through (D)
  8. Close (C) and (D)

  If there is a failure before 6) we revert the changes in (C)
  Chain (D) is only executed if 6) succeded (C is closed on
  crash recovery).

Foreign key errors will be found at the 1) stage.

Additional notes

  - CREATE TABLE without REPLACE and temporary tables is not affected
    by this commit.
    set @@drop_before_create_or_replace=1 can be used to
    get old behaviour where existing tables are dropped
    in CREATE OR REPLACE.

  - CREATE TABLE is reverted if binlogging the query fails.

  - Engines having HTON_EXPENSIVE_RENAME flag set are not affected by
    this commit. Conflicting tables marked with this flag will be
    deleted with CREATE OR REPLACE.

  - Replication execution is not affected by this commit.
    - Replication will first drop the conflicting table and then
      creating the new one.

  - CREATE TABLE .. SELECT XID usage is fixed and now there is no need
    to log DROP TABLE via DDL_CREATE_TABLE_PHASE_LOG (see comments in
    do_postlock()). XID is now correctly updated so it disables
    DDL_LOG_DROP_TABLE_ACTION. Note that binary log is flushed at the
    final stage when the table is ready. So if we have XID in the
    binary log we don't need to drop the table.

  - Three variations of CREATE OR REPLACE handled:

    1. CREATE OR REPLACE TABLE t1 (..);
    2. CREATE OR REPLACE TABLE t1 LIKE t2;
    3. CREATE OR REPLACE TABLE t1 SELECT ..;

  - Test case uses 6 combinations for engines (aria, aria_notrans,
    myisam, ib, lock_tables, expensive_rename) and 2 combinations for
    binlog types (row, stmt). Combinations help to check differences
    between the results. Error failures are tested for the above three
    variations.

  - expensive_rename tests CREATE OR REPLACE without atomic
    replace. The effect should be the same as with the old behaviour
    before this commit.

  - Triggers mechanism is unaffected by this change. This is tested in
    create_replace.test.

  - LOCK TABLES is affected. Lock restoration must be done after new
    table is created or TMP is renamed back to ORIG

  - Moved ddl_log_complete() from send_eof() to finalize_ddl(). This
    checkpoint was not executed before for normal CREATE TABLE but is
    executed now.

  - CREATE TABLE will now rollback also if writing to the binary
    logging failed. See rpl_gtid_strict.test

backup ddl log changes

- In case of a successfull CREATE OR REPLACE we only log
  the CREATE event, not the DROP TABLE event of the old table.

ddl_log.cc changes

  ddl_log_execute_action() now properly return error conditions.
  ddl_log_disable_entry() added to allow one to disable one entry.
  The entry on disk is still reserved until ddl_log_complete() is
  executed.

On XID usage

  Like with all other atomic DDL operations XID is used to avoid
  inconsistency between master and slave in the case of a crash after
  binary log is written and before ddl_log_state_create is closed. On
  recovery XIDs are taken from binary log and corresponding DDL log
  events get disabled.  That is done by
  ddl_log_close_binlogged_events().

On linking two chains together

  Chains are executed in the ascending order of entry_pos of execute
  entries. But entry_pos assignment order is undefined: it may assign
  bigger number for the first chain and then smaller number for the
  second chain. So the execution order in that case will be reverse:
  second chain will be executed first.

  To avoid that we link one chain to another. While the base chain
  (ddl_log_state_create) is active the secondary chain
  (ddl_log_state_rm) is not executed. That is: only one chain can be
  executed in two linked chains.

  The interface ddl_log_link_chains() was defined in "MDEV-22166
  ddl_log_write_execute_entry() extension".

Atomic info parameters in HA_CREATE_INFO

  Many functions in CREATE TABLE pass the same parameters. These
  parameters are part of table creation info and should be in
  HA_CREATE_INFO (or whatever). Passing parameters via single
  structure is much easier for adding new data and
  refactoring.

InnoDB changes
  Added ha_innobase::can_be_renamed_to_backup() to check if
  a table with foreign keys can be renamed.

Aria changes:
- Fixed issue in Aria engine with CREATE + locked tables
  that data was not properly commited in some cases in
  case of crashes.

Other changes:
- Removed some auto variables in log.cc for better code readability.
- Fixed old bug that CREATE ... SELECT would not be able to auto repair
  a table that is part of the SELECT.
- Marked MyISAM that it does not support ROLLBACK (not required but
  done for better consistency with other engines).

Known issues:
- InnoDB tables with foreign key definitions are not fully supported
  with atomic create and replace:
  - ha_innobase::can_be_renamed_to_backup() can detect some cases
    where InnoDB does not support renaming table with foreign key
    constraints.  In this case MariaDB will drop the old table before
    creating the new one.
    The detected cases are:
    - The new and old table is using the same foreign key constraint
      name.
    - The old table has self referencing constraints.
  - If the old and new table uses the same name for a constraint the
    create of the new table will fail. The orignal table will be
    restored in this case.
  - The above issues will be fixed in a future commit.
- CREATE OR REPLACE TEMPORARY table is not full atomic. Any conflicting
  table will always be dropped before creating a new one. (Old behaviour).

Bug fixes related to this MDEV:

MDEV-36435 Assertion failure in finalize_locked_tables()
MDEV-36439 Assertion `thd_arg->lex->sql_command != SQLCOM_CREATE_SEQUENCE...
MDEV-36498 Failed CoR in non-atomic mode no longer generates DROP in RBR...
MDEV-36508 Temporary files #sql-create-....frm occasionally stay after
          crash recovery
MDEV-38479 Crash in CREATE OR REPLACE SEQUENCE when new sequence cannot
          be created
MDEV-36497 Assertion failure after atomic CoR with Aria under lock in
          transactional context
MDEV-36501 EITS data is lost after failed attempt to CREATE OR REPLACE
          table
MDEV-36493 Atomic CREATE OR REPLACE ... SELECT blocks InnoDB purge
MDEV-39367 MSAN/valgrind errors in temp_file_size_cb_func,
          main.tmp_space_usage fails
MDEV-39446 Atomic CREATE OR REPLACE fails if a table cannot be decrypted

InnoDB related changes:
- ha_innodb::rename_table() does not handle foreign key constraint
  when renaming an normal table to internal tempory tables. This
  causes problems for CREATE OR REPLACE as the old constraints causes
  failure when creating a new table with the same constraints.
  This is fixed inside InnoDB by not threating tempfiles (#sql-create-..),
  created as part of CREATE OR REPLACE, as temporary files.
- In ha_innobase::delete_table(), ignore checking of constraints when
  dropping a #sql-create temporary table.
- In tablename_to_filename() and filename_to_tablename(), don't do
  filename conversion for internal temporary tables (#sql-...)

Other things:
- maria_create_trn_for_mysql() does not register a new transaction
  handler for commits. This was needed to ensure create or replace
  will not end with an active transaction.
- We do not get anymore warnings about "Engine not supporting atomic
  create" when doing a legal CREATE OR REPLACE on a table with
  foreign key constraints.
- Updated VIDEX engine flags to disable CREATE SEQUENCE.

Reverted commits:
MDEV-36685 "CREATE-SELECT may lose in binlog side-effects of
stored-routine" as it did not take into account that it safe to clear
binlogs if the created table is non transactional and there are no
other non transactional tables used.
- This was done because it caused extra logging when it is not needed
  (not using any non transactional tables) and it also did not solve
  side effects when using statement based loggging.

Other things:
- EITS data is preserved if create or replace fails if
  drop_before_create_or_replace=OFF. If ON, then create or replace
  will drop EITS before the drop of the original table (as before).
- Using CREATE OR REPLACE on a encrypted table that the user cannot
  decrypt will fail instead of replacing the encrypted table.
  The encrypted table will unchanged.
Sergei Petrunia
Review input 2: re-word comments
bsrikanth-mariadb
MDEV-39226: Push whole multi-table update/delete down into engines

Give storage engines a way to take over an entire multi-table
UPDATE/DELETE, the way they can already take over a SELECT. Without it the
join, the row matching and every modification run in the SQL layer even
when an engine could do the whole statement itself in one step; a
single-table UPDATE/DELETE already avoids this via
direct_update_rows()/direct_delete_rows(), but a multi-table statement has
no primary handler object to drive that path.

This adds a generic, engine-agnostic pushdown interface: the SQL layer
offers the statement to the engine, and if the engine accepts it, it
performs the whole thing and reports only the row counts.

- Split select_handler into a pushdown_handler base with select_handler
  (result set) and a new multi_upddel_handler (runs a whole UPDATE/DELETE,
  reports row counts, reported as PUSHED UPDATE/PUSHED DELETE); add
  handlerton::create_multi_upddel, looked up in Sql_cmd_dml::execute_inner().
- multi_update/multi_delete gain direct_update_delete_done(), which records
  the engine's counts so send_eof() binlogs and replies without the
  SQL-layer loop; it forces statement-format binlogging so the change still
  replicates under binlog_format=ROW, and errors out instead of silently
  dropping counts for an unsupported result object.
- FederatedX implements the interface as the reference engine used to test
  correctness: it prints the statement back and runs it remotely, passes
  the engine's error code/SQLSTATE through, reads the matched count from the
  remote info string, executes IGNORE locally, and only pushes down when all
  tables share one remote server (same as SELECT/derived/unit pushdown).

Test: federated.federatedx_pushdown_upd_del.
Vladislav Vaintroub
MDEV-22991 Support SSL over named pipes

Update libmariadb, to remove named pipe SSL check.
Update client.c to remove that check as well.

Add MTR test for named pipe + SSL:
- for Connector/C client, new test named_pipe_ssl
- for in-server clent, run mariabackup with user created as
  "IDENTIFIED WITH named_pipe REQUIRE SSL"
Georgi (Joro) Kodinov
Merge remote-tracking branch 'origin/main' into main-mdbf-1224
forkfun
MDEV-40303 SIGSEGV in Item_field::type_handler() on PS re-execution

Re-executing a prepared statement that reads a derived-table column
holding a scalar UNION subquery crashed the server: between executions
st_select_lex_unit::cleanup() frees union_result and sets 'cleaned' but
leaves 'prepared' set, so on re-fix set_row() walks a stale item_list
whose Item_field::field are NULL.

Fix: prepare() re-prepares a cleaned unit instead of treating it as
still prepared.
Monty
Added proper cleanup of main.cte_update_delete.test
Sergei Petrunia
Review input 3: don't check for is_supported_update_delete()

* we only get into create_federatedx_multi_upddel_handler for UPDATE/DELETE

* Use of is_supported_update_delete() implies that this checks
  for UPDATEs/DELETEs of which only some are supported.
  The function actually returned TRUE for any UPDATE/DELETE.
Marko Mäkelä
squash! 208ce0ae56e1ddd990555028b82aabd96909484f

InnoDB_backup::commit(): Enqueue the remaining log.

InnoDB_backup::checkpoint_complete(): If backup is running and
commit() has not been called, add each completed innodb_archive_log=ON
file to InnoDB_backup::queue. Else, skip or delete, as appropriate.
Monty
MDEV-40765 Assertion `"unexpected references" == 0' failed upon failing CREATE OR REPLACE

There was a few reasons for this failure:
- There was no timeout for the mdl_lock in the ddl log in
  execute_rename_table. This is needed to protect against a concurrent
  purge in InnoDB. If the purge would be running, the ddl would call
  rename_table without a ddl protection the assert could happen.
- InnoDB locked the original table name in purge, not the temporary
  name used to cache the table in case of rollback. The effect is
  that if a purge happens during rename the assert could happen.
- The table was a partitioned and the detection of partitioned tables
  did not take into account temporary #sql-create- prefixed tables.

Other things:
- The code in execute_rename_table() assumes that InnoDB never holds
  a MDL lock for more more than a few milliseconds. I am using a timeout
  for 10 seconds, just in case. In practice this timeout should be very
  short.
- Added a 10 second timeout for the MDL lock in execute_drop_table()
- Removed mysql_mutex_unlock(&LOCK_gdl) / mysql_mutex_lock(&LOCK_gdl)
  around calls to binlog as these are unsafe. The binlog code uses
  global variables that needs protection from other caller.
- Fixed a few compiler warnings related to tmp_file_prefix.
Alexander Barkov
MDEV-39518 Allow prepared statements in stored functions in assignment right hand

TODO: ps_in_func-stack3-pfp.test crashes with InnoDB!!!

Allowing prepared statements in stored functions when
a stored function is used in an assignment right hand.

Both DEFAULT clause of a variable initialization and
the right side of the SET statement are supported:

  CREATE PROCEDURE p1()
  BEGIN
    -- case 1: DEFAULT clause
    DECLARE spvar1 INT DEFAULT f1_with_ps(); -- OK

    -- case 2: SP variable assignment statement
    DECLARE spvar2 INT;
    SET spvar= f1_with_ps(); -- OK
  END;

- The parser now does not reject PS statements in stored functions.
  PS applicability in stored functions is now detected at run time.
  Note, PS statements in triggers are still prohibited by the parser.

- Functions with PS do not acquire MDL locks on tables.
  They work like procedures in terms of table opening.

- Functions with PS are not replicated as a single `SELECT f1()` call.
  They are replicated per-statement, like procedures.

- Only bare function calls are supported for now. Using a function in
  an expression does not make it PS-safe yet yet:
    SET v= f1()+0;
Aleksey Midenkov
MDEV-20865 Fix tests and system scripts to foreign key requirements

Storing foreign key metadata in TABLE_SHARE makes opening a foreign table
preopen its referenced tables, so a referenced table must be created
before the foreign one and dropped after it. Rearrange the order of
CREATE/DROP commands in tests and system scripts accordingly.

Affected: the help tables in mariadb_system_tables{,_fix}.sql (with
system_mysql_db_fix* results), fetch_first, union, insert_notembedded,
instant_alter_index_rename and opt_context_store_ddls; some also index a
referenced column or add a missing referenced table, and
opt_context_store_ddls re-records the now-shown MyISAM foreign key.
Aleksey Midenkov
MDEV-34392 Check foreign key column nullability in the server layer

Making a foreign key column NOT NULL must be refused when a referential
action can still write NULL into it -- ON UPDATE/DELETE SET NULL, or ON
UPDATE CASCADE from a NULLable parent. Until now this was enforced through a
per-column nullability bitmap in FK_info (fields_nullable) that the storage
engine allocated and populated, and that the server read back while checking
an ALTER.

Since MDEV-20865 the server keeps the foreign key definitions in TABLE_SHARE,
so this round trip through the engine is redundant. Perform the check
directly in mysql_prepare_alter_table() from the stored referential actions
and the old/new column definitions, and drop the FK_info bitmap along with
its assign_nullable()/set_nullable()/is_nullable() helpers. Incompatible
changes are still rejected with ER_FK_COLUMN_NOT_NULL.

Fixes foreign_null test.
Aleksey Midenkov
MDEV-20865 Store foreign key info in TABLE_SHARE

1. Access foreign keys via TABLE_SHARE::foreign_keys and
  TABLE_SHARE::referenced_keys;

  foreign_keys and referenced_keys are lists in TABLE_SHARE.

2. Remove handler FK interface:

  - get_foreign_key_list()
  - get_parent_foreign_key_list()
  - referenced_by_foreign_key()

3. Invalidate referenced shares on:

  - RENAME TABLE
  - DROP TABLE
  - RENAME COLUMN
  - ADD FOREIGN KEY

  When foreign table is created or altered by the above operations
  all referenced shares are closed. This blocks the operation while
  any referenced shares are used (when at least one its TABLE
  instance is locked).

4. Update referenced shares on:

  - CREATE TABLE

  On CREATE TABLE add items to referenced_keys of referenced
  shares. States of referenced shares are restored in case of errors.

5. Invalidate foreign shares on:

  - RENAME TABLE
  - RENAME COLUMN

  The above-mentioned blocking takes effect.

6. Check foreign/referenced shares consistency on:

  - CHECK TABLE

7. Temporary change until MDEV-21051:

  InnoDB fill foreign key info at handler open().

FOREIGN_KEY_INFO is refactored to FK_info holding Lex_cstring.

On first TABLE open FK_info is loaded from storage engine into
TABLE_SHARE. All referenced shares (if any exists) are closed. This
leads to blocking of first time foreign table open while referenced
tables are used.

MDEV-21311 Converge Foreign_key and supplemental generated Key together

mysql_prepare_create_table() does data validation and such utilities
as automatic name generation. But it does that only for indexes and
ignores Foreign_key objects. Now as Foreign_key data needs to be
stored in FRM files as well this processing must be done for it like
for any other Key objects.

Replace Key::FOREIGN_KEY type with Key::foreign flag of type
Key::MULTIPLE and Key::generated set to true. Construct one object
with Key::foreign == true instead of two objects of type
Key::FOREIGN_KEY and Key::MULTIPLE.

MDEV-21051 datadict refactorings

- Move read_extra2() to datadict.cc
- Refactored extra2_fields to Extra2_info
- build_frm_image() readability

MDEV-21051 build_table_shadow_filename() refactoring

mysql_prepare_alter_table() leaks fixes

MDEV-21051 amend system tables locking restriction

Table mysql.help_relation has foreign key to mysql.help_keyword. On
bootstrap when help_relation is opened, it preopens help_keyword for
READ and fails in lock_tables_check().

If system table is opened for write then fk references are opened for
write.

Related to: Bug#25422, WL#3984
Tests: main.lock

MDEV-21051 Store and read foreign key info into/from FRM files

1. Introduce Foreign_key_io class which creates/parses binary stream
containing foreign key structures. Referenced tables store there only
hints about foreign tables (their db and name), they restore full info
from the corresponding tables.

Foreign_key_io is stored under new EXTRA2_FOREIGN_KEY_INFO field in
extra2 section of FRM file.

2. Modify mysql_prepare_create_table() to generate names for foreign
keys. Until InnoDB storage of foreign keys is removed, FK names must
be unique across the database: the FK name must be based on table
name.

3. Keep stored data in sync on DDL changes. Referenced tables update
their foreign hints after following operations on foreign tables:

  - RENAME TABLE
  - DROP TABLE
  - CREATE TABLE
  - ADD FOREIGN KEY
  - DROP FOREIGN KEY

Foreign tables update their foreign info after following operations on
referenced tables:

  - RENAME TABLE
  - RENAME COLUMN

4. To achieve 3. there must be ability to rewrite extra2 section of
FRM file without full reparse. FRM binary is built from primary
structures like HA_CREATE_INFO and cannot be built from TABLE_SHARE.

Use shadow write and rename like fast_alter_partition_table()
does. Shadow FRM is new FRM file that replaces the old one.

CREATE TABLE workflow:

  1. Foreign_key is constructed in parser, placed into
    alter_info->key_list;

  2. mysql_prepare_create_table() translates them to FK_info, assigns
    foreign_id if needed;

  3. build_frm_image() writes two FK_info lists into FRM's extra2
    section, for referenced keys it stores only table names (hints);

  4. init_from_binary_frm_image() parses extra2 section and fills
    foreign_keys and referenced_keys of TABLE_SHARE.

    It restores referenced_keys by reading hint list of table names,
    opening corresponding shares and restoring FK_info from their
    foreign_keys. Hints resolution is done only when initializing
    non-temporary shares. Usually temporary share has different
    (temporary) name and it is impossible to resolve foreign keys by
    that name (as we identify them by both foreign and referenced
    table names). Another not unimportant reason is performance: this
    saves spare share acquisitions.

ALTER TABLE workflow:

  1. Foreign_key is constructed in parser, placed into
    alter_info->key_list;

  2. mysql_prepare_alter_table() prepares action lists and share list
    of foreigns/references;

  3. mysql_prepare_alter_table() locks list of foreigns/references by
    MDL_INTENTION_EXCLUSIVE, acquires shares;

  4. prepare_create_table() converts key_list into FK_list, assigns
    foreign_id;

  5. shadow FRM of altered table is created;

  6. data is copied;

  7. altered table is locked by MDL_EXCLUSIVE;

  8. fk_handle_alter() processes action lists, creates FK backups,
    modifies shares, writes shadow FRMs;

  9. altered table is closed;

  10. shadow FRMs are installed;

  11. altered table is renamed, FRM backup deleted;

  12. (TBD in MDEV-21053) shadow FRMs installation log closed, backups
      deleted;

On FK backup system:

In case of failed DDL operation all shares that was modified must be
restored into original state. This is done by FK_ddl_backup (CREATE,
DROP), FK_rename_backup (RENAME), FK_alter_backup (ALTER).

On STL usage:

STL is used for utility not performance-critical algorithms, core
structures hold native List. A wrapper was made to convert STL
exception into bool error status or NULL value.

MDEV-20865 fk_check_consistency() in CHECK TABLE

Self-refs fix

Test table_flags fix: "debug" deviation is now gone.

FIXMEs: +16 -1
Sergei Petrunia
Review input 1: simplify code in EXPLAIN handling

remove is_pushed_down_select_type(), keep just
get_pushed_down_select_text().
Aleksey Midenkov
MDEV-20865 extra2_fields refactored to Extra2_info class

read_extra2() is now Extra2_info::read()
Additional assertions for checking size consistency.

Extra2_info::write() is used by further MDEV-20865 development.
Georgi (Joro) Kodinov
Merge remote-tracking branch 'origin/main' into main-mdbf-1224
Aleksey Midenkov
MDEV-20865 Extra2_info::write(): gate fields by length, not str

store_size() counts a field only when its length is non-zero, but write()
emitted a field whenever its str was set. A {str != NULL, length == 0} field
would then reach extra2_write_len(0) (bogus zero length / assert) and break
the write_size == store_size() invariant. Gate the writes on length so both
sides agree.

FIXME: fixup to 8f08b2994a3
Monty
MDEV-40765 Assertion `"unexpected references" == 0' failed upon failing CREATE OR REPLACE

There was a few reasons for this failure:
- There was no timeout for the mdl_lock in the ddl log in
  execute_rename_table. This is needed to protect against a concurrent
  purge in InnoDB. If the purge would be running, the ddl would call
  rename_table without a ddl protection the assert could happen.
- InnoDB locked the original table name in purge, not the temporary
  name used to cache the table in case of rollback. The effect is
  that if a purge happens during rename the assert could happen.
- The table was a partitioned and the detection of partitioned tables
  did not take into account temporary #sql-create- prefixed tables.

Other things:
- The code in execute_rename_table() assumes that InnoDB never holds
  a MDL lock for more more than a few milliseconds. I am using a timeout
  for 10 seconds, just in case. In practice this timeout should be very
  short.
- Added a 10 second timeout for the MDL lock in execute_drop_table()
- Removed mysql_mutex_unlock(&LOCK_gdl) / mysql_mutex_lock(&LOCK_gdl)
  around calls to binlog as these are unsafe. The binlog code uses
  global variables that needs protection from other caller.
- Fixed a few compiler warnings related to tmp_file_prefix.
Alessandro Vetere
MDEV-38056 Assertion 'bpage->state() >= buf_page_t::UNFIXED' in buf_page_get_zip()

Three callers reconstruct old versions of a clustered index record, derive
secondary index entries from them, and dereference the BLOB pointers of
externally stored columns: row_vers_impl_x_locked_low() for the implicit lock
check, row_undo_mod_sec_is_unsafe() for rollback, and row_check_index() for
CHECK TABLE ... EXTENDED. purge_sys.view is what keeps those pages allocated,
but purge_sys_t::view_guard froze it only for the duration of
trx_undo_prev_version_build(), and the dereference happens after. On
ROW_FORMAT=COMPRESSED this trips the assertion above; in a release build the
freed page is read anyway, which is silent corruption.

All three now hold purge_sys.latch across the dereference, and only where the
version has externally stored columns, since row_build() dereferences nothing
otherwise. In row_check_index() the freeze spans the whole comparison, which
fetches one field at a time and may evaluate a virtual column expression; for
the latest version it is also confined to a delete-marked record, the only
kind that need not own what it points at. Freezing that late means the view
may have advanced since the walk decided a version was reachable, so each
caller re-establishes that under the freeze; trx_undo_prev_version() states
the condition and why testing the oldest writer applied suffices.

The first two callers can treat that test as an invariant and end the walk
where it fails. row_check_index() cannot, because it decides reachability from
the lagging purge_sys.end_view on purpose, and that lag is how it finds orphan
secondary index records: where the test fails it stops and reports nothing
rather than treating the record as an orphan, and no report is lost for good,
end_view eventually reaching the same point. Its two purge_sys.is_purgeable()
tests now read the frozen view wherever a fetch follows, which makes them
atomic with the fetch they guard. purge_sys_t::view_guard and
purge_sys_t::is_purgeable() assert that the thread holds no purge_sys.latch,
because a nested shared acquisition would hang behind a waiting writer.

trx_undo_prev_version(): remove the gate that was meant to stop CHECK TABLE
... EXTENDED from fetching BLOBs it may no longer own, and
view_guard::is_extended() with it, which no guard mode could satisfy. That
decision belongs to the caller, the only one that knows whether it will
dereference anything.

row_log_table_get_pk(): document why the online ALTER path may dereference
without a freeze.

Debug-only keywords. purge_hold_cleanup parks a purge batch between its last
purged record and purge_sys_t::batch_cleanup(), the window in which a reader
that goes by purge_sys.end_view can still reach history the batch has removed;
a batch opens and closes it without ever returning to the test.
purge_no_blob_freeze sends a version that does have externally stored columns
down the path one without any takes, which is what all four call sites did
before this change, and reproduces the assertion above.
row_vers_impl_x_locked_purgeable, row_undo_mod_sec_is_unsafe_purgeable and
row_check_index_purgeable force the re-validation to fail, reaching exits that
purge_sys.view advancing mid-walk otherwise produces. The first reports no
implicit lock for a row that a live transaction still holds, so a test may
only check that nothing breaks; the other two make the server more cautious.

Tests. old_blob and old_blob_updel cover the implicit lock check,
old_blob_rollback the rollback, old_blob_check CHECK TABLE ... EXTENDED. Each
parks a walk at a dereference, makes the BLOB freeable, and asserts that the
counter of purged update records, which is what would free it, stays at zero
while parked and advances once the walk is over. old_blob_updel covers an undo
log record that stores only the 20-byte reference and needs purge_hold_cleanup
to reach it; old_blob_rollback parks at a reference that the version merely
inherited. All four fail with the original assertion under
debug_dbug=+d,purge_no_blob_freeze, and skip above a 16k page size, which
ROW_FORMAT=COMPRESSED requires. old_blob_purgeable drives the three
re-validation exits, needs no synchronisation, and runs at every page size.
Aleksey Midenkov
MDEV-20865 extra2_fields refactored to Extra2_info class

read_extra2() is now Extra2_info::read()
Additional assertions for checking size consistency.

Extra2_info::write() is used by further MDEV-20865 development.
Aleksey Midenkov
MDEV-40799 Runtime plugin/UDF load errors lost under --silent-startup

Regression from MDEV-32745 (7828fb475b0), which guarded the
plugin-load my_error() calls with opt_silent_startup.  That option is
a lifetime global, set once at startup and never reset, so the guard
suppressed the SQL error for the whole server lifetime, not just
during startup.  Runtime operations (INSTALL PLUGIN, CREATE FUNCTION
... SONAME) then skipped my_error(), never set the diagnostics area
and wrongly succeeded - e.g. main.ps's "call proc_1()" no longer
failed with ER_CANT_OPEN_LIBRARY.

Startup callers pass MYF(ME_ERROR_LOG); runtime callers pass MYF(0).
Gate the silencing on that flag via silent_plugin_startup() so it
applies only to the startup error-log path, and runtime errors always
reach the client.

No new test case: the runtime failure path is already covered by
existing tests (e.g. main.ps's ER_CANT_OPEN_LIBRARY check).  The
regression stayed invisible only because stock MTR does not start
servers with --silent-startup.  A dedicated test would have to restart
the server with --silent-startup solely to assert that a startup-only
option does not affect runtime, which adds little over the restored
invariant.
Monty
MDEV-40776 Atomic CREATE OR REPLACE silently breaks the foreign key

Give an error if one tries to drop a table referenced by a foreign keys
This is needed as innodb will keep the reference to the origina table
even when it is renamed to a temporary name as part of create or replace.

Other things:
- Changed the error message for  ER_TRUNCATE_ILLEGAL_FK to say
  "Cannot drop or truncate a table ..."
Georgi (Joro) Kodinov
Days work: started doing grouping and doxygen conversion of plugin.h
Aleksey Midenkov
MDEV-20865 extra2_fields refactored to Extra2_info class

read_extra2() is now Extra2_info::read()
Additional assertions for checking size consistency.

Extra2_info::write() is used by further MDEV-20865 development.
Andrei Elkin
explore bb-13.2-MDEV-22992:rpl.rpl_xa
Marko Mäkelä
squash! b625be27ca0a0b24730dd19d37d8bbf85917156c

InnoDB_backup::context: Remove the pointer indirection and do not allow
two overlapping backup operations.

InnoDB_backup::init(): Wait for a possible previous BACKUP SERVER
operation to reach the very end of InnoDB_backup::context::cleanup()
so that the context can be safely reused.
Georgi (Joro) Kodinov
day's work: added more plugin types and clarified the top level group structure
Aleksey Midenkov
MDEV-20865 Refactor Share_acquire::fk_error() into acquire()

Share_acquire::fk_error() inspected thd->is_error() after the acquisition to
decide whether a missing referenced table is tolerable.  That breaks when an
outer Internal_error_handler consumes the error first, e.g. the
Postponed_error_handler installed by mysql_rename_tables() (MDEV-27027).

Move the decision into Share_acquire::acquire(): push a
No_such_table_error_handler for the acquisition when foreign key checks are
not enforced, so ER_NO_SUCH_TABLE is trapped above any outer handler rather
than deferred or leaked into SHOW WARNINGS.  The non-tolerated error outcome
is recorded in Share_acquire::error; the four consumers check it instead of
calling fk_error().

FIXME: Squash into the main "MDEV-20865 Store foreign key info in
TABLE_SHARE" commit.
Alessandro Vetere
MDEV-38056 Assertion 'bpage->state() >= buf_page_t::UNFIXED' in buf_page_get_zip()

Three callers reconstruct old versions of a clustered index record, derive
secondary index entries from them, and dereference the BLOB pointers of
externally stored columns: row_vers_impl_x_locked_low() for the implicit lock
check, row_undo_mod_sec_is_unsafe() for rollback, and row_check_index() for
CHECK TABLE ... EXTENDED. purge_sys.view is what keeps those pages allocated,
but purge_sys_t::view_guard froze it only for the duration of
trx_undo_prev_version_build(), and the dereference happens after. On
ROW_FORMAT=COMPRESSED this trips the assertion above; in a release build the
freed page is read anyway, which is silent corruption.

All three now hold purge_sys.latch across the dereference, and only where the
version has externally stored columns, since row_build() dereferences nothing
otherwise. In row_check_index() the freeze spans the whole comparison, which
fetches one field at a time and may evaluate a virtual column expression; for
the latest version it is also confined to a delete-marked record, the only
kind that need not own what it points at. Freezing that late means the view
may have advanced since the walk decided a version was reachable, so each
caller re-establishes that under the freeze; trx_undo_prev_version() states
the condition and why testing the oldest writer applied suffices.

The first two callers can treat that test as an invariant and end the walk
where it fails. row_check_index() cannot, because it decides reachability from
the lagging purge_sys.end_view on purpose, and that lag is how it finds orphan
secondary index records: where the test fails it stops and reports nothing
rather than treating the record as an orphan, and no report is lost for good,
end_view eventually reaching the same point. Its two purge_sys.is_purgeable()
tests now read the frozen view wherever a fetch follows, which makes them
atomic with the fetch they guard. purge_sys_t::view_guard and
purge_sys_t::is_purgeable() assert that the thread holds no purge_sys.latch,
because a nested shared acquisition would hang behind a waiting writer.

trx_undo_prev_version(): remove the gate that was meant to stop CHECK TABLE
... EXTENDED from fetching BLOBs it may no longer own, and
view_guard::is_extended() with it, which no guard mode could satisfy. That
decision belongs to the caller, the only one that knows whether it will
dereference anything.

row_log_table_get_pk(): document why the online ALTER path may dereference
without a freeze.

Debug-only keywords. purge_hold_cleanup parks a purge batch between its last
purged record and purge_sys_t::batch_cleanup(), the window in which a reader
that goes by purge_sys.end_view can still reach history the batch has removed;
a batch opens and closes it without ever returning to the test.
purge_no_blob_freeze sends a version that does have externally stored columns
down the path one without any takes, which is what all four call sites did
before this change, and reproduces the assertion above.
row_vers_impl_x_locked_purgeable, row_undo_mod_sec_is_unsafe_purgeable and
row_check_index_purgeable force the re-validation to fail, reaching exits that
purge_sys.view advancing mid-walk otherwise produces. The first reports no
implicit lock for a row that a live transaction still holds, so a test may
only check that nothing breaks; the other two make the server more cautious.

Tests. old_blob and old_blob_updel cover the implicit lock check,
old_blob_rollback the rollback, old_blob_check CHECK TABLE ... EXTENDED. Each
parks a walk at a dereference, makes the BLOB freeable, and asserts that the
counter of purged update records, which is what would free it, stays at zero
while parked and advances once the walk is over. old_blob_updel covers an undo
log record that stores only the 20-byte reference and needs purge_hold_cleanup
to reach it; old_blob_rollback parks at a reference that the version merely
inherited. All four fail with the original assertion under
debug_dbug=+d,purge_no_blob_freeze, and skip above a 16k page size, which
ROW_FORMAT=COMPRESSED requires. old_blob_purgeable drives the three
re-validation exits, needs no synchronisation, and runs at every page size.
Aleksey Midenkov
MDEV-40311 Fix mysqldump-nl leaving slave connection behind

The trailing CHANGE MASTER left master.info in the datadir, so a
later test that restarts the server auto-started the slave and broke
its check-testcase. Use RESET SLAVE ALL to drop the connection.

MTR's internal check of the test case 'sys_vars.default_master_connection_basic' failed.
This means that the test case does not preserve the state that existed
before the test case was executed.  Most likely the test case did not
do a proper clean-up. It could also be caused by the previous test run
by this thread, if the server wasn't restarted.
This is the diff of the states of the servers before and after the
test case was executed:
-Slave_IO_Running No
-Slave_SQL_Running No
+Slave_IO_Running Connecting
+Slave_SQL_Running Yes
...
-Last_IO_Errno 0
-Last_IO_Error
+Last_IO_Errno 1045
+Last_IO_Error error connecting to master '[email protected]:3306' - retry-time: 60  maximum-retries: 100000  message: Access denied for user 'root'@'localhost' (using password: NO)
Aleksey Midenkov
MDEV-20865 Fix tests and system scripts to foreign key requirements

Storing foreign key metadata in TABLE_SHARE makes opening a foreign table
preopen its referenced tables, so a referenced table must be created
before the foreign one and dropped after it. Rearrange the order of
CREATE/DROP commands in tests and system scripts accordingly.

Affected: the help tables in mariadb_system_tables{,_fix}.sql (with
system_mysql_db_fix* results), fetch_first, union, insert_notembedded,
instant_alter_index_rename and opt_context_store_ddls; some also index a
referenced column or add a missing referenced table, and
opt_context_store_ddls re-records the now-shown MyISAM foreign key.
Georgi (Joro) Kodinov
MDEV-40909: main.user_limits is unstable

The cleanup of the old connection goes on in the background.
It can take longer on a busy server and this trigger the active sessions
warning in DROP user.
Stabilized the test by disabling warnings around DROP USER.
Aleksey Midenkov
MDEV-20865 Reuse share in fk_handle_drop()

mysql_rm_table_no_locks() already opens the dropped table's share, so
pass it to fk_handle_drop() instead of re-acquiring it with
GTS_FK_SHALLOW_HINTS.

Skip fk_handle_drop() when the share is unreadable, letting the engine
report the error. This restores ER_GET_ERRNO for
main.partition_not_blackhole, which clear_error() used to mask by
zeroing my_errno.

New Share_acquire::inexistent_t lets DROP keep seeing ER_NO_SUCH_TABLE
independent of foreign_key_checks (INEXISTENT_ALWAYS).

Fixes main.partition_not_blackhole

FIXME: Squash into the main "MDEV-20865 Store foreign key info in
TABLE_SHARE" commit.
Aleksey Midenkov
MDEV-20865 Fix tests and system scripts to foreign key requirements

Storing foreign key metadata in TABLE_SHARE makes opening a foreign table
preopen its referenced tables, so a referenced table must be created
before the foreign one and dropped after it. Rearrange the order of
CREATE/DROP commands in tests and system scripts accordingly.

Affected: the help tables in mariadb_system_tables{,_fix}.sql (with
system_mysql_db_fix* results), fetch_first, union, insert_notembedded,
instant_alter_index_rename and opt_context_store_ddls; some also index a
referenced column or add a missing referenced table, and
opt_context_store_ddls re-records the now-shown MyISAM foreign key.