Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Monty
MDEV-25292 Atomic CREATE OR REPLACE TABLE

Atomic CREATE OR REPLACE allows to keep an old table intact if the
command fails or during the crash. That is done by renaming the
original table to temporary name, as a backup and restoring it if the
CREATE fails. When the command is complete and logged the backup
table is deleted.

Atomic replace algorithm

  Two DDL chains are used for CREATE OR REPLACE:
  ddl_log_state_create (C) and ddl_log_state_rm (D).

  1. (C) Log rename of ORIG to TMP table (Rename TMP to original).
  2. Rename orignal to TMP.
  3. (C) Log CREATE_TABLE_ACTION of ORIG (drops ORIG);
  4. Do everything with ORIG (like insert data)
  5. (D) Log drop of TMP
  6. Write query to binlog (this marks (C) to be closed in
    case of failure)
  7. Execute drop of TMP through (D)
  8. Close (C) and (D)

  If there is a failure before 6) we revert the changes in (C)
  Chain (D) is only executed if 6) succeded (C is closed on
  crash recovery).

Foreign key errors will be found at the 1) stage.

Additional notes

  - CREATE TABLE without REPLACE and temporary tables is not affected
    by this commit.
    set @@drop_before_create_or_replace=1 can be used to
    get old behaviour where existing tables are dropped
    in CREATE OR REPLACE.

  - CREATE TABLE is reverted if binlogging the query fails.

  - Engines having HTON_EXPENSIVE_RENAME flag set are not affected by
    this commit. Conflicting tables marked with this flag will be
    deleted with CREATE OR REPLACE.

  - Replication execution is not affected by this commit.
    - Replication will first drop the conflicting table and then
      creating the new one.

  - CREATE TABLE .. SELECT XID usage is fixed and now there is no need
    to log DROP TABLE via DDL_CREATE_TABLE_PHASE_LOG (see comments in
    do_postlock()). XID is now correctly updated so it disables
    DDL_LOG_DROP_TABLE_ACTION. Note that binary log is flushed at the
    final stage when the table is ready. So if we have XID in the
    binary log we don't need to drop the table.

  - Three variations of CREATE OR REPLACE handled:

    1. CREATE OR REPLACE TABLE t1 (..);
    2. CREATE OR REPLACE TABLE t1 LIKE t2;
    3. CREATE OR REPLACE TABLE t1 SELECT ..;

  - Test case uses 6 combinations for engines (aria, aria_notrans,
    myisam, ib, lock_tables, expensive_rename) and 2 combinations for
    binlog types (row, stmt). Combinations help to check differences
    between the results. Error failures are tested for the above three
    variations.

  - expensive_rename tests CREATE OR REPLACE without atomic
    replace. The effect should be the same as with the old behaviour
    before this commit.

  - Triggers mechanism is unaffected by this change. This is tested in
    create_replace.test.

  - LOCK TABLES is affected. Lock restoration must be done after new
    table is created or TMP is renamed back to ORIG

  - Moved ddl_log_complete() from send_eof() to finalize_ddl(). This
    checkpoint was not executed before for normal CREATE TABLE but is
    executed now.

  - CREATE TABLE will now rollback also if writing to the binary
    logging failed. See rpl_gtid_strict.test

backup ddl log changes

- In case of a successfull CREATE OR REPLACE we only log
  the CREATE event, not the DROP TABLE event of the old table.

ddl_log.cc changes

  ddl_log_execute_action() now properly return error conditions.
  ddl_log_disable_entry() added to allow one to disable one entry.
  The entry on disk is still reserved until ddl_log_complete() is
  executed.

On XID usage

  Like with all other atomic DDL operations XID is used to avoid
  inconsistency between master and slave in the case of a crash after
  binary log is written and before ddl_log_state_create is closed. On
  recovery XIDs are taken from binary log and corresponding DDL log
  events get disabled.  That is done by
  ddl_log_close_binlogged_events().

On linking two chains together

  Chains are executed in the ascending order of entry_pos of execute
  entries. But entry_pos assignment order is undefined: it may assign
  bigger number for the first chain and then smaller number for the
  second chain. So the execution order in that case will be reverse:
  second chain will be executed first.

  To avoid that we link one chain to another. While the base chain
  (ddl_log_state_create) is active the secondary chain
  (ddl_log_state_rm) is not executed. That is: only one chain can be
  executed in two linked chains.

  The interface ddl_log_link_chains() was defined in "MDEV-22166
  ddl_log_write_execute_entry() extension".

Atomic info parameters in HA_CREATE_INFO

  Many functions in CREATE TABLE pass the same parameters. These
  parameters are part of table creation info and should be in
  HA_CREATE_INFO (or whatever). Passing parameters via single
  structure is much easier for adding new data and
  refactoring.

InnoDB changes
  Added ha_innobase::can_be_renamed_to_backup() to check if
  a table with foreign keys can be renamed.

Aria changes:
- Fixed issue in Aria engine with CREATE + locked tables
  that data was not properly commited in some cases in
  case of crashes.

Other changes:
- Removed some auto variables in log.cc for better code readability.
- Fixed old bug that CREATE ... SELECT would not be able to auto repair
  a table that is part of the SELECT.
- Marked MyISAM that it does not support ROLLBACK (not required but
  done for better consistency with other engines).

Known issues:
- InnoDB tables with foreign key definitions are not fully supported
  with atomic create and replace:
  - ha_innobase::can_be_renamed_to_backup() can detect some cases
    where InnoDB does not support renaming table with foreign key
    constraints.  In this case MariaDB will drop the old table before
    creating the new one.
    The detected cases are:
    - The new and old table is using the same foreign key constraint
      name.
    - The old table has self referencing constraints.
  - If the old and new table uses the same name for a constraint the
    create of the new table will fail. The orignal table will be
    restored in this case.
  - The above issues will be fixed in a future commit.
- CREATE OR REPLACE TEMPORARY table is not full atomic. Any conflicting
  table will always be dropped before creating a new one. (Old behaviour).

Bug fixes related to this MDEV:

MDEV-36435 Assertion failure in finalize_locked_tables()
MDEV-36439 Assertion `thd_arg->lex->sql_command != SQLCOM_CREATE_SEQUENCE...
MDEV-36498 Failed CoR in non-atomic mode no longer generates DROP in RBR...
MDEV-36508 Temporary files #sql-create-....frm occasionally stay after
          crash recovery
MDEV-38479 Crash in CREATE OR REPLACE SEQUENCE when new sequence cannot
          be created
MDEV-36497 Assertion failure after atomic CoR with Aria under lock in
          transactional context
MDEV-36501 EITS data is lost after failed attempt to CREATE OR REPLACE
          table
MDEV-36493 Atomic CREATE OR REPLACE ... SELECT blocks InnoDB purge
MDEV-39367 MSAN/valgrind errors in temp_file_size_cb_func,
          main.tmp_space_usage fails
MDEV-39446 Atomic CREATE OR REPLACE fails if a table cannot be decrypted

InnoDB related changes:
- ha_innodb::rename_table() does not handle foreign key constraint
  when renaming an normal table to internal tempory tables. This
  causes problems for CREATE OR REPLACE as the old constraints causes
  failure when creating a new table with the same constraints.
  This is fixed inside InnoDB by not threating tempfiles (#sql-create-..),
  created as part of CREATE OR REPLACE, as temporary files.
- In ha_innobase::delete_table(), ignore checking of constraints when
  dropping a #sql-create temporary table.
- In tablename_to_filename() and filename_to_tablename(), don't do
  filename conversion for internal temporary tables (#sql-...)

Other things:
- maria_create_trn_for_mysql() does not register a new transaction
  handler for commits. This was needed to ensure create or replace
  will not end with an active transaction.
- We do not get anymore warnings about "Engine not supporting atomic
  create" when doing a legal CREATE OR REPLACE on a table with
  foreign key constraints.
- Updated VIDEX engine flags to disable CREATE SEQUENCE.

Reverted commits:
MDEV-36685 "CREATE-SELECT may lose in binlog side-effects of
stored-routine" as it did not take into account that it safe to clear
binlogs if the created table is non transactional and there are no
other non transactional tables used.
- This was done because it caused extra logging when it is not needed
  (not using any non transactional tables) and it also did not solve
  side effects when using statement based loggging.

Other things:
- EITS data is preserved if create or replace fails if
  drop_before_create_or_replace=OFF. If ON, then create or replace
  will drop EITS before the drop of the original table (as before).
- Using CREATE OR REPLACE on a encrypted table that the user cannot
  decrypt will fail instead of replacing the encrypted table.
  The encrypted table will unchanged.
forkfun
Merge branch '11.4' into '11.8'
Oleg Smirnov
MDEV-39491 Add test harness

This commit implements pseudo-parallel execution of SELECTs
which allows to test the correctness of parallel algorithms.

Eligible InnoDB tables that were planned to be full-scanned
by the clustered index, are switched to the pseudo-parallel execution.
That means the primary index is split into chunks, and those
chunks are processed one after another by a single thread.
This thread mimics the parallel execution by calling the
parallel handler API and acting as both the coordinator and
the worker.

This mode is activated automatically, there is no need to
set any variables before that. If an InnoDB table is set
to be fully scanned by primary index, the pseudo-parallel
mode is employed.

This harness allows to run MTR tests to catch possible bugs
in the parallel logic implementation.
Sergei Golubchik
MDEV-40589 default exclude list for secure-file-priv

don't allow to access /proc if secure-file-priv="",
set secure-file-priv=/ to access everything and disable the exclude list
Oleg Smirnov
MDEV-39491 Parallel Query: cache the chunk-boundary verdict per leaf page

The chunk-boundary clamp added with the InnoDB parallel scan compares every
record returned by row_search_mvcc() against the chunk's exclusive upper
bound. That is one cmp_dtuple_rec() per fetched row, wasted on every row but
the last one of the chunk.

Records on a leaf page are sorted, so the boundary can only fall inside the
page whose last user record is >= the boundary. On any earlier page all
records are in range and no per-record comparison is needed. Take the verdict
once per leaf page by comparing the end tuple against the page's last user
record, and cache it in prebuilt.

- row_pscan_reached_chunk_end() (row0sel.cc): per-page verdict plus the
  per-record fallback when the boundary does lie on the current page.
- row_prebuilt_t::m_pscan_clamp_{page,clock,in_range}: the cached verdict,
  keyed by page number and modify_clock, so a page that was split, merged or
  reorganized in the meantime is re-examined.
- row_prebuilt_t::set_pscan_end_tuple() replaces direct assignment of
  m_pscan_end_tuple, so switching to the next chunk cannot leave a stale
  verdict behind.

The shortcut is confined to consistent reads of records living on the page
the cursor is positioned on: a record read at READ UNCOMMITTED may have been
inserted onto the page after the verdict was taken (a plain insert does not
bump modify_clock), and an old version built in the heap is not covered by
any page's verdict. Those cases fall back to the per-record comparison.
forkfun
Merge branch '13.0' into 'main'
Oleg Smirnov
MDEV-39491 Parallel Query: InnoDB clustered-index partitioning for parallel scan

Implement the InnoDB side of the handler parallel-scan API: partition the
clustered index into disjoint key-range chunks and serve them to the SQL layer
as pull-based scan jobs. Each chunk is read through the normal
row_search_mvcc() path, so MVCC visibility, AHI and the prefetch cache keep
working unchanged. InnoDB spawns no threads of its own.

- Parallel_coordinator (row0pcoord.{h,cc}): adapted from MySQL's
  Parallel_reader, reduced to partitioning and job distribution. Walks the
  index top-down into Exec_ctx chunks bounded by clustered-key tuples.
- ha_innobase: implements parallel_{init,end}_coordinator,
  parallel_get_worker_context, parallel_{init,end}_worker and
  parallel_get_next_row.
- Chunk-boundary clamp: row_prebuilt_t::m_pscan_end_tuple (NULL = unbounded)
  makes row_search_mvcc() stop before prefetching past the chunk's exclusive
  upper bound, treated as end-of-range so the next chunk is pulled.
- btr_pcur_open_on_user_rec(page_cur_t) overload to anchor chunk boundaries.
- Build: row0pcoord.cc added to CMakeLists.txt and auto_event_names[].
Oleg Smirnov
MDEV-39491 Add test harness

This commit implements pseudo-parallel execution of SELECTs
which allows to test the correctness of parallel algorithms.

Eligible InnoDB tables that were planned to be full-scanned
by the clustered index, are switched to the pseudo-parallel execution.
That means the primary index is split into chunks, and those
chunks are processed one after another by a single thread.
This thread mimics the parallel execution by calling the
parallel handler API and acting as both the coordinator and
the worker.

This harness allows to run MTR tests to catch possible bugs
in the parallel logic implementation
Monty
MDEV-23298 Assertion `table_list->prelocking_placeholder == TABLE_LIST::PRELOCK_NONE' failed in check_lock_and_start_stmt on CREATE OR REPLACE TABLE

Fixed by removing wrong assert

Review: Sanja Byelkin
Sergei Golubchik
MDEV-40571 insufficient validation of frm data when opening a table

numerous checks that the frm is valid, no OOB reads,
values make sense (number of keyparts not less than number of keys,
no keys means no keyparts, number of long unique fields is not larger than
number of fields, fields values in the record don't overlap and don't
go over record ends, and so on). most asserts were changed to if()'s
ParadoxV5
MDEV-39788: Remove added line in `master.info` format

The line-count lines in `master.info` and `relay-log.info`
have been inconsistent (off by one) since their introduction.
MDEV-37530 “fixed” this with its common code merger by chance,
changing `master.info` to use `relay-log.info`’s line-count definition.
This change, howëver, affected backward compatibility,
as `master.info` now expects an ignored MySQL-only line
where the first `key=value` option, `master_use_gtid`, is.

Since this legacy text-based format has limitations that make
it due for replacement, only code reüsablility is valuable,
and its consistency does not outweigh its compatibility.
Therefore, this commit solves this problem without reverting code by:
* Changing the writing code to be compatible with both interpretations
  (albeit inconsistent with the reading code)
* Adding a shim entry to `master.info`’s list
  to emulate prior versions’ reading behaviour
  * Although this solution can only restore upgrade compatibility with
    versions 10.0+, versions before MariaDB 10 have long been EOL.

While here, this commit also fixes code and
comments that contradict the actual effect.

[P.S.] The test for this regression is pushed to 10.11 in PR #5147.

Reviewed-by: Brandon Nesterenko <[email protected]>
Sergei Golubchik
cleanup: sys_vars.secure_file_priv test
Sergei Golubchik
fixup! cleanup: sys_vars.secure_file_priv test
Monty
Added --debug-dbug option to mysqltest.cc

This was to get rid of warnings when using mtr --debug
Oleg Smirnov
MDEV-39491 Parallel Query: cache the chunk-boundary verdict per leaf page

The chunk-boundary clamp added with the InnoDB parallel scan compares every
record returned by row_search_mvcc() against the chunk's exclusive upper
bound. That is one cmp_dtuple_rec() per fetched row, wasted on every row but
the last one of the chunk.

Records on a leaf page are sorted, so the boundary can only fall inside the
page whose last user record is >= the boundary. On any earlier page all
records are in range and no per-record comparison is needed. Take the verdict
once per leaf page by comparing the end tuple against the page's last user
record, and cache it in prebuilt.

- row_pscan_reached_chunk_end() (row0sel.cc): per-page verdict plus the
  per-record fallback when the boundary does lie on the current page.
- row_prebuilt_t::m_pscan_clamp_{page,clock,in_range}: the cached verdict,
  keyed by page number and modify_clock, so a page that was split, merged or
  reorganized in the meantime is re-examined.
- row_prebuilt_t::set_pscan_end_tuple() replaces direct assignment of
  m_pscan_end_tuple, so switching to the next chunk cannot leave a stale
  verdict behind.

The shortcut is confined to consistent reads of records living on the page
the cursor is positioned on: a record read at READ UNCOMMITTED may have been
inserted onto the page after the verdict was taken (a plain insert does not
bump modify_clock), and an old version built in the heap is not covered by
any page's verdict. Those cases fall back to the per-record comparison.
Vladislav Vaintroub
Support multi-factor authentication MySQL way

- Allow to specify password2/password3 via
mysql_optionsv(mysql,MYSQL_OPT_USER_PASSWORD, factor,N)

- Handle 0x2 (AuthNextFactor) server packet
Switch password according to factor

- make sure TLS "trust" for self-signed certificate works
if any of the MFA factors is password-based (e.g
gssapi + mysql_native_password would use second factor's
verification)
Sergei Golubchik
MDEV-40571 insufficient validation of frm data when opening a table

numerous checks that the frm is valid, no OOB reads,
values make sense (number of keyparts not less than number of keys,
no keys means no keyparts, number of long unique fields is not larger than
number of fields, fields values in the record don't overlap and don't
go over record ends, and so on). most asserts were changed to if()'s
Sergei Golubchik
cleanup: sys_vars.secure_file_priv test
Sergei Golubchik
MDEV-40589 default exclude list for secure-file-priv

don't allow to access /proc if secure-file-priv="",
set secure-file-priv=/ to access everything and disable the exclude list

remove test for a conditon that can no longer happen
Sergei Golubchik
MDEV-40589 default exclude list for secure-file-priv

don't allow to access /proc if secure-file-priv="",
set secure-file-priv=/ to access everything and disable the exclude list

remove test for a conditon that can no longer happen
Sergei Golubchik
MDEV-40571 insufficient validation of frm data when opening a table

numerous checks that the frm is valid, no OOB reads,
values make sense (number of keyparts not less than number of keys,
no keys means no keyparts, number of long unique fields is not larger than
number of fields, fields values in the record don't overlap and don't
go over record ends, and so on). most asserts were changed to if()'s
Sergei Golubchik
MDEV-40589 default exclude list for secure-file-priv

don't allow to access /proc if secure-file-priv="",
set secure-file-priv=/ to access everything and disable the exclude list

remove test for a conditon that can no longer happen
Alexey (Holyfoot) Botchkov
MDEV-39654 No warning after CAST( AS xmltype).

Warning added.
Marko Mäkelä
MDEV-40596 clang-23 reports unused global variables

Let us remove a number of unused variables to suppress
-Wunused-but-set-global and other warnings.

test_thread(): Instead of incrementing a global counter in a race
condition prone fashion, invoke MY_RELAX_CPU() in order to spend some time.
Arcadiy Ivanov
MDEV-40591 Unexpected ER_NOT_KEYFILE or MSAN error in heap_check_heap

`ha_heap::external_lock()` verifies the table with `heap_check_heap()` at
`F_UNLCK`.  That is safe on the ordinary unlock path, where
`mysql_unlock_tables()` calls `unlock_external()` before `thr_multi_unlock()`
and the THR_LOCK is still held.  It is not safe on either path that unlocks
after a *failed* lock attempt, where the caller holds nothing at all while
another connection is writing:

1. `mysql_lock_tables()` calls `unlock_external()` to balance the external
  locks it already took, because `thr_multi_lock()` timed out.
2. `lock_external()` unwinds the tables it has already locked, because a
  later table refused -- all before `thr_multi_lock()` runs at all.
  `ha_partition::external_lock()` unwinds its partitions the same way.

MEMORY has no row-level concurrency control, so a scan taken outside the
THR_LOCK sees a writer's intermediate state by construction:
`hp_alloc_from_tail()` publishes `total_records` at allocation time, before
the slot is written, while the checker scans `[0, total_records + deleted)`
and reads every slot's flags byte.  Under MSAN that is a use of
uninitialised `my_malloc()` memory; otherwise it is a spurious
`total_records` mismatch.

`heap_check_heap()` ends with `heap_mark_crashed()`, which sets
`HEAP_STATE_CRASHED` in the **shared** `HP_SHARE`, so one bogus mid-write
observation poisons a healthy table for every connection using it -- the
reported `ER_NOT_KEYFILE`.

MDEV-21373 disabled this check in 2021 for exactly this reason, by gating it
on `EXTRA_DEBUG`.  MDEV-38975 changed the gate to `EXTRA_HEAP_DEBUG` and
defined that for every debug build, reviving the race.

Rather than switch the check off wholesale again, ask whether the handle
actually holds the lock.  The requested lock type cannot answer that on its
own, because `ha_heap::store_lock()` records it at `get_lock_data()` time,
before anything is locked: on the second path it is set while nothing is
held.  So HEAP now records the grant itself:

- `hp_lock_granted()`, registered as the `THR_LOCK` `get_status` callback,
  sets `HP_INFO::lock_granted` when `thr_lock()` gives the lock to the
  handle;
- `hp_lock_request_begin()` clears it from `ha_heap::external_lock()`, which
  the SQL layer always reaches just before it tries to take the THR_LOCK;
- `hp_lock_is_held()` requires both the grant and a lock type that
  `thr_unlock()` has not reset, the latter covering the lock that
  `thr_multi_lock()` takes and then rolls back when a later table times out.

Deriving this in the engine rather than repairing `lock_external()` also
covers `ha_partition`, which reimplements the same unwind.

`hp_may_check_heap_on_unlock()` gates the verification on that, and the
redemption of parked blob chains -- which puts records back on the shared
free list -- is gated on it too.

`ha_heap::reset()` gains an assertion that nothing is parked without the
lock.  It cannot be violated: chains are parked only by `heap_delete()` and
`heap_update()`, only for a table that is not internal, and always under the
lock; they are redeemed before it is released.

`hp_test_unlock_check-t` reproduces all of this deterministically, by driving
`thr_multi_lock()`/`thr_multi_unlock()` directly instead of racing.
Sergei Golubchik
MDEV-40571 insufficient validation of frm data when opening a table

numerous checks that the frm is valid, no OOB reads,
values make sense (number of keyparts not less than number of keys,
no keys means no keyparts, number of long unique fields is not larger than
number of fields, fields values in the record don't overlap and don't
go over record ends, and so on). most asserts were changed to if()'s
Monty
MDEV-25292 Atomic CREATE OR REPLACE TABLE

Atomic CREATE OR REPLACE allows to keep an old table intact if the
command fails or during the crash. That is done by renaming the
original table to temporary name, as a backup and restoring it if the
CREATE fails. When the command is complete and logged the backup
table is deleted.

Atomic replace algorithm

  Two DDL chains are used for CREATE OR REPLACE:
  ddl_log_state_create (C) and ddl_log_state_rm (D).

  1. (C) Log rename of ORIG to TMP table (Rename TMP to original).
  2. Rename orignal to TMP.
  3. (C) Log CREATE_TABLE_ACTION of ORIG (drops ORIG);
  4. Do everything with ORIG (like insert data)
  5. (D) Log drop of TMP
  6. Write query to binlog (this marks (C) to be closed in
    case of failure)
  7. Execute drop of TMP through (D)
  8. Close (C) and (D)

  If there is a failure before 6) we revert the changes in (C)
  Chain (D) is only executed if 6) succeded (C is closed on
  crash recovery).

Foreign key errors will be found at the 1) stage.

Additional notes

  - CREATE TABLE without REPLACE and temporary tables is not affected
    by this commit.
    set @@drop_before_create_or_replace=1 can be used to
    get old behaviour where existing tables are dropped
    in CREATE OR REPLACE.

  - CREATE TABLE is reverted if binlogging the query fails.

  - Engines having HTON_EXPENSIVE_RENAME flag set are not affected by
    this commit. Conflicting tables marked with this flag will be
    deleted with CREATE OR REPLACE.

  - Replication execution is not affected by this commit.
    - Replication will first drop the conflicting table and then
      creating the new one.

  - CREATE TABLE .. SELECT XID usage is fixed and now there is no need
    to log DROP TABLE via DDL_CREATE_TABLE_PHASE_LOG (see comments in
    do_postlock()). XID is now correctly updated so it disables
    DDL_LOG_DROP_TABLE_ACTION. Note that binary log is flushed at the
    final stage when the table is ready. So if we have XID in the
    binary log we don't need to drop the table.

  - Three variations of CREATE OR REPLACE handled:

    1. CREATE OR REPLACE TABLE t1 (..);
    2. CREATE OR REPLACE TABLE t1 LIKE t2;
    3. CREATE OR REPLACE TABLE t1 SELECT ..;

  - Test case uses 6 combinations for engines (aria, aria_notrans,
    myisam, ib, lock_tables, expensive_rename) and 2 combinations for
    binlog types (row, stmt). Combinations help to check differences
    between the results. Error failures are tested for the above three
    variations.

  - expensive_rename tests CREATE OR REPLACE without atomic
    replace. The effect should be the same as with the old behaviour
    before this commit.

  - Triggers mechanism is unaffected by this change. This is tested in
    create_replace.test.

  - LOCK TABLES is affected. Lock restoration must be done after new
    table is created or TMP is renamed back to ORIG

  - Moved ddl_log_complete() from send_eof() to finalize_ddl(). This
    checkpoint was not executed before for normal CREATE TABLE but is
    executed now.

  - CREATE TABLE will now rollback also if writing to the binary
    logging failed. See rpl_gtid_strict.test

backup ddl log changes

- In case of a successfull CREATE OR REPLACE we only log
  the CREATE event, not the DROP TABLE event of the old table.

ddl_log.cc changes

  ddl_log_execute_action() now properly return error conditions.
  ddl_log_disable_entry() added to allow one to disable one entry.
  The entry on disk is still reserved until ddl_log_complete() is
  executed.

On XID usage

  Like with all other atomic DDL operations XID is used to avoid
  inconsistency between master and slave in the case of a crash after
  binary log is written and before ddl_log_state_create is closed. On
  recovery XIDs are taken from binary log and corresponding DDL log
  events get disabled.  That is done by
  ddl_log_close_binlogged_events().

On linking two chains together

  Chains are executed in the ascending order of entry_pos of execute
  entries. But entry_pos assignment order is undefined: it may assign
  bigger number for the first chain and then smaller number for the
  second chain. So the execution order in that case will be reverse:
  second chain will be executed first.

  To avoid that we link one chain to another. While the base chain
  (ddl_log_state_create) is active the secondary chain
  (ddl_log_state_rm) is not executed. That is: only one chain can be
  executed in two linked chains.

  The interface ddl_log_link_chains() was defined in "MDEV-22166
  ddl_log_write_execute_entry() extension".

Atomic info parameters in HA_CREATE_INFO

  Many functions in CREATE TABLE pass the same parameters. These
  parameters are part of table creation info and should be in
  HA_CREATE_INFO (or whatever). Passing parameters via single
  structure is much easier for adding new data and
  refactoring.

InnoDB changes
  Added ha_innobase::can_be_renamed_to_backup() to check if
  a table with foreign keys can be renamed.

Aria changes:
- Fixed issue in Aria engine with CREATE + locked tables
  that data was not properly commited in some cases in
  case of crashes.

Other changes:
- Removed some auto variables in log.cc for better code readability.
- Fixed old bug that CREATE ... SELECT would not be able to auto repair
  a table that is part of the SELECT.
- Marked MyISAM that it does not support ROLLBACK (not required but
  done for better consistency with other engines).

Known issues:
- InnoDB tables with foreign key definitions are not fully supported
  with atomic create and replace:
  - ha_innobase::can_be_renamed_to_backup() can detect some cases
    where InnoDB does not support renaming table with foreign key
    constraints.  In this case MariaDB will drop the old table before
    creating the new one.
    The detected cases are:
    - The new and old table is using the same foreign key constraint
      name.
    - The old table has self referencing constraints.
  - If the old and new table uses the same name for a constraint the
    create of the new table will fail. The orignal table will be
    restored in this case.
  - The above issues will be fixed in a future commit.
- CREATE OR REPLACE TEMPORARY table is not full atomic. Any conflicting
  table will always be dropped before creating a new one. (Old behaviour).

Bug fixes related to this MDEV:

MDEV-36435 Assertion failure in finalize_locked_tables()
MDEV-36439 Assertion `thd_arg->lex->sql_command != SQLCOM_CREATE_SEQUENCE...
MDEV-36498 Failed CoR in non-atomic mode no longer generates DROP in RBR...
MDEV-36508 Temporary files #sql-create-....frm occasionally stay after
          crash recovery
MDEV-38479 Crash in CREATE OR REPLACE SEQUENCE when new sequence cannot
          be created
MDEV-36497 Assertion failure after atomic CoR with Aria under lock in
          transactional context
MDEV-36501 EITS data is lost after failed attempt to CREATE OR REPLACE
          table
MDEV-36493 Atomic CREATE OR REPLACE ... SELECT blocks InnoDB purge
MDEV-39367 MSAN/valgrind errors in temp_file_size_cb_func,
          main.tmp_space_usage fails
MDEV-39446 Atomic CREATE OR REPLACE fails if a table cannot be decrypted

InnoDB related changes:
- ha_innodb::rename_table() does not handle foreign key constraint
  when renaming an normal table to internal tempory tables. This
  causes problems for CREATE OR REPLACE as the old constraints causes
  failure when creating a new table with the same constraints.
  This is fixed inside InnoDB by not threating tempfiles (#sql-create-..),
  created as part of CREATE OR REPLACE, as temporary files.
- In ha_innobase::delete_table(), ignore checking of constraints when
  dropping a #sql-create temporary table.
- In tablename_to_filename() and filename_to_tablename(), don't do
  filename conversion for internal temporary tables (#sql-...)

Other things:
- maria_create_trn_for_mysql() does not register a new transaction
  handler for commits. This was needed to ensure create or replace
  will not end with an active transaction.
- We do not get anymore warnings about "Engine not supporting atomic
  create" when doing a legal CREATE OR REPLACE on a table with
  foreign key constraints.
- Updated VIDEX engine flags to disable CREATE SEQUENCE.

Reverted commits:
MDEV-36685 "CREATE-SELECT may lose in binlog side-effects of
stored-routine" as it did not take into account that it safe to clear
binlogs if the created table is non transactional and there are no
other non transactional tables used.
- This was done because it caused extra logging when it is not needed
  (not using any non transactional tables) and it also did not solve
  side effects when using statement based loggging.

Other things:
- EITS data is preserved if create or replace fails if
  drop_before_create_or_replace=OFF. If ON, then create or replace
  will drop EITS before the drop of the original table (as before).
- Using CREATE OR REPLACE on a encrypted table that the user cannot
  decrypt will fail instead of replacing the encrypted table.
  The encrypted table will unchanged.
forkfun
Merge branch '11.8' into '12.3'
Sergei Golubchik
MDEV-40571 insufficient validation of frm data when opening a table

numerous checks that the frm is valid, no OOB reads,
values make sense (number of keyparts not less than number of keys,
no keys means no keyparts, number of long unique fields is not larger than
number of fields, fields values in the record don't overlap and don't
go over record ends, and so on). most asserts were changed to if()'s
Oleg Smirnov
MDEV-39491 Parallel Query: InnoDB clustered-index partitioning for parallel scan

Implement the InnoDB side of the handler parallel-scan API: partition the
clustered index into disjoint key-range chunks and serve them to the SQL layer
as pull-based scan jobs. Each chunk is read through the normal
row_search_mvcc() path, so MVCC visibility, AHI and the prefetch cache keep
working unchanged. InnoDB spawns no threads of its own.

- Parallel_coordinator (row0pcoord.{h,cc}): adapted from MySQL's
  Parallel_reader, reduced to partitioning and job distribution. Walks the
  index top-down into Exec_ctx chunks bounded by clustered-key tuples.
- ha_innobase: implements parallel_{init,end}_coordinator,
  parallel_get_worker_context, parallel_{init,end}_worker and
  parallel_get_next_row.
- Chunk-boundary clamp: row_prebuilt_t::m_pscan_end_tuple (NULL = unbounded)
  makes row_search_mvcc() stop before prefetching past the chunk's exclusive
  upper bound, treated as end-of-range so the next chunk is pulled.
- btr_pcur_open_on_user_rec(page_cur_t) overload to anchor chunk boundaries.
- Build: row0pcoord.cc added to CMakeLists.txt and auto_event_names[].
Alexey (Holyfoot) Botchkov
MDEV-39683 Numeric aggregates should end up with an error for xmltype.

Appropriate xxx_fix_length_and_dec() added to the Type_handler_xmltype.
Alexey Botchkov
MDEV-39750 ExtractValue does not control recursion depth.

Stack exhaustive test shouldn't be ran with the ASAN/UBSAN.
Sergei Golubchik
MDEV-40589 default exclude list for secure-file-priv

don't allow to access /proc if secure-file-priv="",
set secure-file-priv=/ to access everything and disable the exclude list
Sergei Golubchik
cleanup: sys_vars.secure_file_priv test
forkfun
Merge branch '12.3' into '13.0'
Sergei Golubchik
MDEV-40589 default exclude list for secure-file-priv

don't allow to access /proc if secure-file-priv="",
set secure-file-priv=/ to access everything and disable the exclude list
forkfun
Merge branch '10.11' into '11.4'
Daniel Black
MDEV-39113 MSAN/ADDR addr2line stack resolver detrimental

MSAN/ASAN test environment, the addr2line was so high in memory
utilization that it was the pick of the OOM killer to resolve the OOM
situation. Once this occurred there wasn't a saved core or gdb backtrace
of the core to resolve the issue.

To resolve this, make stack-trace default to 0 (off) for the addr2line
base implementation under memory sanitizer and address sanitizer.

MariaDB-backup also forces the enabling of stack-trace. Disabling this
unconditionally reduces the risk of a user operational impact if a
lengthy stack trace starting in a mariadb-backup critical locked period.

The mysqld--help test now excludes the stack-trace as its result is
environment dependant. The "Defaults to..." output for suppressed
variables, currently only stack-trace, is excluded.

Since thread-stack is an excluded varable, the ubsan/asan exclusions
from commits dfa6fba9595a and dfa6fba9595a aren't required.
Monty
ha_table_exists() cleanup and improvement

This is part of MDEV-25292 Atomic CREATE OR REPLACE TABLE.

Removed default values for arguments, added flags argument to specify
filename flags (FN_TO_IS_TMP, FN_FROM_IS_TMP) and forward the flag to
build_table_name().

Original patch from: Aleksey Midenkov <[email protected]>
Oleg Smirnov
WIP: Clamp InnoDB pages