Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Yuchen Pei
MDEV-24813 [wip] Add an innodb switch table_lock_on_full_scan

And check LIMIT is ULONG_MAX (likely no LIMIT).

Updated tests that have different results. Some still failing.
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Monty
MDEV-38865 Simplify testing if binary logging is enabled

In the MariaDB server most of the code uses different expressions to
test if binary logging should be done for the current query.

In many cases the current code does a lot of not needed work before
noticing that binary logging is not needed, which affects performance.
For example, in some cases early decision for preparing to do binary
logging was based on current_stmt_binlog_format (which is by default
statement) without checking if binary logging was enable.d

There is also a lot of different variables that affects if binary logging
should be used:
- (thd->variables.option_bits & OPTION_BIN_LOG)
- (thd->variables.option_bits & OPTION_BIN_TMP_LOG_OFF)
- WSREP(thd) && wsrep_emulate_bin_log && WSREP_PROVIDER_EXISTS_
- mysql_bin_log.is_open()
- xxx_binlog_local_stmt_filter()
- binlog_filter
- thd->variables.sql_log_bin
- thd->binlog_evt_union.do_union

The goal is to move all states to a single variable (for performance,
simplicity and easier debugging). We should ignore all possible
extra work if binary logging is not needed for a particular query.

In most cases we can now use one the following functions to check if
we binary logging is needed.  We need different version to be able
to shortcut code if binary logging is not enabled (with different
code paths if Galera is enabled or not).

- binlog_ready()
- binlog_ready_with_wsrep()
- binlog_ready_no_wsrep()
- binlog_ready_precheck()
- binlog_ready_later()

The state of binary logging is stored in thd->binlog_state.
There are a few bits that shows why binary logging is needed and
one bit for every different reasons for disabling binary logging.
By printing this variable (gdb prints all bits with there enum name)
one can exactly see why binary logging is not happening.

The initial bits are set in THD::THD() and THD::init() and verified in
THD::decide_logging_format().

In this commit all testing of OPTION_BIN_LOG and most testing of
mysql_bin_log.is_open() is removed.
We should over time remove all testing of mysql_bin_log.is_open() from the
main code, except in binlog commit, as the value can change 'anytime' if
binlog rotation fails, which is very likely to result in crashes.
(The patch fixes some of these cases).

THD::binlog_state is a new variable that simplifies testing if replication
is permanently off, is on or temporarly off (for a sub statement).

We also set current_stmt_binlog_format to BINLOG_FORMAT_UNSPEC if binlog is
off.

The above changes allows some code changes:

One does not need to test for mysql_bin_log.is_open() if one also tests
for thd->is_current_stmt_binlog_format_stmt() or
thd->is_current_stmt_binlog_format_row().

The change also allows the following transformations:

(WSREP(thd) && wsrep_emulate_bin_log) ||
(mysql_bin_log.is_open()) && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready()

(WSREP_NNULL(thd) && wsrep_emulate_bin_log) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

(WSREP_EMULATE_BINLOG(thd) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

mysql_bin_log.is_open() && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready_no_wsrep()

Other transformation are easy to do by using the bits set in binlog_state
set by decide_binlog_format()

Other things:
- Implement THD::tmp_disable_binlog() and THD::reenable_binlog() using
  the new binlog_state framework.
  The old code had no effect for row logging, which spider and mroonga
  assumed. Now handler::binlog_log_row() disables row logging properly if
  tmp_disable_binlog() is used
- THD::variables.wsrep_on is now updated by THD:enable_wsrep(),
  THD::disable_wsrep() or THD::set_wsrep(). I added an assert for
  (WSREP_PROVIDER_EXISTS_ && WSREP_ON_) when setting wsrep_on=1
- Reset sql_log_bin in THD::init(). This is to ensure that change user
  will not get affected by the previous users sql_log_bin state.
- Replaced OPTION_BIN_TMP_LOG_OFF with
  (thd->binlog_state & BINLOG_STATE_TMP_DISABLED)
- Added TABLE->disabled_rowlogging() and TABLE->enable_rowlogging() to
  be used when one want to only disable logging for single write, update
  or delete call.

Things to do:
- We should consider removing OPTION_BIN_LOG and instead use
  binlog_state. This would allow us to remove reset and setting it in
  decide_binlog_format(). The code is not anymore using the value of
  OPTION_BIN_LOG anywhere.
- Update binlog_state with BINLOG_STATE_FILTER only when database or filter
  is changed.  This means we do not have to call binlog_filter->db_ok() for
  every statement in decide_binlog_format().
- Remove clear/reset xxx_binlog_local_stmt_filter() as this is now handled
  by binlog_state.
- Remove 'silent' option from mysql_create_db_internal and instead use
  tmp_disable_binlog() / renenable_binlog()
- Remove testing if binary log is desabled in:
  MYSQL_BIN_LOG::write(Log_event *event_info, my_bool *with_annotate):
    if ((!(option_bin_log_flag)) ||
        (thd->lex->sql_command != SQLCOM_ROLLBACK_TO_SAVEPOINT &&
        thd->lex->sql_command != SQLCOM_SAVEPOINT &&
        !binlog_filter->db_ok(local_db)))
      DBUG_RETURN(0);
- Remove all testing of mysql_bin_log.is_open(). Instead test for
  binlog_ready() in main code and add testing of is_open() when
  trying to commit the binary log.  This is needed as currently
  mysql_bin_log.is_open() is tested without a mutex which makes
  it unreliable.
- Remove testing of WSREP_PROVIDER_EXISTS_ in WSREP_NULL() as this
  is guaranteed by THD::enable_wsrep()
- BINLOG_STATE_FILTER need a bit more work. It is currently only used
  in decide_binlog_format()
- Remove BINLOG_STATE_BYPASS (Not needed as some other BINLOG_STATE disable
  bit is set if BYPASS is set).  BYPASS was added mostly to simplify
  testing of the new code.
Yuchen Pei
MDEV-24813 [wip] Add an innodb switch table_lock_on_full_scan

And check LIMIT is ULONG_MAX (likely no LIMIT).

Updated tests that have different results. Tests still failing:

innodb.deadlock_victim_race 'table_lock'
innodb.deadlock_wait_lock_race 'table_lock'
innodb.deadlock_wait_thr_race 'table_lock'
Monty
Remove for (auto []) constructs from rpl_master_info.h

There is a lot of more work needed to make this code compatitble with
the rest of the MariaDB server.

- All std:: usage should be removed
- Most templatest should be removed
- Usage of <unordered_map>, <string_view>, <optional>, <unordered_set>
  should be removed and replaced with my_sys, sql_string functions

It would be nice of one could also have default values for all the
CHANGE MASTER variables in my.cnf.  This would allow one to have
much shorter CHANGE MASTER commands
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
Monty
MDEV-38865 Simplify testing if binary logging is enabled

In the MariaDB server most of the code uses different expressions to
test if binary logging should be done for the current query.

In many cases the current code does a lot of not needed work before
noticing that binary logging is not needed, which affects performance.
For example, in some cases early decision for preparing to do binary
logging was based on current_stmt_binlog_format (which is by default
statement) without checking if binary logging was enable.d

There is also a lot of different variables that affects if binary logging
should be used:
- (thd->variables.option_bits & OPTION_BIN_LOG)
- (thd->variables.option_bits & OPTION_BIN_TMP_LOG_OFF)
- WSREP(thd) && wsrep_emulate_bin_log && WSREP_PROVIDER_EXISTS_
- mysql_bin_log.is_open()
- xxx_binlog_local_stmt_filter()
- binlog_filter
- thd->variables.sql_log_bin
- thd->binlog_evt_union.do_union

The goal is to move all states to a single variable (for performance,
simplicity and easier debugging). We should ignore all possible
extra work if binary logging is not needed for a particular query.

In most cases we can now use one the following functions to check if
we binary logging is needed.  We need different version to be able
to shortcut code if binary logging is not enabled (with different
code paths if Galera is enabled or not).

- binlog_ready()
- binlog_ready_with_wsrep()
- binlog_ready_no_wsrep()
- binlog_ready_precheck()
- binlog_ready_later()

The state of binary logging is stored in thd->binlog_state.
There are a few bits that shows why binary logging is needed and
one bit for every different reasons for disabling binary logging.
By printing this variable (gdb prints all bits with there enum name)
one can exactly see why binary logging is not happening.

The initial bits are set in THD::THD() and THD::init() and verified in
THD::decide_logging_format().

In this commit all testing of OPTION_BIN_LOG and most testing of
mysql_bin_log.is_open() is removed.
We should over time remove all testing of mysql_bin_log.is_open() from the
main code, except in binlog commit, as the value can change 'anytime' if
binlog rotation fails, which is very likely to result in crashes.
(The patch fixes some of these cases).

THD::binlog_state is a new variable that simplifies testing if replication
is permanently off, is on or temporarly off (for a sub statement).

We also set current_stmt_binlog_format to BINLOG_FORMAT_UNSPEC if binlog is
off.

The above changes allows some code changes:

One does not need to test for mysql_bin_log.is_open() if one also tests
for thd->is_current_stmt_binlog_format_stmt() or
thd->is_current_stmt_binlog_format_row().

The change also allows the following transformations:

(WSREP(thd) && wsrep_emulate_bin_log) ||
(mysql_bin_log.is_open()) && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready()

(WSREP_NNULL(thd) && wsrep_emulate_bin_log) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

(WSREP_EMULATE_BINLOG(thd) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

mysql_bin_log.is_open() && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready_no_wsrep()

Other transformation are easy to do by using the bits set in binlog_state
set by decide_binlog_format()

Other things:
- Implement THD::tmp_disable_binlog() and THD::reenable_binlog() using
  the new binlog_state framework.
  The old code had no effect for row logging, which spider and mroonga
  assumed. Now handler::binlog_log_row() disables row logging properly if
  tmp_disable_binlog() is used
- THD::variables.wsrep_on is now updated by THD:enable_wsrep(),
  THD::disable_wsrep() or THD::set_wsrep(). I added an assert for
  (WSREP_PROVIDER_EXISTS_ && WSREP_ON_) when setting wsrep_on=1
- Reset sql_log_bin in THD::init(). This is to ensure that change user
  will not get affected by the previous users sql_log_bin state.
- Replaced OPTION_BIN_TMP_LOG_OFF with
  (thd->binlog_state & BINLOG_STATE_TMP_DISABLED)
- Added TABLE->disabled_rowlogging() and TABLE->enable_rowlogging() to
  be used when one want to only disable logging for single write, update
  or delete call.

Things to do:
- We should consider removing OPTION_BIN_LOG and instead use
  binlog_state. This would allow us to remove reset and setting it in
  decide_binlog_format(). The code is not anymore using the value of
  OPTION_BIN_LOG anywhere.
- Update binlog_state with BINLOG_STATE_FILTER only when database or filter
  is changed.  This means we do not have to call binlog_filter->db_ok() for
  every statement in decide_binlog_format().
- Remove clear/reset xxx_binlog_local_stmt_filter() as this is now handled
  by binlog_state.
- Remove 'silent' option from mysql_create_db_internal and instead use
  tmp_disable_binlog() / renenable_binlog()
- Remove testing if binary log is desabled in:
  MYSQL_BIN_LOG::write(Log_event *event_info, my_bool *with_annotate):
    if ((!(option_bin_log_flag)) ||
        (thd->lex->sql_command != SQLCOM_ROLLBACK_TO_SAVEPOINT &&
        thd->lex->sql_command != SQLCOM_SAVEPOINT &&
        !binlog_filter->db_ok(local_db)))
      DBUG_RETURN(0);
- Remove all testing of mysql_bin_log.is_open(). Instead test for
  binlog_ready() in main code and add testing of is_open() when
  trying to commit the binary log.  This is needed as currently
  mysql_bin_log.is_open() is tested without a mutex which makes
  it unreliable.
- Remove testing of WSREP_PROVIDER_EXISTS_ in WSREP_NULL() as this
  is guaranteed by THD::enable_wsrep()
- BINLOG_STATE_FILTER need a bit more work. It is currently only used
  in decide_binlog_format()
- Remove BINLOG_STATE_BYPASS (Not needed as some other BINLOG_STATE disable
  bit is set if BYPASS is set).  BYPASS was added mostly to simplify
  testing of the new code.
Oleksandr Byelkin
first pass of bootstrap
Oleg Smirnov
MDEV-38045 Expand warnings text, fix review comments
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Yuchen Pei
MDEV-24813 [wip] Add an innodb switch table_lock_on_full_scan

And check LIMIT is ULONG_MAX (likely no LIMIT).

Updated tests that have different results. Some still failing.
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Monty
Fixing Galera test results
Raghunandan Bhat
MDEV-38864: Use mmap for MEMORY engine allocations

MEMORY engine's data and indexes blocks are allocated using `my_malloc`.
This created internal fragmentation within the system allocator, causing
gradual memory growth and leading to OOM server crash.

This patch introduces memory mapping for MEMORY engine's allocations.
- Implements `my_vmalloc` and `my_vmrelease` functions.
- These functions wrap `my_virtual_mem_alloc` & `my_virtual_mem_release`
  to provide error handling, interface with Performance Schema(PSI) and
  global memory accounting.
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Monty
MDEV-38865 Simplify testing if binary logging is enabled

In the MariaDB server most of the code uses different expressions to
test if binary logging should be done for the current query.

In many cases the current code does a lot of not needed work before
noticing that binary logging is not needed, which affects performance.
For example, in some cases early decision for preparing to do binary
logging was based on current_stmt_binlog_format (which is by default
statement) without checking if binary logging was enable.d

There is also a lot of different variables that affects if binary logging
should be used:
- (thd->variables.option_bits & OPTION_BIN_LOG)
- (thd->variables.option_bits & OPTION_BIN_TMP_LOG_OFF)
- WSREP(thd) && wsrep_emulate_bin_log && WSREP_PROVIDER_EXISTS_
- mysql_bin_log.is_open()
- xxx_binlog_local_stmt_filter()
- binlog_filter
- thd->variables.sql_log_bin
- thd->binlog_evt_union.do_union

The goal is to move all states to a single variable (for performance,
simplicity and easier debugging). We should ignore all possible
extra work if binary logging is not needed for a particular query.

In most cases we can now use one the following functions to check if
we binary logging is needed.  We need different version to be able
to shortcut code if binary logging is not enabled (with different
code paths if Galera is enabled or not).

- binlog_ready()
- binlog_ready_with_wsrep()
- binlog_ready_no_wsrep()
- binlog_ready_precheck()
- binlog_ready_later()

The state of binary logging is stored in thd->binlog_state.
There are a few bits that shows why binary logging is needed and
one bit for every different reasons for disabling binary logging.
By printing this variable (gdb prints all bits with there enum name)
one can exactly see why binary logging is not happening.

The initial bits are set in THD::THD() and THD::init() and verified in
THD::decide_logging_format().

In this commit all testing of OPTION_BIN_LOG and most testing of
mysql_bin_log.is_open() is removed.
We should over time remove all testing of mysql_bin_log.is_open() from the
main code, except in binlog commit, as the value can change 'anytime' if
binlog rotation fails, which is very likely to result in crashes.
(The patch fixes some of these cases).

THD::binlog_state is a new variable that simplifies testing if replication
is permanently off, is on or temporarly off (for a sub statement).

We also set current_stmt_binlog_format to BINLOG_FORMAT_UNSPEC if binlog is
off.

The above changes allows some code changes:

One does not need to test for mysql_bin_log.is_open() if one also tests
for thd->is_current_stmt_binlog_format_stmt() or
thd->is_current_stmt_binlog_format_row().

The change also allows the following transformations:

(WSREP(thd) && wsrep_emulate_bin_log) ||
(mysql_bin_log.is_open()) && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready()

(WSREP_NNULL(thd) && wsrep_emulate_bin_log) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

(WSREP_EMULATE_BINLOG(thd) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

mysql_bin_log.is_open() && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready_no_wsrep()

Other transformation are easy to do by using the bits set in binlog_state
set by decide_binlog_format()

Other things:
- Implement THD::tmp_disable_binlog() and THD::reenable_binlog() using
  the new binlog_state framework.
  The old code had no effect for row logging, which spider and mroonga
  assumed. Now handler::binlog_log_row() disables row logging properly if
  tmp_disable_binlog() is used
- THD::variables.wsrep_on is now updated by THD:enable_wsrep(),
  THD::disable_wsrep() or THD::set_wsrep(). I added an assert for
  (WSREP_PROVIDER_EXISTS_ && WSREP_ON_) when setting wsrep_on=1
- Reset sql_log_bin in THD::init(). This is to ensure that change user
  will not get affected by the previous users sql_log_bin state.
- Replaced OPTION_BIN_TMP_LOG_OFF with
  (thd->binlog_state & BINLOG_STATE_TMP_DISABLED)
- Added TABLE->disabled_rowlogging() and TABLE->enable_rowlogging() to
  be used when one want to only disable logging for single write, update
  or delete call.

Things to do:
- We should consider removing OPTION_BIN_LOG and instead use
  binlog_state. This would allow us to remove reset and setting it in
  decide_binlog_format(). The code is not anymore using the value of
  OPTION_BIN_LOG anywhere.
- Update binlog_state with BINLOG_STATE_FILTER only when database or filter
  is changed.  This means we do not have to call binlog_filter->db_ok() for
  every statement in decide_binlog_format().
- Remove clear/reset xxx_binlog_local_stmt_filter() as this is now handled
  by binlog_state.
- Remove 'silent' option from mysql_create_db_internal and instead use
  tmp_disable_binlog() / renenable_binlog()
- Remove testing if binary log is desabled in:
  MYSQL_BIN_LOG::write(Log_event *event_info, my_bool *with_annotate):
    if ((!(option_bin_log_flag)) ||
        (thd->lex->sql_command != SQLCOM_ROLLBACK_TO_SAVEPOINT &&
        thd->lex->sql_command != SQLCOM_SAVEPOINT &&
        !binlog_filter->db_ok(local_db)))
      DBUG_RETURN(0);
- Remove all testing of mysql_bin_log.is_open(). Instead test for
  binlog_ready() in main code and add testing of is_open() when
  trying to commit the binary log.  This is needed as currently
  mysql_bin_log.is_open() is tested without a mutex which makes
  it unreliable.
- Remove testing of WSREP_PROVIDER_EXISTS_ in WSREP_NULL() as this
  is guaranteed by THD::enable_wsrep()
- BINLOG_STATE_FILTER need a bit more work. It is currently only used
  in decide_binlog_format()
- Remove BINLOG_STATE_BYPASS (Not needed as some other BINLOG_STATE disable
  bit is set if BYPASS is set).  BYPASS was added mostly to simplify
  testing of the new code.
Georgi (Joro) Kodinov
MDEV-38673: Focus the pick-a-task wording in CONTRIBUTING.md
The wording on how to pick a task in CONTRIBUTING.md is a bit weak.
A more focused message is needed to highlight the tasks that
are available for people to work on.
Georgi (Joro) Kodinov
MDEV-38673: Focus the pick-a-task wording in CONTRIBUTING.md
The wording on how to pick a task in CONTRIBUTING.md is a bit weak.
A more focused message is needed to highlight the tasks that
are available for people to work on.
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
Monty
Ensure we test WSREP_NNULL() before calling wsrep_thd_is_local()

This is to ensure we follow the protocol defined for wsrep_thd_is_local()

Other things:
- Replace WSREP(thd) with WSREP_NNULL(thd) when we are sure thd cannot
  be NULL.
- Removed DBUG_ASSERT() in wsrep_mysqld.cc as it was already ensured
  by previous code a few lines up.
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Marko Mäkelä
MDEV-37412/MDEV-38289 fixup: Allow more FILE_CHECKPOINT

The test was occasionally failing on Microsoft Windows, with the message
saying 12514, which was exactly 16 more than the maximum value that our
regular expression tolerated. Let us add some more slack, up to 12690.
Georgi Kodinov
Merge branch '10.11' into 10.6-mdev-38673
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Oleg Smirnov
MDEV-38045 Expand warnings text, fix review comments
Alexey Botchkov
MDEV-37262 XMLTYPE: validation.
Alexander Barkov
MDEV-19635 System package SYS.DBMS_SQL

In progress
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Monty
MDEV-38865 Simplify testing if binary logging is enabled

In the MariaDB server most of the code uses different expressions to
test if binary logging should be done for the current query.

In many cases the current code does a lot of not needed work before
noticing that binary logging is not needed, which affects performance.
For example, in some cases early decision for preparing to do binary
logging was based on current_stmt_binlog_format (which is by default
statement) without checking if binary logging was enable.d

There is also a lot of different variables that affects if binary logging
should be used:
- (thd->variables.option_bits & OPTION_BIN_LOG)
- (thd->variables.option_bits & OPTION_BIN_TMP_LOG_OFF)
- WSREP(thd) && wsrep_emulate_bin_log && WSREP_PROVIDER_EXISTS_
- mysql_bin_log.is_open()
- xxx_binlog_local_stmt_filter()
- binlog_filter
- thd->variables.sql_log_bin
- thd->binlog_evt_union.do_union

The goal is to move all states to a single variable (for performance,
simplicity and easier debugging). We should ignore all possible
extra work if binary logging is not needed for a particular query.

In most cases we can now use one the following functions to check if
we binary logging is needed.  We need different version to be able
to shortcut code if binary logging is not enabled (with different
code paths if Galera is enabled or not).

- binlog_ready()
- binlog_ready_with_wsrep()
- binlog_ready_no_wsrep()
- binlog_ready_precheck()
- binlog_ready_later()

The state of binary logging is stored in thd->binlog_state.
There are a few bits that shows why binary logging is needed and
one bit for every different reasons for disabling binary logging.
By printing this variable (gdb prints all bits with there enum name)
one can exactly see why binary logging is not happening.

The initial bits are set in THD::THD() and THD::init() and verified in
THD::decide_logging_format().

In this commit all testing of OPTION_BIN_LOG and most testing of
mysql_bin_log.is_open() is removed.
We should over time remove all testing of mysql_bin_log.is_open() from the
main code, except in binlog commit, as the value can change 'anytime' if
binlog rotation fails, which is very likely to result in crashes.
(The patch fixes some of these cases).

THD::binlog_state is a new variable that simplifies testing if replication
is permanently off, is on or temporarly off (for a sub statement).

We also set current_stmt_binlog_format to BINLOG_FORMAT_UNSPEC if binlog is
off.

The above changes allows some code changes:

One does not need to test for mysql_bin_log.is_open() if one also tests
for thd->is_current_stmt_binlog_format_stmt() or
thd->is_current_stmt_binlog_format_row().

The change also allows the following transformations:

(WSREP(thd) && wsrep_emulate_bin_log) ||
(mysql_bin_log.is_open()) && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready()

(WSREP_NNULL(thd) && wsrep_emulate_bin_log) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

(WSREP_EMULATE_BINLOG(thd) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

mysql_bin_log.is_open() && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready_no_wsrep()

Other transformation are easy to do by using the bits set in binlog_state
set by decide_binlog_format()

Other things:
- Implement THD::tmp_disable_binlog() and THD::reenable_binlog() using
  the new binlog_state framework.
  The old code had no effect for row logging, which spider and mroonga
  assumed. Now handler::binlog_log_row() disables row logging properly if
  tmp_disable_binlog() is used
- THD::variables.wsrep_on is now updated by THD:enable_wsrep(),
  THD::disable_wsrep() or THD::set_wsrep(). I added an assert for
  (WSREP_PROVIDER_EXISTS_ && WSREP_ON_) when setting wsrep_on=1
- Reset sql_log_bin in THD::init(). This is to ensure that change user
  will not get affected by the previous users sql_log_bin state.
- Replaced OPTION_BIN_TMP_LOG_OFF with
  (thd->binlog_state & BINLOG_STATE_TMP_DISABLED)
- Added TABLE->disabled_rowlogging() and TABLE->enable_rowlogging() to
  be used when one want to only disable logging for single write, update
  or delete call.

Things to do:
- We should consider removing OPTION_BIN_LOG and instead use
  binlog_state. This would allow us to remove reset and setting it in
  decide_binlog_format(). The code is not anymore using the value of
  OPTION_BIN_LOG anywhere.
- Update binlog_state with BINLOG_STATE_FILTER only when database or filter
  is changed.  This means we do not have to call binlog_filter->db_ok() for
  every statement in decide_binlog_format().
- Remove clear/reset xxx_binlog_local_stmt_filter() as this is now handled
  by binlog_state.
- Remove 'silent' option from mysql_create_db_internal and instead use
  tmp_disable_binlog() / renenable_binlog()
- Remove testing if binary log is desabled in:
  MYSQL_BIN_LOG::write(Log_event *event_info, my_bool *with_annotate):
    if ((!(option_bin_log_flag)) ||
        (thd->lex->sql_command != SQLCOM_ROLLBACK_TO_SAVEPOINT &&
        thd->lex->sql_command != SQLCOM_SAVEPOINT &&
        !binlog_filter->db_ok(local_db)))
      DBUG_RETURN(0);
- Remove all testing of mysql_bin_log.is_open(). Instead test for
  binlog_ready() in main code and add testing of is_open() when
  trying to commit the binary log.  This is needed as currently
  mysql_bin_log.is_open() is tested without a mutex which makes
  it unreliable.
- Remove testing of WSREP_PROVIDER_EXISTS_ in WSREP_NULL() as this
  is guaranteed by THD::enable_wsrep()
- BINLOG_STATE_FILTER need a bit more work. It is currently only used
  in decide_binlog_format()
- Remove BINLOG_STATE_BYPASS (Not needed as some other BINLOG_STATE disable
  bit is set if BYPASS is set).  BYPASS was added mostly to simplify
  testing of the new code.
Monty
Cleanup binary logging API

- Add is_current_stmt_binlog_format_stmt()
- Replace !is_current_stmt_binlog_format_row() with
  is_current_stmt_binlog_format_stmt(). This is in prepartion for using
  BINLOG_FORMAT_UNSPEC if no binary logging.
- Removed printing of temporary_tables info in
  reset_current_stmt_binlog_format_row() as this is not relevant anymore.
- Added testing of (thd->variables.option_bits & OPTION_BIN_LOG) when
  binlogging create procedure.
Alexander Barkov
MDEV-19635 System package SYS.DBMS_SQL

In progress