Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Yuchen Pei
MDEV-24813 [wip] Add an innodb switch table_lock_on_full_scan

And check LIMIT is ULONG_MAX (likely no LIMIT).

Updated tests that have different results. Some still failing.
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Monty
MDEV-38865 Simplify testing if binary logging is enabled

In the MariaDB server most of the code uses different expressions to
test if binary logging should be done for the current query.

In many cases the current code does a lot of not needed work before
noticing that binary logging is not needed, which affects performance.
For example, in some cases early decision for preparing to do binary
logging was based on current_stmt_binlog_format (which is by default
statement) without checking if binary logging was enable.d

There is also a lot of different variables that affects if binary logging
should be used:
- (thd->variables.option_bits & OPTION_BIN_LOG)
- (thd->variables.option_bits & OPTION_BIN_TMP_LOG_OFF)
- WSREP(thd) && wsrep_emulate_bin_log && WSREP_PROVIDER_EXISTS_
- mysql_bin_log.is_open()
- xxx_binlog_local_stmt_filter()
- binlog_filter
- thd->variables.sql_log_bin
- thd->binlog_evt_union.do_union

The goal is to move all states to a single variable (for performance,
simplicity and easier debugging). We should ignore all possible
extra work if binary logging is not needed for a particular query.

In most cases we can now use one the following functions to check if
we binary logging is needed.  We need different version to be able
to shortcut code if binary logging is not enabled (with different
code paths if Galera is enabled or not).

- binlog_ready()
- binlog_ready_with_wsrep()
- binlog_ready_no_wsrep()
- binlog_ready_precheck()
- binlog_ready_later()

The state of binary logging is stored in thd->binlog_state.
There are a few bits that shows why binary logging is needed and
one bit for every different reasons for disabling binary logging.
By printing this variable (gdb prints all bits with there enum name)
one can exactly see why binary logging is not happening.

The initial bits are set in THD::THD() and THD::init() and verified in
THD::decide_logging_format().

In this commit all testing of OPTION_BIN_LOG and most testing of
mysql_bin_log.is_open() is removed.
We should over time remove all testing of mysql_bin_log.is_open() from the
main code, except in binlog commit, as the value can change 'anytime' if
binlog rotation fails, which is very likely to result in crashes.
(The patch fixes some of these cases).

THD::binlog_state is a new variable that simplifies testing if replication
is permanently off, is on or temporarly off (for a sub statement).

We also set current_stmt_binlog_format to BINLOG_FORMAT_UNSPEC if binlog is
off.

The above changes allows some code changes:

One does not need to test for mysql_bin_log.is_open() if one also tests
for thd->is_current_stmt_binlog_format_stmt() or
thd->is_current_stmt_binlog_format_row().

The change also allows the following transformations:

(WSREP(thd) && wsrep_emulate_bin_log) ||
(mysql_bin_log.is_open()) && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready()

(WSREP_NNULL(thd) && wsrep_emulate_bin_log) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

(WSREP_EMULATE_BINLOG(thd) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

mysql_bin_log.is_open() && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready_no_wsrep()

Other transformation are easy to do by using the bits set in binlog_state
set by decide_binlog_format()

Other things:
- Implement THD::tmp_disable_binlog() and THD::reenable_binlog() using
  the new binlog_state framework.
  The old code had no effect for row logging, which spider and mroonga
  assumed. Now handler::binlog_log_row() disables row logging properly if
  tmp_disable_binlog() is used
- THD::variables.wsrep_on is now updated by THD:enable_wsrep(),
  THD::disable_wsrep() or THD::set_wsrep(). I added an assert for
  (WSREP_PROVIDER_EXISTS_ && WSREP_ON_) when setting wsrep_on=1
- Reset sql_log_bin in THD::init(). This is to ensure that change user
  will not get affected by the previous users sql_log_bin state.
- Replaced OPTION_BIN_TMP_LOG_OFF with
  (thd->binlog_state & BINLOG_STATE_TMP_DISABLED)
- Added TABLE->disabled_rowlogging() and TABLE->enable_rowlogging() to
  be used when one want to only disable logging for single write, update
  or delete call.

Things to do:
- We should consider removing OPTION_BIN_LOG and instead use
  binlog_state. This would allow us to remove reset and setting it in
  decide_binlog_format(). The code is not anymore using the value of
  OPTION_BIN_LOG anywhere.
- Update binlog_state with BINLOG_STATE_FILTER only when database or filter
  is changed.  This means we do not have to call binlog_filter->db_ok() for
  every statement in decide_binlog_format().
- Remove clear/reset xxx_binlog_local_stmt_filter() as this is now handled
  by binlog_state.
- Remove 'silent' option from mysql_create_db_internal and instead use
  tmp_disable_binlog() / renenable_binlog()
- Remove testing if binary log is desabled in:
  MYSQL_BIN_LOG::write(Log_event *event_info, my_bool *with_annotate):
    if ((!(option_bin_log_flag)) ||
        (thd->lex->sql_command != SQLCOM_ROLLBACK_TO_SAVEPOINT &&
        thd->lex->sql_command != SQLCOM_SAVEPOINT &&
        !binlog_filter->db_ok(local_db)))
      DBUG_RETURN(0);
- Remove all testing of mysql_bin_log.is_open(). Instead test for
  binlog_ready() in main code and add testing of is_open() when
  trying to commit the binary log.  This is needed as currently
  mysql_bin_log.is_open() is tested without a mutex which makes
  it unreliable.
- Remove testing of WSREP_PROVIDER_EXISTS_ in WSREP_NULL() as this
  is guaranteed by THD::enable_wsrep()
- BINLOG_STATE_FILTER need a bit more work. It is currently only used
  in decide_binlog_format()
- Remove BINLOG_STATE_BYPASS (Not needed as some other BINLOG_STATE disable
  bit is set if BYPASS is set).  BYPASS was added mostly to simplify
  testing of the new code.
Yuchen Pei
MDEV-24813 [wip] Add an innodb switch table_lock_on_full_scan

And check LIMIT is ULONG_MAX (likely no LIMIT).

Updated tests that have different results. Tests still failing:

innodb.deadlock_victim_race 'table_lock'
innodb.deadlock_wait_lock_race 'table_lock'
innodb.deadlock_wait_thr_race 'table_lock'
Marko Mäkelä
MDEV-38958: Core dump contains buffer pool in release builds

buf_pool_t::create(), buf_pool_t::resize(): After
my_virtual_mem_commit() successfully invoked mmap(MAP_FIXED), invoke
ut_dontdump() so that the buffer pool will continue to be excluded
from any core dump as expected on platforms that implement this
functionality.

This was manually tested on Linux and FreeBSD by executing
killall -ABRT mariadbd
and checking the size of the core dump file, while the following
test case was executing:
--source include/have_innodb.inc
set global innodb_buffer_pool_size=10737418240;
sleep 3600;

This fixes up the following changes:
commit b6923420f326ac030e4f3ef89a2acddb45eccb30 (MDEV-29445)
commit 072c7dc774e7f31974eaa43ec1cbb3b742a1582e (MDEV-38671)
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
Monty
MDEV-38865 Simplify testing if binary logging is enabled

In the MariaDB server most of the code uses different expressions to
test if binary logging should be done for the current query.

In many cases the current code does a lot of not needed work before
noticing that binary logging is not needed, which affects performance.
For example, in some cases early decision for preparing to do binary
logging was based on current_stmt_binlog_format (which is by default
statement) without checking if binary logging was enable.d

There is also a lot of different variables that affects if binary logging
should be used:
- (thd->variables.option_bits & OPTION_BIN_LOG)
- (thd->variables.option_bits & OPTION_BIN_TMP_LOG_OFF)
- WSREP(thd) && wsrep_emulate_bin_log && WSREP_PROVIDER_EXISTS_
- mysql_bin_log.is_open()
- xxx_binlog_local_stmt_filter()
- binlog_filter
- thd->variables.sql_log_bin
- thd->binlog_evt_union.do_union

The goal is to move all states to a single variable (for performance,
simplicity and easier debugging). We should ignore all possible
extra work if binary logging is not needed for a particular query.

In most cases we can now use one the following functions to check if
we binary logging is needed.  We need different version to be able
to shortcut code if binary logging is not enabled (with different
code paths if Galera is enabled or not).

- binlog_ready()
- binlog_ready_with_wsrep()
- binlog_ready_no_wsrep()
- binlog_ready_precheck()
- binlog_ready_later()

The state of binary logging is stored in thd->binlog_state.
There are a few bits that shows why binary logging is needed and
one bit for every different reasons for disabling binary logging.
By printing this variable (gdb prints all bits with there enum name)
one can exactly see why binary logging is not happening.

The initial bits are set in THD::THD() and THD::init() and verified in
THD::decide_logging_format().

In this commit all testing of OPTION_BIN_LOG and most testing of
mysql_bin_log.is_open() is removed.
We should over time remove all testing of mysql_bin_log.is_open() from the
main code, except in binlog commit, as the value can change 'anytime' if
binlog rotation fails, which is very likely to result in crashes.
(The patch fixes some of these cases).

THD::binlog_state is a new variable that simplifies testing if replication
is permanently off, is on or temporarly off (for a sub statement).

We also set current_stmt_binlog_format to BINLOG_FORMAT_UNSPEC if binlog is
off.

The above changes allows some code changes:

One does not need to test for mysql_bin_log.is_open() if one also tests
for thd->is_current_stmt_binlog_format_stmt() or
thd->is_current_stmt_binlog_format_row().

The change also allows the following transformations:

(WSREP(thd) && wsrep_emulate_bin_log) ||
(mysql_bin_log.is_open()) && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready()

(WSREP_NNULL(thd) && wsrep_emulate_bin_log) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

(WSREP_EMULATE_BINLOG(thd) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

mysql_bin_log.is_open() && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready_no_wsrep()

Other transformation are easy to do by using the bits set in binlog_state
set by decide_binlog_format()

Other things:
- Implement THD::tmp_disable_binlog() and THD::reenable_binlog() using
  the new binlog_state framework.
  The old code had no effect for row logging, which spider and mroonga
  assumed. Now handler::binlog_log_row() disables row logging properly if
  tmp_disable_binlog() is used
- THD::variables.wsrep_on is now updated by THD:enable_wsrep(),
  THD::disable_wsrep() or THD::set_wsrep(). I added an assert for
  (WSREP_PROVIDER_EXISTS_ && WSREP_ON_) when setting wsrep_on=1
- Reset sql_log_bin in THD::init(). This is to ensure that change user
  will not get affected by the previous users sql_log_bin state.
- Replaced OPTION_BIN_TMP_LOG_OFF with
  (thd->binlog_state & BINLOG_STATE_TMP_DISABLED)
- Added TABLE->disabled_rowlogging() and TABLE->enable_rowlogging() to
  be used when one want to only disable logging for single write, update
  or delete call.

Things to do:
- We should consider removing OPTION_BIN_LOG and instead use
  binlog_state. This would allow us to remove reset and setting it in
  decide_binlog_format(). The code is not anymore using the value of
  OPTION_BIN_LOG anywhere.
- Update binlog_state with BINLOG_STATE_FILTER only when database or filter
  is changed.  This means we do not have to call binlog_filter->db_ok() for
  every statement in decide_binlog_format().
- Remove clear/reset xxx_binlog_local_stmt_filter() as this is now handled
  by binlog_state.
- Remove 'silent' option from mysql_create_db_internal and instead use
  tmp_disable_binlog() / renenable_binlog()
- Remove testing if binary log is desabled in:
  MYSQL_BIN_LOG::write(Log_event *event_info, my_bool *with_annotate):
    if ((!(option_bin_log_flag)) ||
        (thd->lex->sql_command != SQLCOM_ROLLBACK_TO_SAVEPOINT &&
        thd->lex->sql_command != SQLCOM_SAVEPOINT &&
        !binlog_filter->db_ok(local_db)))
      DBUG_RETURN(0);
- Remove all testing of mysql_bin_log.is_open(). Instead test for
  binlog_ready() in main code and add testing of is_open() when
  trying to commit the binary log.  This is needed as currently
  mysql_bin_log.is_open() is tested without a mutex which makes
  it unreliable.
- Remove testing of WSREP_PROVIDER_EXISTS_ in WSREP_NULL() as this
  is guaranteed by THD::enable_wsrep()
- BINLOG_STATE_FILTER need a bit more work. It is currently only used
  in decide_binlog_format()
- Remove BINLOG_STATE_BYPASS (Not needed as some other BINLOG_STATE disable
  bit is set if BYPASS is set).  BYPASS was added mostly to simplify
  testing of the new code.
Oleksandr Byelkin
first pass of bootstrap
Oleg Smirnov
MDEV-38045 Expand warnings text, fix review comments
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Alexander Barkov
MDEV-19635 System package SYS.DBMS_SQL

In progress
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
Monty
Fixing Galera test results
Raghunandan Bhat
MDEV-38864: Use mmap for MEMORY engine allocations

MEMORY engine's data and indexes blocks are allocated using `my_malloc`.
This created internal fragmentation within the system allocator, causing
gradual memory growth and leading to OOM server crash.

This patch introduces memory mapping for MEMORY engine's allocations.
- Implements `my_vmalloc` and `my_vmrelease` functions.
- These functions wrap `my_virtual_mem_alloc` & `my_virtual_mem_release`
  to provide error handling, interface with Performance Schema(PSI) and
  global memory accounting.
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Yuchen Pei
MDEV-24813 [wip] Add an innodb switch table_lock_on_full_scan

And check LIMIT is ULONG_MAX (likely no LIMIT).

Updated tests that have different results.
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
Marko Mäkelä
MDEV-38947 Reimplement SET GLOBAL innodb_buffer_pool_size

We deprecate and ignore the parameter innodb_buffer_pool_chunk_size
and let the buffer pool size to be changed in arbitrary 1-megabyte
increments.

innodb_buffer_pool_size_max: A new read-only startup parameter
that specifies the maximum innodb_buffer_pool_size. On 64-bit
systems other than IBM AIX the default is 8 TiB and the minimum
8 MiB. On other systems, the default and minimum are 0, and
the value 0 will be replaced with the initial innodb_buffer_pool_size
rounded up to the allocation unit (2 MiB or 8 MiB).  The maximum value
is 4GiB-2MiB on 32-bit systems and 16EiB-8MiB on 64-bit systems.
This maximum is very likely to be limited further by the operating system.

The status variable Innodb_buffer_pool_resize_status will reflect
the status of shrinking the buffer pool. When no shrinking is in
progress, the string will be empty.

Unlike before, the execution of SET GLOBAL innodb_buffer_pool_size
will block until the requested buffer pool size change has been
implemented, or the execution is interrupted by a KILL statement
a client disconnect, or server shutdown.  If the
buf_flush_page_cleaner() thread notices that we are running out of
memory, the operation may fail with ER_WRONG_USAGE.

SET GLOBAL innodb_buffer_pool_size will be refused
if the server was started with --large-pages (even if
no HugeTLB pages were successfully allocated). This functionality
is somewhat exercised by the test main.large_pages, which now runs
also on Microsoft Windows.  On Linux, explicit HugeTLB mappings are
apparently excluded from the reported Resident Set Size (RSS), and
apparently unshrinkable between mmap(2) and munmap(2).

The buffer pool will be mapped to a contiguous virtual memory area
that will be aligned and partitioned into extents of 8 MiB on
64-bit systems and 2 MiB on 32-bit systems.

Within an extent, the first few innodb_page_size blocks contain
buf_block_t objects that will cover the page frames in the rest
of the extent.  The number of such frames is precomputed in the
array first_page_in_extent[] for each innodb_page_size.
In this way, there is a trivial mapping between
page frames and block descriptors and we do not need any
lookup tables like buf_pool.zip_hash or buf_pool_t::chunk_t::map.

We will always allocate the same number of block descriptors for
an extent, even if we do not need all the buf_block_t in the last
extent in case the innodb_buffer_pool_size is not an integer multiple
of the of extents size.

The minimum innodb_buffer_pool_size is 256*5/4 pages.  At the default
innodb_page_size=16k this corresponds to 5 MiB.  However, now that the
innodb_buffer_pool_size includes the memory allocated for the block
descriptors, the minimum would be innodb_buffer_pool_size=6m.

my_virtual_mem_reserve(), my_virtual_mem_commit(),
my_virtual_mem_decommit(), my_virtual_mem_release():
New interface mostly by Vladislav Vaintroub, to separately
reserve and release virtual address space, as well as to
commit and decommit memory within it.

The function my_virtual_mem_reserve() is only defined for Microsoft Windows.
Other platforms should invoke my_large_virtual_alloc() instead.

my_large_virtual_alloc(): A new function, similar to my_large_malloc(),
for other platforms than Microsoft Windows.
For regular page size allocations, do not specify MAP_NORESERVE nor
MAP_POPULATE, to preserve compatibility with my_large_malloc().

After my_virtual_mem_decommit(), the virtual memory range will be
inaccessible.

opt_super_large_pages: Declare only on Solaris. Actually, this is
specific to the SPARC implementation of Solaris, but because we
lack access to a Solaris development environment, we will not revise
this for other MMU and ISA.

buf_pool_t::chunk_t::create(): Remove.

buf_pool_t::create(): Initialize all n_blocks of the buf_pool.free list.

buf_pool_t::allocate(): Renamed from buf_LRU_get_free_only().

buf_pool_t::LRU_warned: Changed to Atomic_relaxed<bool>,
only to be modified by the buf_flush_page_cleaner() thread.

buf_pool_t::shrink(): Attempt to shrink the buffer pool.
There are 3 possible outcomes: SHRINK_DONE (success),
SHRINK_IN_PROGRESS (the caller may keep trying),
and SHRINK_ABORT (we seem to be running out of buffer pool).
While traversing buf_pool.LRU, release the contended
buf_pool.mutex once in every 32 iterations in order to
reduce starvation. Use lru_scan_itr for efficient traversal,
similar to buf_LRU_free_from_common_LRU_list().
When relocating a buffer page, invalidate the page identifier
of the original page so that buf_pool_t::page_guess()
will not accidentally match it.

buf_pool_t::shrunk(): Update the reduced size of the buffer pool
in a way that is compatible with buf_pool_t::page_guess(),
and invoke my_virtual_mem_decommit().

buf_pool_t::resize(): Before invoking shrink(), run one batch of
buf_flush_page_cleaner() in order to prevent LRU_warn().
Abort if shrink() recommends it, or no blocks were withdrawn in
the past 15 seconds, or the execution of the statement
SET GLOBAL innodb_buffer_pool_size was interrupted.
After successfully shrinking the buffer pool, announce the success.
The size had already been updated in shrunk().  After failing to
shrink the buffer pool, re-enable the adaptive hash index
if it had been enabled before the resizing.

buf_pool_t::first_to_withdraw: The first block descriptor that is
out of the bounds of the shrunk buffer pool.

buf_pool_t::withdrawn: The list of withdrawn blocks.
If buf_pool_t::resize() is aborted before shrink() completes,
we must be able to resurrect the withdrawn blocks in the free list.

buf_pool_t::contains_zip(): Added a parameter for the
number of least significant pointer bits to disregard,
so that we can find any pointers to within a block
that is supposed to be free.

buf_pool_t::is_shrinking(): Return the total number or blocks that
were withdrawn or are to be withdrawn.

buf_pool_t::to_withdraw(): Return the number of blocks that will need to
be withdrawn.

buf_pool_t::usable_size(): Number of usable pages, considering possible
in-progress attempt at shrinking the buffer pool.

buf_pool_t::page_guess(): Try to buffer-fix a guessed block pointer.
Always check that the pointer is within the current buffer pool size
before dereferencing it.

buf_pool_t::get_info(): Replaces buf_stats_get_pool_info().

innodb_init_param(): Refactored. We must first compute
srv_page_size_shift and then determine the valid bounds of
innodb_buffer_pool_size.

buf_buddy_shrink(): Replaces buf_buddy_realloc().
Part of the work is deferred to buf_buddy_condense_free(),
which is being executed when we are not holding any
buf_pool.page_hash latch.

buf_buddy_condense_free(): Do not relocate blocks.

buf_buddy_free_low(): Do not care about buffer pool shrinking.
This will be handled by buf_buddy_shrink() and
buf_buddy_condense_free().

buf_buddy_alloc_zip(): Assert !buf_pool.contains_zip()
when we are allocating from the binary buddy system.
Previously we were asserting this on multiple recursion levels.

buf_buddy_block_free(), buf_buddy_free_low():
Assert !buf_pool.contains_zip().

buf_buddy_alloc_from(): Remove the redundant parameter j.

buf_flush_LRU_list_batch(): Add the parameter to_withdraw
to keep track of buf_pool.n_blocks_to_withdraw.
Keep evicting as long as the buffer pool is being shrunk,
for at most innodb_lru_scan_depth extra blocks.
Disregard the flush limit for pages that are marked as freed in files.

buf_flush_LRU_to_withdraw(): Update the to_withdraw target during
buf_flush_LRU_list_batch().

buf_pool_t::will_be_withdrawn(): Allow also ptr=nullptr (the condition
will not hold for it).

buf_flush_sync_for_checkpoint(): Wait for pending writes, in order
to guarantee progress even if the scheduler is unfair.

buf_do_LRU_batch(): Skip buf_free_from_unzip_LRU_list_batch()
if we are shrinking the buffer pool. In that case, we want
to minimize the page relocations and just finish as quickly
as possible.

buf_LRU_check_size_of_non_data_objects(): Avoid a crash when the
buffer pool is being shrunk.

trx_purge_attach_undo_recs(): Limit purge_sys.n_pages_handled()
in every iteration, in case the buffer pool is being shrunk
in the middle of a purge batch.

recv_sys_t::wait_for_pool(): Also wait for pending writes, so that
previously written blocks can be evicted and reused.

This ports the following changes from the 10.11 branch:
commit b6923420f326ac030e4f3ef89a2acddb45eccb30 (MDEV-29445)
commit 027d815546d45513ec597b490f2fa45b567802ba
commit 58a36773090223c97d814a07d57ab35ebf803cc5
commit a096f12ff75595ce51fedf879b71640576f70e52
commit df83d3dd8b6551f1d21c9acd0dbcbd4b358d5134
commit 669f719cc21286020c95eec11f0d09b74f96639e (MDEV-36489)
commit f1a8b7fe95399ebe2a1c4a370e332d61dbf6891a (MDEV-36646)
commit 8fb09426b98583916ccfd4f8c49741adc115bac3 (MDEV-36759)
commit 56e0be34bc5d1e967ad610a9b8e24c3f5553bdd8 (MDEV-36780)
commit bb48d7bc812baf7cbd71c9e41b29fac6288cec97 (MDEV-36781)
commit 7b4b759f136f25336fdc12a5a705258a5846d224 (MDEV-36868)
commit cedfe8eca49506c6b4d2d6868f1014c72caaab36 (MDEV-37250)
commit 55e0c34f4f00ca70ad8d6f0522efa94bb81f74fb (MDEV-37263)
commit 21bb6a3e348f89c5cf23d4ee688c57f6078c7b02 (MDEV-37447)
commit 072c7dc774e7f31974eaa43ec1cbb3b742a1582e (MDEV-38671)
commit d4c09187659211eb902e57fc8a9cda2d9df20d07 (MDEV-38958)
Marko Mäkelä
MDEV-38989 main.ctype_utf16le SEGV in Ubuntu 26.04 (x86-64-v3)

my_utf16le_uni(), my_lengthsp_utf16le(): Instead of wrongly claiming
aligned access by invoking uint2korr(), inform the compiler of
unaligned access by invoking memcpy(), which will be optimized away.
Monty
Ensure we test WSREP_NNULL() before calling wsrep_thd_is_local()

This is to ensure we follow the protocol defined for wsrep_thd_is_local()

Other things:
- Replace WSREP(thd) with WSREP_NNULL(thd) when we are sure thd cannot
  be NULL.
- Removed DBUG_ASSERT() in wsrep_mysqld.cc as it was already ensured
  by previous code a few lines up.
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
Mohammad Tafzeel Shams
MDEV-38958: Core dump contains buffer pool in release builds

Issue:
Calling mmap() with MAP_FIXED resets the previously applied
madvise(MADV_DONTDUMP) flag on the affected memory region. As a result,
the buffer pool memory becomes eligible for inclusion in core dumps in
release builds.

Fix:
Move the madvise(MADV_DONTDUMP) call to happen after
my_virtual_mem_commit(), ensuring the memory region retains the
MADV_DONTDUMP flag.
Marko Mäkelä
MDEV-38989 main.ctype_utf16le SEGV in Ubuntu 26.04 (x86-64-v3)

my_lengthsp_utf16le(): Instead of wrongly claiming aligned access
by invoking uint2korr(), inform the compiler of unaligned access
by invoking memcpy(), which will be optimized away.
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Oleg Smirnov
MDEV-38045 Expand warnings text, fix review comments
forkfun
MDEV-38967: STR_TO_DATE() cannot be used in indexed virtual columns

Update STR_TO_DATE() to accept an optional third argument for the locale.
DATE_FORMAT() and STR_TO_DATE() return NULL when the locale argument is NULL;
issue a warning and fall back to the session @@lc_time_names for invalid locale names.
STR_TO_DATE() can now be used in indexed virtual columns while having all 3 arguments,
but not allowed when 2 as it depends then on the session @@lc_time_names
Alexey Botchkov
MDEV-37262 XMLTYPE: validation.
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Yuchen Pei
MDEV-24813 [wip] Add an innodb switch table_lock_on_full_scan

And check LIMIT is ULONG_MAX (likely no LIMIT).

Updated tests that have different results.
Marko Mäkelä
Merge 10.6 into 10.11
Monty
MDEV-38865 Simplify testing if binary logging is enabled

In the MariaDB server most of the code uses different expressions to
test if binary logging should be done for the current query.

In many cases the current code does a lot of not needed work before
noticing that binary logging is not needed, which affects performance.
For example, in some cases early decision for preparing to do binary
logging was based on current_stmt_binlog_format (which is by default
statement) without checking if binary logging was enable.d

There is also a lot of different variables that affects if binary logging
should be used:
- (thd->variables.option_bits & OPTION_BIN_LOG)
- (thd->variables.option_bits & OPTION_BIN_TMP_LOG_OFF)
- WSREP(thd) && wsrep_emulate_bin_log && WSREP_PROVIDER_EXISTS_
- mysql_bin_log.is_open()
- xxx_binlog_local_stmt_filter()
- binlog_filter
- thd->variables.sql_log_bin
- thd->binlog_evt_union.do_union

The goal is to move all states to a single variable (for performance,
simplicity and easier debugging). We should ignore all possible
extra work if binary logging is not needed for a particular query.

In most cases we can now use one the following functions to check if
we binary logging is needed.  We need different version to be able
to shortcut code if binary logging is not enabled (with different
code paths if Galera is enabled or not).

- binlog_ready()
- binlog_ready_with_wsrep()
- binlog_ready_no_wsrep()
- binlog_ready_precheck()
- binlog_ready_later()

The state of binary logging is stored in thd->binlog_state.
There are a few bits that shows why binary logging is needed and
one bit for every different reasons for disabling binary logging.
By printing this variable (gdb prints all bits with there enum name)
one can exactly see why binary logging is not happening.

The initial bits are set in THD::THD() and THD::init() and verified in
THD::decide_logging_format().

In this commit all testing of OPTION_BIN_LOG and most testing of
mysql_bin_log.is_open() is removed.
We should over time remove all testing of mysql_bin_log.is_open() from the
main code, except in binlog commit, as the value can change 'anytime' if
binlog rotation fails, which is very likely to result in crashes.
(The patch fixes some of these cases).

THD::binlog_state is a new variable that simplifies testing if replication
is permanently off, is on or temporarly off (for a sub statement).

We also set current_stmt_binlog_format to BINLOG_FORMAT_UNSPEC if binlog is
off.

The above changes allows some code changes:

One does not need to test for mysql_bin_log.is_open() if one also tests
for thd->is_current_stmt_binlog_format_stmt() or
thd->is_current_stmt_binlog_format_row().

The change also allows the following transformations:

(WSREP(thd) && wsrep_emulate_bin_log) ||
(mysql_bin_log.is_open()) && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready()

(WSREP_NNULL(thd) && wsrep_emulate_bin_log) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

(WSREP_EMULATE_BINLOG(thd) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

mysql_bin_log.is_open() && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready_no_wsrep()

Other transformation are easy to do by using the bits set in binlog_state
set by decide_binlog_format()

Other things:
- Implement THD::tmp_disable_binlog() and THD::reenable_binlog() using
  the new binlog_state framework.
  The old code had no effect for row logging, which spider and mroonga
  assumed. Now handler::binlog_log_row() disables row logging properly if
  tmp_disable_binlog() is used
- THD::variables.wsrep_on is now updated by THD:enable_wsrep(),
  THD::disable_wsrep() or THD::set_wsrep(). I added an assert for
  (WSREP_PROVIDER_EXISTS_ && WSREP_ON_) when setting wsrep_on=1
- Reset sql_log_bin in THD::init(). This is to ensure that change user
  will not get affected by the previous users sql_log_bin state.
- Replaced OPTION_BIN_TMP_LOG_OFF with
  (thd->binlog_state & BINLOG_STATE_TMP_DISABLED)
- Added TABLE->disabled_rowlogging() and TABLE->enable_rowlogging() to
  be used when one want to only disable logging for single write, update
  or delete call.

Things to do:
- We should consider removing OPTION_BIN_LOG and instead use
  binlog_state. This would allow us to remove reset and setting it in
  decide_binlog_format(). The code is not anymore using the value of
  OPTION_BIN_LOG anywhere.
- Update binlog_state with BINLOG_STATE_FILTER only when database or filter
  is changed.  This means we do not have to call binlog_filter->db_ok() for
  every statement in decide_binlog_format().
- Remove clear/reset xxx_binlog_local_stmt_filter() as this is now handled
  by binlog_state.
- Remove 'silent' option from mysql_create_db_internal and instead use
  tmp_disable_binlog() / renenable_binlog()
- Remove testing if binary log is desabled in:
  MYSQL_BIN_LOG::write(Log_event *event_info, my_bool *with_annotate):
    if ((!(option_bin_log_flag)) ||
        (thd->lex->sql_command != SQLCOM_ROLLBACK_TO_SAVEPOINT &&
        thd->lex->sql_command != SQLCOM_SAVEPOINT &&
        !binlog_filter->db_ok(local_db)))
      DBUG_RETURN(0);
- Remove all testing of mysql_bin_log.is_open(). Instead test for
  binlog_ready() in main code and add testing of is_open() when
  trying to commit the binary log.  This is needed as currently
  mysql_bin_log.is_open() is tested without a mutex which makes
  it unreliable.
- Remove testing of WSREP_PROVIDER_EXISTS_ in WSREP_NULL() as this
  is guaranteed by THD::enable_wsrep()
- BINLOG_STATE_FILTER need a bit more work. It is currently only used
  in decide_binlog_format()
- Remove BINLOG_STATE_BYPASS (Not needed as some other BINLOG_STATE disable
  bit is set if BYPASS is set).  BYPASS was added mostly to simplify
  testing of the new code.
Monty
Cleanup binary logging API

- Add is_current_stmt_binlog_format_stmt()
- Replace !is_current_stmt_binlog_format_row() with
  is_current_stmt_binlog_format_stmt(). This is in prepartion for using
  BINLOG_FORMAT_UNSPEC if no binary logging.
- Removed printing of temporary_tables info in
  reset_current_stmt_binlog_format_row() as this is not relevant anymore.
- Added testing of (thd->variables.option_bits & OPTION_BIN_LOG) when
  binlogging create procedure.
Alexander Barkov
MDEV-19635 System package SYS.DBMS_SQL

In progress