Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
sjaakola
MDEV-21935 Mariabackup file limits set incorrectly

Fixed a build time regression happening in x86-debian-12 platform
Thirunarayanan Balathandayuthapani
MDEV-36436  Assertion "unexpected references" == 0 after ALTER IGNORE TABLE

Problem:
=========
An assertion failure occurs in InnoDB during consecutive
ALTER TABLE operations using the COPY algorithm. The crash
happens during the table rename phase because the
source table (dict_table_t::n_ref_count) is non-zero, despite
the thread holding an exclusive Metadata Lock (MDL).

Reason:
========
When ALTER IGNORE TABLE is executed via the COPY algorithm,
it generates undo logs for every row inserted into the
intermediate table (e.g., #sql-alter-...). The background Purge
Thread, responsible for cleaning up these undo logs, attempts
to take an MDL on the table to prevent the table from
being dropped while in use.

Race condition:
==================
First ALTER: Creates #sql-alter-, copies data, and renames it to t1.

Purge Activation: The Purge thread picks up the undo logs from step 1.
It takes an MDL on the temporary name (#sql-alter-) and increments
the table's n_ref_count.

Identity Shift: InnoDB renames the physical table object to t1, but
the Purge thread still holds a reference to this object.

Second ALTER: Starts a new copy process. When it attempts to rename
the "new" t1 to a backup name, it checks if n_ref_count == 0.
Because the Purge thread is still "pinning" the object to
clean up logs from the first ALTER, the count is > 0,
triggering the assertion failure.

Solution:
========
ALTER IGNORE TABLE needs row-level undo logging to easily
roll back the last inserted row in case of duplicate key errors.
By discarding the last undo log record after inserting each row,
purge will not process any log records generated by
ALTER IGNORE TABLE, preventing unexpected access from the purge
subsystem during subsequent DDL operations.

Make skip_alter_undo (1-bit) to (2-bit enum)
to support different ALTER operation modes:

- NO_UNDO (0): Normal mode with standard undo logging
- SKIP_UNDO (1): ALTER mode that skips undo logging
- IGNORE_UNDO (2): ALTER IGNORE mode that rewrites undo blocks

trx_undo_report_row_operation(): Add ALTER IGNORE undo
rewriting logic. Store old undo record info before writing
new records for IGNORE_UNDO mode. Reset undo top_offset
to maintain only latest insert undo

row_ins_clust_index_entry_low(): Prevent ALTER IGNORE from
using bulk insert optimization. since it requires to write
undo log for each row operation that's incompatible with
existing bulk operation.
Mohammad Tafzeel Shams
MDEV-38079: Crash recovery fails after ALTER TABLE…PAGE_COMPRESSED=1

Issue:
- Instant enabling of page_compressed may cause crash recovery failure.
- FSP_SPACE_FLAGS may not match the actual page format during recovery.

Root Cause:
- Table t has pending writes.
- ALTER TABLE t PAGE_COMPRESSED=1 is executed.
- Some pending pages are written in compressed format, but the server may
  crash before all pages are rewritten.
- During recovery, FSP_SPACE_FLAGS may indicate non-compressed while some
  pages are compressed which results in space ID mismatch.

Fix:
- We go for the simple fix of requiring the table to be rebuilt when the
  page_compressed attribute is changed, because:
  - The page_compressed option is used very rarely.
  - The only motivation of page_compressed=1 is to save space, and space may
    only be saved when all pages of the table are rewritten (table is rebuilt).

- alter_options_need_rebuild(): Enabling page_compressed will require a
  full table rebuild.
Marko Mäkelä
MDEV-29445 fixup: Remove a bogus assertion

buf_buddy_shrink(): Remove a redundant assertion
ut_ad(buf_pool.will_be_withdrawn(bpage->zip.data, size))
that is using an incorrect size parameter. In the only caller
buf_pool_t::shrink(size_t size) we already checked with the
correct size that the ROW_FORMAT=COMPRESSED page frame will be
in the being-trimmed portion of the buffer pool. If
p >= memory + size held for a larger size, it would also trivially
hold with the smaller (and incorrect) size.

This code is covered by the test innodb.innodb_buffer_pool_resize,
which uses ROW_FORMAT=COMPRESSED tables.
Monty
Remove for (auto []) constructs from rpl_master_info.h

There is a lot of more work needed to make this code compatitble with
the rest of the MariaDB server.

- All std:: usage should be removed
- Most templatest should be removed
- Usage of <unordered_map>, <string_view>, <optional>, <unordered_set>
  should be removed and replaced with my_sys, sql_string functions

It would be nice of one could also have default values for all the
CHANGE MASTER variables in my.cnf.  This would allow one to have
much shorter CHANGE MASTER commands
Sergei Golubchik
MDEV-38778 mariadbcheck --silent

silence non-errors from admin commands, as it's what --silent
is documented to do
forkfun
MDEV-26112 STR_TO_DATE should work with lc_time_names
nada
MDEV-38624: disallow skipping ER_CONNECTION_KILLED

Prevent ER_CONNECTION_KILLED (1927) from being added to the slave_error_mask bitmap in init_slave_skip_errors(), even when explicitly listed in --slave-skip-errors or when using --slave-skip-errors=all. A warning is emitted in the error log when either is attempted.
ParadoxV5
MDEV-38825 Slave delays when there is "nothing" to replicate

Delayed Replication does not delay on certain metadata events,
but not every type of “non-user” event.
GTID List and Binlog Checkpoint are two of them,
and because the primary sends them after a replica connects,
this replica delays upon START SLAVE even when it’s already up-to-date.

This commit fixes this oddity by changing this condition
set to check the entire non-group event category.
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
sjaakola
MDEV-21935 Mariabackup file limits set incorrectly

When a cluster donor node executes mariabackup SST, it will use same
approximation for max open files limit as was set for the mariadbd
process during the server startup. This may be a problem for installation
where mariabackup execution would need higher open file count, and might
crash for exceeding the too tight open file limit.

The reason for this behavior is that when mariadbd server calculates the
expected max open files count, it will record this open file count approximation
as system ulimit value, both as soft and hard limit. Later, when the node
operates as SST donor, it spawns mariabackup executable, which now inherits the
ulmit setting used for the mariadbd process. Mariabackup tries to enforce
open_files_limit variable value configured in [mariabackup] group in the my.cnf
fle, but this will fail if hard ulimit value is smaller.

The fix in this commit records the approximated max open file count only as soft
ulimit value. If hard ulimit is higher or unlimited, there remains head room for
the mariabackup to use higher open_files_limit configuration.
Sergei Golubchik
MDEV-38868 Assertion `*str != '\0'' failed in my_message_sql on CREATE TABLE

avoid empty error messages in failed CONNECT assisted discovery
Marko Mäkelä
MDEV-37412 fixup: Allow more FILE_CHECKPOINT writes

The test was occasionally failing on Microsoft Windows, with the message
saying 12514, which was exactly 16 more than the maximum value that our
regular expression tolerated. Let us add some more slack.
Thirunarayanan Balathandayuthapani
MDEV-38928 Assertion `undo_no <= 1' failed in trx_t::reset_and_truncate_undo()

Problem:
========
1) ALTER IGNORE TABLE...ALGORITHM=COPY on an InnoDB temporary table
triggers an assertion failure in trx_t::reset_and_truncate_undo().
The IGNORE_UNDO mode rewrites the insert undo log to retain only
the latest record, but temporary tables undo logs are not processed
by the purge thread. Applying this optimization
to a temporary table is incorrect.

2) For partitioned tables with ALTER IGNORE...ALGORITHM=COPY, the
IGNORE_UNDO in trx_undo_report_row_operation() resets trx->undo_no
to 0, but trx->mod_tables retains entries from earlier partitions.
When a different partition (dict_table_t) is inserted into, its
mod_tables entry is created with first=1. After the rewinding, it
writes the undo record with top_undo_no=0, time.valid(0) fails
because first(1) > 0.

3) The savepoint for partial rollback also did not account for
IGNORE_UNDO mode, where trx->undo_no is continuously reset to 0.

Solution:
=========
ha_innobase::extra(): Set skip_alter_undo=IGNORE_UNDO only for
non-temporary tables during HA_EXTRA_BEGIN_ALTER_IGNORE_COPY.
Temporary tables keep NORMAL_UNDO since purge won't process
write temporary table undo logs.

trx_t::reset_and_truncate_undo(): Remove the table parameter
because same transaction could modify multiple tables in
case of partition.

trx_undo_report_row_operation(): After rewinding trx->undo_no to 0
in IGNORE_UNDO mode, clear mod_tables and re-emplace the
table with first=0.

row_insert_for_mysql(): Use savept=0 for IGNORE_UNDO mode. So that
partial rollback target the last inserted undo record.

Change TRX_DML_BULK=2, TRX_DDL_BULK=3 so that bit 1 is set for all
bulk insert modes and for dict_table_t::IGNORE_UNDO. This allows
to replace the savept conditional with a single bit test:
(trx->bulk_insert | table->skip_alter_undo) & 2
Georgi Kodinov
Merge branch '10.11' into 10.6-mdev-38673
bsrikanth-mariadb
MDEV-38805:Store optimizer_context into a IS table

Currently, optimizer context is written as JSON sub-element in the
Optimizer Trace.

In this task, we separate it out from the optimizer trace, and instead
store it in optimizer_context Information Schema table.

The structure of the context is changed to look like below: -
----------------------------------
SET var_name1=val1;

SET var_name2=val2;
.
.
.

CREATE TABLE t1 ( ... );
-- in case it is a constant table
REPLACE INTO t1 VALUES (...);

CREATE TABLE t2 ( ... );
...

set @context='{ JSON with all the captured calls }';
set @optimizer_context='context';

... the original query;
----------------------------------

The IS can be used to read the current stored context, as well as to
dump it to a sql file which can later be replayed in a different
environment.

It is done like below: -
--------------------------------------
set optimizer_record_context=ON;
set optimizer_trace=1

-- sample query

select into outfile '/tmp/captured-context.sql' context from information_schema.OPTIMIZER_CONTEXT;
---------------------------------------

All the existing tests are modified to query OPTIMIZER_CONTEXT IS table
Monty
MDEV-38865 Simplify testing if binary logging is enabled

In the MariaDB server most of the code uses different expressions to
test if binary logging should be done for the current query.

In many cases the current code does a lot of not needed work before
noticing that binary logging is not needed, which affects performance.
For example, in some cases early decision for preparing to do binary
logging was based on current_stmt_binlog_format (which is by default
statement) without checking if binary logging was enable.d

There is also a lot of different variables that affects if binary logging
should be used:
- (thd->variables.option_bits & OPTION_BIN_LOG)
- (thd->variables.option_bits & OPTION_BIN_TMP_LOG_OFF)
- WSREP(thd) && wsrep_emulate_bin_log && WSREP_PROVIDER_EXISTS_
- mysql_bin_log.is_open()
- xxx_binlog_local_stmt_filter()
- binlog_filter
- thd->variables.sql_log_bin
- thd->binlog_evt_union.do_union

The goal is to move all states to a single variable (for performance,
simplicity and easier debugging). We should ignore all possible
extra work if binary logging is not needed for a particular query.

In most cases we can now use one the following functions to check if
we binary logging is needed.  We need different version to be able
to shortcut code if binary logging is not enabled (with different
code paths if Galera is enabled or not).

- binlog_ready()
- binlog_ready_with_wsrep()
- binlog_ready_no_wsrep()
- binlog_ready_precheck()
- binlog_ready_later()

The state of binary logging is stored in thd->binlog_state.
There are a few bits that shows why binary logging is needed and
one bit for every different reasons for disabling binary logging.
By printing this variable (gdb prints all bits with there enum name)
one can exactly see why binary logging is not happening.

The initial bits are set in THD::THD() and THD::init() and verified in
THD::decide_logging_format().

In this commit all testing of OPTION_BIN_LOG and most testing of
mysql_bin_log.is_open() is removed.
We should over time remove all testing of mysql_bin_log.is_open() from the
main code, except in binlog commit, as the value can change 'anytime' if
binlog rotation fails, which is very likely to result in crashes.
(The patch fixes some of these cases).

THD::binlog_state is a new variable that simplifies testing if replication
is permanently off, is on or temporarly off (for a sub statement).

We also set current_stmt_binlog_format to BINLOG_FORMAT_UNSPEC if binlog is
off.

The above changes allows some code changes:

One does not need to test for mysql_bin_log.is_open() if one also tests
for thd->is_current_stmt_binlog_format_stmt() or
thd->is_current_stmt_binlog_format_row().

The change also allows the following transformations:

(WSREP(thd) && wsrep_emulate_bin_log) ||
(mysql_bin_log.is_open()) && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready()

(WSREP_NNULL(thd) && wsrep_emulate_bin_log) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

(WSREP_EMULATE_BINLOG(thd) || mysql_bin_log.is_open())
->
thd->binlog_ready_with_wsrep()

mysql_bin_log.is_open() && (thd->variables.option_bits & OPTION_BIN_LOG)
->
thd->binlog_ready_no_wsrep()

Other transformation are easy to do by using the bits set in binlog_state
set by decide_binlog_format()

Other things:
- Implement THD::tmp_disable_binlog() and THD::reenable_binlog() using
  the new binlog_state framework.
  The old code had no effect for row logging, which spider and mroonga
  assumed. Now handler::binlog_log_row() disables row logging properly if
  tmp_disable_binlog() is used
- THD::variables.wsrep_on is now updated by THD:enable_wsrep(),
  THD::disable_wsrep() or THD::set_wsrep(). I added an assert for
  (WSREP_PROVIDER_EXISTS_ && WSREP_ON_) when setting wsrep_on=1
- Reset sql_log_bin in THD::init(). This is to ensure that change user
  will not get affected by the previous users sql_log_bin state.
- Replaced OPTION_BIN_TMP_LOG_OFF with
  (thd->binlog_state & BINLOG_STATE_TMP_DISABLED)
- Added TABLE->disabled_rowlogging() and TABLE->enable_rowlogging() to
  be used when one want to only disable logging for single write, update
  or delete call.

Things to do:
- We should consider removing OPTION_BIN_LOG and instead use
  binlog_state. This would allow us to remove reset and setting it in
  decide_binlog_format(). The code is not anymore using the value of
  OPTION_BIN_LOG anywhere.
- Update binlog_state with BINLOG_STATE_FILTER only when database or filter
  is changed.  This means we do not have to call binlog_filter->db_ok() for
  every statement in decide_binlog_format().
- Remove clear/reset xxx_binlog_local_stmt_filter() as this is now handled
  by binlog_state.
- Remove 'silent' option from mysql_create_db_internal and instead use
  tmp_disable_binlog() / renenable_binlog()
- Remove testing if binary log is desabled in:
  MYSQL_BIN_LOG::write(Log_event *event_info, my_bool *with_annotate):
    if ((!(option_bin_log_flag)) ||
        (thd->lex->sql_command != SQLCOM_ROLLBACK_TO_SAVEPOINT &&
        thd->lex->sql_command != SQLCOM_SAVEPOINT &&
        !binlog_filter->db_ok(local_db)))
      DBUG_RETURN(0);
- Remove all testing of mysql_bin_log.is_open(). Instead test for
  binlog_ready() in main code and add testing of is_open() when
  trying to commit the binary log.  This is needed as currently
  mysql_bin_log.is_open() is tested without a mutex which makes
  it unreliable.
- Remove testing of WSREP_PROVIDER_EXISTS_ in WSREP_NULL() as this
  is guaranteed by THD::enable_wsrep()
- BINLOG_STATE_FILTER need a bit more work. It is currently only used
  in decide_binlog_format()
- Remove BINLOG_STATE_BYPASS (Not needed as some other BINLOG_STATE disable
  bit is set if BYPASS is set).  BYPASS was added mostly to simplify
  testing of the new code.
Georgi (Joro) Kodinov
MDEV-38673: Focus the pick-a-task wording in CONTRIBUTING.md
The wording on how to pick a task in CONTRIBUTING.md is a bit weak.
A more focused message is needed to highlight the tasks that
are available for people to work on.
Georgi (Joro) Kodinov
MDEV-38673: Focus the pick-a-task wording in CONTRIBUTING.md
The wording on how to pick a task in CONTRIBUTING.md is a bit weak.
A more focused message is needed to highlight the tasks that
are available for people to work on.
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
Sergei Golubchik
MDEV-38827 Assertion `str[strlen(str)-1] != '\n'[ || strlen(str) == 512-1]' failed in my_message_sql

remove trailing '\n' from CONNECT error messages
Thirunarayanan Balathandayuthapani
MDEV-38882 Assertion in diagnostics area on DDL stats

- This issue shares the same root cause of MDEV-38667. This
patch fixes the remaining caller of HA_EXTRA_END_ALTER_COPY by
temporarily setting abort_on_warning to false that prevents
the warning to error escalation.
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
Marko Mäkelä
MDEV-37412/MDEV-38289 fixup: Allow more FILE_CHECKPOINT

The test was occasionally failing on Microsoft Windows, with the message
saying 12514, which was exactly 16 more than the maximum value that our
regular expression tolerated. Let us add some more slack, up to 12690.
Georgi Kodinov
Merge branch '10.11' into 10.6-mdev-38673
Marko Mäkelä
MDEV-38832 test case fixup

The test innodb.stat_tables would fail with a result difference
when using non-default values of the global variables.
Georgi Kodinov
Merge branch '10.11' into 10.6-mdev-38673
Thirunarayanan Balathandayuthapani
Merge branch 10.6 into 10.11
Alexander Barkov
MDEV-19635 System package SYS.DBMS_SQL

In progress
Marko Mäkelä
Merge 10.6 into 10.11
Marko Mäkelä
MDEV-38671: Default to large innodb_buffer_pool_size_max (except on AIX)

my_large_virtual_alloc(): If large page allocation fails, clear
my_use_large_pages. Outside large page allocation (and IBM AIX),
invoke mmap(2) with PROT_NONE so that no page table entries will
have to be allocated. Also, fix the error reporting.

my_virtual_mem_commit(): Do not set the Linux specific flag MAP_POPULATE
because it would introduce a performance regression on startup.
There is a better, system-wide way to disable the overcommit:

sudo sysctl vm.overcommit_memory=2

HAVE_UNACCESSIBLE_AFTER_MEM_DECOMMIT: Remove. We will always make
the virtual memory unaccessible, so that the operating system
can deallocate the page table entries.

buf_pool_t::create(): Skip the my_virtual_mem_commit() on AIX only.

buf_pool_t::page_guess(): Unconditionally check if the pointer is
within the currently allocated buffer pool.

innodb_buffer_pool_size_max: On 64-bit systems other than IBM AIX,
we will default to 8 TiB.

On 32-bit systems, we retain the default innodb_buffer_pool_size_max=0,
that is, by default, SET GLOBAL innodb_buffer_pool_size=... will be
unable to increase from the initial value.

There is no functional change to the startup logic that adjusts
innodb_buffer_pool_size_max to be at least innodb_buffer_pool_size
rounded up to the extent size (2 MiB on 32-bit, 8 MiB on 64-bit).