Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
fe705ac88f4c...
fe705ac88f4c4d04a9636ff19b3a690d0f84c190
Brandon Nesterenko
MDEV-38796 Remove *_binlog_local_stmt_filter()

TODO Addressed:
  - Remove clear/reset xxx_binlog_local_stmt_filter() as this is now handled
      by binlog_state.
ec8283cc2e0e...
ec8283cc2e0e763b3dda2bfd2ab9ac0c116479ce
Aleksey Midenkov
MDEV-27569 Valgrind/MSAN errors in ha_partition::swap_blobs

row_sel_store_mysql_field() does:

3113                    mysql_rec[templ->mysql_null_byte_offset]
3114                            &= static_cast<byte>(~templ->mysql_null_bit_mask);

but mysql_rec[] which is prefetch buffer was allocated without
initialization. Prefetch buffers are allocated by
row_sel_prefetch_cache_init():

3881            ptr = static_cast<byte*>(ut_malloc_nokey(sz));

and then it initializes the buffers with the magic numbers.

The fix initializes the buffers with null bytes as well for nullable
fields.
e983ad20aec6...
e983ad20aec6dfd1a2ea19ad17716bf5965a3430
Brandon Nesterenko
MDEV-38796 Replace mysql_bin_log.is_open with binlog_ready()

Addresses TODO

    - Remove all testing of mysql_bin_log.is_open(). Instead test for
      binlog_ready() in main code and add testing of is_open() when
      trying to commit the binary log.  This is needed as currently
      mysql_bin_log.is_open() is tested without a mutex which makes
      it unreliable.

And also adds tests for the binlog closing for various types of
statements in-between a thd->binlog_ready() check and the commit-time
acquiring of mysql_bin_log.LOCK_log.

Still TODO from the above request:
* WSREP related mysql_bin_log.is_open() changes as a part of MDEV-38865
e55dc5286cca...
e55dc5286ccaf885732ccccf1d1ac3d15cf689f1
Tomáš Mózes
MDEV-39784 test plugins.feedback_os_release fails

The use of un-interpolated quotes resulted in no
bash expansion of the variables of the test causing
test failure.

Review: Daniel Black
d11539dc5b41...
d11539dc5b41daa58342d901dd4a91576a048f16
Yuchen Pei
MDEV-15621 Auto add RANGE COLUMNS partitions by interval

Allow auto partitioning by interval in PARTITION BY RANGE COLUMNS

PARTITION BY RANGE COLUMNS (col_name)
INTERVAL interval [AUTO]
(
  PARTITION partition_name VALUES LESS THAN (value)
  [, PARTITION partition_name VALUES LESS THAN (value) ... ]
)

where

- col_name is the name of one column of type DATE or DATETIME or
  TIMESTAMP

- at least one partition is supplied, and the highest partition cannot
  have MAXVALUE range

- INTERVAL interval is a positive time interval. it can be mariadb
  format or oracle NUMTODSINTERVAL/NUMTOYMINTERVAL format. Like
  versioning, the smallest unit is second, i.e. no subsecond like
  microsecond.

- DATE column cannot have interval with values less than a day

When performing DML on such a table, it will first add partitions
by the specified interval until the partition covers the current time.

Partition addition will not cause an implicit commit like DDL normally
does.

The partitions are named pN.

Otherwise the table behaves exactly the same as a normal RANGE COLUMNS
partitioned table.

Note that TIMESTAMP is not allowed as a type for PARTITION BY RANGE
COLUMNS otherwise.
ce76ee593fd7...
ce76ee593fd7cac5a329d1a41dc6f80a9a699649
Vladislav Vaintroub
MDEV-14443 - address some review comments

Make sure deny_entry_to_json does not fail in memory allocation,
by providing StringBuffer with a large enough statically allocated
size DENY_JSON_ENTRY_BUFSIZE, currently 1230 bytes.

update_deny_entry() - handle json.append() errors
cd0d3ffa10b2...
cd0d3ffa10b2040dc8b45eb1ffdfea861ce36c33
Sergei Petrunia
MDEV-39720: Optimizer Trace doesn't quote string values, may have invalid JSON

- Make Json_writer::add_str() do the quoting with json_escape_to_string().
  = All other string writing methods are handled as they call this one.
- Move json_escape_to_string() from opt_histogram_json.cc to
  my_json_writer.cc. This is needed because JSON writer unit tests do not
  link with opt_histogram_json.
- Histogram code does its own escaping (with handling for special cases).
  Add Json_writer::add_escaped_str() to accept escaped strings.
  This is not intended to be commonly used.
- opt_trace_print_expanded_query() did its own escaping for "expanded_query"
  string. It was done in MDEV-30354. Remove quoting there as otherwise it will
  be double-quoted.

CHERRY PICK INTO OPTIMIZER CONTEXT branch
c0e48fc966d3...
c0e48fc966d3d3926bade68c7c5bfdb9fe88ad3a
Marko Mäkelä
WIP MDEV-39092, and back up non-InnoDB files
c0bb3560296b...
c0bb3560296b39eae6473ec69ac23fb738f4cce6
sjaakola
MDEV-38260 applier hang due to sequence table access

Avoiding to mark sequence table write as a DDL transaction
for galera applying.

Skipping commit for DDL marked GTID log event in applying as this
would lead to double commit, if the transaction has also a real
transaction. Such scnenario could happen if transaction has sequence
table writes together with other innodb table writes.

The commit contains a new mtr test galera.MDEV-38260 for testing
applying of a transaction having sequence table access when using
GTID mode
b8c5023b4fe0...
b8c5023b4fe0ef5c1e96b9cb87beeb50480dd61b
Sergey Vojtovich
Cleanup trx_sys.find() calls

Replaced some trx_sys.find() calls with trx_sys.is_registered(). The
latter is more appropriate in this context as it documents the intention
and may provide more optimal implementation if we ever decide to.
b122852e426e...
b122852e426e90ffbcd78f399ffbcf3dbf9207e7
Brandon Nesterenko
MDEV-38796 Update BINLOG_STATE_FILTER at DB change time

Addresses TODO

    - Update binlog_state with BINLOG_STATE_FILTER only when database or filter
      is changed.  This means we do not have to call binlog_filter->db_ok() for
      every statement in decide_binlog_format().
b05bc3f62d0b...
b05bc3f62d0b1ad8a4fd579140f89eedd58344a6
Rex Johnston
MDEV-39495 Parallel Query: Use temporary work tables to ship results: basic test

Built on top of MDEV-39492, the parallel worker manager creates N
temporary tables and gives them to each parallel worker to populate.
At the conclusion of all worker threads, each row from the above tables
is added to the join_tab->table representing the query result.  To use

MariaDB [test]> set session parallel_worker_threads=10;
Query OK, 0 rows affected (0.001 sec)

MariaDB [test]> select SQL_BUFFER_RESULT * from t1 join seq_1_to_5 on t1.a = seq;
+------+------+-----+
| a    | b    | seq |
+------+------+-----+
|    1 | one  |  1 |
|    2 | two  |  2 |
|    1 | un  |  1 |
|    1 | one  |  1 |
|    2 | two  |  2 |
|    1 | un  |  1 |
|    1 | one  |  1 |
|    2 | two  |  2 |
|    1 | un  |  1 |
|    1 | one  |  1 |
|    2 | two  |  2 |
|    1 | un  |  1 |
|    1 | one  |  1 |
|    2 | two  |  2 |
|    1 | un  |  1 |
|    1 | one  |  1 |
|    2 | two  |  2 |
|    1 | un  |  1 |
|    1 | one  |  1 |
|    2 | two  |  2 |
|    1 | un  |  1 |
|    1 | one  |  1 |
|    2 | two  |  2 |
|    1 | un  |  1 |
|    1 | one  |  1 |
|    2 | two  |  2 |
|    1 | un  |  1 |
|    1 | one  |  1 |
|    2 | two  |  2 |
|    1 | un  |  1 |
+------+------+-----+
30 rows in set (0.010 sec)

Each worker has read the first table with a JT_ALL access method and
replicated the results back to the manager thread, which executes the
rest of the join.  In the above example there are 10 worker threads, so
10 copies of the join result.
aa474c5eff48...
aa474c5eff4864d88535b1155cc9931769b2f36c
Vladislav Vaintroub
x
9962acb15110...
9962acb15110d997cfed9971871a26242c0773e8
Yuchen Pei
MDEV-39789 Fix compiling without perfschema
90b9f0782893...
90b9f07828930aca01bc076ca22bc5e1804dbe6d
Sergei Petrunia
MDEV-39720: Optimizer Trace doesn't quote string values, may have invalid JSON

- Make Json_writer::add_str() do the quoting with json_escape_to_string().
  = All other string writing methods are handled as they call this one.
- Move json_escape_to_string() from opt_histogram_json.cc to
  my_json_writer.cc. This is needed because JSON writer unit tests do not
  link with opt_histogram_json.
- Histogram code does its own escaping (with handling for special cases).
  Add Json_writer::add_escaped_str() to accept escaped strings.
  This is not intended to be commonly used.
- opt_trace_print_expanded_query() did its own escaping for
  "expanded_query" string. It was done in MDEV-30354. Remove escaping
  there to avoid double escaping.
- dump_record_to_trace() also did own escaping when writing table DDLs
  into the optimizer trace. Remove double escaping.
- Adjust my_json_writer unit test.

May 29: CHERRY PICK INTO OPTIMIZER CONTEXT REPLAY branch.
903e00b624c3...
903e00b624c3c2a3a06696a9996a84e0f16445e5
Jan Lindström
MDEV-39783 : Galera test failure on MDEV-35018

Test did echo wsrep_apply_waits+1 value, but this value
is dependent what was value of the wsrep_apply_waits when
test started. If previous test run by same worker
increased the value of wsrep_apply_waits, expected
value 1 in result file would be different than current
value. Removed output as it does not give any real value,
because test already has wait_condition for expected
increase of wsrep_apply_waits value.
8fd382adf638...
8fd382adf6380ee299937b831ffdb4f22eaab37e
Daniel Bartholomew
bump the VERSION
7f207a291cb8...
7f207a291cb8b9903c08a05b0daa0a7eb90c3921
Marko Mäkelä
WIP MDEV-14992 BACKUP SERVER

This introduces a basic driver Sql_cmd_backup, storage engine interfaces,
and basic copying of InnoDB data files.
On Windows, we pass a target directory name; elsewhere, we pass a
target directory handle.

backup_target: A structured data type to represent a directory or a
stream. On Microsoft Windows, we must use directory paths because
there is no variant of CopyFileEx() that would work on file handles.

copy_entire_file(): A file copying service for POSIX systems.

copy_file(): A sparse file-copying service for POSIX systems.

backup_context: An InnoDB backup context, attached to trx->lock.backup
so that context can exist between InnoDB_backup::end(), which is
releasing all locks, and InnoDB_backup::fini() in the same thread,
which is expected to finalize the backup without modifying files
in the server data directory.

fil_space_t::write_or_backup: Keep track of in-flight page writes and
pending backup operation. We must not allow them concurrently, because
that could lead into torn pages in the backup.

fil_space_t::backup_end: The first page number that is not being backed up
(by default 0, to indicate that no backup is in progress).

TRX_STATE_BACKUP: A special InnoDB transaction state indicating association
with BACKUP SERVER, which allows us to pass some context in trx_t from
innodb_backup_end() to innodb_backup_finalize().

log_t::backup: Whether BACKUP SERVER is in progress. The purpose of this
is to make BACKUP SERVER prevent the concurrent execution of
SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size
when innodb_log_archive=OFF.

log_sys.archived_checkpoint: Keep track of the earliest available
checkpoint, corresponding to log_sys.archived_lsn. This reflects
SET GLOBAL innodb_log_recovery_start (which is settable now), for
incremental backup.

buf_flush_list_space(): Check for concurrent backup before writing each
page. This is inefficient, but this function may be invoked from multiple
threads concurrently, and it cannot be changed easily, especially for
fil_crypt_thread().
7923e240f296...
7923e240f29628789edb16406d96875fbffec65c
Yuchen Pei
MDEV-15621 [refactor] Partitioning cleanup

change p_column_list_val::fixed to a bool
remove redundant end label in partition_info::fix_column_value_functions
7696f3d7b9d5...
7696f3d7b9d572c51e9f3c62ed6b4d830a615a81
forkfun
MDEV-24557 mariadb-dump: translate MySQL 8.x user/grant syntax for MariaDB import

  mariadb-dump --system=users (and --system=all) can now capture a
  MySQL 8.0+ source and emit the users, roles and grants as statements
  that load on MariaDB. The dump targets MariaDB only (migration is
  one-way), so each statement is emitted once, in MariaDB form.

  - privileges: mapped to a MariaDB equivalent where one exists
    (e.g. BINLOG_ADMIN -> BINLOG REPLAY, BINLOG ADMIN), otherwise dropped
    with a # WARNING;
  - CREATE USER clauses, that MariaDB has no equivalent for (PASSWORD HISTORY,
    PASSWORD REUSE INTERVAL, DEFAULT ROLE, ...): dropped;
  - auth plugins, that MariaDB does not ship (e.g. sha256_password): the
    IDENTIFIED clause is dropped and the account is forced ACCOUNT LOCK;
  - roles and the role-admin hierarchy, default roles, and partial
    REVOKE (no MariaDB equivalent -> # WARNING).
6ec59dcc0a1c...
6ec59dcc0a1c09530c16b57f8d63f38cb9d2c53a
Sergei Petrunia
[No MDEV#] Invalid conversion is done when reading JSON

When reading JSON strings. we call json_unescape_to_string()
with a StringBuffer with "binary" charset.

That causes conversion from utf8mb4 (data in JSON is in this
charset) to "wide characters".

Create the StringBuffer with utf8mb4 to make sure no conversion
is done.

Now with unit test.
65823a0279c5...
65823a0279c5e0e9222b6c998575562151c5993c
bsrikanth-mariadb
MDEV-39791: Handle count aggregate optimization for replay purpose

Separately dump into the optimizer context, the count of records for
every count aggregate along with the call_number that represents
how many times opt_sum_query() was invoked as part of the original
query execution.
During replay, hook the recorded count value when the hook is invoked
from opt_sum_query().
5f4ea851ced5...
5f4ea851ced582d3a29d8721bbf51cd66518abcd
bsrikanth-mariadb
MDEV-39791: Handle count aggregate optimization for replay purpose

Separately dump into the optimizer context, the count of records for
every count aggregate along with the call_number that represents
how many times opt_sum_query() was invoked as part of the original
query execution.
During replay, hook the recorded count value when the hook is invoked
from opt_sum_query().
5c539706a852...
5c539706a85285a817440e667081a722b10093a4
Yuchen Pei
MDEV-15621 Auto add RANGE COLUMNS partitions by interval

Allow auto partitioning by interval in PARTITION BY RANGE COLUMNS

PARTITION BY RANGE COLUMNS (col_name)
INTERVAL interval [AUTO]
(
  PARTITION partition_name VALUES LESS THAN (value)
  [, PARTITION partition_name VALUES LESS THAN (value) ... ]
)

where

- col_name is the name of one column of type DATE or DATETIME or
  TIMESTAMP

- at least one partition is supplied, and the highest partition cannot
  have MAXVALUE range

- INTERVAL interval is a positive time interval. it can be mariadb
  format or oracle NUMTODSINTERVAL/NUMTOYMINTERVAL format. Like
  versioning, the smallest unit is second, i.e. no subsecond like
  microsecond.

- DATE column cannot have interval with values less than a day

When performing DML on such a table, it will first add partitions
by the specified interval until the partition covers the current time.

Partition addition will not cause an implicit commit like DDL normally
does.

The partitions are named pN.

Otherwise the table behaves exactly the same as a normal RANGE COLUMNS
partitioned table.

Note that TIMESTAMP is not allowed as a type for PARTITION BY RANGE
COLUMNS otherwise.
5acf866df442...
5acf866df44237e3132cd122910d4f16eb6912a4
bsrikanth-mariadb
MDEV-39791: Handle count aggregate optimization for replay purpose

Separately dump into the optimizer context, the count of records for
every count aggregate along with the call_number that represents
how many times opt_sum_query() was invoked as part of the original
query execution.
During replay, hook the recorded count value when the hook is invoked
from opt_sum_query().
5a0b74a31c14...
5a0b74a31c146969b1012082d62fb3efc4edf5a2
Yuchen Pei
MDEV-15621 [refactor] Partitioning cleanup

change p_column_list_val::fixed to a bool
remove redundant end label in partition_info::fix_column_value_functions
3e67de1b0bd2...
3e67de1b0bd2125f87943936cf62b3099baa389e
Thirunarayanan Balathandayuthapani
MDEV-34358  Encryption threads consume CPU and deadlock with DROP TABLE/purge

Problem:
========
1. Encryption threads busy-wait when no work is available:

When reaching fil_system.space_list.end(), fil_crypt_return_iops() is called
with wake=true, causing pthread_cond_broadcast() to wake all threads
unnecessarily, leading to CPU waste.

2. Tablespaces with CLOSING/STOPPING flags skipped during iteration:

Since DDL completion doesn't wake encryption threads, these spaces may never
be encrypted if threads sleep indefinitely.

3. For default_encrypt_list iteration, when spaces exist but none are
acquirable, threads need to wake others for cooperative retry, but this
case was not distinguished from fil_system.space_list.end().

4. IOPS are allocated before searching for tablespaces, wasting resources
during iteration when no I/O occurs.

5. Encryption threads use fil_crypt_threads_cond for two different purposes:
waiting for encryption work and waiting for IOPS allocation. When
fil_crypt_return_iops() or fil_crypt_realloc_iops() broadcasts after
releasing IOPS, it wakes ALL threads including those correctly waiting
for work, causing spurious wakeups and CPU waste.

6. When innodb_encrypt_tables or innodb_encryption_rotate_key_age is changed
during encryption thread iteration, threads continue with stale configuration
values, potentially missing tablespaces that should be encrypted or rotated
under the new settings.

7. The InnoDB encryption thread could deadlock with DROP TABLE and purge
operations in a three-way deadlock scenario:

- DROP TABLE thread holds lock_sys.latch and waits for the tablespace
pending reference count to reach zero before dropping the space.

- Encryption thread holds a tablespace reference and waits to acquire
the tablespace allocation latch in exclusive mode to call
fseg_page_is_allocated() for checking if a page is allocated before
encrypting it.

- Purge coordinator thread holds the tablespace allocation latch
in exclusive mode and waits to acquire lock_sys.latch
in shared mode for record lock operations.

This creates a circular dependency and leads to deadlock.

Solution:
=========
1. Implement timed wait with exponential backoff:

When space == fil_system.space_list.end() (applies to both default_encrypt_list
and space_list iteration when no acquirable spaces are found):
- First timeout: 5 seconds
- Subsequent timeouts: (timed_wait_count + 1) * 5 seconds (10s, 20s, 40s, 60s)
- After 5 consecutive timeouts (~135 seconds total), continue with 60-second
timed waits to ensure threads periodically recheck for state changes
- Timeout counter resets to 0 when woken by signal or when work is found

2. Move IOPS allocation from before tablespace search to after finding a space
that needs rotation. If allocation fails, set recheck=true to skip waiting
and immediately try next space.

Encryption threads would hold space references while waiting in
fil_crypt_alloc_iops(), blocking DROP TABLE. To prevent this,
encyption should do release-wait-reacquire pattern.

fil_crypt_alloc_iops(): Added nowait parameter (default false). When
nowait=true, returns immediately if IOPS not available instead of waiting.

fil_crypt_thread(): Try non-blocking IOPS allocation first with nowait=true.
Only if IOPS not immediately available:
- Save space ID and release space reference
- Wait for IOPS with nowait=false
- Reacquire space using fil_space_get_by_id() and space->acquire()
- If space dropped or stopping, release IOPS and skip

This ensures encryption threads never hold space references while waiting
for IOPS, allowing DROP TABLE operations to proceed without deadlock.

3. Introduce separate condition variable fil_crypt_iops_cond specifically for
IOPS allocation synchronization to prevent spurious wakeups:

- fil_crypt_threads_cond: Used in wait_for_work() for waiting when no
tablespaces need encryption. Signaled when settings change, new
tablespaces are created, or thread count changes.

- fil_crypt_iops_cond: Used in fil_crypt_alloc_iops() for waiting
when IOPS limit is reached. Signaled when IOPS are returned via
fil_crypt_return_iops(), released via fil_crypt_realloc_iops(), or
when srv_n_fil_crypt_iops is increased.

4. Added atomic version counter fil_crypt_settings_version that is incremented
whenever innodb_encrypt_tables or innodb_encryption_rotate_key_age changes.
Encryption threads capture the version at iteration start and check for
changes during iteration. If config changed, threads immediately restart
iteration from the beginning to ensure complete coverage with new settings.

New fields:
- fil_crypt_settings_version: Atomic counter to track configuration changes
- rotate_thread_t::timed_wait_count (uint8_t): Counts consecutive timeouts
  for exponential backoff
- rotate_thread_t::wait_for_work(): Implements timed/indefinite wait strategy
- rotate_thread_t::settings_version: Compares with fil_crypt_settings_version
  to restart encryption from the beginning

5. Fix three-way deadlock with trylock mechanism:
fseg_page_is_allocated(): Added optional caller_mtr parameter. Encryption
thread passes an mtr, allocation bitmap page latch is now correctly held
in that mtr until the caller commits it.

fil_crypt_get_page_throttle(): Use fil_space_t::x_lock_try() to acquire
the tablespace allocation latch non-blockingly. If the trylock fails,
back off and retry the same page to avoid deadlock. The bitmap page
S-latch is added to the mtr and held throughout the page read and
encryption operations. Release the tablespace exclusive latch immediately
after the allocation check to minimize contention, while the bitmap page
S-latch remains held in the mtr.

fil_crypt_rotate_page(): After 10 retries on page latch acquisition,
release IOPS, sleep 10ms, then try to reacquire IOPS with nowait=true.
If IOPS not immediately available, exit gracefully by setting first=true.
This periodic IOPS release allows other encryption threads working on
tablespaces being dropped to make progress and release their space
references, helping to break the deadlock scenario.
3c85b12acf22...
3c85b12acf22fd56bcda432f24010ba4123ebd6a
Vladislav Vaintroub
MDEV-14443 add replication test for DENY

Verify that DENY is correctly replicated to slave.
Added to rpl suite, since it runs all 3 modes of replication (stmt,mix,row)
3c435b8e28dc...
3c435b8e28dc67485fbcc290598cdc0b54309283
sjaakola
MDEV-38260 applier hang due to sequence table access

Avoiding to mark sequence table write as a DDL transaction
for galera applying.

Skipping commit for DDL marked GTID log event in applying as this
would lead to double commit, if the transaction has also a real
transaction. Such scnenario could happen if transaction has sequence
table writes together with other innodb table writes.

The commit contains a new mtr test galera.MDEV-38260 for testing
applying of a transaction having sequence table access when using
GTID mode
39a0f4dd0923...
39a0f4dd0923d4c21fd4e11d331105a1cea008bc
Alexander Barkov
MDEV-39518 Allow prepared statements in stored functions in assignment right hand

In progress
33d0ab084cc6...
33d0ab084cc6fd6340b8a8d9b330305e5ac2dd10
Yuchen Pei
MDEV-39789 Fix compiling without perfschema
32f4609d14c7...
32f4609d14c70d27b1ca86dc1f63ac05d46f6bb9
Yuchen Pei
MDEV-15621 Auto add RANGE COLUMNS partitions by interval

Allow auto partitioning by interval in PARTITION BY RANGE COLUMNS

PARTITION BY RANGE COLUMNS (col_name)
INTERVAL interval [AUTO]
(
  PARTITION partition_name VALUES LESS THAN (value)
  [, PARTITION partition_name VALUES LESS THAN (value) ... ]
)

where

- col_name is the name of one column of type DATE or DATETIME or
  TIMESTAMP

- at least one partition is supplied, and the highest partition cannot
  have MAXVALUE range

- INTERVAL interval is a positive time interval. it can be mariadb
  format or oracle NUMTODSINTERVAL/NUMTOYMINTERVAL format. Like
  versioning, the smallest unit is second, i.e. no subsecond like
  microsecond.

- DATE column cannot have interval with values less than a day

When performing DML on such a table, it will first add partitions
by the specified interval until the partition covers the current time.

Partition addition will not cause an implicit commit like DDL normally
does.

The partitions are named pN.

Otherwise the table behaves exactly the same as a normal RANGE COLUMNS
partitioned table.

Note that TIMESTAMP is not allowed as a type for PARTITION BY RANGE
COLUMNS otherwise.
28e67ab8c840...
28e67ab8c84026faad19dc302a7ad088dfb885a2
Thirunarayanan Balathandayuthapani
MDEV-34358  Encryption threads consume CPU and deadlock with DROP TABLE/purge

Problem:
========
1. Encryption threads busy-wait when no work is available:

When reaching fil_system.space_list.end(), fil_crypt_return_iops() is called
with wake=true, causing pthread_cond_broadcast() to wake all threads
unnecessarily, leading to CPU waste.

2. Tablespaces with CLOSING/STOPPING flags skipped during iteration:

Since DDL completion doesn't wake encryption threads, these spaces may never
be encrypted if threads sleep indefinitely.

3. For default_encrypt_list iteration, when spaces exist but none are
acquirable, threads need to wake others for cooperative retry, but this
case was not distinguished from fil_system.space_list.end().

4. IOPS are allocated before searching for tablespaces, wasting resources
during iteration when no I/O occurs.

5. Encryption threads use fil_crypt_threads_cond for two different purposes:
waiting for encryption work and waiting for IOPS allocation. When
fil_crypt_return_iops() or fil_crypt_realloc_iops() broadcasts after
releasing IOPS, it wakes ALL threads including those correctly waiting
for work, causing spurious wakeups and CPU waste.

6. When innodb_encrypt_tables or innodb_encryption_rotate_key_age is changed
during encryption thread iteration, threads continue with stale configuration
values, potentially missing tablespaces that should be encrypted or rotated
under the new settings.

7. The InnoDB encryption thread could deadlock with DROP TABLE and purge
operations in a three-way deadlock scenario:

- DROP TABLE thread holds lock_sys.latch and waits for the tablespace
pending reference count to reach zero before dropping the space.

- Encryption thread holds a tablespace reference and waits to acquire
the tablespace allocation latch in exclusive mode to call
fseg_page_is_allocated() for checking if a page is allocated before
encrypting it.

- Purge coordinator thread holds the tablespace allocation latch
in exclusive mode and waits to acquire lock_sys.latch
in shared mode for record lock operations.

This creates a circular dependency and leads to deadlock.

Solution:
=========
1. Implement timed wait with exponential backoff:

When space == fil_system.space_list.end() (applies to both default_encrypt_list
and space_list iteration when no acquirable spaces are found):
- First timeout: 5 seconds
- Subsequent timeouts: (timed_wait_count + 1) * 5 seconds (10s, 20s, 40s, 60s)
- After 5 consecutive timeouts (~135 seconds total), continue with 60-second
timed waits to ensure threads periodically recheck for state changes
- Timeout counter resets to 0 when woken by signal or when work is found

2. Move IOPS allocation from before tablespace search to after finding a space
that needs rotation. If allocation fails, set recheck=true to skip waiting
and immediately try next space.

Encryption threads would hold space references while waiting in
fil_crypt_alloc_iops(), blocking DROP TABLE. To prevent this,
encyption should do release-wait-reacquire pattern.

fil_crypt_alloc_iops(): Added nowait parameter (default false). When
nowait=true, returns immediately if IOPS not available instead of waiting.

fil_crypt_thread(): Try non-blocking IOPS allocation first with nowait=true.
Only if IOPS not immediately available:
- Save space ID and release space reference
- Wait for IOPS with nowait=false
- Reacquire space using fil_space_get_by_id() and space->acquire()
- If space dropped or stopping, release IOPS and skip

This ensures encryption threads never hold space references while waiting
for IOPS, allowing DROP TABLE operations to proceed without deadlock.

3. Introduce separate condition variable fil_crypt_iops_cond specifically for
IOPS allocation synchronization to prevent spurious wakeups:

- fil_crypt_threads_cond: Used in wait_for_work() for waiting when no
tablespaces need encryption. Signaled when settings change, new
tablespaces are created, or thread count changes.

- fil_crypt_iops_cond: Used in fil_crypt_alloc_iops() for waiting
when IOPS limit is reached. Signaled when IOPS are returned via
fil_crypt_return_iops(), released via fil_crypt_realloc_iops(), or
when srv_n_fil_crypt_iops is increased.

4. Added atomic version counter fil_crypt_settings_version that is incremented
whenever innodb_encrypt_tables or innodb_encryption_rotate_key_age changes.
Encryption threads capture the version at iteration start and check for
changes during iteration. If config changed, threads immediately restart
iteration from the beginning to ensure complete coverage with new settings.

New fields:
- fil_crypt_settings_version: Atomic counter to track configuration changes
- rotate_thread_t::timed_wait_count (uint8_t): Counts consecutive timeouts
  for exponential backoff
- rotate_thread_t::wait_for_work(): Implements timed/indefinite wait strategy
- rotate_thread_t::settings_version: Compares with fil_crypt_settings_version
  to restart encryption from the beginning

5. Fix three-way deadlock with trylock mechanism:
fseg_page_is_allocated(): Added optional caller_mtr parameter. Encryption
thread passes an mtr, allocation bitmap page latch is now correctly held
in that mtr until the caller commits it.

fil_crypt_get_page_throttle(): Use fil_space_t::x_lock_try() to acquire
the tablespace allocation latch non-blockingly. If the trylock fails,
back off and retry the same page to avoid deadlock. The bitmap page
S-latch is added to the mtr and held throughout the page read and
encryption operations. Release the tablespace exclusive latch immediately
after the allocation check to minimize contention, while the bitmap page
S-latch remains held in the mtr.

fil_crypt_rotate_page(): After 10 retries on page latch acquisition,
release IOPS, sleep 10ms, then try to reacquire IOPS with nowait=true.
If IOPS not immediately available, exit gracefully by setting first=true.
This periodic IOPS release allows other encryption threads working on
tablespaces being dropped to make progress and release their space
references, helping to break the deadlock scenario.
243e7cd44fd6...
243e7cd44fd6f21145dee90012f0674f825ff36e
Sergei Petrunia
MDEV-39720: Optimizer Trace doesn't quote string values, may have invalid JSON

- Make Json_writer::add_str() do the quoting with json_escape_to_string().
  = All other string writing methods are handled as they call this one.
- Move json_escape_to_string() from opt_histogram_json.cc to
  my_json_writer.cc. This is needed because JSON writer unit tests do not
  link with opt_histogram_json.
- Histogram code does its own escaping (with handling for special cases).
  Add Json_writer::add_escaped_str() to accept escaped strings.
  This is not intended to be commonly used.
- opt_trace_print_expanded_query() did its own escaping for
  "expanded_query" string. It was done in MDEV-30354. Remove escaping
  there to avoid double escaping.
- dump_record_to_trace() also did own escaping when writing table DDLs
  into the optimizer trace. Remove double escaping.
- Adjust my_json_writer unit test.
23c821529de7...
23c821529de7001d0b9e9101ee53c60c46bf9994
sjaakola
MDEV-38260 applier hang due to sequence table access

Avoiding to mark sequence table write as a DDL transaction
for galera applying.

Skipping commit for DDL marked GTID log event in applying as this
would lead to double commit, if the transaction has also a real
transaction. Such scnenario could happen if transaction has sequence
table writes together with other innodb table writes.

The commit contains a new mtr test galera.MDEV-38260 for testing
applying of a transaction having sequence table access when using
GTID mode
2247f9d138d5...
2247f9d138d571573c3a119d4bd838c5c3035cb6
Brandon Nesterenko
MDEV-38796 Remove silent option from mysql_create_db_internal

TODO addressed
    - Remove 'silent' option from mysql_create_db_internal and instead use
      tmp_disable_binlog() / reenable_binlog()
1d75fff2e606...
1d75fff2e6069e0ef90bcce30a05763532d7d66c
Sergei Petrunia
MDEV-39720: Optimizer Trace doesn't quote string values, may have invalid JSON

- Make Json_writer::add_str() do the quoting with json_escape_to_string().
  = All other string writing methods are handled as they call this one.
- Move json_escape_to_string() from opt_histogram_json.cc to
  my_json_writer.cc. This is needed because JSON writer unit tests do not
  link with opt_histogram_json.
- Histogram code does its own escaping (with handling for special cases).
  Add Json_writer::add_escaped_str() to accept escaped strings.
  This is not intended to be commonly used.
- opt_trace_print_expanded_query() did its own escaping for "expanded_query"
  string. It was done in MDEV-30354. Remove quoting there as otherwise it will
  be double-quoted.
111587b8bef8...
111587b8bef8187e3166a1e3d088ce131b0f75fc
Yuchen Pei
MDEV-39789 Fix compiling without perfschema
039fe52fd5cd...
039fe52fd5cd00848e12ee35aae67edf4cea1c44
Sergei Petrunia
[No MDEV#] Invalid conversion is done when reading JSON

When reading JSON strings. we call json_unescape_to_string()
with a StringBuffer with "binary" charset.

That causes conversion from utf8mb4 (data in JSON is in this
charset) to "wide characters".

Create the StringBuffer with utf8mb4 to make sure no conversion
occurs.
02a7d6593643...
02a7d6593643f4f1af49464feb38aabed94a6dd2
Yuchen Pei
MDEV-15621 Auto add RANGE COLUMNS partitions by interval

Allow auto partitioning by interval in PARTITION BY RANGE COLUMNS

PARTITION BY RANGE COLUMNS (col_name)
INTERVAL interval [AUTO]
(
  PARTITION partition_name VALUES LESS THAN (value)
  [, PARTITION partition_name VALUES LESS THAN (value) ... ]
)

where

- col_name is the name of one column of type DATE or DATETIME or
  TIMESTAMP

- at least one partition is supplied, and the highest partition cannot
  have MAXVALUE range

- INTERVAL interval is a positive time interval. it can be mariadb
  format or oracle NUMTODSINTERVAL/NUMTOYMINTERVAL format. Like
  versioning, the smallest unit is second, i.e. no subsecond like
  microsecond.

- DATE column cannot have interval with values less than a day

When performing DML on such a table, it will first add partitions
by the specified interval until the partition covers the current time.

Partition addition will not cause an implicit commit like DDL normally
does.

The partitions are named pN.

Otherwise the table behaves exactly the same as a normal RANGE COLUMNS
partitioned table.

Note that TIMESTAMP is not allowed as a type for PARTITION BY RANGE
COLUMNS otherwise.
BuildBot (0.8.9) working for the MariaDB project.
Page built: Fri 29 May 2026 17:32:30 (UTC)