Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
fd9e396c0cea...
fd9e396c0cea654f023ff31520e4d81cbf2a804f
Brandon Nesterenko
MDEV-40768: Slave cannot apply a fragmented row event written with log_bin_compress=ON

MDEV-32570 added fragmentation of large row events. When binlog
compression is enabled (log_bin_compress=ON), the compression is
bypassed, and these binary log events remain as regular row events, when
they should be compressed row events.

This is because the compression happens during the Log_event::write()
function, whereas row data fragmentation happens before
Log_event::write() is called. The Rows_log_event super-class stores the
row data buffers to be written to disk. The regular Rows_log_event
sub-classes's implementations of Log_event::write() write this data
as-is. The compressed sub-classes's implementation of ::write()
over-write these buffers with the compressed rows data before writing to
disk.

To fragment a large row event, the server fragments the Rows_log_event's
row data buffers, to be written by multiple Partial_rows_log_event's,
and each Partial_rows_log_event::write() call writes its portion of the
row data buffer to disk. However, this happens before compression ever
has a chance to take place, and thereby, an event that should be
compressed, never is.

MDEV-39762 added event structure validation for compressed events, and
discovered that these events carry the compressed event type, but are
not actually compressed. Event validation thereby fails.

This patch adds a workaround to override the event type of the
fragmented row events to be regular row events, to be consistent with
the actual on-disk content. The underlying problem still needs to be
addressed though, and is tracked by MDEV-40851.

Signed-off-by: Brandon Nesterenko <[email protected]>
fcca8d1edb0a...
fcca8d1edb0a2e789861eb6b30fec975e15248e1
Arcadiy Ivanov
MDEV-40781 Duplicate row despite DISTINCT when tmp table converts

`create_internal_tmp_table_from_heap()` writes the pending `record[0]`,
the row whose write filled the in-memory table, into the new table.
Since **MDEV-40376** (`636f154bb49`) that write happens *before*
`ha_end_bulk_insert()` rather than after it.

`ha_maria::start_bulk_insert()` disables **all** indexes of an internal
temporary table that is about to receive at least
`MARIA_MIN_ROWS_TO_DISABLE_INDEXES` (100) rows:

```c
if (file->open_flags & HA_OPEN_INTERNAL_TABLE)
{
  /* Internal table; If we get a duplicate something is very wrong */
  file->update|= HA_STATE_CHANGED;
  index_disabled= share->base.keys > 0;
  maria_clear_all_keys_active(file->s->state.key_map);
}
```

`maria_write()` then skips `_ma_check_unique()` entirely, so the unique
constraint that implements `DISTINCT` for a key too wide to be an index
is not enforced. The rows copied out of the in-memory table are already
distinct and need no checking against each other, but the pending row is
exactly the row whose duplicate status is unknown, and it was written
inside that window. A `SELECT DISTINCT` over wide columns could
therefore return a duplicate row.

Note that the justification given in `636f154bb49` is not the mechanism
at work here. It refers to the bulk insert key *tree*, a different
branch of `ha_maria::start_bulk_insert()`; setting
`bulk_insert_buffer_size=0` does not avoid the problem.

The fix splits the copy in two:

1. `Tmp_table_row_copier` gains a second virtual,
  `write_pending_row()`, defaulting to a no-op.
2. `copy_rows()` now only copies the rows the in-memory table holds.
3. `create_internal_tmp_table_from_heap()` calls
  `ha_end_bulk_insert()` and then `write_pending_row()`, so the
  pending row is written with the indexes of the new table back in
  place and a duplicate of an already copied row is detected.

`Window_rowid_remapper` keeps writing its pending row within
`copy_rows()` and inherits the no-op default. Its new position is only
known once the rows before it have been written, and nothing is lost by
writing it with the indexes still disabled: it replaces a row that is
already in the table rather than adding one, and an update of a window
function value cannot collide with another row, as a deduplicating key
is not built on the columns it changes.

The new test covers `SELECT DISTINCT`, `SELECT DISTINCT ... ORDER BY`,
`GROUP BY`, `UNION` and `INSERT ... SELECT DISTINCT`, and asserts that
the conversion actually happened so that a future sizing change cannot
silently void the coverage.
f860f2a937d5...
f860f2a937d526497d206a7fb27c723118d7e278
Oleksandr Byelkin
Merge branch '11.8' into 12.3
f787ac45ba70...
f787ac45ba703f1edaa70b605ee50dae66949d17
Monty
MDEV-40776 Atomic CREATE OR REPLACE silently breaks the foreign key

Give an error if one tries to drop a table referenced by foreign keys if
foreign_key_checks=1

Other things:
- Changed the error message for  ER_TRUNCATE_ILLEGAL_FK to say
  "Cannot drop or truncate a table ..."
f6243d365037...
f6243d3650370c88562a722b7186186e431252f7
Sergei Petrunia
Cleanup: move parallel cost functions from sql_parallel_workers.cc

Now they are in sql_parallel_execution.cc, but probably should go
to something like sql_parallel_optimization.cc
f59b0bd08336...
f59b0bd083363fad9b6c10ee2e1591d19ec0c5cf
bsrikanth-mariadb
MDEV-39226: Push whole multi-table update/delete down into engines

Give storage engines a way to take over an entire multi-table
UPDATE/DELETE, the way they can already take over a SELECT. Without it the
join, the row matching and every modification run in the SQL layer even
when an engine could do the whole statement itself in one step; a
single-table UPDATE/DELETE already avoids this via
direct_update_rows()/direct_delete_rows(), but a multi-table statement has
no primary handler object to drive that path.

This adds a generic, engine-agnostic pushdown interface: the SQL layer
offers the statement to the engine, and if the engine accepts it, it
performs the whole thing and reports only the row counts.

- Split select_handler into a pushdown_handler base with select_handler
  (result set) and a new multi_upddel_handler (runs a whole UPDATE/DELETE,
  reports row counts, reported as PUSHED UPDATE/PUSHED DELETE); add
  handlerton::create_multi_upddel, looked up in Sql_cmd_dml::execute_inner().
- multi_update/multi_delete gain direct_update_delete_done(), which records
  the engine's counts so send_eof() binlogs and replies without the
  SQL-layer loop; it forces statement-format binlogging so the change still
  replicates under binlog_format=ROW, and errors out instead of silently
  dropping counts for an unsupported result object.
- FederatedX implements the interface as the reference engine used to test
  correctness: it prints the statement back and runs it remotely, passes
  the engine's error code/SQLSTATE through, reads the matched count from the
  remote info string, executes IGNORE locally, and only pushes down when all
  tables share one remote server (same as SELECT/derived/unit pushdown).

Test: federated.federatedx_pushdown_upd_del.
f09e3b61e3a0...
f09e3b61e3a03bc4a1773f98a901acd11740c583
Monty
Removed some not needed checks and add a DBUG_ASSERT() for not covered code

- In ha_partition.cc:check_parallel_search(), remove check if
  item_field->field is null. This is not needed as the function is run
  after fix_field() which guarnatees that the field is always set.
- Added DBUG_ASSERT(new_field) to Item_field::fix_fields() to check if a
  select-list item, found by name or alias when resolving ORDER BY/GROUP
  BY/HAVING, can have field == 0. This error path is not covered by any
  mtr test.
e6ff5aafc8f8...
e6ff5aafc8f809042a51f1df0b336bd83d567cfa
Monty
Fixed internal temporary buffer sizes to use tmp_memory_table_size

tmp_memory_table_size is limiting the size of internal temporary memory
tables. max_heap_table_size is there to limiting the size of explictely
created memory tables. max_heap_table_size can be much larger than
tmp_memory_table_size as the memory used by temporary tables is in the
control of the user.

This commit changes the usage of max_heap_table_size for internal buffers
to min(max_heap_table_size, tmp_memory_table_size), like we do for
internal temporary tables.

This changes the in memory buffer allocations for:
- GROUP_CONCAT()
- Calculating the cost for scanning memory tables (the original code was
  wrong here as it used the wrong size for memory tables).
- ANALYZE TABLE buffer sizes for calculating distinct column values

Other things:
- Add THD::ram_limitation() to provide consistent memory limitations
  in all code that used variables.tmp_memory_table_size as buffers.
  If tmp_memory_table_size == 0, then 8192 is used.
  This replaces Item_sum::ram_limitation which used 1024 as min buffer,
  which is way to little for any practical case.
- Added security guard in heap_prepare_hp_create_info to ensure that
  max_table_size is calculated same way as in MariaDB server.
- Fixed initial memory allocations for Item_func_group::concat which
  allocated 'max allowed memory' at start. Now it allocates only 1/16
  of that memory at start.
dd0e3b90d35c...
dd0e3b90d35cbc892987dafe8df349581e7c0f6e
Vladislav Vaintroub
MDEV-22992 Refactor VIO into layered transports and filters

Replace the function-pointer VIO implementation with an abstract C++
interface while retaining the procedural C entry points.

Implement socket and named-pipe transports and composable filters for
client read-ahead, Windows thread-pool prefetch, and TLS. OpenSSL uses a
custom BIO, while wolfSSL uses callbacks that perform I/O through the VIO
below the TLS filter. This keeps waits and timeouts in the transport layer.

Keep sockets nonblocking and implement timed I/O with transport waits.
Named pipes use overlapped I/O for timeout-aware waits and report blocking
waits through the same scheduler callbacks as sockets. Semi-sync
temporarily changes the real VIO read timeout instead of copying VIO state.

Hide transport and TLS implementation state behind accessors. Expose the
TLS handle opaquely and update callers that previously accessed VIO fields
directly. Compile the VIO implementations as C++ and retain PSI memory
accounting for VIO allocations.

Adapt Windows thread-pool pre-read to a Prefetched_vio filter inserted
above the transport so both plain and TLS connections consume prefetched
bytes through the same layered VIO path.
cec8da508f6b...
cec8da508f6bd1019ba38590983040ce47ea095f
Brandon Nesterenko
MDEV-40768: Slave cannot apply a fragmented row event written with log_bin_compress=ON

MDEV-32570 added fragmentation of large row events. When binlog
compression is enabled (log_bin_compress=ON), the compression is
bypassed, and these binary log events remain as regular row events, when
they should be compressed row events.

This is because the compression happens during the Log_event::write()
function, whereas row data fragmentation happens before
Log_event::write() is called. The Rows_log_event super-class stores the
row data buffers to be written to disk. The regular Rows_log_event
sub-classes's implementations of Log_event::write() write this data
as-is. The compressed sub-classes's implementation of ::write()
over-write these buffers with the compressed rows data before writing to
disk.

To fragment a large row event, the server fragments the Rows_log_event's
row data buffers, to be written by multiple Partial_rows_log_event's,
and each Partial_rows_log_event::write() call writes its portion of the
row data buffer to disk. However, this happens before compression ever
has a chance to take place, and thereby, an event that should be
compressed, never is.

MDEV-39762 added event structure validation for compressed events, and
discovered that these events carry the compressed event type, but are
not actually compressed. Event validation thereby fails.

This patch adds a workaround to override the event type of the
fragmented row events to be regular row events, to be consistent with
the actual on-disk content. The underlying problem still needs to be
addressed though, and is tracked by MDEV-40851.

Signed-off-by: Brandon Nesterenko <[email protected]>
ce7fba6acd61...
ce7fba6acd611bf9efe68c989aab3bda74929c4d
Marko Mäkelä
squash! 43946bab13fe78cdc3f55f0b2f2af887dfceddb3

log_t::backup_start(): If we were running with innodb_log_archive=ON,
ensure that the latest file is a valid recovery starting point.
That is, wait for the latest log checkpoint to be within the file.
bc330f38980b...
bc330f38980b378b1d1f77d3bfb87ef37b005e59
Oleksandr Byelkin
Merge branch 'br-11.8-merge' into bb-12.3-release-bad
bab03b0fc44e...
bab03b0fc44e7aeac869b20b0055c6a8f2083eb6
Monty
Added proper cleanup of main.cte_update_delete.test
b628112b90c2...
b628112b90c2ec7aa9ede30dbc7e55092355c50c
Yuchen Pei
MDEV-40805 Do not call lock_rec_convert_impl_to_expl if a table S-lock is held

lock_clust_rec_read_check_and_lock() skipped the implicit-to-explicit
conversion only under a table LOCK_X. When a table LOCK_S is held the
conversion is equally pointless: no other transaction can hold an
implicit X-lock on the record, because modifying a row requires a
table LOCK_IX and LOCK_IX is incompatible with our LOCK_S.
lock_table_has() matches stronger modes, so testing LOCK_S subsumes
the old LOCK_X test.
accf3eff6fcb...
accf3eff6fcbb6a6640ce5a659d6958e4e2724ae
Rex Johnston
MDEV-38801 Item_sum & Item_cache implement deep_copy()

{Item_cache,Item_cache_row,Item_sum}::deep_copy() currently call
shallow_copy_with_checks().  This causes issues when a proper
independent copy is required, e.g. in add_key_part called on a key
with a value containing an item inherited from Item_cache.
We implement a proper deep copy that shares no nodes with the source,
and add a check to ensure this.
9edbb6b12f9f...
9edbb6b12f9f6f376339d9b869ba8bef8b6805b3
Rex Johnston
MDEV-40012 Parallel Query: execute the join in the worker threads

Each parallel worker now runs the join and the WHERE over its own chunk of the
driving table and ships the base-table columns of every row that qualified. The
manager copies those columns back into the fields they came from in its own
table instances, so that after a row is drained its records hold what a serial
scan would have left there, and then evaluates the select list against them and
sends the row. This replaces the model where the workers shipped raw source
records and the manager ran the join.

The transport carries columns rather than projected values because anything the
manager does with a row beyond sending it reads a record, and not all of those
reads go through Items that could be re-pointed: create_tmp_table() builds
Copy_field pairs holding raw Field pointers into the base tables.

make_join_readinfo()'s gate, can_run_query_in_workers(), chooses the worker-side
path for an inner select-project[-join] over a parallel-scannable driving table:
no tmp table, no LIMIT/SQL_CALC_FOUND_ROWS/procedure/aggregate, no outer join,
no semijoin strategy the worker's plain nested loop cannot honour, and every
non-driving table reached by eq_ref, ref or full scan. Every expression it hands
a worker has to deep-copy into something that shares no node with the original
and reads no table outside the join, or the copy would carry the manager's own
items onto a worker's tables. do_select() then runs run_worker_side_join();
anything ineligible runs serially.

Each worker opens a private copy of every non-const table, deep-clones and
field-rebinds the conditions and the shipped column list, rebuilds each ref,
scans its chunk, runs its own join and ships the columns it read for each
full match. It runs under the session's query id, so an item holding a
value for the statement recomputes it rather than handing out the one it was
cloned with.

sql_parallel_execution.cc holds all of the above; sql_parallel_workers.cc keeps
the worker threads, the batch channel and their lifetime.

Tests: parallel_query_worker_side, parallel_query_join and parallel_query_clone
compare a parallel result set against a serial one; parallel_query and
parallel_query_oom moved to a plain SELECT and were re-recorded.
.claude/commits/MDEV-40012.md records what came from the 13.0 branch verbatim,
what was adapted to this tree, and what is new here.

This commit was prepared with Claude Code: it ported the work from the 13.0
branch onto this tree's refactored structures and renamed scan API, found and
fixed the two holes this tree's deeper Item copying opens -- an expression
reaching a table the worker does not have, and a statement-lifetime item cached
against query id 0 -- split the file, and ran the suites.
98999c7b5a9f...
98999c7b5a9f1abd9a595b2e1877cf36be073dd7
Yuchen Pei
MDEV-40486 [fixup] Clamp max_length at MAX_FIELD_VARCHARLENGTH in Item_func_vec_fromtext::fix_length_and_dec

And move the length check in Item_func_vec_fromtext::val_str to later

This allows

create table t1 (v vector(64) not null);
insert into t1 select vec_fromtext(concat('[',group_concat(1),']')) from seq_1_to_64;

which was banned in the previous fix
bb0ac437015dec04fbee226745a8eb2bb4825917, though this also introduces
the inconsistency(?) where

create table t1 as select vec_fromtext(concat('[',group_concat(1),']')) from seq_1_to_64;

still fails ER_TRUNCATED_WRONG_VALUE
96f7c9702a48...
96f7c9702a48dd34320f4b044b6de65044735397
Rex Johnston
MDEV-39492 PQ workers must read the manager's transaction snapshot

A parallel worker runs in its own THD, so it runs in its own transaction and
opened its own read view at its first read. Nothing tied that view to the one
the manager holds: the workers and the manager could each read a different
version of the table, and the chunk boundaries the engine partitioned under
the manager's view did not even belong to the snapshot the workers scanned.
Both directions were visible. Under REPEATABLE READ a worker returned rows
another session had committed after the manager's snapshot was taken, and rows
the manager's own transaction had written but not committed went missing,
because to the worker's view the manager was just another active transaction.

InnoDB implements it by copying the source transaction's read view.
ReadView::clone() installs the source's ReadViewBase state and opens the copy.
It inherits the source's m_creator_trx_id rather than keeping the worker's own,
so rows written by the sharing transaction itself stay visible through the copy
-- that is what changes_visible() uses the creator id for -- and it inherits
the source's isolation level, so a READ UNCOMMITTED source is still read
without consulting a view at all. The source view is read under its m_mutex,
the same protection the purge coordinator takes in ReadView::append_to(), and
from_thd's handlerton data under its LOCK_thd_data.
937af9d8bd0f...
937af9d8bd0f3ad228d4d856e33fd86c2c3645ec
Vladislav Vaintroub
MDEV-22992 Refactor VIO into layered transports and filters

Replace the function-pointer VIO implementation with an abstract C++
interface while retaining the procedural C entry points.

Implement socket and named-pipe transports and composable filters for
client read-ahead, Windows thread-pool prefetch, and TLS. OpenSSL uses a
custom BIO, while wolfSSL uses callbacks that perform I/O through the VIO
below the TLS filter. This keeps waits and timeouts in the transport layer.

Keep sockets nonblocking and implement timed I/O with transport waits.
Named pipes use overlapped I/O for timeout-aware waits and report blocking
waits through the same scheduler callbacks as sockets. Semi-sync
temporarily changes the real VIO read timeout instead of copying VIO state.

Hide transport and TLS implementation state behind accessors. Expose the
TLS handle opaquely and update callers that previously accessed VIO fields
directly. Compile the VIO implementations as C++ and retain PSI memory
accounting for VIO allocations.

Adapt Windows thread-pool pre-read to a Prefetched_vio filter inserted
above the transport so both plain and TLS connections consume prefetched
bytes through the same layered VIO path.
9299f18da02a...
9299f18da02a7dbeff77d8f0d68ac8c114111913
Oleksandr Byelkin
Merge branch '12.3' into br-12.3-merge
9133cd5d8f8d...
9133cd5d8f8d21868d599f5c8fac5b774c7739ad
Vladislav Vaintroub
MDEV-22992 Refactor VIO into layered transports and filters

Replace the function-pointer VIO implementation with an abstract C++
interface while retaining the procedural C entry points.

Implement socket and named-pipe transports and composable filters for
client read-ahead, Windows thread-pool prefetch, and TLS. OpenSSL uses a
custom BIO, while wolfSSL uses callbacks that perform I/O through the VIO
below the TLS filter. This keeps waits and timeouts in the transport layer.

Keep sockets nonblocking, as they have already been for a long time, and
remove the switch to block/nonblock machinery. Socket vio is now
nonblocking already after construction and this never changes.

Hide transport and TLS implementation state behind accessors. Expose the
TLS handle opaquely and update callers that previously accessed VIO fields
directly. Compile the VIO implementations as C++ and retain PSI memory
accounting for VIO allocations.

WolfSSL/OpenSSL headers are no longer included into VIO public headers.
The files that actually use SSL, SSL_CTX, X509 get the headers via
ssl_compat.h

Adapt Windows thread-pool AIO pre-read to a Prefetched_vio filter,
also for TLS (previously TLS was readiness-only, inefficient use of AIO)
83e909fc2a0d...
83e909fc2a0dbc394b4b683fb3fa2d7dcf26cc5e
Brandon Nesterenko
MDEV-40768: Slave cannot apply a fragmented row event written with log_bin_compress=ON

MDEV-32570 added fragmentation of large row events. When binlog
compression is enabled (log_bin_compress=ON), the compression is
bypassed, and these binary log events remain as regular row events, when
they should be compressed row events.

This is because the compression happens during the Log_event::write()
function, whereas row data fragmentation happens before
Log_event::write() is called. The Rows_log_event super-class stores the
row data buffers to be written to disk. The regular Rows_log_event
sub-classes's implementations of Log_event::write() write this data
as-is. The compressed sub-classes's implementation of ::write()
over-write these buffers with the compressed rows data before writing to
disk.

To fragment a large row event, the server fragments the Rows_log_event's
row data buffers, to be written by multiple Partial_rows_log_event's,
and each Partial_rows_log_event::write() call writes its portion of the
row data buffer to disk. However, this happens before compression ever
has a chance to take place, and thereby, an event that should be
compressed, never is.

MDEV-39762 added event structure validation for compressed events, and
discovered that these events carry the compressed event type, but are
not actually compressed. Event validation thereby fails.

This patch adds a workaround to override the event type of the
fragmented row events to be regular row events, to be consistent with
the actual on-disk content. The underlying problem still needs to be
addressed though, and is tracked by MDEV-40851.

Signed-off-by: Brandon Nesterenko <[email protected]>
702076ee0f83...
702076ee0f839834a5c11f067bb4f246e2ad6eeb
Brandon Nesterenko
MDEV-40635 fixup
6a6d6751736f...
6a6d6751736f8b1d4912f2329e70fc533e4e0656
Rex Johnston
MDEV-39492 PQ framework for manager and worker threads

Introduces parallel_worker_threads variable to control the number
of worker threads created by a parallel execution query.

2 new files, sql_parallel_workers.h sql_parallel_workers.cc which
contain structures for the creation, management and deletion of
parallel worker threads (pwt_ in the name).  Main management
class created in the stack in JOIN::exec, implemented for the
top level select.

Threads are registed in server_threads, so are visible in
information_schema.processlist and the show processlist command.

We check that a kill query on a parallel worker is passed onto it's
manager and the query is properly aborted, and that a kill connection
is handled properly in parallel_worker.test.
688bcacb2025...
688bcacb20256281999d1056d92c9611cf0a25f7
ParadoxV5
MDEV-40636 merge fix

reäpply missed merge fixes

Co-authored-by: Brandon Nesterenko <[email protected]>
6803fb454891...
6803fb4548912e89af275cff41e7301e71ff8e6b
Brandon Nesterenko
MDEV-40768: Slave cannot apply a fragmented row event written with log_bin_compress=ON

MDEV-32570 added fragmentation of large row events. When binlog
compression is enabled (log_bin_compress=ON), the compression is
bypassed, and these binary log events remain as regular row events, when
they should be compressed row events.

This is because the compression happens during the Log_event::write()
function, whereas row data fragmentation happens before
Log_event::write() is called. The Rows_log_event super-class stores the
row data buffers to be written to disk. The regular Rows_log_event
sub-classes's implementations of Log_event::write() write this data
as-is. The compressed sub-classes's implementation of ::write()
over-write these buffers with the compressed rows data before writing to
disk.

To fragment a large row event, the server fragments the Rows_log_event's
row data buffers, to be written by multiple Partial_rows_log_event's,
and each Partial_rows_log_event::write() call writes its portion of the
row data buffer to disk. However, this happens before compression ever
has a chance to take place, and thereby, an event that should be
compressed, never is.

MDEV-39762 added event structure validation for compressed events, and
discovered that these events carry the compressed event type, but are
not actually compressed. Event validation thereby fails.

This patch adds a workaround to override the event type of the
fragmented row events to be regular row events, to be consistent with
the actual on-disk content. The underlying problem still needs to be
addressed though, and is tracked by MDEV-40851.

Signed-off-by: Brandon Nesterenko <[email protected]>
677451704df6...
677451704df6c439f5530b86418a3ddc49675395
Yuchen Pei
MDEV-40751 Make sure that VEC_FROMTEXT results in a length of multiple of 4
6415902490b1...
6415902490b1b0a9a500cee9bc0a41c342190605
Yuchen Pei
MDEV-40805 Do not call lock_rec_convert_impl_to_expl if a table S-lock is held

lock_clust_rec_read_check_and_lock() skipped the implicit-to-explicit
conversion only under a table LOCK_X. When a table LOCK_S is held the
conversion is equally pointless: no other transaction can hold an
implicit X-lock on the record, because modifying a row requires a
table LOCK_IX and LOCK_IX is incompatible with our LOCK_S.
lock_table_has() matches stronger modes, so testing LOCK_S subsumes
the old LOCK_X test.
5c0f61990c2e...
5c0f61990c2ef3c6ddf7ee573cc8c59b664a769d
Marko Mäkelä
fixup! a2020ca6c21691a3727fa2e6a5b8524242e93836
5301674b0151...
5301674b0151284552efc9690e9ce9fc9d93480c
Monty
Limit the memory used by GROUP_CONCAT() with ORDER BY

GROUP_CONCAT() with ORDER BY collects all rows of the group in a TREE
and only cuts it down in repack_tree(). The repack was triggered by

  (tree_len >> GCONCAT_REPACK_FACTOR) > thd->gconcat_max_len()

with GCONCAT_REPACK_FACTOR 10, that is when the rows in the tree had
produced 1024 * group_concat_max_len bytes, or 1G with the default
settings. On top of that tree_len only counted the length of the
strings, while the tree costs sizeof(TREE_ELEMENT) + reclength per row.
For GROUP_CONCAT(int_col ORDER BY int_col) that is about 40 bytes per
row against 6 bytes of result, so the tree had grown to several GB
before the first repack. In practice the server ran out of memory
first and the repack code was close to never used.

The tree is now limited by the memory it has really allocated,
tree->allocated, instead of by the length of the strings it holds.
The limit is MY_MAX(thd->ram_limitation(), thd->gconcat_max_len()) and
is never set so low that the tree can not hold a few rows.

repack_tree() builds a new tree while the old one is still in memory,
so the peak usage is the size we start the repack at plus the size we
copy to. To keep the sum within the limit it is split into
GCONCAT_TREE_PARTS parts; the repack starts when
GCONCAT_TREE_REPACK_PARTS of them are used and copies to the remaining
part. The part we do not copy to is also the room the tree has to grow
before the next repack, which keeps the repacks amortized.

Other changes:

- tree_len is removed. It was only read by the old trigger.

- repack_tree() decided that it had run out of memory by testing
  st.len <= st.maxlen after the walk. That test was only valid because
  the old trigger guaranteed that a complete copy had to overshoot
  st.maxlen. A repack triggered by memory can complete the walk with
  st.len far below st.maxlen, which would have failed the query with a
  wrong out of memory error. There is now an explicit flag for it.

- The length that decides which rows to keep now also counts the
  separator that is put between two rows, so that it matches what
  val_str() will produce.

- When the memory limit stops the copy, the result becomes shorter
  than group_concat_max_len. dump_leaf_key() can not detect this, as
  the result never reaches the maximum length. This is now remembered
  in result_cut and reported to the user.

- All cut value reporting is moved to val_str(); dump_leaf_key() only
  marks that the result was cut. This removes the need to clear the
  truncated flag of table->blob_storage to avoid a duplicated warning,
  and gives one warning per group also when val_str() is called more
  than once for the same group, which repeated the warning before.

- Added a function comment for repack_tree() that describes where the
  rows are cut away and why building a copy frees memory.
4e681204d7e8...
4e681204d7e8bdbe2058ffeaf55afeeecd34f686
Arcadiy Ivanov
MDEV-40802 COUNT(DISTINCT <blob>) fails when its tmp table converts

`COUNT(DISTINCT)` collects the distinct values in a temporary table
with a unique constraint over the aggregate's arguments, and treats a
duplicate key error from the write as "value already seen":

```c
if (!table->file->is_fatal_error(error, HA_CHECK_DUP))
  return FALSE;                          // duplicate, not an error
```

For a blob argument the record holds only a pointer to the value, so
`Aggregator_distinct::setup()` cannot use the `Unique` tree, which
compares raw record bytes, and every value goes through that write
instead.

When such a write overflows the in-memory table,
`create_internal_tmp_table_from_heap()` copies the stored rows to an
on-disk table and then writes the row that overflowed, which until
then was held in `record[0]` alone. Whether a duplicate key error on
that last write is fatal is decided by the caller's
`ignore_last_dupp_key_error` argument, and `Aggregator_distinct::add()`
passed **0** three lines below the code that ignores the very same
condition. The statement failed with

    ERROR 1169 (23000): Can't write, because of unique constraint,
    to table '(temporary)'

Pass **1** instead, so that a duplicate arriving through the conversion
is discarded exactly like one arriving through the ordinary write. The
result is `table->file->stats.records` of that table, so not storing
the duplicate is what makes the count right.

The argument is the same upstream, where it is unreachable: a
temporary table with a blob column was created on the on-disk engine
to begin with, so the conversion was never entered for the only
tables whose pending row can be a duplicate. Supporting blob columns
in the in-memory engine made the table start in memory and convert.

New tests `heap.count_distinct_blob_convert` and
`heap.count_distinct_blob_convert_debug`.

A write rejected as a duplicate returns its record to the free list and
never reaches the allocation of the blob value, so only the first copy
of a value makes the in-memory table grow, and the write that finds it
full is the second copy of the value stored last. That holds only while
a record slot is what the table runs out of first. Blob values come out
of the same space, and only a write that is not a duplicate ever
allocates one, so when a blob allocation is the one that hits the limit,
the pending row is not a duplicate at all.

Which of the two runs out first follows from how records and blob values
pack together, not from any threshold on the value width. Of 24 measured
combinations of width and `max_heap_table_size`, 20 convert but only 8
reach a duplicate pending row, so asserting that the table was converted
does not establish that the ignored duplicate was reached.

The first test uses widths measured to overflow on a record slot. The
second removes the dependency on that measurement, injecting the
duplicate through a new debug point in
`Tmp_table_default_copier::copy_rows()`, beside the one the row copy
loop already carries. Every value is present twice, so whichever copy
the injected duplicate discards, the other one is still written and the
count does not depend on which write overflowed.

The status counter is read with the in-memory limit restored. The status
table is materialized into a temporary table of its own, and its
VARIABLE_VALUE column is wide enough to be stored as a blob, so under
the shrunken limit that table can overflow and be converted as well, and
would then report its own conversion.
4a03ee87ce1a...
4a03ee87ce1a0046629ec99ede18823141585880
Brandon Nesterenko
MDEV-40768: Slave cannot apply a fragmented row event written with log_bin_compress=ON

MDEV-32570 added fragmentation of large row events. When binlog
compression is enabled (log_bin_compress=ON), the compression is
bypassed, and these binary log events remain as regular row events, when
they should be compressed row events.

This is because the compression happens during the Log_event::write()
function, whereas row data fragmentation happens before
Log_event::write() is called. The Rows_log_event super-class stores the
row data buffers to be written to disk. The regular Rows_log_event
sub-classes's implementations of Log_event::write() write this data
as-is. The compressed sub-classes's implementation of ::write()
over-write these buffers with the compressed rows data before writing to
disk.

To fragment a large row event, the server fragments the Rows_log_event's
row data buffers, to be written by multiple Partial_rows_log_event's,
and each Partial_rows_log_event::write() call writes its portion of the
row data buffer to disk. However, this happens before compression ever
has a chance to take place, and thereby, an event that should be
compressed, never is.

MDEV-39762 added event structure validation for compressed events, and
discovered that these events carry the compressed event type, but are
not actually compressed. Event validation thereby fails.

This patch adds a workaround to override the event type of the
fragmented row events to be regular row events, to be consistent with
the actual on-disk content. The underlying problem still needs to be
addressed though, and is tracked by MDEV-40851.

Signed-off-by: Brandon Nesterenko <[email protected]>
3cd3e74b735c...
3cd3e74b735cacf4c202628126bbac0b4cddd478
Vladislav Vaintroub
MDEV-22992 Refactor VIO into layered transports and filters

Replace the function-pointer VIO implementation with an abstract C++
interface while retaining the procedural C entry points.

Implement socket and named-pipe transports and composable filters for
client read-ahead, Windows thread-pool prefetch, and TLS. OpenSSL uses a
custom BIO, while wolfSSL uses callbacks that perform I/O through the VIO
below the TLS filter. This keeps waits and timeouts in the transport layer.

Keep sockets nonblocking and implement timed I/O with transport waits.
Named pipes use overlapped I/O for timeout-aware waits and report blocking
waits through the same scheduler callbacks as sockets. Semi-sync
temporarily changes the real VIO read timeout instead of copying VIO state.

Hide transport and TLS implementation state behind accessors. Expose the
TLS handle opaquely and update callers that previously accessed VIO fields
directly. Compile the VIO implementations as C++ and retain PSI memory
accounting for VIO allocations.

Adapt Windows thread-pool pre-read to a Prefetched_vio filter inserted
above the transport so both plain and TLS connections consume prefetched
bytes through the same layered VIO path.
37077ccef15c...
37077ccef15cae879095ffdf311e663769936679
Monty
Limit the memory used by GROUP_CONCAT() with ORDER BY

GROUP_CONCAT() with ORDER BY collects all rows of the group in a TREE
and only cuts it down in repack_tree(). The repack was triggered by

  (tree_len >> GCONCAT_REPACK_FACTOR) > thd->gconcat_max_len()

with GCONCAT_REPACK_FACTOR 10, that is when the rows in the tree had
produced 1024 * group_concat_max_len bytes, or 1G with the default
settings. On top of that tree_len only counted the length of the
strings, while the tree costs sizeof(TREE_ELEMENT) + reclength per row.
For GROUP_CONCAT(int_col ORDER BY int_col) that is about 40 bytes per
row against 6 bytes of result, so the tree had grown to several GB
before the first repack. In practice the server ran out of memory
first and the repack code was close to never used.

The tree is now limited by the memory it has really allocated,
tree->allocated, instead of by the length of the strings it holds.
The limit is MY_MAX(thd->ram_limitation(), thd->gconcat_max_len()) and
is never set so low that the tree can not hold a few rows.

repack_tree() builds a new tree while the old one is still in memory,
so the peak usage is the size we start the repack at plus the size we
copy to. To keep the sum within the limit it is split into
GCONCAT_TREE_PARTS parts; the repack starts when
GCONCAT_TREE_REPACK_PARTS of them are used and copies to the remaining
part. The part we do not copy to is also the room the tree has to grow
before the next repack, which keeps the repacks amortized.

Other changes:

- tree_len is removed. It was only read by the old trigger.

- repack_tree() decided that it had run out of memory by testing
  st.len <= st.maxlen after the walk. That test was only valid because
  the old trigger guaranteed that a complete copy had to overshoot
  st.maxlen. A repack triggered by memory can complete the walk with
  st.len far below st.maxlen, which would have failed the query with a
  wrong out of memory error. There is now an explicit flag for it.

- The length that decides which rows to keep now also counts the
  separator that is put between two rows, so that it matches what
  val_str() will produce.

- When the memory limit stops the copy, the result becomes shorter
  than group_concat_max_len. dump_leaf_key() can not detect this, as
  the result never reaches the maximum length. This is now remembered
  in result_cut and reported to the user.

- All cut value reporting is moved to val_str(); dump_leaf_key() only
  marks that the result was cut. This removes the need to clear the
  truncated flag of table->blob_storage to avoid a duplicated warning,
  and gives one warning per group also when val_str() is called more
  than once for the same group, which repeated the warning before.

- Added a function comment for repack_tree() that describes where the
  rows are cut away and why building a copy frees memory.
2c0f583a065b...
2c0f583a065bdfcaf20b451922fb6caeadd72aa4
Monty
MDEV-40776 Atomic CREATE OR REPLACE silently breaks the foreign key

Give an error if one tries to drop a table referenced by a foreign keys
This is needed as innodb will keep the reference to the origina table
even when it is renamed to a temporary name as part of create or replace.

Other things:
- Changed the error message for  ER_TRUNCATE_ILLEGAL_FK to say
  "Cannot drop or truncate a table ..."
29bf6dcf9035...
29bf6dcf90359ef3768e8fb6e6c55d0549fbca63
ParadoxV5
MDEV-40366 merge fix

reäpply missed merge fixes

Co-authored-by: Brandon Nesterenko <[email protected]>
1fd028f70603...
1fd028f7060367582ec11cae7e3db603e8920be8
Brandon Nesterenko
MDEV-40768: Slave cannot apply a fragmented row event written with log_bin_compress=ON

MDEV-32570 added fragmentation of large row events. When binlog
compression is enabled (log_bin_compress=ON), the compression is
bypassed, and these binary log events remain as regular row events, when
they should be compressed row events.

This is because the compression happens during the Log_event::write()
function, whereas row data fragmentation happens before
Log_event::write() is called. The Rows_log_event super-class stores the
row data buffers to be written to disk. The regular Rows_log_event
sub-classes's implementations of Log_event::write() write this data
as-is. The compressed sub-classes's implementation of ::write()
over-write these buffers with the compressed rows data before writing to
disk.

To fragment a large row event, the server fragments the Rows_log_event's
row data buffers, to be written by multiple Partial_rows_log_event's,
and each Partial_rows_log_event::write() call writes its portion of the
row data buffer to disk. However, this happens before compression ever
has a chance to take place, and thereby, an event that should be
compressed, never is.

MDEV-39762 added event structure validation for compressed events, and
discovered that these events carry the compressed event type, but are
not actually compressed. Event validation thereby fails.

This patch adds a workaround to override the event type of the
fragmented row events to be regular row events, to be consistent with
the actual on-disk content. The underlying problem still needs to be
addressed though, and is tracked by MDEV-40851.

Signed-off-by: Brandon Nesterenko <[email protected]>
126107bfa0f9...
126107bfa0f9285fb95d036440302bab8e66fd41
Yuchen Pei
MDEV-40805 Do not call lock_rec_convert_impl_to_expl if a table S-lock is held

lock_clust_rec_read_check_and_lock() skipped the implicit-to-explicit
conversion only under a table LOCK_X. When a table LOCK_S is held the
conversion is equally pointless: no other transaction can hold an
implicit X-lock on the record, because modifying a row requires a
table LOCK_IX and LOCK_IX is incompatible with our LOCK_S.
lock_table_has() matches stronger modes, so testing LOCK_S subsumes
the old LOCK_X test.
04df838bc4f8...
04df838bc4f8a223818fe4b7ebf291168548504d
ParadoxV5
MDEV-40366 merge fix

reäpply missed merge fixes

Co-authored-by: Brandon Nesterenko <[email protected]>
BuildBot (0.8.9) working for the MariaDB project.
Page built: Thu 20 Aug 2026 06:47:46 (UTC)