Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
fd9e396c0cea...
fd9e396c0cea654f023ff31520e4d81cbf2a804f
Brandon Nesterenko
MDEV-40768: Slave cannot apply a fragmented row event written with log_bin_compress=ON

MDEV-32570 added fragmentation of large row events. When binlog
compression is enabled (log_bin_compress=ON), the compression is
bypassed, and these binary log events remain as regular row events, when
they should be compressed row events.

This is because the compression happens during the Log_event::write()
function, whereas row data fragmentation happens before
Log_event::write() is called. The Rows_log_event super-class stores the
row data buffers to be written to disk. The regular Rows_log_event
sub-classes's implementations of Log_event::write() write this data
as-is. The compressed sub-classes's implementation of ::write()
over-write these buffers with the compressed rows data before writing to
disk.

To fragment a large row event, the server fragments the Rows_log_event's
row data buffers, to be written by multiple Partial_rows_log_event's,
and each Partial_rows_log_event::write() call writes its portion of the
row data buffer to disk. However, this happens before compression ever
has a chance to take place, and thereby, an event that should be
compressed, never is.

MDEV-39762 added event structure validation for compressed events, and
discovered that these events carry the compressed event type, but are
not actually compressed. Event validation thereby fails.

This patch adds a workaround to override the event type of the
fragmented row events to be regular row events, to be consistent with
the actual on-disk content. The underlying problem still needs to be
addressed though, and is tracked by MDEV-40851.

Signed-off-by: Brandon Nesterenko <[email protected]>
fcca8d1edb0a...
fcca8d1edb0a2e789861eb6b30fec975e15248e1
Arcadiy Ivanov
MDEV-40781 Duplicate row despite DISTINCT when tmp table converts

`create_internal_tmp_table_from_heap()` writes the pending `record[0]`,
the row whose write filled the in-memory table, into the new table.
Since **MDEV-40376** (`636f154bb49`) that write happens *before*
`ha_end_bulk_insert()` rather than after it.

`ha_maria::start_bulk_insert()` disables **all** indexes of an internal
temporary table that is about to receive at least
`MARIA_MIN_ROWS_TO_DISABLE_INDEXES` (100) rows:

```c
if (file->open_flags & HA_OPEN_INTERNAL_TABLE)
{
  /* Internal table; If we get a duplicate something is very wrong */
  file->update|= HA_STATE_CHANGED;
  index_disabled= share->base.keys > 0;
  maria_clear_all_keys_active(file->s->state.key_map);
}
```

`maria_write()` then skips `_ma_check_unique()` entirely, so the unique
constraint that implements `DISTINCT` for a key too wide to be an index
is not enforced. The rows copied out of the in-memory table are already
distinct and need no checking against each other, but the pending row is
exactly the row whose duplicate status is unknown, and it was written
inside that window. A `SELECT DISTINCT` over wide columns could
therefore return a duplicate row.

Note that the justification given in `636f154bb49` is not the mechanism
at work here. It refers to the bulk insert key *tree*, a different
branch of `ha_maria::start_bulk_insert()`; setting
`bulk_insert_buffer_size=0` does not avoid the problem.

The fix splits the copy in two:

1. `Tmp_table_row_copier` gains a second virtual,
  `write_pending_row()`, defaulting to a no-op.
2. `copy_rows()` now only copies the rows the in-memory table holds.
3. `create_internal_tmp_table_from_heap()` calls
  `ha_end_bulk_insert()` and then `write_pending_row()`, so the
  pending row is written with the indexes of the new table back in
  place and a duplicate of an already copied row is detected.

`Window_rowid_remapper` keeps writing its pending row within
`copy_rows()` and inherits the no-op default. Its new position is only
known once the rows before it have been written, and nothing is lost by
writing it with the indexes still disabled: it replaces a row that is
already in the table rather than adding one, and an update of a window
function value cannot collide with another row, as a deduplicating key
is not built on the columns it changes.

The new test covers `SELECT DISTINCT`, `SELECT DISTINCT ... ORDER BY`,
`GROUP BY`, `UNION` and `INSERT ... SELECT DISTINCT`, and asserts that
the conversion actually happened so that a future sizing change cannot
silently void the coverage.
f787ac45ba70...
f787ac45ba703f1edaa70b605ee50dae66949d17
Monty
MDEV-40776 Atomic CREATE OR REPLACE silently breaks the foreign key

Give an error if one tries to drop a table referenced by foreign keys if
foreign_key_checks=1

Other things:
- Changed the error message for  ER_TRUNCATE_ILLEGAL_FK to say
  "Cannot drop or truncate a table ..."
f6243d365037...
f6243d3650370c88562a722b7186186e431252f7
Sergei Petrunia
Cleanup: move parallel cost functions from sql_parallel_workers.cc

Now they are in sql_parallel_execution.cc, but probably should go
to something like sql_parallel_optimization.cc
f09e3b61e3a0...
f09e3b61e3a03bc4a1773f98a901acd11740c583
Monty
Removed some not needed checks and add a DBUG_ASSERT() for not covered code

- In ha_partition.cc:check_parallel_search(), remove check if
  item_field->field is null. This is not needed as the function is run
  after fix_field() which guarnatees that the field is always set.
- Added DBUG_ASSERT(new_field) to Item_field::fix_fields() to check if a
  select-list item, found by name or alias when resolving ORDER BY/GROUP
  BY/HAVING, can have field == 0. This error path is not covered by any
  mtr test.
e6ff5aafc8f8...
e6ff5aafc8f809042a51f1df0b336bd83d567cfa
Monty
Fixed internal temporary buffer sizes to use tmp_memory_table_size

tmp_memory_table_size is limiting the size of internal temporary memory
tables. max_heap_table_size is there to limiting the size of explictely
created memory tables. max_heap_table_size can be much larger than
tmp_memory_table_size as the memory used by temporary tables is in the
control of the user.

This commit changes the usage of max_heap_table_size for internal buffers
to min(max_heap_table_size, tmp_memory_table_size), like we do for
internal temporary tables.

This changes the in memory buffer allocations for:
- GROUP_CONCAT()
- Calculating the cost for scanning memory tables (the original code was
  wrong here as it used the wrong size for memory tables).
- ANALYZE TABLE buffer sizes for calculating distinct column values

Other things:
- Add THD::ram_limitation() to provide consistent memory limitations
  in all code that used variables.tmp_memory_table_size as buffers.
  If tmp_memory_table_size == 0, then 8192 is used.
  This replaces Item_sum::ram_limitation which used 1024 as min buffer,
  which is way to little for any practical case.
- Added security guard in heap_prepare_hp_create_info to ensure that
  max_table_size is calculated same way as in MariaDB server.
- Fixed initial memory allocations for Item_func_group::concat which
  allocated 'max allowed memory' at start. Now it allocates only 1/16
  of that memory at start.
e11647f17bcd...
e11647f17bcd98d55f7b2924ed4174838d0e4ad1
Vladislav Vaintroub
MDEV-22992 Refactor VIO into layered transports and filters

Replace the function-pointer VIO implementation with an abstract C++
interface while retaining the procedural C entry points.

Implement socket and named-pipe transports and composable filters for
client read-ahead, Windows thread-pool prefetch, and TLS. OpenSSL uses a
custom BIO, while wolfSSL uses callbacks that perform I/O through the VIO
below the TLS filter. This keeps waits and timeouts in the transport layer.

Keep sockets nonblocking and implement timed I/O with transport waits.
Named pipes use overlapped I/O for timeout-aware waits and report blocking
waits through the same scheduler callbacks as sockets. Semi-sync
temporarily changes the real VIO read timeout instead of copying VIO state.

Hide transport and TLS implementation state behind accessors. Expose the
TLS handle opaquely and update callers that previously accessed VIO fields
directly. Compile the VIO implementations as C++ and retain PSI memory
accounting for VIO allocations.

Adapt Windows thread-pool pre-read to a Prefetched_vio filter inserted
above the transport so both plain and TLS connections consume prefetched
bytes through the same layered VIO path.
c9b1ca9c20ac...
c9b1ca9c20ac6f0c0754bb34b262d074a3c683e2
Dmitry Shulga
MDEV-40091: ASAN: heap-use-after-free with concurrent create/drop system trigger

In case there are 'on shutdown' triggers it could result in abnormal
server termination if at the moment server shutdown is in progress
the server received the DROP TRIGGER statement for one of 'ON SHUTDOWN'
system triggers being already executed as part shutdown process. Another
words, there is the race condition between running triggers on shutdown
and execution of DROP TRIGGER for system triggers ON SHUTDOWN event.

To fix the issue protect running on shutdown triggers and drop/create
of system triggers under the lock to avoid race condition.
Check under the new lock for the flag that shutdown is in progress and
don't add/remove a trigger instance into/from internal array as part of
handling CREATE/DROP TRIGGER for ON SHUTDOWN event. That is, add/drop
metadata about the trigger but don't modify the internal runtime data
structures.
c06b0ab60dc1...
c06b0ab60dc19352b0126c19cc23e154c471277e
Sergei Golubchik
fix debian versioning
bab03b0fc44e...
bab03b0fc44e7aeac869b20b0055c6a8f2083eb6
Monty
Added proper cleanup of main.cte_update_delete.test
b9f155841c1b...
b9f155841c1b0be1f8e4afc6409fff7876c20f0e
Vladislav Vaintroub
MDEV-22992 Refactor VIO into layered transports and filters

Replace the function-pointer VIO implementation with an abstract C++
interface while retaining the procedural C entry points.

Implement socket and named-pipe transports and composable filters for
client read-ahead, Windows thread-pool prefetch, and TLS. OpenSSL uses a
custom BIO, while wolfSSL uses callbacks that perform I/O through the VIO
below the TLS filter. This keeps waits and timeouts in the transport layer.

Keep sockets nonblocking and implement timed I/O with transport waits.
Named pipes use overlapped I/O for timeout-aware waits and report blocking
waits through the same scheduler callbacks as sockets. Semi-sync
temporarily changes the real VIO read timeout instead of copying VIO state.

Hide transport and TLS implementation state behind accessors. Expose the
TLS handle opaquely and update callers that previously accessed VIO fields
directly. Compile the VIO implementations as C++ and retain PSI memory
accounting for VIO allocations.

Adapt Windows thread-pool pre-read to a Prefetched_vio filter inserted
above the transport so both plain and TLS connections consume prefetched
bytes through the same layered VIO path.
a657340547b5...
a657340547b581b42d8692814b159c44292b8d5e
Alexander Barkov
MDEV-39563 Implement UPDATE ... RETURNING ... INTO

Adding support for UPDATE .. RETURNING .. INTO queries.

For example:

  UPDATE t1 SET a=10,b=20 RETURNING a,b INTO va,vb;
  UPDATE t1 SET a=10,b=20 RETURNING a,b INTO @a,@b;

Note, ANALYZE UPDATE .. RETURNING .. INTO queries work,
ignoring the INTO clause.

  ANALYZE UPDATE t1 SET a=10,b=20 RETURNING a,b INTO va,vb;

These types of queries:
- REPLACE .. RETURNING .. INTO
- DELETE .. RETURNING .. INTO
do not work - they return an error.
They will be implemented separately, when needed.
a3572b952bb3...
a3572b952bb3cf648ff407e3604ac6be07264345
Marko Mäkelä
fixup! b5501deaccc06941352c9980361266e47c09c53f
987a2a28977f...
987a2a28977f63f1d485b46ef6e3026df53e98aa
Oleksandr Byelkin
Merge branch 'br-11.4-merge' into bb-11.8-release
937af9d8bd0f...
937af9d8bd0f3ad228d4d856e33fd86c2c3645ec
Vladislav Vaintroub
MDEV-22992 Refactor VIO into layered transports and filters

Replace the function-pointer VIO implementation with an abstract C++
interface while retaining the procedural C entry points.

Implement socket and named-pipe transports and composable filters for
client read-ahead, Windows thread-pool prefetch, and TLS. OpenSSL uses a
custom BIO, while wolfSSL uses callbacks that perform I/O through the VIO
below the TLS filter. This keeps waits and timeouts in the transport layer.

Keep sockets nonblocking and implement timed I/O with transport waits.
Named pipes use overlapped I/O for timeout-aware waits and report blocking
waits through the same scheduler callbacks as sockets. Semi-sync
temporarily changes the real VIO read timeout instead of copying VIO state.

Hide transport and TLS implementation state behind accessors. Expose the
TLS handle opaquely and update callers that previously accessed VIO fields
directly. Compile the VIO implementations as C++ and retain PSI memory
accounting for VIO allocations.

Adapt Windows thread-pool pre-read to a Prefetched_vio filter inserted
above the transport so both plain and TLS connections consume prefetched
bytes through the same layered VIO path.
92d72ec2c6b8...
92d72ec2c6b8b7b8a35473068a3df84f237ac66a
Kristian Nielsen
MDEV-40729: Re-use the existing Xid_log_event and my_xid

Avoid introducing yet another transaction identifier, instead re-use the
internal XID we already use in Xid_log_event and for two-phase commit
between binlog and engines.

Extend the Xid_log_event so it contains both the connection_id and the
commit_id part of my_xid.
Use the Xid_log_event to implement client trx_status() searching for a
transaction in the binlog.

A ToDo for this patch is to put in the new-style transaction ID into
thd->transaction->implicit_xid in trans_register_ha(). In this
proof-of-concept patch, the transaction id is still supplied by the client
and the code falls back to old (query_id, server_id) when not supplied; this
should be changed.

Note that this change to Xid_log_event will break downgrades if they happen
on a crashed server. The new version will write the connection_id into the
first part of the Xid_log_event, which is no longer unique amongst all
transactions. If the server is restarted on old version after a crash, the
old version will only read the first part of the transaction id into
old-style my_xid, and it may identify the wrong transaction as committed.

Signed-off-by: Kristian Nielsen <[email protected]>
83c5d8746480...
83c5d87464801eb23e23ff4bcbc6b7c6f9b7edd4
Daniel Black
MDEV-40750 gcc-16.1.0 on ppc64 causes innodb to fail to compile

Assembler comes up with the error:
unrecognized opcode: `dcbstps'

dcbstps is a Power 10 instruction. The default target arch on most
platforms is Power 8 or 9.

Added the target power10 to the function pmem_phwsync. The execution
of this function is gated on the ISA 3.1 in pmem_persist_init so
there's no chance of a SIGILL.

clang supports this target as arch=pwr10 and gcc as cpu=power10.
Revert back to using opcodes for old versions.
833c76e091c6...
833c76e091c6cab30b48b82d811be8b61fb973da
Monty
MDEV-40729 Add functionality to verify if a transaction is committed or not after failure

In progress.
702076ee0f83...
702076ee0f839834a5c11f067bb4f246e2ad6eeb
Brandon Nesterenko
MDEV-40635 fixup
69f99014c9d4...
69f99014c9d4ad5d772aecab696165670a3d7b08
Kristian Nielsen
MDEV-40729: Add functionality to verify if a transaction is committed or not after failure

Basic proof-of-concept patch, only implements the basics to put trx_id into
the binlog file and search for it with trx_status().

A test case rpl.rpl_client_trx_id demonstrates the functionality.

Some limitations/considerations from this patch:

There is no support for MARIADB_TRX_IN_PROGRESS (do we want/need such
functionality?).

Based on 10.11, however pushing to stable 10.11 will be controversial as it
changes GTID event format.

Since 10.11 has no GTID indexes, linear scan of at least one entire binlog
file will be required. An in-memory cache may be needed to handle client
reconnect-storm after a crash or network outage that caused many ongoing
commits to fail.

When starting gtid is not specified for trx_status(), it is not possible to
distinguish between MARIADB_TRX_ABORTED or MARIADB_TRX_UNKNOWN. In this
case, we return MARIADB_TRX_ABORTED.

Knowing where to start scanning binlogs is important, for performance (to
avoid scanning _entire_ binlog history), and to distinguish
MARIADB_TRX_ABORTED from MARIADB_TRX_UNKNOWN. It is somewhat tricky
though:

- A GTID position is multi-dimensional. The starting GTID _must_ be with
  the same domain_id as the transaction being searched for, otherwise
  searching on a slave may start too late in the slave's binlog and wrongly
  return MARIADB_TRX_ABORTED for a committed transaction.

- The client will need to obtain a starting GTID for the very first
  transaction done on the connection. One possible way could be to
  SELECT @@GLOBAL.gtid_binlog_pos and pick out the one with the domain_id
  which will be used for subsequent transactions.

Signed-off-by: Kristian Nielsen <[email protected]>
688bcacb2025...
688bcacb20256281999d1056d92c9611cf0a25f7
ParadoxV5
MDEV-40636 merge fix

reäpply missed merge fixes

Co-authored-by: Brandon Nesterenko <[email protected]>
6803fb454891...
6803fb4548912e89af275cff41e7301e71ff8e6b
Brandon Nesterenko
MDEV-40768: Slave cannot apply a fragmented row event written with log_bin_compress=ON

MDEV-32570 added fragmentation of large row events. When binlog
compression is enabled (log_bin_compress=ON), the compression is
bypassed, and these binary log events remain as regular row events, when
they should be compressed row events.

This is because the compression happens during the Log_event::write()
function, whereas row data fragmentation happens before
Log_event::write() is called. The Rows_log_event super-class stores the
row data buffers to be written to disk. The regular Rows_log_event
sub-classes's implementations of Log_event::write() write this data
as-is. The compressed sub-classes's implementation of ::write()
over-write these buffers with the compressed rows data before writing to
disk.

To fragment a large row event, the server fragments the Rows_log_event's
row data buffers, to be written by multiple Partial_rows_log_event's,
and each Partial_rows_log_event::write() call writes its portion of the
row data buffer to disk. However, this happens before compression ever
has a chance to take place, and thereby, an event that should be
compressed, never is.

MDEV-39762 added event structure validation for compressed events, and
discovered that these events carry the compressed event type, but are
not actually compressed. Event validation thereby fails.

This patch adds a workaround to override the event type of the
fragmented row events to be regular row events, to be consistent with
the actual on-disk content. The underlying problem still needs to be
addressed though, and is tracked by MDEV-40851.

Signed-off-by: Brandon Nesterenko <[email protected]>
668c6c4bb850...
668c6c4bb85064d775138191152522dadca5ec72
Alexander Barkov
MDEV-39563 Implement UPDATE ... RETURNING ... INTO

Adding support for UPDATE .. RETURNING .. INTO queries.

For example:

  UPDATE t1 SET a=10,b=20 RETURNING a,b INTO va,vb;
  UPDATE t1 SET a=10,b=20 RETURNING a,b INTO @a,@b;

Note, ANALYZE UPDATE .. RETURNING .. INTO queries work,
ignoring the INTO clause.

  ANALYZE UPDATE t1 SET a=10,b=20 RETURNING a,b INTO va,vb;

These types of queries:
- REPLACE .. RETURNING .. INTO
- DELETE .. RETURNING .. INTO
do not work - they return an error.
They will be implemented separately, when needed.
6312f5fe95d6...
6312f5fe95d6a0c5f2d9f8120479b183bb1f04c4
Arcadiy Ivanov
MDEV-40802 COUNT(DISTINCT <blob>) fails when its tmp table converts

`COUNT(DISTINCT)` collects the distinct values in a temporary table
with a unique constraint over the aggregate's arguments, and treats a
duplicate key error from the write as "value already seen":

```c
if (!table->file->is_fatal_error(error, HA_CHECK_DUP))
  return FALSE;                          // duplicate, not an error
```

For a blob argument the record holds only a pointer to the value, so
`Aggregator_distinct::setup()` cannot use the `Unique` tree, which
compares raw record bytes, and every value goes through that write
instead.

When such a write overflows the in-memory table,
`create_internal_tmp_table_from_heap()` copies the stored rows to an
on-disk table and then writes the row that overflowed, which until
then was held in `record[0]` alone. Whether a duplicate key error on
that last write is fatal is decided by the caller's
`ignore_last_dupp_key_error` argument, and `Aggregator_distinct::add()`
passed **0** three lines below the code that ignores the very same
condition. The statement failed with

    ERROR 1169 (23000): Can't write, because of unique constraint,
    to table '(temporary)'

Pass **1** instead, so that a duplicate arriving through the conversion
is discarded exactly like one arriving through the ordinary write. The
result is `table->file->stats.records` of that table, so not storing
the duplicate is what makes the count right.

The argument is the same upstream, where it is unreachable: a
temporary table with a blob column was created on the on-disk engine
to begin with, so the conversion was never entered for the only
tables whose pending row can be a duplicate. Supporting blob columns
in the in-memory engine made the table start in memory and convert.

New tests `heap.count_distinct_blob_convert` and
`heap.count_distinct_blob_convert_debug`.

A write rejected as a duplicate returns its record to the free list and
never reaches the allocation of the blob value, so only the first copy
of a value makes the in-memory table grow, and the write that finds it
full is the second copy of the value stored last. That holds only while
a record slot is what the table runs out of first. Blob values come out
of the same space, and only a write that is not a duplicate ever
allocates one, so when a blob allocation is the one that hits the limit,
the pending row is not a duplicate at all.

Which of the two runs out first follows from how records and blob values
pack together, not from any threshold on the value width. Of 24 measured
combinations of width and `max_heap_table_size`, 20 convert but only 8
reach a duplicate pending row, so asserting that the table was converted
does not establish that the ignored duplicate was reached.

The first test uses widths measured to overflow on a record slot. The
second removes the dependency on that measurement, injecting the
duplicate through a new debug point in
`Tmp_table_default_copier::copy_rows()`, beside the one the row copy
loop already carries. Every value is present twice, so whichever copy
the injected duplicate discards, the other one is still written and the
count does not depend on which write overflowed.

The status counter is read with the in-memory limit restored. The status
table is materialized into a temporary table of its own, and its
VARIABLE_VALUE column is wide enough to be stored as a blob, so under
the shrunken limit that table can overflow and be converted as well, and
would then report its own conversion.
61307aec0c6a...
61307aec0c6ae2ab312793a500abfef220c2d7a1
Sergei Golubchik
disable DBUG in plugins, better deb version detection

dbug can be enabled with -UDBUG_OFF, if *really* needed
5f6f81c1ed08...
5f6f81c1ed080d3469c92da8e12d28cc3e8a10d9
Marko Mäkelä
Fix InnoDB_backup::context::de_hardlink()
53b9b3441464...
53b9b34414648df8a2b067e6aa31629c8e844f6b
Georgi (Joro) Kodinov
day's work:  the beginning of the top level structure
5301674b0151...
5301674b0151284552efc9690e9ce9fc9d93480c
Monty
Limit the memory used by GROUP_CONCAT() with ORDER BY

GROUP_CONCAT() with ORDER BY collects all rows of the group in a TREE
and only cuts it down in repack_tree(). The repack was triggered by

  (tree_len >> GCONCAT_REPACK_FACTOR) > thd->gconcat_max_len()

with GCONCAT_REPACK_FACTOR 10, that is when the rows in the tree had
produced 1024 * group_concat_max_len bytes, or 1G with the default
settings. On top of that tree_len only counted the length of the
strings, while the tree costs sizeof(TREE_ELEMENT) + reclength per row.
For GROUP_CONCAT(int_col ORDER BY int_col) that is about 40 bytes per
row against 6 bytes of result, so the tree had grown to several GB
before the first repack. In practice the server ran out of memory
first and the repack code was close to never used.

The tree is now limited by the memory it has really allocated,
tree->allocated, instead of by the length of the strings it holds.
The limit is MY_MAX(thd->ram_limitation(), thd->gconcat_max_len()) and
is never set so low that the tree can not hold a few rows.

repack_tree() builds a new tree while the old one is still in memory,
so the peak usage is the size we start the repack at plus the size we
copy to. To keep the sum within the limit it is split into
GCONCAT_TREE_PARTS parts; the repack starts when
GCONCAT_TREE_REPACK_PARTS of them are used and copies to the remaining
part. The part we do not copy to is also the room the tree has to grow
before the next repack, which keeps the repacks amortized.

Other changes:

- tree_len is removed. It was only read by the old trigger.

- repack_tree() decided that it had run out of memory by testing
  st.len <= st.maxlen after the walk. That test was only valid because
  the old trigger guaranteed that a complete copy had to overshoot
  st.maxlen. A repack triggered by memory can complete the walk with
  st.len far below st.maxlen, which would have failed the query with a
  wrong out of memory error. There is now an explicit flag for it.

- The length that decides which rows to keep now also counts the
  separator that is put between two rows, so that it matches what
  val_str() will produce.

- When the memory limit stops the copy, the result becomes shorter
  than group_concat_max_len. dump_leaf_key() can not detect this, as
  the result never reaches the maximum length. This is now remembered
  in result_cut and reported to the user.

- All cut value reporting is moved to val_str(); dump_leaf_key() only
  marks that the result was cut. This removes the need to clear the
  truncated flag of table->blob_storage to avoid a duplicated warning,
  and gives one warning per group also when val_str() is called more
  than once for the same group, which repeated the warning before.

- Added a function comment for repack_tree() that describes where the
  rows are cut away and why building a copy frees memory.
4e681204d7e8...
4e681204d7e8bdbe2058ffeaf55afeeecd34f686
Arcadiy Ivanov
MDEV-40802 COUNT(DISTINCT <blob>) fails when its tmp table converts

`COUNT(DISTINCT)` collects the distinct values in a temporary table
with a unique constraint over the aggregate's arguments, and treats a
duplicate key error from the write as "value already seen":

```c
if (!table->file->is_fatal_error(error, HA_CHECK_DUP))
  return FALSE;                          // duplicate, not an error
```

For a blob argument the record holds only a pointer to the value, so
`Aggregator_distinct::setup()` cannot use the `Unique` tree, which
compares raw record bytes, and every value goes through that write
instead.

When such a write overflows the in-memory table,
`create_internal_tmp_table_from_heap()` copies the stored rows to an
on-disk table and then writes the row that overflowed, which until
then was held in `record[0]` alone. Whether a duplicate key error on
that last write is fatal is decided by the caller's
`ignore_last_dupp_key_error` argument, and `Aggregator_distinct::add()`
passed **0** three lines below the code that ignores the very same
condition. The statement failed with

    ERROR 1169 (23000): Can't write, because of unique constraint,
    to table '(temporary)'

Pass **1** instead, so that a duplicate arriving through the conversion
is discarded exactly like one arriving through the ordinary write. The
result is `table->file->stats.records` of that table, so not storing
the duplicate is what makes the count right.

The argument is the same upstream, where it is unreachable: a
temporary table with a blob column was created on the on-disk engine
to begin with, so the conversion was never entered for the only
tables whose pending row can be a duplicate. Supporting blob columns
in the in-memory engine made the table start in memory and convert.

New tests `heap.count_distinct_blob_convert` and
`heap.count_distinct_blob_convert_debug`.

A write rejected as a duplicate returns its record to the free list and
never reaches the allocation of the blob value, so only the first copy
of a value makes the in-memory table grow, and the write that finds it
full is the second copy of the value stored last. That holds only while
a record slot is what the table runs out of first. Blob values come out
of the same space, and only a write that is not a duplicate ever
allocates one, so when a blob allocation is the one that hits the limit,
the pending row is not a duplicate at all.

Which of the two runs out first follows from how records and blob values
pack together, not from any threshold on the value width. Of 24 measured
combinations of width and `max_heap_table_size`, 20 convert but only 8
reach a duplicate pending row, so asserting that the table was converted
does not establish that the ignored duplicate was reached.

The first test uses widths measured to overflow on a record slot. The
second removes the dependency on that measurement, injecting the
duplicate through a new debug point in
`Tmp_table_default_copier::copy_rows()`, beside the one the row copy
loop already carries. Every value is present twice, so whichever copy
the injected duplicate discards, the other one is still written and the
count does not depend on which write overflowed.

The status counter is read with the in-memory limit restored. The status
table is materialized into a temporary table of its own, and its
VARIABLE_VALUE column is wide enough to be stored as a blob, so under
the shrunken limit that table can overflow and be converted as well, and
would then report its own conversion.
4a03ee87ce1a...
4a03ee87ce1a0046629ec99ede18823141585880
Brandon Nesterenko
MDEV-40768: Slave cannot apply a fragmented row event written with log_bin_compress=ON

MDEV-32570 added fragmentation of large row events. When binlog
compression is enabled (log_bin_compress=ON), the compression is
bypassed, and these binary log events remain as regular row events, when
they should be compressed row events.

This is because the compression happens during the Log_event::write()
function, whereas row data fragmentation happens before
Log_event::write() is called. The Rows_log_event super-class stores the
row data buffers to be written to disk. The regular Rows_log_event
sub-classes's implementations of Log_event::write() write this data
as-is. The compressed sub-classes's implementation of ::write()
over-write these buffers with the compressed rows data before writing to
disk.

To fragment a large row event, the server fragments the Rows_log_event's
row data buffers, to be written by multiple Partial_rows_log_event's,
and each Partial_rows_log_event::write() call writes its portion of the
row data buffer to disk. However, this happens before compression ever
has a chance to take place, and thereby, an event that should be
compressed, never is.

MDEV-39762 added event structure validation for compressed events, and
discovered that these events carry the compressed event type, but are
not actually compressed. Event validation thereby fails.

This patch adds a workaround to override the event type of the
fragmented row events to be regular row events, to be consistent with
the actual on-disk content. The underlying problem still needs to be
addressed though, and is tracked by MDEV-40851.

Signed-off-by: Brandon Nesterenko <[email protected]>
43946bab13fe...
43946bab13fe78cdc3f55f0b2f2af887dfceddb3
Marko Mäkelä
fixup! 5f6f81c1ed080d3469c92da8e12d28cc3e8a10d9

Partially revert a non-working fix.
3b8db4540386...
3b8db4540386791bb0fd2fbf53f397f9ad8c19a0
Oleksandr Byelkin
Merge branch '11.4' into 11.8
37077ccef15c...
37077ccef15cae879095ffdf311e663769936679
Monty
Limit the memory used by GROUP_CONCAT() with ORDER BY

GROUP_CONCAT() with ORDER BY collects all rows of the group in a TREE
and only cuts it down in repack_tree(). The repack was triggered by

  (tree_len >> GCONCAT_REPACK_FACTOR) > thd->gconcat_max_len()

with GCONCAT_REPACK_FACTOR 10, that is when the rows in the tree had
produced 1024 * group_concat_max_len bytes, or 1G with the default
settings. On top of that tree_len only counted the length of the
strings, while the tree costs sizeof(TREE_ELEMENT) + reclength per row.
For GROUP_CONCAT(int_col ORDER BY int_col) that is about 40 bytes per
row against 6 bytes of result, so the tree had grown to several GB
before the first repack. In practice the server ran out of memory
first and the repack code was close to never used.

The tree is now limited by the memory it has really allocated,
tree->allocated, instead of by the length of the strings it holds.
The limit is MY_MAX(thd->ram_limitation(), thd->gconcat_max_len()) and
is never set so low that the tree can not hold a few rows.

repack_tree() builds a new tree while the old one is still in memory,
so the peak usage is the size we start the repack at plus the size we
copy to. To keep the sum within the limit it is split into
GCONCAT_TREE_PARTS parts; the repack starts when
GCONCAT_TREE_REPACK_PARTS of them are used and copies to the remaining
part. The part we do not copy to is also the room the tree has to grow
before the next repack, which keeps the repacks amortized.

Other changes:

- tree_len is removed. It was only read by the old trigger.

- repack_tree() decided that it had run out of memory by testing
  st.len <= st.maxlen after the walk. That test was only valid because
  the old trigger guaranteed that a complete copy had to overshoot
  st.maxlen. A repack triggered by memory can complete the walk with
  st.len far below st.maxlen, which would have failed the query with a
  wrong out of memory error. There is now an explicit flag for it.

- The length that decides which rows to keep now also counts the
  separator that is put between two rows, so that it matches what
  val_str() will produce.

- When the memory limit stops the copy, the result becomes shorter
  than group_concat_max_len. dump_leaf_key() can not detect this, as
  the result never reaches the maximum length. This is now remembered
  in result_cut and reported to the user.

- All cut value reporting is moved to val_str(); dump_leaf_key() only
  marks that the result was cut. This removes the need to clear the
  truncated flag of table->blob_storage to avoid a duplicated warning,
  and gives one warning per group also when val_str() is called more
  than once for the same group, which repeated the warning before.

- Added a function comment for repack_tree() that describes where the
  rows are cut away and why building a copy frees memory.
3559681489c1...
3559681489c153c8ded4f951991ee71ff3ddcd5d
Kristian Nielsen
MDEV-40729: Revert extension to GTID event with new transaction id.

Since we switched to using the Xid_log_event to have the transaction ID, do
not extend the GTID event with the id.

Kept as a separate commit for now to better isolate the different changes
for easier review and working.

Signed-off-by: Kristian Nielsen <[email protected]>
3471f6a5d33e...
3471f6a5d33ebff282c965bd47f7992abbea46e9
Arcadiy Ivanov
MDEV-40781 Duplicate row despite DISTINCT when tmp table converts

`create_internal_tmp_table_from_heap()` writes the pending `record[0]`,
the row whose write filled the in-memory table, into the new table.
Since **MDEV-40376** (`636f154bb49`) that write happens *before*
`ha_end_bulk_insert()` rather than after it.

`ha_maria::start_bulk_insert()` disables **all** indexes of an internal
temporary table that is about to receive at least
`MARIA_MIN_ROWS_TO_DISABLE_INDEXES` (100) rows:

```c
if (file->open_flags & HA_OPEN_INTERNAL_TABLE)
{
  /* Internal table; If we get a duplicate something is very wrong */
  file->update|= HA_STATE_CHANGED;
  index_disabled= share->base.keys > 0;
  maria_clear_all_keys_active(file->s->state.key_map);
}
```

`maria_write()` then skips `_ma_check_unique()` entirely, so the unique
constraint that implements `DISTINCT` for a key too wide to be an index
is not enforced. The rows copied out of the in-memory table are already
distinct and need no checking against each other, but the pending row is
exactly the row whose duplicate status is unknown, and it was written
inside that window. A `SELECT DISTINCT` over wide columns could
therefore return a duplicate row.

Note that the justification given in `636f154bb49` is not the mechanism
at work here. It refers to the bulk insert key *tree*, a different
branch of `ha_maria::start_bulk_insert()`; setting
`bulk_insert_buffer_size=0` does not avoid the problem.

The fix splits the copy in two:

1. `Tmp_table_row_copier` gains a second virtual,
  `write_pending_row()`, defaulting to a no-op.
2. `copy_rows()` now only copies the rows the in-memory table holds.
3. `create_internal_tmp_table_from_heap()` calls
  `ha_end_bulk_insert()` and then `write_pending_row()`, so the
  pending row is written with the indexes of the new table back in
  place and a duplicate of an already copied row is detected.

`Window_rowid_remapper` keeps writing its pending row within
`copy_rows()` and inherits the no-op default. Its new position is only
known once the rows before it have been written, and nothing is lost by
writing it with the indexes still disabled: it replaces a row that is
already in the table rather than adding one, and an update of a window
function value cannot collide with another row, as a deduplicating key
is not built on the columns it changes.

The new test covers `SELECT DISTINCT`, `SELECT DISTINCT ... ORDER BY`,
`UNION` and `INSERT ... SELECT DISTINCT`, and asserts that the
conversion actually happened so that a future sizing change cannot
silently void the coverage.
2c0f583a065b...
2c0f583a065bdfcaf20b451922fb6caeadd72aa4
Monty
MDEV-40776 Atomic CREATE OR REPLACE silently breaks the foreign key

Give an error if one tries to drop a table referenced by a foreign keys
This is needed as innodb will keep the reference to the origina table
even when it is renamed to a temporary name as part of create or replace.

Other things:
- Changed the error message for  ER_TRUNCATE_ILLEGAL_FK to say
  "Cannot drop or truncate a table ..."
04df838bc4f8...
04df838bc4f8a223818fe4b7ebf291168548504d
ParadoxV5
MDEV-40366 merge fix

reäpply missed merge fixes

Co-authored-by: Brandon Nesterenko <[email protected]>
BuildBot (0.8.9) working for the MariaDB project.
Page built: Wed 19 Aug 2026 23:19:37 (UTC)