Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Brandon Nesterenko
MDEV-40768: Slave cannot apply a fragmented row event written with log_bin_compress=ON

MDEV-32570 added fragmentation of large row events. When binlog
compression is enabled (log_bin_compress=ON), the compression is
bypassed, and these binary log events remain as regular row events, when
they should be compressed row events.

This is because the compression happens during the Log_event::write()
function, whereas row data fragmentation happens before
Log_event::write() is called. The Rows_log_event super-class stores the
row data buffers to be written to disk. The regular Rows_log_event
sub-classes's implementations of Log_event::write() write this data
as-is. The compressed sub-classes's implementation of ::write()
over-write these buffers with the compressed rows data before writing to
disk.

To fragment a large row event, the server fragments the Rows_log_event's
row data buffers, to be written by multiple Partial_rows_log_event's,
and each Partial_rows_log_event::write() call writes its portion of the
row data buffer to disk. However, this happens before compression ever
has a chance to take place, and thereby, an event that should be
compressed, never is.

MDEV-39762 added event structure validation for compressed events, and
discovered that these events carry the compressed event type, but are
not actually compressed. Event validation thereby fails.

This patch adds a workaround to override the event type of the
fragmented row events to be regular row events, to be consistent with
the actual on-disk content. The underlying problem still needs to be
addressed though, and is tracked by MDEV-40851.

Signed-off-by: Brandon Nesterenko <[email protected]>
Arcadiy Ivanov
MDEV-40781 Duplicate row despite DISTINCT when tmp table converts

`create_internal_tmp_table_from_heap()` writes the pending `record[0]`,
the row whose write filled the in-memory table, into the new table.
Since **MDEV-40376** (`636f154bb49`) that write happens *before*
`ha_end_bulk_insert()` rather than after it.

`ha_maria::start_bulk_insert()` disables **all** indexes of an internal
temporary table that is about to receive at least
`MARIA_MIN_ROWS_TO_DISABLE_INDEXES` (100) rows:

```c
if (file->open_flags & HA_OPEN_INTERNAL_TABLE)
{
  /* Internal table; If we get a duplicate something is very wrong */
  file->update|= HA_STATE_CHANGED;
  index_disabled= share->base.keys > 0;
  maria_clear_all_keys_active(file->s->state.key_map);
}
```

`maria_write()` then skips `_ma_check_unique()` entirely, so the unique
constraint that implements `DISTINCT` for a key too wide to be an index
is not enforced. The rows copied out of the in-memory table are already
distinct and need no checking against each other, but the pending row is
exactly the row whose duplicate status is unknown, and it was written
inside that window. A `SELECT DISTINCT` over wide columns could
therefore return a duplicate row.

Note that the justification given in `636f154bb49` is not the mechanism
at work here. It refers to the bulk insert key *tree*, a different
branch of `ha_maria::start_bulk_insert()`; setting
`bulk_insert_buffer_size=0` does not avoid the problem.

The fix splits the copy in two:

1. `Tmp_table_row_copier` gains a second virtual,
  `write_pending_row()`, defaulting to a no-op.
2. `copy_rows()` now only copies the rows the in-memory table holds.
3. `create_internal_tmp_table_from_heap()` calls
  `ha_end_bulk_insert()` and then `write_pending_row()`, so the
  pending row is written with the indexes of the new table back in
  place and a duplicate of an already copied row is detected.

`Window_rowid_remapper` keeps writing its pending row within
`copy_rows()` and inherits the no-op default. Its new position is only
known once the rows before it have been written, and nothing is lost by
writing it with the indexes still disabled: it replaces a row that is
already in the table rather than adding one, and an update of a window
function value cannot collide with another row, as a deduplicating key
is not built on the columns it changes.

The new test covers `SELECT DISTINCT`, `SELECT DISTINCT ... ORDER BY`,
`GROUP BY`, `UNION` and `INSERT ... SELECT DISTINCT`, and asserts that
the conversion actually happened so that a future sizing change cannot
silently void the coverage.
Monty
MDEV-40776 Atomic CREATE OR REPLACE silently breaks the foreign key

Give an error if one tries to drop a table referenced by foreign keys if
foreign_key_checks=1

Other things:
- Changed the error message for  ER_TRUNCATE_ILLEGAL_FK to say
  "Cannot drop or truncate a table ..."
Sergei Petrunia
Cleanup: move parallel cost functions from sql_parallel_workers.cc

Now they are in sql_parallel_execution.cc, but probably should go
to something like sql_parallel_optimization.cc
Monty
Removed some not needed checks and add a DBUG_ASSERT() for not covered code

- In ha_partition.cc:check_parallel_search(), remove check if
  item_field->field is null. This is not needed as the function is run
  after fix_field() which guarnatees that the field is always set.
- Added DBUG_ASSERT(new_field) to Item_field::fix_fields() to check if a
  select-list item, found by name or alias when resolving ORDER BY/GROUP
  BY/HAVING, can have field == 0. This error path is not covered by any
  mtr test.
Monty
Fixed internal temporary buffer sizes to use tmp_memory_table_size

tmp_memory_table_size is limiting the size of internal temporary memory
tables. max_heap_table_size is there to limiting the size of explictely
created memory tables. max_heap_table_size can be much larger than
tmp_memory_table_size as the memory used by temporary tables is in the
control of the user.

This commit changes the usage of max_heap_table_size for internal buffers
to min(max_heap_table_size, tmp_memory_table_size), like we do for
internal temporary tables.

This changes the in memory buffer allocations for:
- GROUP_CONCAT()
- Calculating the cost for scanning memory tables (the original code was
  wrong here as it used the wrong size for memory tables).
- ANALYZE TABLE buffer sizes for calculating distinct column values

Other things:
- Add THD::ram_limitation() to provide consistent memory limitations
  in all code that used variables.tmp_memory_table_size as buffers.
  If tmp_memory_table_size == 0, then 8192 is used.
  This replaces Item_sum::ram_limitation which used 1024 as min buffer,
  which is way to little for any practical case.
- Added security guard in heap_prepare_hp_create_info to ensure that
  max_table_size is calculated same way as in MariaDB server.
- Fixed initial memory allocations for Item_func_group::concat which
  allocated 'max allowed memory' at start. Now it allocates only 1/16
  of that memory at start.
Vladislav Vaintroub
MDEV-22992 Refactor VIO into layered transports and filters

Replace the function-pointer VIO implementation with an abstract C++
interface while retaining the procedural C entry points.

Implement socket and named-pipe transports and composable filters for
client read-ahead, Windows thread-pool prefetch, and TLS. OpenSSL uses a
custom BIO, while wolfSSL uses callbacks that perform I/O through the VIO
below the TLS filter. This keeps waits and timeouts in the transport layer.

Keep sockets nonblocking and implement timed I/O with transport waits.
Named pipes use overlapped I/O for timeout-aware waits and report blocking
waits through the same scheduler callbacks as sockets. Semi-sync
temporarily changes the real VIO read timeout instead of copying VIO state.

Hide transport and TLS implementation state behind accessors. Expose the
TLS handle opaquely and update callers that previously accessed VIO fields
directly. Compile the VIO implementations as C++ and retain PSI memory
accounting for VIO allocations.

Adapt Windows thread-pool pre-read to a Prefetched_vio filter inserted
above the transport so both plain and TLS connections consume prefetched
bytes through the same layered VIO path.
Dmitry Shulga
MDEV-40091: ASAN: heap-use-after-free with concurrent create/drop system trigger

In case there are 'on shutdown' triggers it could result in abnormal
server termination if at the moment server shutdown is in progress
the server received the DROP TRIGGER statement for one of 'ON SHUTDOWN'
system triggers being already executed as part shutdown process. Another
words, there is the race condition between running triggers on shutdown
and execution of DROP TRIGGER for system triggers ON SHUTDOWN event.

To fix the issue protect running on shutdown triggers and drop/create
of system triggers under the lock to avoid race condition.
Check under the new lock for the flag that shutdown is in progress and
don't add/remove a trigger instance into/from internal array as part of
handling CREATE/DROP TRIGGER for ON SHUTDOWN event. That is, add/drop
metadata about the trigger but don't modify the internal runtime data
structures.
Sergei Golubchik
fix debian versioning
Monty
Added proper cleanup of main.cte_update_delete.test
Vladislav Vaintroub
MDEV-22992 Refactor VIO into layered transports and filters

Replace the function-pointer VIO implementation with an abstract C++
interface while retaining the procedural C entry points.

Implement socket and named-pipe transports and composable filters for
client read-ahead, Windows thread-pool prefetch, and TLS. OpenSSL uses a
custom BIO, while wolfSSL uses callbacks that perform I/O through the VIO
below the TLS filter. This keeps waits and timeouts in the transport layer.

Keep sockets nonblocking and implement timed I/O with transport waits.
Named pipes use overlapped I/O for timeout-aware waits and report blocking
waits through the same scheduler callbacks as sockets. Semi-sync
temporarily changes the real VIO read timeout instead of copying VIO state.

Hide transport and TLS implementation state behind accessors. Expose the
TLS handle opaquely and update callers that previously accessed VIO fields
directly. Compile the VIO implementations as C++ and retain PSI memory
accounting for VIO allocations.

Adapt Windows thread-pool pre-read to a Prefetched_vio filter inserted
above the transport so both plain and TLS connections consume prefetched
bytes through the same layered VIO path.
Alexander Barkov
MDEV-39563 Implement UPDATE ... RETURNING ... INTO

Adding support for UPDATE .. RETURNING .. INTO queries.

For example:

  UPDATE t1 SET a=10,b=20 RETURNING a,b INTO va,vb;
  UPDATE t1 SET a=10,b=20 RETURNING a,b INTO @a,@b;

Note, ANALYZE UPDATE .. RETURNING .. INTO queries work,
ignoring the INTO clause.

  ANALYZE UPDATE t1 SET a=10,b=20 RETURNING a,b INTO va,vb;

These types of queries:
- REPLACE .. RETURNING .. INTO
- DELETE .. RETURNING .. INTO
do not work - they return an error.
They will be implemented separately, when needed.
Marko MƤkelƤ
fixup! b5501deaccc06941352c9980361266e47c09c53f
Oleksandr Byelkin
Merge branch 'br-11.4-merge' into bb-11.8-release
Vladislav Vaintroub
MDEV-22992 Refactor VIO into layered transports and filters

Replace the function-pointer VIO implementation with an abstract C++
interface while retaining the procedural C entry points.

Implement socket and named-pipe transports and composable filters for
client read-ahead, Windows thread-pool prefetch, and TLS. OpenSSL uses a
custom BIO, while wolfSSL uses callbacks that perform I/O through the VIO
below the TLS filter. This keeps waits and timeouts in the transport layer.

Keep sockets nonblocking and implement timed I/O with transport waits.
Named pipes use overlapped I/O for timeout-aware waits and report blocking
waits through the same scheduler callbacks as sockets. Semi-sync
temporarily changes the real VIO read timeout instead of copying VIO state.

Hide transport and TLS implementation state behind accessors. Expose the
TLS handle opaquely and update callers that previously accessed VIO fields
directly. Compile the VIO implementations as C++ and retain PSI memory
accounting for VIO allocations.

Adapt Windows thread-pool pre-read to a Prefetched_vio filter inserted
above the transport so both plain and TLS connections consume prefetched
bytes through the same layered VIO path.
Kristian Nielsen
MDEV-40729: Re-use the existing Xid_log_event and my_xid

Avoid introducing yet another transaction identifier, instead re-use the
internal XID we already use in Xid_log_event and for two-phase commit
between binlog and engines.

Extend the Xid_log_event so it contains both the connection_id and the
commit_id part of my_xid.
Use the Xid_log_event to implement client trx_status() searching for a
transaction in the binlog.

A ToDo for this patch is to put in the new-style transaction ID into
thd->transaction->implicit_xid in trans_register_ha(). In this
proof-of-concept patch, the transaction id is still supplied by the client
and the code falls back to old (query_id, server_id) when not supplied; this
should be changed.

Note that this change to Xid_log_event will break downgrades if they happen
on a crashed server. The new version will write the connection_id into the
first part of the Xid_log_event, which is no longer unique amongst all
transactions. If the server is restarted on old version after a crash, the
old version will only read the first part of the transaction id into
old-style my_xid, and it may identify the wrong transaction as committed.

Signed-off-by: Kristian Nielsen <[email protected]>
Daniel Black
MDEV-40750 gcc-16.1.0 on ppc64 causes innodb to fail to compile

Assembler comes up with the error:
unrecognized opcode: `dcbstps'

dcbstps is a Power 10 instruction. The default target arch on most
platforms is Power 8 or 9.

Added the target power10 to the function pmem_phwsync. The execution
of this function is gated on the ISA 3.1 in pmem_persist_init so
there's no chance of a SIGILL.

clang supports this target as arch=pwr10 and gcc as cpu=power10.
Revert back to using opcodes for old versions.
Monty
MDEV-40729 Add functionality to verify if a transaction is committed or not after failure

In progress.
Brandon Nesterenko
MDEV-40635 fixup
Kristian Nielsen
MDEV-40729: Add functionality to verify if a transaction is committed or not after failure

Basic proof-of-concept patch, only implements the basics to put trx_id into
the binlog file and search for it with trx_status().

A test case rpl.rpl_client_trx_id demonstrates the functionality.

Some limitations/considerations from this patch:

There is no support for MARIADB_TRX_IN_PROGRESS (do we want/need such
functionality?).

Based on 10.11, however pushing to stable 10.11 will be controversial as it
changes GTID event format.

Since 10.11 has no GTID indexes, linear scan of at least one entire binlog
file will be required. An in-memory cache may be needed to handle client
reconnect-storm after a crash or network outage that caused many ongoing
commits to fail.

When starting gtid is not specified for trx_status(), it is not possible to
distinguish between MARIADB_TRX_ABORTED or MARIADB_TRX_UNKNOWN. In this
case, we return MARIADB_TRX_ABORTED.

Knowing where to start scanning binlogs is important, for performance (to
avoid scanning _entire_ binlog history), and to distinguish
MARIADB_TRX_ABORTED from MARIADB_TRX_UNKNOWN. It is somewhat tricky
though:

- A GTID position is multi-dimensional. The starting GTID _must_ be with
  the same domain_id as the transaction being searched for, otherwise
  searching on a slave may start too late in the slave's binlog and wrongly
  return MARIADB_TRX_ABORTED for a committed transaction.

- The client will need to obtain a starting GTID for the very first
  transaction done on the connection. One possible way could be to
  SELECT @@GLOBAL.gtid_binlog_pos and pick out the one with the domain_id
  which will be used for subsequent transactions.

Signed-off-by: Kristian Nielsen <[email protected]>
ParadoxV5
MDEV-40636 merge fix

reƤpply missed merge fixes

Co-authored-by: Brandon Nesterenko <[email protected]>
Brandon Nesterenko
MDEV-40768: Slave cannot apply a fragmented row event written with log_bin_compress=ON

MDEV-32570 added fragmentation of large row events. When binlog
compression is enabled (log_bin_compress=ON), the compression is
bypassed, and these binary log events remain as regular row events, when
they should be compressed row events.

This is because the compression happens during the Log_event::write()
function, whereas row data fragmentation happens before
Log_event::write() is called. The Rows_log_event super-class stores the
row data buffers to be written to disk. The regular Rows_log_event
sub-classes's implementations of Log_event::write() write this data
as-is. The compressed sub-classes's implementation of ::write()
over-write these buffers with the compressed rows data before writing to
disk.

To fragment a large row event, the server fragments the Rows_log_event's
row data buffers, to be written by multiple Partial_rows_log_event's,
and each Partial_rows_log_event::write() call writes its portion of the
row data buffer to disk. However, this happens before compression ever
has a chance to take place, and thereby, an event that should be
compressed, never is.

MDEV-39762 added event structure validation for compressed events, and
discovered that these events carry the compressed event type, but are
not actually compressed. Event validation thereby fails.

This patch adds a workaround to override the event type of the
fragmented row events to be regular row events, to be consistent with
the actual on-disk content. The underlying problem still needs to be
addressed though, and is tracked by MDEV-40851.

Signed-off-by: Brandon Nesterenko <[email protected]>
Alexander Barkov
MDEV-39563 Implement UPDATE ... RETURNING ... INTO

Adding support for UPDATE .. RETURNING .. INTO queries.

For example:

  UPDATE t1 SET a=10,b=20 RETURNING a,b INTO va,vb;
  UPDATE t1 SET a=10,b=20 RETURNING a,b INTO @a,@b;

Note, ANALYZE UPDATE .. RETURNING .. INTO queries work,
ignoring the INTO clause.

  ANALYZE UPDATE t1 SET a=10,b=20 RETURNING a,b INTO va,vb;

These types of queries:
- REPLACE .. RETURNING .. INTO
- DELETE .. RETURNING .. INTO
do not work - they return an error.
They will be implemented separately, when needed.
Arcadiy Ivanov
MDEV-40802 COUNT(DISTINCT <blob>) fails when its tmp table converts

`COUNT(DISTINCT)` collects the distinct values in a temporary table
with a unique constraint over the aggregate's arguments, and treats a
duplicate key error from the write as "value already seen":

```c
if (!table->file->is_fatal_error(error, HA_CHECK_DUP))
  return FALSE;                          // duplicate, not an error
```

For a blob argument the record holds only a pointer to the value, so
`Aggregator_distinct::setup()` cannot use the `Unique` tree, which
compares raw record bytes, and every value goes through that write
instead.

When such a write overflows the in-memory table,
`create_internal_tmp_table_from_heap()` copies the stored rows to an
on-disk table and then writes the row that overflowed, which until
then was held in `record[0]` alone. Whether a duplicate key error on
that last write is fatal is decided by the caller's
`ignore_last_dupp_key_error` argument, and `Aggregator_distinct::add()`
passed **0** three lines below the code that ignores the very same
condition. The statement failed with

    ERROR 1169 (23000): Can't write, because of unique constraint,
    to table '(temporary)'

Pass **1** instead, so that a duplicate arriving through the conversion
is discarded exactly like one arriving through the ordinary write. The
result is `table->file->stats.records` of that table, so not storing
the duplicate is what makes the count right.

The argument is the same upstream, where it is unreachable: a
temporary table with a blob column was created on the on-disk engine
to begin with, so the conversion was never entered for the only
tables whose pending row can be a duplicate. Supporting blob columns
in the in-memory engine made the table start in memory and convert.

New tests `heap.count_distinct_blob_convert` and
`heap.count_distinct_blob_convert_debug`.

A write rejected as a duplicate returns its record to the free list and
never reaches the allocation of the blob value, so only the first copy
of a value makes the in-memory table grow, and the write that finds it
full is the second copy of the value stored last. That holds only while
a record slot is what the table runs out of first. Blob values come out
of the same space, and only a write that is not a duplicate ever
allocates one, so when a blob allocation is the one that hits the limit,
the pending row is not a duplicate at all.

Which of the two runs out first follows from how records and blob values
pack together, not from any threshold on the value width. Of 24 measured
combinations of width and `max_heap_table_size`, 20 convert but only 8
reach a duplicate pending row, so asserting that the table was converted
does not establish that the ignored duplicate was reached.

The first test uses widths measured to overflow on a record slot. The
second removes the dependency on that measurement, injecting the
duplicate through a new debug point in
`Tmp_table_default_copier::copy_rows()`, beside the one the row copy
loop already carries. Every value is present twice, so whichever copy
the injected duplicate discards, the other one is still written and the
count does not depend on which write overflowed.

The status counter is read with the in-memory limit restored. The status
table is materialized into a temporary table of its own, and its
VARIABLE_VALUE column is wide enough to be stored as a blob, so under
the shrunken limit that table can overflow and be converted as well, and
would then report its own conversion.
Sergei Golubchik
disable DBUG in plugins, better deb version detection

dbug can be enabled with -UDBUG_OFF, if *really* needed
Marko MƤkelƤ
Fix InnoDB_backup::context::de_hardlink()
Georgi (Joro) Kodinov
day's work:  the beginning of the top level structure
Monty
Limit the memory used by GROUP_CONCAT() with ORDER BY

GROUP_CONCAT() with ORDER BY collects all rows of the group in a TREE
and only cuts it down in repack_tree(). The repack was triggered by

  (tree_len >> GCONCAT_REPACK_FACTOR) > thd->gconcat_max_len()

with GCONCAT_REPACK_FACTOR 10, that is when the rows in the tree had
produced 1024 * group_concat_max_len bytes, or 1G with the default
settings. On top of that tree_len only counted the length of the
strings, while the tree costs sizeof(TREE_ELEMENT) + reclength per row.
For GROUP_CONCAT(int_col ORDER BY int_col) that is about 40 bytes per
row against 6 bytes of result, so the tree had grown to several GB
before the first repack. In practice the server ran out of memory
first and the repack code was close to never used.

The tree is now limited by the memory it has really allocated,
tree->allocated, instead of by the length of the strings it holds.
The limit is MY_MAX(thd->ram_limitation(), thd->gconcat_max_len()) and
is never set so low that the tree can not hold a few rows.

repack_tree() builds a new tree while the old one is still in memory,
so the peak usage is the size we start the repack at plus the size we
copy to. To keep the sum within the limit it is split into
GCONCAT_TREE_PARTS parts; the repack starts when
GCONCAT_TREE_REPACK_PARTS of them are used and copies to the remaining
part. The part we do not copy to is also the room the tree has to grow
before the next repack, which keeps the repacks amortized.

Other changes:

- tree_len is removed. It was only read by the old trigger.

- repack_tree() decided that it had run out of memory by testing
  st.len <= st.maxlen after the walk. That test was only valid because
  the old trigger guaranteed that a complete copy had to overshoot
  st.maxlen. A repack triggered by memory can complete the walk with
  st.len far below st.maxlen, which would have failed the query with a
  wrong out of memory error. There is now an explicit flag for it.

- The length that decides which rows to keep now also counts the
  separator that is put between two rows, so that it matches what
  val_str() will produce.

- When the memory limit stops the copy, the result becomes shorter
  than group_concat_max_len. dump_leaf_key() can not detect this, as
  the result never reaches the maximum length. This is now remembered
  in result_cut and reported to the user.

- All cut value reporting is moved to val_str(); dump_leaf_key() only
  marks that the result was cut. This removes the need to clear the
  truncated flag of table->blob_storage to avoid a duplicated warning,
  and gives one warning per group also when val_str() is called more
  than once for the same group, which repeated the warning before.

- Added a function comment for repack_tree() that describes where the
  rows are cut away and why building a copy frees memory.
Arcadiy Ivanov
MDEV-40802 COUNT(DISTINCT <blob>) fails when its tmp table converts

`COUNT(DISTINCT)` collects the distinct values in a temporary table
with a unique constraint over the aggregate's arguments, and treats a
duplicate key error from the write as "value already seen":

```c
if (!table->file->is_fatal_error(error, HA_CHECK_DUP))
  return FALSE;                          // duplicate, not an error
```

For a blob argument the record holds only a pointer to the value, so
`Aggregator_distinct::setup()` cannot use the `Unique` tree, which
compares raw record bytes, and every value goes through that write
instead.

When such a write overflows the in-memory table,
`create_internal_tmp_table_from_heap()` copies the stored rows to an
on-disk table and then writes the row that overflowed, which until
then was held in `record[0]` alone. Whether a duplicate key error on
that last write is fatal is decided by the caller's
`ignore_last_dupp_key_error` argument, and `Aggregator_distinct::add()`
passed **0** three lines below the code that ignores the very same
condition. The statement failed with

    ERROR 1169 (23000): Can't write, because of unique constraint,
    to table '(temporary)'

Pass **1** instead, so that a duplicate arriving through the conversion
is discarded exactly like one arriving through the ordinary write. The
result is `table->file->stats.records` of that table, so not storing
the duplicate is what makes the count right.

The argument is the same upstream, where it is unreachable: a
temporary table with a blob column was created on the on-disk engine
to begin with, so the conversion was never entered for the only
tables whose pending row can be a duplicate. Supporting blob columns
in the in-memory engine made the table start in memory and convert.

New tests `heap.count_distinct_blob_convert` and
`heap.count_distinct_blob_convert_debug`.

A write rejected as a duplicate returns its record to the free list and
never reaches the allocation of the blob value, so only the first copy
of a value makes the in-memory table grow, and the write that finds it
full is the second copy of the value stored last. That holds only while
a record slot is what the table runs out of first. Blob values come out
of the same space, and only a write that is not a duplicate ever
allocates one, so when a blob allocation is the one that hits the limit,
the pending row is not a duplicate at all.

Which of the two runs out first follows from how records and blob values
pack together, not from any threshold on the value width. Of 24 measured
combinations of width and `max_heap_table_size`, 20 convert but only 8
reach a duplicate pending row, so asserting that the table was converted
does not establish that the ignored duplicate was reached.

The first test uses widths measured to overflow on a record slot. The
second removes the dependency on that measurement, injecting the
duplicate through a new debug point in
`Tmp_table_default_copier::copy_rows()`, beside the one the row copy
loop already carries. Every value is present twice, so whichever copy
the injected duplicate discards, the other one is still written and the
count does not depend on which write overflowed.

The status counter is read with the in-memory limit restored. The status
table is materialized into a temporary table of its own, and its
VARIABLE_VALUE column is wide enough to be stored as a blob, so under
the shrunken limit that table can overflow and be converted as well, and
would then report its own conversion.
Brandon Nesterenko
MDEV-40768: Slave cannot apply a fragmented row event written with log_bin_compress=ON

MDEV-32570 added fragmentation of large row events. When binlog
compression is enabled (log_bin_compress=ON), the compression is
bypassed, and these binary log events remain as regular row events, when
they should be compressed row events.

This is because the compression happens during the Log_event::write()
function, whereas row data fragmentation happens before
Log_event::write() is called. The Rows_log_event super-class stores the
row data buffers to be written to disk. The regular Rows_log_event
sub-classes's implementations of Log_event::write() write this data
as-is. The compressed sub-classes's implementation of ::write()
over-write these buffers with the compressed rows data before writing to
disk.

To fragment a large row event, the server fragments the Rows_log_event's
row data buffers, to be written by multiple Partial_rows_log_event's,
and each Partial_rows_log_event::write() call writes its portion of the
row data buffer to disk. However, this happens before compression ever
has a chance to take place, and thereby, an event that should be
compressed, never is.

MDEV-39762 added event structure validation for compressed events, and
discovered that these events carry the compressed event type, but are
not actually compressed. Event validation thereby fails.

This patch adds a workaround to override the event type of the
fragmented row events to be regular row events, to be consistent with
the actual on-disk content. The underlying problem still needs to be
addressed though, and is tracked by MDEV-40851.

Signed-off-by: Brandon Nesterenko <[email protected]>
Marko MƤkelƤ
fixup! 5f6f81c1ed080d3469c92da8e12d28cc3e8a10d9

Partially revert a non-working fix.
Oleksandr Byelkin
Merge branch '11.4' into 11.8
Monty
Limit the memory used by GROUP_CONCAT() with ORDER BY

GROUP_CONCAT() with ORDER BY collects all rows of the group in a TREE
and only cuts it down in repack_tree(). The repack was triggered by

  (tree_len >> GCONCAT_REPACK_FACTOR) > thd->gconcat_max_len()

with GCONCAT_REPACK_FACTOR 10, that is when the rows in the tree had
produced 1024 * group_concat_max_len bytes, or 1G with the default
settings. On top of that tree_len only counted the length of the
strings, while the tree costs sizeof(TREE_ELEMENT) + reclength per row.
For GROUP_CONCAT(int_col ORDER BY int_col) that is about 40 bytes per
row against 6 bytes of result, so the tree had grown to several GB
before the first repack. In practice the server ran out of memory
first and the repack code was close to never used.

The tree is now limited by the memory it has really allocated,
tree->allocated, instead of by the length of the strings it holds.
The limit is MY_MAX(thd->ram_limitation(), thd->gconcat_max_len()) and
is never set so low that the tree can not hold a few rows.

repack_tree() builds a new tree while the old one is still in memory,
so the peak usage is the size we start the repack at plus the size we
copy to. To keep the sum within the limit it is split into
GCONCAT_TREE_PARTS parts; the repack starts when
GCONCAT_TREE_REPACK_PARTS of them are used and copies to the remaining
part. The part we do not copy to is also the room the tree has to grow
before the next repack, which keeps the repacks amortized.

Other changes:

- tree_len is removed. It was only read by the old trigger.

- repack_tree() decided that it had run out of memory by testing
  st.len <= st.maxlen after the walk. That test was only valid because
  the old trigger guaranteed that a complete copy had to overshoot
  st.maxlen. A repack triggered by memory can complete the walk with
  st.len far below st.maxlen, which would have failed the query with a
  wrong out of memory error. There is now an explicit flag for it.

- The length that decides which rows to keep now also counts the
  separator that is put between two rows, so that it matches what
  val_str() will produce.

- When the memory limit stops the copy, the result becomes shorter
  than group_concat_max_len. dump_leaf_key() can not detect this, as
  the result never reaches the maximum length. This is now remembered
  in result_cut and reported to the user.

- All cut value reporting is moved to val_str(); dump_leaf_key() only
  marks that the result was cut. This removes the need to clear the
  truncated flag of table->blob_storage to avoid a duplicated warning,
  and gives one warning per group also when val_str() is called more
  than once for the same group, which repeated the warning before.

- Added a function comment for repack_tree() that describes where the
  rows are cut away and why building a copy frees memory.
Kristian Nielsen
MDEV-40729: Revert extension to GTID event with new transaction id.

Since we switched to using the Xid_log_event to have the transaction ID, do
not extend the GTID event with the id.

Kept as a separate commit for now to better isolate the different changes
for easier review and working.

Signed-off-by: Kristian Nielsen <[email protected]>
Arcadiy Ivanov
MDEV-40781 Duplicate row despite DISTINCT when tmp table converts

`create_internal_tmp_table_from_heap()` writes the pending `record[0]`,
the row whose write filled the in-memory table, into the new table.
Since **MDEV-40376** (`636f154bb49`) that write happens *before*
`ha_end_bulk_insert()` rather than after it.

`ha_maria::start_bulk_insert()` disables **all** indexes of an internal
temporary table that is about to receive at least
`MARIA_MIN_ROWS_TO_DISABLE_INDEXES` (100) rows:

```c
if (file->open_flags & HA_OPEN_INTERNAL_TABLE)
{
  /* Internal table; If we get a duplicate something is very wrong */
  file->update|= HA_STATE_CHANGED;
  index_disabled= share->base.keys > 0;
  maria_clear_all_keys_active(file->s->state.key_map);
}
```

`maria_write()` then skips `_ma_check_unique()` entirely, so the unique
constraint that implements `DISTINCT` for a key too wide to be an index
is not enforced. The rows copied out of the in-memory table are already
distinct and need no checking against each other, but the pending row is
exactly the row whose duplicate status is unknown, and it was written
inside that window. A `SELECT DISTINCT` over wide columns could
therefore return a duplicate row.

Note that the justification given in `636f154bb49` is not the mechanism
at work here. It refers to the bulk insert key *tree*, a different
branch of `ha_maria::start_bulk_insert()`; setting
`bulk_insert_buffer_size=0` does not avoid the problem.

The fix splits the copy in two:

1. `Tmp_table_row_copier` gains a second virtual,
  `write_pending_row()`, defaulting to a no-op.
2. `copy_rows()` now only copies the rows the in-memory table holds.
3. `create_internal_tmp_table_from_heap()` calls
  `ha_end_bulk_insert()` and then `write_pending_row()`, so the
  pending row is written with the indexes of the new table back in
  place and a duplicate of an already copied row is detected.

`Window_rowid_remapper` keeps writing its pending row within
`copy_rows()` and inherits the no-op default. Its new position is only
known once the rows before it have been written, and nothing is lost by
writing it with the indexes still disabled: it replaces a row that is
already in the table rather than adding one, and an update of a window
function value cannot collide with another row, as a deduplicating key
is not built on the columns it changes.

The new test covers `SELECT DISTINCT`, `SELECT DISTINCT ... ORDER BY`,
`UNION` and `INSERT ... SELECT DISTINCT`, and asserts that the
conversion actually happened so that a future sizing change cannot
silently void the coverage.
Monty
MDEV-40776 Atomic CREATE OR REPLACE silently breaks the foreign key

Give an error if one tries to drop a table referenced by a foreign keys
This is needed as innodb will keep the reference to the origina table
even when it is renamed to a temporary name as part of create or replace.

Other things:
- Changed the error message for  ER_TRUNCATE_ILLEGAL_FK to say
  "Cannot drop or truncate a table ..."
ParadoxV5
MDEV-40366 merge fix

reƤpply missed merge fixes

Co-authored-by: Brandon Nesterenko <[email protected]>