Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Arcadiy Ivanov
Restore `hp_flush_pending_blob_free()` to entry of `heap_write()`

ee463d926d8 moved the flush from entry to after `hp_write_blobs()` to
protect against a theoretical use-after-free in `UPDATE ... FOR PORTION
OF` on HEAP tables with blob columns and application-time periods.

This caused a concrete regression: deferred blob chain records are not
freed until after `hp_write_blobs()` allocates fresh continuation records
from the tail, preventing free-list reuse.  `last_allocated` grows
monotonically instead of recycling freed slots.  The
`hp_test_freelist` unit test (Tests 1-2) catches this as assertions 8,
16, 20, 24 all fail with `last_allocated` doubling on each cycle.

Move the flush back to entry of `heap_write()`, before
`next_free_record_pos()`.  This restores free-list reuse: deferred
chain records are freed and available (on the delete list or reclaimed
by `hp_shrink_tail()`) before the new blob allocation begins.

The FOR PORTION OF scenario (HEAP + blobs + application-time periods +
`UPDATE ... FOR PORTION OF SET blob_col = ...`) remains theoretical
with no test coverage and should be addressed separately, if at all,
by materializing blob data at the SQL layer before the write.
Monty
Small performance update of previous commit
Arcadiy Ivanov
Add test for user variable geometry assignment in GROUP_CONCAT

`Item_func_set_user_var::create_tmp_field_ex()` calls
`create_tmp_field_ex_from_handler()` directly with raw `type_handler()`,
bypassing `type_handler_for_tmp_table()`. When the argument is geometry,
`fix_fields` preserves `Type_handler_geometry` (item_func.cc:4824), so
the geometry check in `create_tmp_field_ex_from_handler()` is the only
thing preventing a `Field_geom` from being created on the GROUP_CONCAT
temp table.
Arcadiy Ivanov
fixup! 6e9ae08b7ad79885425652bc648f8fc56f7b1177

Fix 32-bit compile: use `sizeof(void*)` for blob pointer memcpy

`portable_sizeof_char_ptr` is always 8 (the record slot width), but
actual pointer size is 4 on 32-bit. The production code (`hp_hash.c`)
uses `HP_PTR_SIZE = sizeof(void*)` for memcpy of pointer values.
The test must do the same to avoid `-Werror=stringop-overflow`.
Monty
Review of "Replace `hp_blob_run_format()` enum with direct bit testing"

Change 'uchar format' to 'uint8 format' to make it clear that we use
format as a number.
Arcadiy Ivanov
Use SHA256 integrity checks in blob_fallback test

Store SHA2(blob, 256) on insert and compare on retrieval instead of
direct string comparison. Use non-repeat blob patterns to make
corruption detection more meaningful.
Monty
Set blob pointer to null with bzero()

This is to avoid storing a pointer into an unaligned address.
Arcadiy Ivanov
fixup! c7643a97878d170f43856d4839500f59e4d6cc2f

Fix typo: "where" -> "were" in hp_shrink_tail() comment
Arcadiy Ivanov
MDEV-39732: fix row-based replication of DELETE on HEAP blob tables

`handler::ha_delete_row()` calls `binlog_log_row()` AFTER
`delete_row()` returns. For HEAP blob tables, `heap_delete()` was
freeing continuation chains immediately, overwriting chain records
with free-list `del_link` pointers. The zero-copy pointers in the
record buffer (used by the binlog to read blob data) became dangling,
causing corrupted replication events.

**Fix**: split the blob free path by table type:

- **Internal temporary tables** (`share->internal`): free chains
  immediately via `hp_free_blobs()` -- these are never binlogged.

- **User-created MEMORY tables**: defer the free by saving chain
  head pointers into `HP_INFO::pending_blob_chains` (one per blob
  column, allocated once at `heap_open_from_share()`). The flush
  runs on the next mutating operation (`heap_write`, `heap_update`,
  `heap_delete`) or on `heap_reset`/`heap_close`.

`heap_clear()` (TRUNCATE) invalidates `has_pending_blob_free`
without flushing, since the HP_BLOCK tree is already destroyed.

Also fixes `hp_free_blobs()` to skip zero-length blobs (avoids
chasing a NULL chain pointer) and `heap_update()` to NULL the
chain pointer slot for unchanged zero-length blobs (prevents
stale SQL-layer pointers from being misread as chain heads).
Monty
MDEV-39703 mroonga/storage.fulltext_order_natural_language_mode_different
Removed unnessary show status like "Created_tmp%" test
Alexander Barkov
Adding a new method Type_handler::make_and_init_table_field_ex()
Arcadiy Ivanov
Replace internal Case A/B/C terminology with descriptive names in blob_find_unique test
Michael Widenius
Added documentation for how internal temporary tables works
Arcadiy Ivanov
Add unit test for concurrent blob del_link race condition

Two threads share one `HP_SHARE` via separate `HP_INFO` handles and
do insert/delete cycles with blobs. A `pthread_mutex` serializes all
operations including `hp_flush_pending_blob_free()`, matching what
`ha_heap::external_lock(F_UNLCK)` does under `thr_lock`.

Moving the flush outside the mutex (simulating the unfixed code path
where the flush ran in `heap_reset()` after `thr_lock` release)
reproduces the SIGSEGV crash from concurrent `del_link` corruption.
bsrikanth-mariadb
When warnings are shown, the captured_opt_ctx is getting cleared

This is a problem when "explain extended" is used, as it shows the
resulting query text after optimization, as a warning.
Yuchen Pei
MDEV-15621 Auto add RANGE COLUMNS partitions by interval

Allow auto partitioning by interval in PARTITION BY RANGE COLUMNS

PARTITION BY RANGE COLUMNS (col_name)
INTERVAL interval [AUTO]
(
  PARTITION partition_name VALUES LESS THAN (value)
  [, PARTITION partition_name VALUES LESS THAN (value) ... ]
)

where

- col_name is the name of one column of type DATE or DATETIME or
  TIMESTAMP

- at least one partition is supplied, and the highest partition cannot
  have MAXVALUE range

- INTERVAL interval is a positive time interval. it can be mariadb
  format or oracle NUMTODSINTERVAL/NUMTOYMINTERVAL format. Like
  versioning, the smallest unit is second, i.e. no subsecond like
  microsecond.

- DATE column cannot have interval with values less than a day

When performing DML on such a table, it will first add partitions
by the specified interval until the partition covers the current time.

Partition addition will not cause an implicit commit like DDL normally
does.

The partitions are named pN.

Otherwise the table behaves exactly the same as a normal RANGE COLUMNS
partitioned table.

Note that TIMESTAMP is not allowed as a type for PARTITION BY RANGE
COLUMNS otherwise.
Arcadiy Ivanov
MDEV-39825 Fix blob data corruption on REPLACE into HEAP table

REPLACE on a HEAP table with blob columns corrupts blob data because
`heap_write()` flushes the deferred blob chain free (from the preceding
delete) before writing the new row's blobs. The flush overwrites the
HP_BLOCK records that zero-copy blob pointers in the record buffer
still reference, destroying the source data.

**Root cause**: two layers of corruption:
1. `hp_flush_pending_blob_free()` -> `hp_free_run_chain()` overwrites
  the first `sizeof(uchar*)` bytes of each freed record with del_link
  pointers, corrupting zero-copy source data
2. `hp_write_one_blob()` reuses those freed records as write
  destinations, creating overlapping `memcpy` ranges (the ASAN report)

**Fix**: use the existing `HA_EXTRA_WRITE_CAN_REPLACE` mechanism to
disable zero-copy blob reads during REPLACE statements.  The SQL layer
already calls `extra(HA_EXTRA_WRITE_CAN_REPLACE)` before REPLACE and
`extra(HA_EXTRA_WRITE_CANNOT_REPLACE)` after.  HEAP now handles these
calls by setting `HP_SHARE::write_can_replace`, which is visible to
all handlers sharing the table.  When the flag is set,
`hp_read_blobs()` forces all blob reads into `blob_buff` (copy mode)
instead of returning zero-copy pointers into HP_BLOCK records.  This
ensures that when `heap_write()` flushes deferred blob chains, no
record buffer holds dangling pointers into the freed chain records.

The flag is on `HP_SHARE` (not `HP_INFO`) because the zero-copy
pointers that cause corruption may originate from a different handler
instance (e.g., the SELECT handler in `REPLACE INTO t SELECT * FROM t`)
than the one performing the write.  Since all handlers share the same
`HP_SHARE`, setting the flag once affects all concurrent reads.

The flag is NOT cleared by `heap_reset()` because `ha_reset()` runs
after the table lock is released (in `close_thread_tables()`), which
would race with another connection's REPLACE setting the flag.  The
flag lifecycle is managed entirely by the bracketed
`HA_EXTRA_WRITE_CAN_REPLACE` / `HA_EXTRA_WRITE_CANNOT_REPLACE` calls,
which execute under the table's write lock.

Additionally, `hp_materialize_one_blob()` (used by hash key comparison)
now uses its own `HP_INFO::key_blob_buff` instead of sharing
`blob_buff` with `hp_read_blobs()`.  Previously, a key comparison
during `heap_update()` or `hp_delete_key()` could overwrite `blob_buff`
while record pointers still referenced it, corrupting blob data during
REPLACE's UPDATE path.

Internal temporary tables are unaffected: they never execute REPLACE,
so the flag is never set and zero-copy remains enabled.

**Files changed**:
- `heap.h`: new `write_can_replace` field in `HP_SHARE`; new
  `key_blob_buff` / `key_blob_buff_len` in `HP_INFO`; removed
  `write_blob_buf` from `HP_INFO`
- `heapdef.h`: removed `hp_materialize_write_blobs()` declaration and
  `hp_free_write_blob_buf()` inline
- `hp_extra.c`: handle `HA_EXTRA_WRITE_CAN_REPLACE` /
  `HA_EXTRA_WRITE_CANNOT_REPLACE`; free `key_blob_buff` in
  `heap_reset()`
- `hp_blob.c`: `hp_read_blobs()` checks `write_can_replace` to force
  copy mode for Cases A/B; `hp_materialize_one_blob()` uses
  `key_blob_buff`; removed `hp_materialize_write_blobs()`
- `hp_write.c`: removed write-side materialization logic
- `hp_delete.c`, `hp_update.c`: removed `hp_free_write_blob_buf()`
- `hp_close.c`: free `key_blob_buff`; removed `write_blob_buf` free

**Tests**:
- New `heap/blob_replace_overlap`: blob integrity after REPLACE with
  single/multiple blob columns at various sizes (single-record,
  multi-record, bulk copy paths) + INSERT ON DUPLICATE KEY UPDATE
- Extended `heap/blob_replication` Test 6: REPLACE with blobs under
  row-based replication -- self-referencing, VALUES, multi-blob
Arcadiy Ivanov
MDEV-39782: fix row-based replication of UPDATE on HEAP blob tables

`handler::ha_update_row()` calls `binlog_log_row()` AFTER
`update_row()` returns. For HEAP blob tables, `heap_update()` was
freeing old continuation chains immediately via `hp_free_run_chain()`,
overwriting chain records with `del_link` pointers. The zero-copy
pointers in the before-image record (used by the binlog to read blob
data) became dangling, corrupting the replication event. The slave
then failed to locate the row with `ER_KEY_NOT_FOUND`.

Two corruption vectors:

1. **Dangling zero-copy pointers**: `hp_free_run_chain()` zeroes chain
  records that `record[1]` (before-image) still references via
  zero-copy (Case A/B) pointers.

2. **`blob_buff` reallocation**: `hp_read_blobs()` called to refresh
  `record[0]` (after-image) could `realloc` `info->blob_buff`,
  invalidating Case C materialized pointers in `record[1]`.

**Fix**:

- Split the chain-free path by `share->internal`: internal temporary
  tables (never binlogged) free chains immediately; user-created tables
  defer via `pending_blob_chains`, matching the DELETE path from
  MDEV-39732.

- Gate `hp_read_blobs()` refresh on `share->internal`: user tables
  skip the refresh since the SQL layer re-materializes on the next
  fetch, avoiding the `blob_buff` reallocation hazard.

- Move `hp_flush_pending_blob_free()` in `heap_write()` from entry to
  after `hp_write_blobs()` completes, so that blob data in the record
  buffer remains readable during the write. This prevents use-after-free
  when `UPDATE ... FOR PORTION OF` inserts a row whose blob pointers
  reference deferred chains.
Monty
Review commit for Overflow-to-Aria on `ha_update_tmp_row()`...

- Updated test to print number created temporary files for all
  test parts.
- Fixed long comment in hp_blob.c
Arcadiy Ivanov
MDEV-39723, MDEV-39724: fix `hp_key_cmp()` blob packlength

`hp_key_cmp()` hardcoded `packlength=4` for the record side of blob
segment comparison. This is wrong when `Field_blob_key` preserves
the original blob subtype's packlength (1-4) via
`set_pack_length(blob_handler->length_bytes())`.

Use `seg->bit_start` (the actual packlength) instead. Update comments
in `hp_key_cmp()` and `hp_make_key()` to document the asymmetry
between record-side format (packlength-byte length prefix) and
key-side format (always 4-byte normalized length from `hp_make_key()`).
Arcadiy Ivanov
Fix stale comments, test bugs, and expand test coverage

**Comment fixes** (source):
- `hp_create.c`: fix "VARTEXT2" -> "VARTEXT4", "Paclength" typo, replace
  stale 8-line `bit_start` derivation comment with accurate 3-line version
- `sql_select.cc`: fix `make_sort_key()` -> `make_sort_key_part()` in
  `remove_duplicates()` comment; clarify HEAP packed-format comment
- `field.h`: fix "inc record" -> "in a record" typo

**Comment fixes** (tests):
- `hp_test_key_setup-t.cc`: replace stale Phase 1 file header and function
  comment describing `key_part->length` widening with accurate blob segment
  normalization description
- `hp_test_hash-t.c`: fix swapped field names in mixed-key record layout
  comment; remove stale "hp_hashnr is static" comment; add missing
  `bit_length=2` for VARTEXT1 segment in `setup_mixed_keydef`
- `blob_big3.test`: fix "without" -> "with", "dicrectly" -> "directly"
- `blob_big.inc`: fix "in both runs" -> "when HEAP is used"

**Test bug fixes**:
- `blob_sj_test`: change `semijoin=off` to `semijoin=on,firstmatch=off,
  loosescan=off` so the test actually exercises DuplicateWeedout SJ strategy;
  add optimizer_switch save/restore
- `blob_fallback`: replace MD5-based integrity checks with direct
  `b = repeat(...)` comparisons
- `blob_stress`: replace 4 useless `check table` (HEAP returns "not
  supported") with echo comments
- `blob_big.inc`: add save/restore for `max_sort_length` and
  `sort_buffer_size`

**Test coverage expansion** (unit tests -- `hp_test_hash-t.c`):
- Add packlength 1, 3, 4 hash/comparison tests (12 assertions)
- Add blob+blob multi-segment key test (6 assertions)
- Plan: 49 -> 67

**Test coverage expansion** (MTR -- `blob.test`):
- INSERT ON DUPLICATE KEY UPDATE with blobs (conflict, no-conflict,
  NULL transitions)
- JSON column CRUD on MEMORY table
- LONGBLOB at uint16 `run_rec_count` split boundary (1,048,549 / 1,048,550
  / 2MB)
- Case A/B/C exact boundary blob sizes (5B, 6B, 10KB, 50KB) with
  cross-case UPDATE
- BTREE+blob rejection (both BTREE and HASH explicit blob keys rejected;
  BTREE on non-blob column with blob data works)
- Table-full error: verify no partial rows from failed inserts (row count,
  corruption check, scan count)

**Test coverage expansion** (MTR -- `blob_stress.test`):
- NULL->non-NULL and non-NULL->NULL blob UPDATE operations in the 200-cycle
  stored procedure

**Build**:
- Wire `hp_test_key_setup-t.cc` into CMake build; remove Phase 1-only
  tests (`test_rebuild_key_from_group_buff_mixed`,
  `test_varchar_promoted_to_blob`); update assertions for current blob
  segment normalization; plan: 47 -> 34
Monty
squash! 4888cdb69fd115986625da2a44b78a6a3898983c

Fixed that heap_prepare_hp_create_info() honors tmp_memory_table_size
Fixed memory overrun error in hp_test_hash-t
Added DBUG_ASSERT to ensure that we are not used converted heap
keys with index_read()
Monty
Review fixes for "Batch tail allocation for blob continuation chains"

- Optimized setting is_only_run (avoiding one/two if's for each loop)
- Removed test for run_rec_count == 1 in a context where it was guaranteed
  to be 1.
- Removed not needed variable prev_pos (as it was always identical to
  run_start). This also fixes one of the bugs fixed by the next commit.
Monty
Ensure that 'param' is always set for type_handler_for_tmp_table()
Monty
Remove pack_length_no_ptr()

Replaced pack_length_no_ptr() with existing length_size() that does the
same thing.
Monty
Review of reclaim tail records on failed blob allocation

- Simplify loop in hp_shrink_tail to avoid multiplcations
- Moved assert from callers of hp_shrink_tail() to hp_shrink_tail()
Monty
Cleanup of previous commit
- Added virtual function Type_handler_geometry::type_handler_for_tmp_table()
  to be able to remove testing of MYSQL_TYPE_GEOMETRY in
  Item::tmp_table_field_from_field_type()

Co-author: Alexander Barkov <[email protected]>
Arcadiy Ivanov
Fix CI failures: 32-bit/big-endian test portability and non-deterministic ordering

- `hp_test_helpers.h`: use `sizeof(blob_data)` instead of `portable_sizeof_char_ptr`
  for `memcpy` of pointer values, fixing `-Werror=stringop-overread` on 32-bit
- `hp_test_hash-t.c`: use `int2store()` for blob length (endian-safe) fixing
  s390x big-endian failures in DISTINCT/GROUP BY key tests; use
  `portable_sizeof_char_ptr` for blob+blob key offset (32-bit blob2 length read);
  use `sizeof(varchar_data)` for pointer copies
- `heap.blob_big2`: reduce `tmp_memory_table_size` 65536->32768 so UNION DISTINCT
  overflows to disk on 32-bit (smaller recbuffer = more rows fit in memory)
- `main.type_bit`: add `--disable_cursor_protocol` around metadata-sensitive
  DISTINCT/GROUP BY queries (cursor mode loses `UNIQUE_KEY_FLAG` on temp table)
- mroonga fulltext_order tests: add `, title` tiebreaker to ORDER BY for
  equal MATCH scores; mask `Created_tmp_tables` counter with `--replace_column`
Monty
Review commit of Avoid double blob materialization
Michael Widenius
Small optimization for MDEV-39825 Fix blob data corruption...
Monty
Review fixes for "Code review feedback: `hp_update.c` cleanup, test renames, style fixes"

- Optimized struct layouts in heap.h even more by moving variables
  according to size
- Replaced max_heap_table_size with tmp_memory_table_size in tests
- In ha_heap::records_in_range, removed not needed variable blob_count
- In hp_blob.cc:hp_write_run_data() replaced loop with one memcpy
- In heap_write() replaced loop with jump to avoid code duplication

Other things:
- Removed extra empty lines
- Reformatted rows to fit 80 characters
- Removed not needed braces from blocks with no varaibles.
Arcadiy Ivanov
MDEV-39761: fix `Field_geom::store()` assertion in GROUP_CONCAT with geometry expressions

`Field_geom::store()` asserts `!table->blob_storage`, but GROUP_CONCAT
with ORDER BY or DISTINCT sets `blob_storage` on its internal temp table.
When geometry expressions (IF, CASE, COALESCE, DEFAULT, etc.) appear as
GROUP_CONCAT arguments, the type handler field creation path creates
`Field_geom` directly, bypassing the existing downgrade in
`Field_blob::make_new_field()` that converts `Field_geom` to plain
`Field_blob` for GROUP_CONCAT temp tables.

Fix: downgrade geometry to `Field_blob` (via `type_handler_long_blob`) in
both field creation paths that expressions can take:
- `create_tmp_field_ex_from_handler()` for `Item_result_field` items
- `tmp_table_field_from_field_type()` for `Item_default_value` and other
  callers that bypass the handler path
Arcadiy Ivanov
fixup! 07846d3ef329caca03d26e58e200f3f2d1e4f04b

Replace exact `Created_tmp_%` counts with ON/OFF checks in blob_big tests

Exact temp table counts vary between 32-bit and 64-bit platforms
because view protocol creates additional temp tables whose overflow
behavior depends on pointer size and internal structure sizes.

Replace `SHOW STATUS LIKE 'Created_tmp%'` with a query that reports
ON/OFF for `CREATED_TMP_TABLES` and `CREATED_TMP_DISK_TABLES`. This
preserves the key assertions (disk tables created vs not created)
while being platform-independent.
Arcadiy Ivanov
Replace `hp_blob_run_format()` enum with direct bit testing

Remove `enum hp_blob_format` and `hp_blob_run_format()` indirection.
Add `HP_ROW_MULTIPLE_REC` (bit 5) so all three blob storage formats
have a dedicated flag bit. Add named inline predicates
`hp_is_single_rec()`, `hp_is_zerocopy()`, `hp_is_multi_run()` matching
the existing `hp_is_active()`/`hp_has_cont()`/`hp_is_cont()` pattern.

Change `hp_write_run_data()` format parameter from enum to `uchar`
receiving bit constants directly; simplify flags byte assignment from
ternary to bitwise OR.

Addresses review feedback F127-F128, F130, F132-F134.
Monty
Fixed potential race condition in heap that could cause crash

In case of "create table(..) engine=heap" there is this possible scenaro:
- T1 deletes a row with a blob, that is stored for a later call to
  hp_flush_pending_blob_free(), and then ends the statement.
- T2 does a write of a new row
- T1 calls hp_close() that calls hp_flush_pending_blob_free()
Now T1 and T2 are both working on the same delete list, which can
cause a crash.

Fix is to call hp_flush_pending_blob_free() at external_unlock
(end of statement) so that T1's blobs are freed before T2 can
get a lock on the table.
Arcadiy Ivanov
fixup! 45718801fb0755ea2092d8382bf29f7490667dcb

Disable ps_protocol in blob_update_overflow status checks

PS protocol creates extra internal temp tables, inflating
Created_tmp_tables counters. Disable all protocol variants
around FLUSH STATUS / SHOW STATUS sections.
Monty
fixup! 13e1fdfb6d3deb492f81e07caacadc2c4fa75dfb

Fixed duplicate key error when converting HEAP table Aria
Arcadiy Ivanov
Avoid double blob materialization in `find_unique_row()`

For blob tables, `find_unique_row()` previously materialized blobs
twice: once during `hp_rec_key_cmp()` (per-segment via
`hp_materialize_one_blob()`) and again via `hp_read_blobs()` after
the match was found.

Reorder the blob path to materialize-then-compare: save the input
record, copy the stored candidate, call `hp_read_blobs()` once to
materialize all blobs, then compare via `hp_rec_key_cmp()` with
`info=NULL` since both records now have direct data pointers.

Non-blob tables keep the original fast path unchanged.
Arcadiy Ivanov
Add missing `hp_shrink_tail()` after immediate blob chain frees

`hp_free_run_chain()` puts freed continuation records onto the delete
list but does not reclaim tail-positioned records. `hp_shrink_tail()`
must be called afterward to shrink `block.last_allocated` when freed
records happen to be at the tail, preventing monotonic HP_BLOCK growth.

The deferred-free path (`hp_flush_pending_blob_free_impl`) already
calls `hp_shrink_tail()`, but three immediate-free sites did not:

- `hp_free_blobs()`: used by DELETE on internal temporary tables
- `heap_update()` internal-table path: immediate free of old chains
- `heap_update()` rollback path: free of partially-written new chains

Expose `hp_shrink_tail()` in `heapdef.h` (was `static` in `hp_blob.c`)
and add the call at all three sites.
Arcadiy Ivanov
fixup! 6e9ae08b7ad79885425652bc648f8fc56f7b1177

Fix MSVC C4244 warning: split chained assignment to avoid uint16-to-uint8 narrowing