Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
fd3dfb8dcb8c...
fd3dfb8dcb8c6e62803457b058be2fd2e0457e0a
Oleg Smirnov
MDEV-39491 Parallel Query: distribute parallel work more evenly

Lower SPLIT_THRESHOLD from 3 to 2, enabling re-partitioning of chunks
for shallower trees (one root node, one or more levels of internal nodes
and a level of leaf nodes).

Improve the detection of amount of work left: empty run queue does not
always mean the scan is over, it might be that a context pulled
for re-splitting has not enqueued its sub-contexts yet.
fcf31ea916f5...
fcf31ea916f55451b4fb073d90ee04ae301b4d76
Sergei Golubchik
use export target
f5e9abfebec5...
f5e9abfebec53f6b4080a8d10215b8ba9e9e6823
Kristian Nielsen
MDEV-35691: Invalid access, use-after-free, on rli->description_event_for_exec

This commit rewrites the rpl_master_has_bug() mechanism to solve a problem
with invalid memory access. The rpl_master_has_bug() mechanism detects
certain bugs depending on the master version, and uses that to enable
specific work-arounds on the slave. The problem was that
rpl_master_has_bug() accessed Relay_log_info::description_event_for_exec
that is not valid to access from concurrent parallel replication worker
threads, only from the SQL driver thread. Thus it could use the wrong event
or access invalid/freed memory.

This patch instead computes a bitmask of detected bugs when the SQL driver
thread processes the format description event, and reads that bitmask with
an atomic load from the worker threads. The bitmask of bugs can only change
when the master restarts with a new version, and we do not replicate events
concurrently across a format description event from a master restart. Thus,
the bitmask is safe to read concurrently from the Relay_log_info object
without locking.

This also avoids an expensive match of each entry in the bug list against
the master server version done for every single call to
rpl_master_has_bug(), which could be quite expensive when done eg. per field
in row events as in Field_string::compatible_field_size().

Also remove redundant conditional in table_def::compatible_with().

Thanks to Andrei Elkin for the idea to safely read the bitmask
concurrently from the Relay_log_info.

Reviewed-by: Andrei Elkin <[email protected]>
Signed-off-by: Kristian Nielsen <[email protected]>
eb3a216a0175...
eb3a216a01758ffede9eeab6319c7fdd57a0e6d8
Oleg Smirnov
MDEV-39491 Parallel Query: distribute parallel work more evenly

Lower SPLIT_THRESHOLD from 3 to 2, enabling re-partitioning of chunks
for shallower trees (one root node, one or more levels of internal nodes
and a level of leaf nodes).

Improve the detection of amount of work left: empty run queue does not
always mean the scan is over, it might be that a context pulled
for re-splitting has not enqueued its sub-contexts yet.
e296dfa5493e...
e296dfa5493e8198aba184050a6e4abd643f902e
Marko Mäkelä
squash! e61cabc0a2ae1a58c401fc24bb9f25caab6dc9b5

Remove traces of HTON_CHECK_NEEDED_FOR_CREATE_OR_REPLACE
and handler::can_be_renamed_to_backup() that were made redundant by
MDEV-28933 (commit cffbb17480a6fba6bf8cb42d943833cca214b34a)
d557ea8809f3...
d557ea8809f389670c71d8b2769f19d95b80e7ba
ParadoxV5
MDEV-40996 Support `--sync_with_master 0, $variable` in mysqltest

`--sync_with_master` uses `get_string()`,
which has `$variable` support, but it only uses the read buffer,
which is written with the unexpanded string and not the variable value.

Reviewed-by: KhaledR57 <[email protected]>
d515d66c10cb...
d515d66c10cb956cf5adb3865690bcd8289339e9
Arcadiy Ivanov
MDEV-21879 GROUP_CONCAT(DISTINCT ORDER BY) is wrong when Unique spills

`Item_func_group_concat::add()` decided whether a row was a duplicate
by checking whether `Unique::elements_in_tree()` had grown after
`unique_add()`:

    uint count= unique_filter->elements_in_tree();
    unique_filter->unique_add(get_record_pointer());
    if (count == unique_filter->elements_in_tree())
      row_eligible= FALSE;

`Unique` flushes its whole in-memory tree to disk when it runs out of
memory, and `elements_in_tree()` only counts what is still in memory.
After the first flush the test says nothing about the rows that were
already spilled.

**MDEV-11563** made this harmless for `GROUP_CONCAT(DISTINCT x)` by
building the result in `val_str()` from `unique_filter->walk()`, which
merges the spilled parts back in. It left the `ORDER BY` case alone.
There the result comes from the sort tree, which `add()` fills gated by
`row_eligible`, so the defect is still fully live.

Both directions of the failure are reachable, depending on how often
the filter flushes relative to the insert:

1. Duplicates reach the result. 100 rows holding 50 distinct values
  give all 100 values back.
2. Rows are lost. 30 distinct rows of 2000 bytes give one value back.

`JSON_ARRAYAGG(DISTINCT x ORDER BY y)` fails in the same way.

Fixed by not filling the sort tree from `add()` when `DISTINCT` is
used. `val_str()` now walks the merged `unique_filter` into the sort
tree and then walks the sort tree, so the rows are sorted after the
duplicate filtering is complete instead of during it.

`Unique::walk()` merges everything it flushed, so the sort tree can be
handed more rows than fit in memory. `insert_to_order_tree()` repacks
it on the same memory budget `add()` used, and a walk that runs out of
memory sets `result_cut`, so the user gets a cut value warning rather
than a silently short result.

**Behaviour change.** `ORDER BY` does not order rows that tie on the
ordering expression, and which of them comes first changes here. It
used to follow the order the rows were read in; it now follows the
order the duplicate filter keeps them in. Unlike the old order, the new
one depends on neither the memory available nor the physical row order.
`main.gconcat_distinct_spill` checks that, and `main.func_gconcat`
records one such tie.
d3452c70012b...
d3452c70012b6f5324de2a1f96e680d339c8f75e
Yuchen Pei
MDEV-40168 [wip] Add multi valued index over fulltext

TODOs on top of those in the patch diff:

- EXPLAIN output should not say fulltext
- check type match to avoid false negative / positive bugs in mysql
- transcode the value into the index charset in mvi_encode_key
c43d9b0989c4...
c43d9b0989c430d5c6496da8e29e6b6bcf800ecb
Sergei Petrunia
MDEV-40168: JSON-over-fulltext: add estimates.

Add records_in_range-like estimates for fulltext index
c3ddbaf16491...
c3ddbaf16491bc0814a484f42e1df4b9e7b38723
Sergei Golubchik
fix include paths again
c2f630911fc1...
c2f630911fc1944211e96eccaff4e28794ee1a63
Monty
MDEV-40454 UBSAN: maria.aria_pack_mdev invalid-shift-exponent

Shifting with 64 is no-op in the the code (no ill effects)

Added a test to not do anything if shift with 64 would happen.
Tested with ma_test_all that test aria_pack.
c16a29b0da32...
c16a29b0da32404f9b03b385e931f712586bba5d
Alessandro Vetere
MDEV-32286 Reuse remembered clustered leaves in secondary-index scans

Row_sel_get_clust_rec_for_mysql::operator() descends the clustered B-tree
from the root for every row of a non-covering secondary-index scan, although
consecutive rows often land on the same clustered leaf page. ANALYZE
FORMAT=JSON charges each descent its full height, and those descents are
nearly the whole cost: the secondary index is charged its own descent and
one page for each further leaf, and nothing per row, because the position
that its cursor holds between two rows is restored optimistically, which
latches the leaf again without counting an access. The range estimate that
ha_innobase::records_in_range() makes for the optimizer is charged over the
same leaves, once more. So pages_accessed is the row count times the height
of the clustered index, plus that handful. 1000 rows over a 2-level
clustered index cost 2006, of which 3 are the scan of the secondary index
and 3 the range estimate, and 750 rows over a 3-level one cost 2291, of
which 27 and 14, where a full table scan of the same data costs 23 and 110.

Remember, in the new row_prebuilt_t::clust_leaf_hint, the
CLUST_LEAF_HINT_SLOTS (4) clustered leaves that the lookups of this
statement reached, most recently used first. Each slot names one leaf: its
page number, copies of its first and last user record truncated to the key
fields, which bound the key range the leaf held when it was remembered, and
the rec_get_offsets() of both. The copies are needed because the page is
unlatched between two lookups, and the offsets spare a lookup the parsing of
them. Several slots serve the scans that alternate between a few leaves,
which one slot cannot serve at all, and a descent refreshes the slot of a
leaf that is remembered already rather than spend a second one on the same
page. A slot owns its key buffers and grows them only when a longer key
arrives, so a row allocates nothing. The used-slot count and the miss
counter are reset per statement in ha_innobase::reset(), matching
autoinc_last_value.

A lookup first compares its key against the remembered ranges, so an
uncorrelated scan settles its misses in memory, with no buffer pool access
and no pages_accessed. Only a covering range is probed, through the new
btr_cur_t::try_leaf_hint(), which acquires the page with buf_page_try_get():
a hint is never derived from a latched parent page, so by the time it is
tried it may precede the caller's already-latched secondary-index leaf in
the latching order, where a blocking wait can deadlock. A stale range costs
a wasted probe or a needless descent, never a wrong result, because the
checks that try_leaf_hint() makes on the latched page remain the sole
authority, and the ranges therefore need no invalidation protocol.

After CLUST_LEAF_HINT_MAX_MISSES (8) consecutive unanswered lookups, a scan
gives the slots up: row_sel_clust_leaf_hint_armed() stops both the test of
the slots and the copies that refresh them, which are the larger half of
their cost. One lookup in CLUST_LEAF_HINT_RETRY (1024) starts the count
again, so a scan whose order becomes correlated only later recovers, and the
trial that this begins refreshes the slots as it goes.

The run is short because a hit saves little where the pages above the leaf
are resident: one buffer pool access and one page-local search for each
level. Measured against the same tree built without the hints, at 16k with a
resident working set and no adaptive hash index, a wholly correlated scan
runs a quarter faster over half the page accesses, a scan that answers three
lookups in five runs level with it over 30% fewer, one whose locality
appears only half way through runs an eighth faster over a quarter fewer,
and a scan that answers nothing stays within the noise. A run of 8 is what
keeps that last one there. A clustered index small enough to stay in cache
is the exception that the run does not catch: it answers often, so the count
never builds, and it saves nothing, because the descent that a hit replaces
costs almost nothing there. Such a scan pays about a tenth.

Where the adaptive hash index is enabled, the hints are neither used nor
collected: its guess solves the same problem better, landing on the record
with no page-local search and no page access to charge. It is off by
default, so the hints are active in a default configuration.

innodb.non_covering_sec_idx_scan measures pages_accessed over key orders
that differ in how closely the secondary order tracks the clustered one, and
seven further tables check query results over the record formats and key
shapes that a clustered-index lookup has to read, down to the metadata
pseudo-record of instant ALTER TABLE and to leaves that split and merge
while a locking read walks them. non_covering_sec_idx_scan_debug runs the
same body with the hints turned off, through a debug switch that returns
before a lookup tests or refreshes the slots, so a diff of the two .result
files is what the hints save: 2006 to 1028 (2-level clustered index), 2291
to 1007 (3-level), 20020 to 15780 (decorrelated), 4006 to 2012 (two
interleaved key ranges), 20020 to 20006 (shuffled) and 12016 to 9289
(locality in the second half alone). The last of those pins the retry:
without it the count is 11934.
main.rowid_filter_innodb: 90 to 81, and its ahi combination unchanged.
bdd3c3a42df9...
bdd3c3a42df9170cfbce88c71699e0f217899dd2
Sergei Golubchik
windows
bcd0a1a63a9c...
bcd0a1a63a9c506b3edd90929f34c9999b315e4a
Khaled Riyad
MDEV-40551 Copy/Paste friendly output format for MariaDB Command Line Client

Copy/paste friendly output was only reachable by starting the client with
--silent --skip-column-names, which cannot be done from a running
interactive session.

Add \S, a statement terminator which prints the result of one statement in
the tab separated format without column names.

com_silent() sets output_plain, opt_silent and column_names around
com_go(), then restores them, the same way com_ego() handles vertical.
output_plain selects print_tab_data() ahead of the vertical and table
branches, so \S gives the same output whether the session was started
plainly or with --table, --vertical or --silent. --html and --xml still
win, matching \G.
bcbf5d4875d7...
bcbf5d4875d7559cef7f42504dd39a8fc5a03267
Monty
MDEV-39949 Add per-table "SQL_CACHE" option

Add SQL_CACHE=0|1 optioon for tables to disable/enable query caches if
table is part of the query.
SQL_CACHE option can be disable for a table by using SQL_CACHE=DEFAULT

Also provide a workaround for the problem described at:
MDEV-6631 "No cache directive(SQL_CACHE) query still wait for
query cache lock" on query_cache_type=DEMAND mode"
by adding DEMAND_STRICT mode.
The new mode is needed as modifying how DEMAND works would made the
change backward incompatible.

Query_cache_type has now following options (0-2 are same as before)
OFF    (0) = Query cache off
ON    (1) = Query cache on, all queries are cached except queries marked
            with SQL_NO_CACHE
DEMAND (2) = Query cache on, only queries marked with SQL_CACHE are cached
DEMAND_STRICT
      (3) = Query cache on, only queries starting with SELECT SQL_CACHE
            are cached. This avoids a query cache lock+lookup for every
            query.
TABLES (4) = Query cache on, only cache queries marked with SQL_CACHE or
            where all used tables are using the create option SQL_CACHE=1

Note that if any of the tables used in the query has the create option
SQL_CACHE=0 or is a system table, temporary table or other non
cacheable table, then the query will not be cached.

SHOW CREATE TABLE prints SQL_CACHE=0|1 when the option is set. For tables
where the option can never apply (temporary tables, tables in the mysql
database, and engines that do not support the query cache) it is printed
as /* SQL_CACHE=1 */.

co-author: Seunguck Lee
- Took some changes related to MDEV-6631 from his patch.
bc5d6a45ebd9...
bc5d6a45ebd9d71588baa54a2f50306b31376edc
Sergei Golubchik
bintar plugin names
b8c880ce9932...
b8c880ce9932227a825bc4b0f4b48cf989f02d12
Alessandro Vetere
MDEV-32286 Reuse remembered clustered leaves in secondary-index scans

Row_sel_get_clust_rec_for_mysql::operator() descends the clustered B-tree
from the root for every row whose clustered-index record a secondary-index
scan must read, although consecutive rows often land on the same clustered
leaf page. A non-covering scan needs one for every row, a locking read needs
one whatever the secondary index holds, because an exclusive select lock
type makes ha_innobase::build_template() build its template against the
clustered index, and a covering scan needs one for every row of a secondary
leaf whose PAGE_MAX_TRX_ID its read view cannot see. ANALYZE FORMAT=JSON
charges each descent its full height, and those descents are nearly the
whole cost: the secondary index is charged its own descent and one page for
each further leaf, and nothing per row, because the position that its cursor
holds between two rows is restored optimistically, which latches the leaf
again without counting an access. The range estimate that
ha_innobase::records_in_range() makes for the optimizer is charged over the
same leaves, once more. So pages_accessed is the row count times the height
of the clustered index, plus that handful. 1000 rows over a 2-level
clustered index cost 2006, of which 3 are the scan of the secondary index
and 3 the range estimate, and 750 rows over a 3-level one cost 2291, of
which 27 and 14, where a full table scan of the same data costs 23 and 110.

Remember, in the new row_prebuilt_t::clust_leaf_hint, the
CLUST_LEAF_HINT_SLOTS (4) clustered leaves that the lookups of this
statement reached, most recently used first. Each slot names one leaf: its
page number, copies of its first and last user record truncated to the key
fields, which bound the key range the leaf held when it was remembered, and
the rec_get_offsets() of both. The copies are needed because the page is
unlatched between two lookups, and the offsets spare a lookup the parsing of
them. Several slots serve the scans that alternate between a few leaves,
which one slot cannot serve at all, and a descent refreshes the slot of a
leaf that is remembered already rather than spend a second one on the same
page. A slot owns its key buffers and grows them only when a longer key
arrives, so a row allocates nothing. The used-slot count and the miss
counter are reset per statement in ha_innobase::reset(), matching
autoinc_last_value.

A lookup first compares its key against the remembered ranges, so an
uncorrelated scan settles its misses in memory, with no buffer pool access
and no pages_accessed. Only a covering range is probed, through the new
btr_cur_t::try_leaf_hint(), which acquires the page with buf_page_try_get():
a hint is never derived from a latched parent page, so by the time it is
tried it may precede the caller's already-latched secondary-index leaf in
the latching order, where a blocking wait can deadlock. A stale range costs
a wasted probe or a needless descent, never a wrong result, because the
checks that try_leaf_hint() makes on the latched page remain the sole
authority, and the ranges therefore need no invalidation protocol.

After CLUST_LEAF_HINT_MAX_MISSES (8) consecutive unanswered lookups, a scan
gives the slots up: row_sel_clust_leaf_hint_armed() stops both the test of
the slots and the copies that refresh them, which are the larger half of
their cost. One lookup in CLUST_LEAF_HINT_RETRY (1024) starts the count
again, so a scan whose order becomes correlated only later recovers, and the
trial that this begins refreshes the slots as it goes.

The run is short because a hit saves little where the pages above the leaf
are resident: one buffer pool access and one page-local search for each
level. Measured against the same tree built without the hints, at 16k with a
resident working set and no adaptive hash index, a wholly correlated scan
runs a quarter faster over half the page accesses, a scan that answers three
lookups in five runs level with it over 30% fewer, one whose locality
appears only half way through runs an eighth faster over a quarter fewer,
and a scan that answers nothing stays within the noise. A run of 8 is what
keeps that last one there. A clustered index small enough to stay in cache
is the exception that the run does not catch: it answers often, so the count
never builds, and it saves nothing, because the descent that a hit replaces
costs almost nothing there. Such a scan pays about a tenth.

Where the adaptive hash index is enabled, the hints are neither used nor
collected: its guess solves the same problem better, landing on the record
with no page-local search and no page access to charge. It is off by
default, so the hints are active in a default configuration.

innodb.non_covering_sec_idx_scan measures pages_accessed over key orders
that differ in how closely the secondary order tracks the clustered one, and
eight further tables check query results over the record formats and key
shapes that a clustered-index lookup has to read, down to the metadata
pseudo-record of instant ALTER TABLE, to leaves that split and merge while a
locking read walks them, and to a record that a remembered leaf supplies for
a scan that must then rebuild an older version of it.
non_covering_sec_idx_scan_debug runs the same body with the hints turned
off, through a debug switch that returns before a lookup tests or refreshes
the slots, so a diff of the two .result files is what the hints save: 2006
to 1028 (2-level clustered index), 2291 to 1007 (3-level), 20020 to 15780
(decorrelated), 4006 to 2012 (two interleaved key ranges), 20020 to 20006
(shuffled), 12016 to 9289 (locality in the second half alone) and 20020 to
10042 for a covering scan that FOR UPDATE makes non-covering, where the same
scan without FOR UPDATE costs 20 in both files. The locality count pins the
retry: without it the scan would keep only the 82 hits it makes before it
gives up, and the count would be 11934.
main.rowid_filter_innodb: 90 to 81, and its ahi combination unchanged.
b655aec30937...
b655aec30937a6d62f070503b180f51365830a00
Kristian Nielsen
MDEV-22848: SET GLOBAL gtid_slave_pos leaves dangling partial transaction

When AUTOCOMMIT=0, SET GLOBAL gtid_slave_pos did not properly commit the
(full) transaction, leaving the InnoDB hton registrered in the ha_list. This
could then later assert when InnoDB was called upon to eg. prepare() a
transaction that it does not participate in.

This patch makes rpl_slave_state::load() properly commit the (full)
transaction to solve the issue.

Reviewed-by: Brandon Nesterenko <[email protected]>
Reviewed-by: Monty <[email protected]>
Signed-off-by: Kristian Nielsen <[email protected]>
b2a8c2234dbc...
b2a8c2234dbc276fe3947633c9ffea454badf48f
Sergei Golubchik
MDEV-34805 post-review fixes

* keep `vec_len >= subdist_part * 2` logic in one place only
* keep "distance-greater-than" mode logic in one place only
* simplify VECTOR_DIMENSIONS (no need to have a special ctx->vec_len
  path if the other one always works)
* new plugin = maturity beta
* remove redundant casts, etc
* moved vector_indexes_fields_enum to the global scope to use it
  for setting schema->idx_field1/schema->idx_field2
* open the hlindex graph table, if needed, otherwise most values
  are unknown unless a user did vector search before
* added TABLE_CATALOG column
* remove CACHE_OVERFLOWS column, doesn't work as implemented,
  the fix is complex and isn't worth it
* add privilege checks (MDEV-40793)

in the test:
* prefer query_vertical for readability
* select all columns at least once
* select INDEX_SIZE even if engine-dependent, use rdiff files
* test how get_all_tables only open one specific table, and
  even only .frm file, if possible
a9c2f6f1dcae...
a9c2f6f1dcaee85a584da1db68fa68ca701d3d2d
Monty
MDEV-40454 UBSAN: maria.aria_pack_mdev invalid-shift-exponent

Shifting with 64 is no-op in the the code (no ill effects).

Added a test to not do anything if shift with 64 would happen.
Tested with ma_test_all that test aria_pack.
8ad0910a61da...
8ad0910a61daf097e1da31e5bbc54a81c3d55c0d
Marko Mäkelä
InnoDB review changes

table_name_t::is_create_or_replace(): A new predicate to check for
CREATE OR REPLACE TABLE will rename an old table to
and eventually drop after creating the replacement.

dict_table_t::parse_name(): Do acquire MDL on #sql-create- names
for partitioned tables.

dict_table_rename_in_cache(): On CREATE OR REPLACE TABLE ... SELECT,
forget the original dict_table_t::mdl_name so that purge will
acquire MDL on the #sql-create- name instead. In this way, the
MDL_EXCLUSIVE that the CREATE OR REPLACE TABLE holds on the
user-visible name will not unnecessarily block any purge of old history
until the very end when the #sql-create- table will be dropped.

ha_innobase::delete_table(): Do not check FOREIGN KEY consistency
when dropping an #sql-create- table.

row_rename_table_for_mysql(): Update SYS_FOREIGN.ID also
when renaming to #sql-create- in order to avoid any
duplicate key error when CREATE OR REPLACE TABLE is
creating some FOREIGN KEY constraints by names
that existed in the old table.
7fd0ed6419c9...
7fd0ed6419c9219fb8de3029f8110b5c4ff9dd4d
Georg Richter
CONC-850: Fix out-of-bounds reads in parse_server_packet (GSSAPI)

Replace strncpy/strnlen parsing in plugins/auth/auth_gssapi_client.c
with bounded memchr/memcpy calls to prevent two security vulnerabilities:

1. Stack Buffer Over-Read: Oversized SPN inputs without a NUL byte caused
  strncpy to omit NUL-termination, leading to OOB reads in strlen().
2. Packet Over-Read: Malformed packets missing a NUL terminator caused
  mechanism parsing to read past the end of the packet buffer.

Thanks to Aisle Research for reporting this issue.
7a41993850c9...
7a41993850c983f0589c8d0d22d6f01d56cd75b4
Marko Mäkelä
squash! e61cabc0a2ae1a58c401fc24bb9f25caab6dc9b5

Table_specification_st::end_create_table(): If or_replace(),
acquire MDL_EXCLUSIVE on the backup table name to keep
the InnoDB purge out while we are dropping the table.

(To prevent MDEV-36493 we must defer the acquisition until
it is really necessary.)
77afbfd57812...
77afbfd57812422ef25b5e634dc0060e4652824b
Monty
Speed up the query_cache interface

- Do checks inline before calling query_cache.send_result_to_client()

Things tried that failed:
- I tried removing calling lex_start() and reset_for_next_command() if
  query is cached, but this is would cause duplicated, hard to maintain
  code and would require double resets of things if query cache was
  not used. I have documented in sql_parse.cc what would need to be
  reset for this approach to work.
75b93afcec62...
75b93afcec6238f1c03b3e00886d73d0cd8e1260
Marko Mäkelä
Revert all InnoDB changes
6d7c9d455ad6...
6d7c9d455ad626770b709372cf0c8a6c6ba17fd9
Oleg Smirnov
MDEV-39491 Parallel Query: distribute parallel work more evenly

Lower SPLIT_THRESHOLD from 3 to 2, enabling re-partitioning of chunks
for shallower trees (one root node, one or more levels of internal nodes
and a level of leaf nodes).

Improve the detection of amount of work left: empty run queue does not
always mean the scan is over, it might be that a context pulled
for re-splitting has not enqueued its sub-contexts yet.
66951c20a5ac...
66951c20a5ac169beb123710c902bb2745a9eaf0
Georg Richter
CONC-842: Fix incomplete source bounds check and state desync in mthd_my_read_rows

Commit 85c322f70e47 introduced source buffer bounds validation against end_cp,
but contained edge-case flaws allowing heap buffer over-reads and data leaks:

1. Pointer Underflow: net_field_length() advances 'cp'. If 'cp' advances past
  'end_cp', (end_cp - cp) underflows into a large positive unsigned integer,
  causing 'len > (ulong)(end_cp - cp)' to evaluate to false and bypass
  memcpy bounds enforcement. Fix this by explicitly checking 'cp > end_cp'.

2. State Desynchronization: When 'cp >= end_cp', the previous fix set
  remaining field pointers to NULL and continued loop execution. On truncated
  or malformed packets, completing the row allows corrupted or stale heap
  data from previous queries to be returned. Fix this by failing immediately
  with CR_MALFORMED_PACKET when 'cp >= end_cp'.

3. EOF Status Read Bounds: Added validation before reading warning_count
  and server_status from EOF status packets to prevent out-of-bounds reads
  on short EOF frames.

Thanks to fg0x0 for analyzing and reporting the guard bypass.
5eeea51e83be...
5eeea51e83bebe498427a8987d548cc314070484
Oleg Smirnov
MDEV-39491 Parallel Query: distribute parallel work more evenly

Lower SPLIT_THRESHOLD from 3 to 2, enabling re-partitioning of chunks
for shallower trees (one root node, one or more levels of internal nodes
and a level of leaf nodes).

Improve the detection of amount of work left: empty run queue does not
always mean the scan is over, it might be that a context pulled
for re-splitting has not enqueued its sub-contexts yet.
5ae99c3831fa...
5ae99c3831fae0efdf7f716a049b872d1941d70b
ParadoxV5
MDEV-39880: Reïmplement MDEV-37146 to include MDEV-39519

MDEV-39519 added MySQL 8 compatibility to `mariadb-dump --dump-slave`
by attempting `SHOW REPLICA STATUS` first and then `SHOW SLAVE STATUS`.
This conflicted with MDEV-37146, where `mariadb-dump --dump-slave`
queries either `SELECT … FROM information_schema.SLAVE_STATUS`
or `SHOW ALL SLAVES STATUS` depending on the server version.

This commit merges MDEV-37146 and MDEV-39519:
* Use MDEV-39519’s strategy based on syntax error handling.
* Use MDEV-37146’s preference order:
  1. `SELECT … FROM information_schema.SLAVE_STATUS`
  2. `SHOW ALL SLAVES STATUS`
  3. `SHOW REPLICA STATUS` (for MySQL compatibility _only_)
* Send `STOP`/`START REPLICA SQL_THREAD FOR CHANNEL '…'`
  commands for both MariaDB 10.7+ and MySQL.
* Refactor column indices to variables set when a query succeeds.
* Partially revert MDEV-37146’s removal of `--dump-slave`’s support for
  pre-GTID & pre-multi-source, but tailored for MySQL compatibility;
  coverage for MariaDB pre-10.0 is not fully restored.

Reviewed-by: Brandon Nesterenko <[email protected]>
59e04c3aa079...
59e04c3aa079b61e86ebf8021279ea53f1d6bd0c
ParadoxV5
MDEV-39880: Reïmplement MDEV-37146 to include MDEV-39519

MDEV-39519 added MySQL 8 compatibility to `mariadb-dump --dump-slave`
by attempting `SHOW REPLICA STATUS` first and then `SHOW SLAVE STATUS`.
This conflicted with MDEV-37146, where `mariadb-dump --dump-slave`
queries either `SELECT … FROM information_schema.SLAVE_STATUS`
or `SHOW ALL SLAVES STATUS` depending on the server version.

This commit merges MDEV-37146 and MDEV-39519:
* Use MDEV-39519’s strategy based on syntax error handling.
* Use MDEV-37146’s preference order:
  1. `SELECT … FROM information_schema.SLAVE_STATUS`
  2. `SHOW ALL SLAVES STATUS`
  3. `SHOW REPLICA STATUS` (for MySQL compatibility _only_)
* Send `STOP`/`START REPLICA SQL_THREAD FOR CHANNEL '…'`
  commands for both MariaDB 10.7+ and MySQL.
* Refactor column indices to variables set when a query succeeds.
* Partially revert MDEV-37146’s removal of `--dump-slave`’s support for
  pre-GTID & pre-multi-source, but tailored for MySQL compatibility;
  coverage for MariaDB pre-10.0 is not fully restored.

Reviewed-by: Brandon Nesterenko <[email protected]>
3716d0c58dd4...
3716d0c58dd4a2abdef84cee36d919b9c916802d
Alexander Barkov
MDEV-39518 Allow prepared statements in stored functions in assignment right hand

Allowing prepared statements in stored functions when
a stored function is used in an assignment right hand.

Both DEFAULT clause of a variable initialization and
the right side of the SET statement are supported:

  CREATE PROCEDURE p1()
  BEGIN
    -- case 1: DEFAULT clause
    DECLARE spvar1 INT DEFAULT f1_with_ps(); -- OK

    -- case 2: SP variable assignment statement
    DECLARE spvar2 INT;
    SET spvar2= f1_with_ps(); -- OK
  END;

- Only assignments to SP variables works for now:
  * SET spvar= func_with_ps(); -- OK
  * SET @uvar= func_with_ps(); -- Error

- Only bare function calls are supported for now. Using a function in
  an expression does not make it PS-safe yet:
    SET v= f1()+0;

- The parser now does not reject PS statements in stored functions.
  PS applicability in stored functions is now detected at run time.
  Note, PS statements in triggers are still prohibited by the parser.

- Functions with PS do not acquire MDL locks on tables, and no MDL is
  taken on the routines themselves either. They work like procedures in
  terms of table opening and routine locking: a concurrent DROP FUNCTION
  can complete while such a function is executing.

- Functions with PS are not replicated as a single `SELECT f1()` call.
  They are replicated per-statement, like procedures.

Helper changes:
- Changing the return result for LEX::sp_variable_declarations_init()
  from void to bool to catch errors in the caller properly.

Misc:
- This patch incorporates fixes for the following bugs found during debugging:
  MDEV-39518,MDEV-40224,MDEV-40225,MDEV-40226,MDEV-40227,MDEV-40240,
  MDEV-40285,MDEV-40288,MDEV-40315,MDEV-40318,MDEV-40890,MDEV-40900,
  MDEV-40901,MDEV-40913,MDEV-40914

Assisted-by: Claude - reviews and minor clean-ups
31914fb74029...
31914fb7402925bb95f13394bcb529aae4c9329c
Sergei Petrunia
Tmp: added comments
2362eb264048...
2362eb264048e930dc9eb42da444086d8fce464b
Arcadiy Ivanov
MDEV-41007 Warn when GROUP_CONCAT(DISTINCT) loses rows silently

Give a warning when `GROUP_CONCAT(DISTINCT x)` or
`JSON_ARRAYAGG(DISTINCT x)` returns only part of a group, or nothing at
all, because the walk of the duplicate filter failed. The result is
wrong rather than deliberately cut, and nothing was said about it.

Both build their result in `val_str()` by walking `unique_filter`, and
threw the walk's return value away. `Unique::walk()` reports its own
failures through it, from allocating the merge buffer to reading back
the chunks it merged, so a failure gave a short result, or an empty
one, in silence.

The return value cannot be used on its own. `dump_leaf_key()` also
stops the walk, for two reasons that are not failures: it cuts the
result at `group_concat_max_len`, which it already reports by setting
`result_cut`, and it stops without losing anything once the `LIMIT` is
used up. Reporting every non-zero return as a cut warns about
`GROUP_CONCAT(DISTINCT a LIMIT 5)` returning exactly the five rows
that were asked for.

`dump_leaf_key()` now records that it was the one that stopped the
walk, so `val_str()` asks for the cut value warning only when the walk
itself failed.

The debug keyword `unique_walk_merge_fail` fails the merging walk, and
`main.gconcat_distinct_walk_fail` uses it. The `LIMIT` case needs no
debug build and is checked in `main.gconcat_distinct_spill`.
1ade6437e3b6...
1ade6437e3b6a215e610f93e3b3d9d24d1581df9
Marko Mäkelä
Merge main
12c540ddd4bb...
12c540ddd4bb6e724d02abf69ddbb6f2e07a4c7e
Thirunarayanan Balathandayuthapani
MDEV-19574: innodb_stats_method is not honored when innodb_stats_persistent=ON

Problem:
=======
When persistent statistics are enabled (innodb_stats_persistent=ON),
the innodb_stats_method setting is not properly utilized during
statistics calculation.

The statistics collection functions always use a hardcoded default
behavior for NULL value comparison instead of respecting the
configured stats method. This affects the accuracy of
n_diff_key_vals (distinct key count), particularly for
indexes with nullable columns containing NULL values.

Moreover, stat_n_non_null_key_vals[] was never computed for
persistent statistics; it stayed at the 0 that
dict_stats_empty_index() assigns.

With innodb_stats_method=nulls_ignored, innodb_rec_per_key()
therefore always found n_diff <= n_null and reported one record
per key for every index. This impacts the query optimizer,
which makes decisions based on inaccurate cardinality estimates.

Solution:
========
Introduced IndexLevelStats to collect statistics at a specific
B-tree level during index analysis.

Introduced PageStats to collect statistics for leaf page analysis.

Refactored the following functions:
dict_stats_analyze_index_level() to IndexLevelStats::analyze_level()
dict_stats_analyze_index_for_n_prefix() to IndexLevelStats::sample_leaf_pages()
dict_stats_analyze_index_below_cur() to PageStats::scan_below()
dict_stats_scan_page() to PageStats::scan()

The innodb_stats_method value is read once per table in
dict_stats_update_persistent() and passed down, so that all
indexes of a table are analyzed with the same method.

Add the stats method name to stat_description when
innodb_stats_method has a non-default value. The suffix is
dropped when the description is already full.

Added the new stat name n_nonnull_fld01, n_nonnull_fld02, etc.
with a stats description, to indicate how many non-null values
exist for the nth field of the index. This value is retrieved
and stored in the index statistics
in dict_stats_fetch_index_stats_step(). The counts are per
column, not per n-column prefix.

rec_get_n_blob_pages(): Calculate the number of
externally stored pages for a record, using ceiling division
by the usable BLOB page payload (blob_part_size), which differs
between ROW_FORMAT=COMPRESSED (zip_size minus FIL_PAGE_DATA) and
the other formats (srv_page_size minus the BLOB header and the
page trailer). For ROW_FORMAT=COMPRESSED the length in
the field reference is the uncompressed length, so the result is an
upper bound.

When the leaf level is scanned in full, the number of leaf pages that
were scanned is reported as n_leaf_pages for a multi level index.
Before, result.n_leaf_pages was overwritten with
index->stat_n_leaf_pages, which dict_stats_empty_index() had
just set to 1, so every index that took the full scan path
reported n_leaf_pages=1. Single page indexes report 1.
This changes cardinality estimates and
therefore leads to multiple changes in existing test cases.

Non-null values are counted only at the leaf level, since only leaf
pages hold actual records. A full scan of the leaf level counts them
exactly. When the level is sampled, the per column count is derived
from the sampled leaves with the same formula as n_diff:

  n_ordinary_leaf_pages * n_non_null_all_analyzed_pages
                        / n_leaf_pages_to_analyze

This is an estimate for NOT NULL columns as well: the sampled leaves
may hold fewer or more records than the average, and a dive that
stops at a boring page contributes nothing to the sum while still
counting in the divisor.

innodb_rec_per_key(): stat_n_non_null_key_vals[i] holds the
number of records in which the i-th indexed column alone is
not NULL, while what has to be excluded here is the number
of records whose first i+1 columns are all not NULL,
because that is the population which the n-column
prefix statistic stat_n_diff_key_vals[i] has to be
corrected against when innodb_stats_method=nulls_ignored:
with NULLs compared as unequal, every record carrying a NULL
anywhere in the prefix adds a distinct value of its own to n_diff.

PageStats::scan(): n_non_null is accumulated and assigned only
for leaf pages, so that a non-leaf scan cannot leave a node
pointer count behind when scan_below() stops at a boring page
without reaching a leaf.

IndexLevelStats::reset_for_level() also clears n_diff[], and
dict_stats_analyze_index() zero initializes the buffer backing it, so
that a level scan which finds no records (a failed
btr_pcur_open_level(), or a non-leaf page whose first record is not
marked as the leftmost one on the level) leaves n_diff[] at 0 instead
of stale values.

IndexLevelStats::sample_leaf_pages() returns early when the group
boundaries for the prefix are empty, which is the same condition.

IndexLevelStats::analyze_level(): Instead of copying the last record
of the page, retain the latch on the page until the record has been
compared with the first record of the next page

dict_stats_fetch_index_stats_step() no longer resets
stat_n_non_null_key_vals[] while processing an n_diff_pfxNN row:
dict_stats_empty_table() has already cleared the array before the
fetch, and with n_nonnull_fldNN rows now being read too,
that reset would make the result depend on the order in which the
rows arrive.

dict_stats_save(): now static function in dict0stats.cc that
takes the innodb_stats_method value, and is removed from dict0stats.h.
dict_stats_update_persistent() saves the statistics itself, so its
callers no longer have to.

Replaced btr_rec_get_externally_stored_len() with
rec_get_n_blob_pages() in dict0stats.cc.

btr_rec_get_field_ref_offs() and btr_rec_get_field_ref(),
together with the BTR_BLOB_HDR_* macros, were moved from btr0cur.cc
to btr0cur.h so that rec_get_n_blob_pages() can reuse them;

btr_rec_get_field_ref_offs() is now a noexcept function
returning size_t.

Changed stat_n_diff_key_vals and stat_n_non_null_key_vals from
ib_uint64_t* to uint64_t*

len_is_stored(): simplified to a single comparison, which is
equivalent for the unsigned lengths that it is used with.

Removed the unused UNIV_STATS_DEBUG build macro (univ.i) and
turned the DEBUG_PRINTF() helper in dict0stats.cc into an
unconditional no-op
02da3aca98b7...
02da3aca98b7118d58c06b29209de4e2756998b4
Alessandro Vetere
MDEV-32286 Reuse remembered clustered leaves in secondary-index scans

Row_sel_get_clust_rec_for_mysql::operator() descends the clustered B-tree
from the root for every row whose clustered-index record a secondary-index
scan must read, although consecutive rows often land on the same clustered
leaf page. A non-covering scan needs one for every row, a locking read needs
one whatever the secondary index holds, because an exclusive select lock
type makes ha_innobase::build_template() build its template against the
clustered index, and a covering scan needs one for every row of a secondary
leaf whose PAGE_MAX_TRX_ID its read view cannot see. ANALYZE FORMAT=JSON
charges each descent its full height, and those descents are nearly the
whole cost: the secondary index is charged its own descent and one page for
each further leaf, and nothing per row, because the position that its cursor
holds between two rows is restored optimistically, which latches the leaf
again without counting an access. The range estimate that
ha_innobase::records_in_range() makes for the optimizer is charged over the
same leaves, once more. So pages_accessed is the row count times the height
of the clustered index, plus that handful. 1000 rows over a 2-level
clustered index cost 2006, of which 3 are the scan of the secondary index
and 3 the range estimate, and 750 rows over a 3-level one cost 2291, of
which 27 and 14, where a full table scan of the same data costs 23 and 110.

Remember, in the new row_prebuilt_t::clust_leaf_hint, the
CLUST_LEAF_HINT_SLOTS (4) clustered leaves that the lookups of this
statement reached, most recently used first. Each slot names one leaf: its
page number, copies of its first and last user record truncated to the key
fields, which bound the key range the leaf held when it was remembered, and
the rec_get_offsets() of both. The copies are needed because the page is
unlatched between two lookups, and the offsets spare a lookup the parsing of
them. Several slots serve the scans that alternate between a few leaves,
which one slot cannot serve at all, and a descent refreshes the slot of a
leaf that is remembered already rather than spend a second one on the same
page. A slot owns its key buffers and grows them only when a longer key
arrives, so a row allocates nothing. The used-slot count and the miss
counter are reset per statement in ha_innobase::reset(), matching
autoinc_last_value.

A lookup first compares its key against the remembered ranges, so an
uncorrelated scan settles its misses in memory, with no buffer pool access
and no pages_accessed. Only a covering range is probed, through the new
btr_cur_t::try_leaf_hint(), which acquires the page with buf_page_try_get():
a hint is never derived from a latched parent page, so by the time it is
tried it may precede the caller's already-latched secondary-index leaf in
the latching order, where a blocking wait can deadlock. A stale range costs
a wasted probe or a needless descent, never a wrong result, because the
checks that try_leaf_hint() makes on the latched page remain the sole
authority, and the ranges therefore need no invalidation protocol.

After CLUST_LEAF_HINT_MAX_MISSES (8) consecutive unanswered lookups, a scan
gives the slots up: row_sel_clust_leaf_hint_armed() stops both the test of
the slots and the copies that refresh them, which are the larger half of
their cost. One lookup in CLUST_LEAF_HINT_RETRY (1024) starts the count
again, so a scan whose order becomes correlated only later recovers, and the
trial that this begins refreshes the slots as it goes.

The run is short because a hit saves little where the pages above the leaf
are resident: one buffer pool access and one page-local search for each
level. Measured against the same tree built without the hints, at 16k with a
resident working set and no adaptive hash index, a wholly correlated scan
runs a quarter faster over half the page accesses, a scan that answers three
lookups in five runs level with it over 30% fewer, one whose locality
appears only half way through runs an eighth faster over a quarter fewer,
and a scan that answers nothing stays within the noise. A run of 8 is what
keeps that last one there. A clustered index small enough to stay in cache
is the exception that the run does not catch: it answers often, so the count
never builds, and it saves nothing, because the descent that a hit replaces
costs almost nothing there. Such a scan pays about a tenth.

Where the adaptive hash index is enabled, the hints are neither used nor
collected: its guess solves the same problem better, landing on the record
with no page-local search and no page access to charge. It is off by
default, so the hints are active in a default configuration.

innodb.non_covering_sec_idx_scan measures pages_accessed over key orders
that differ in how closely the secondary order tracks the clustered one, and
eight further tables check query results over the record formats and key
shapes that a clustered-index lookup has to read, down to the metadata
pseudo-record of instant ALTER TABLE, to leaves that split and merge while a
locking read walks them, and to a record that a remembered leaf supplies for
a scan that must then rebuild an older version of it.
non_covering_sec_idx_scan_debug runs the same body with the hints turned
off, through a debug switch that returns before a lookup tests or refreshes
the slots, so a diff of the two .result files is what the hints save: 2006
to 1028 (2-level clustered index), 2291 to 1007 (3-level), 20020 to 15780
(decorrelated), 4006 to 2012 (two interleaved key ranges), 20020 to 20006
(shuffled), 12016 to 9289 (locality in the second half alone) and 20020 to
10042 for a covering scan that FOR UPDATE makes non-covering, where the same
scan without FOR UPDATE costs 20 in both files. The locality count pins the
retry: without it the scan would keep only the 82 hits it makes before it
gives up, and the count would be 11934.
main.rowid_filter_innodb: 90 to 81, and its ahi combination unchanged.
BuildBot (0.8.9) working for the MariaDB project.
Page built: Thu 03 Sep 2026 00:02:07 (UTC)