Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
fe6da0f4b4bd...
fe6da0f4b4bd2f918ab4eeb705976f2fb67a313a
Alexander Barkov
MDEV-19635 System package SYS.DBMS_SQL

In progress
fe6a0428e9b7...
fe6a0428e9b740d4a2c2a0e9034c2e1d672ab596
Rex Johnston
MDEV-35333 Performance Issue on TPC-H Query 18

Query 18 of the TPC-H benchmark looks like this

select
    c_name,
    c_custkey,
    o_orderkey,
    o_orderdate,
    o_totalprice,
    sum(l_quantity)
from
    ORDERS
    join CUSTOMER on c_custkey = o_custkey
    join LINEITEM on o_orderkey = l_orderkey
where
    o_orderkey in
    (
        select
            l_orderkey as ok
        from
            LINEITEM
        group by
            l_orderkey
        having
            sum(l_quantity) > 314
    )
group by
    c_name,
    c_custkey,
    o_orderkey,
    o_orderdate,
    o_totalprice
order by
    o_totalprice desc,
    o_orderdate
limit
    100;

Our derived table is converted into a semi-join, but the optimizer
doesn't know how many rows to expect before materialization, so
doesn't know where in the join order is best.

We need to be able to estimate the selectivity of
having sum(l_quantity) > 314 as well as the cardinality of l_orderkey
in the table LINEITEM.

Here we introduce a framework to calculate grouped selectivity, and perhaps
other ignored, non-grouping predicates in the future.

We add selectivity_estimate() to our Item class as the main entry point
for our calculations.  This method is implemented for
aggregate functions Item_func_{gt,ge,lt,le,eq,ne}.
we look for a ref item, indicating an aggregate function and a constant item.
If found, our ref item is asked for a likely distribution
of values.  Implemented ref items are Item_sum_{sum,min,max,count}
each returning something we can compare to our constant item.

The class Expected_distribution encapsulates both the collection of the
expected values from our aggregate function, as well as the comparison
operators (currently implemented using field->val_real()).
In order to calculate the probability of any aggregate function
comparison, we first calculate the required group size, then the
probability of this group size using a poisson distribution.

We also remove a call to the old get_post_group_estimate() to a newer
estimate_post_group_cardinality() that includes estimates of
item list cardinality and provides an entry point to our new
selectivity estimate above.
fb2e24b2d868...
fb2e24b2d868fcd1baa618173044455ca9287c46
Dave Gosselin
MDEV-38934: ICP, Reverse scan: range access will scan whole index for range w/o max endpoint

Test to show the bug.
fab2be895a06...
fab2be895a0691ac7c667cfc388e683c1a28d84b
bsrikanth-mariadb
MDEV-38805:Store optimizer_context into a IS table

Currently, optimizer context is written as JSON sub-element in the
Optimizer Trace.

In this task, we separate it out from the optimizer trace, and instead
store it in optimizer_context Information Schema table.

The structure of the context is changed to look like below: -
----------------------------------
CREATE TABLE t1 ( ... );
-- in case it is a constant table
REPLACE INTO t1 VALUES (...);

CREATE TABLE t2 ( ... );
...

set @context='{ JSON with all the captured calls }';
set @optimizer_context='context';

... the original query;
----------------------------------

The IS can be used to read the current stored context, as well as to
dump it to a sql file which can later be replayed in a different
environment.

It is done like below: -
--------------------------------------
set optimizer_record_context=ON;
set optimizer_trace=1

-- sample query

select into outfile '/tmp/captured-context.sql' context from information_schema.OPTIMIZER_CONTEXT;
---------------------------------------

All the existing tests are modified to query OPTIMIZER_CONTEXT IS table
e795ea6c5fff...
e795ea6c5fff3a3a175bed40576ba44f53fddaee
sjaakola
MDEV-21935 Mariabackup file limits set incorrectly

Fixed a build time regression happening in x86-debian-12 platform
dcb3e578355c...
dcb3e578355ca196e8e6daf1a39c3690656a9985
Marko Mäkelä
MDEV-38947 Reimplement SET GLOBAL innodb_buffer_pool_size

We deprecate and ignore the parameter innodb_buffer_pool_chunk_size
and let the buffer pool size to be changed in arbitrary 1-megabyte
increments.

innodb_buffer_pool_size_max: A new read-only startup parameter
that specifies the maximum innodb_buffer_pool_size. On 64-bit
systems other than IBM AIX the default is 8 TiB and the minimum
8 MiB. On other systems, the default and minimum are 0, and
the value 0 will be replaced with the initial innodb_buffer_pool_size
rounded up to the allocation unit (2 MiB or 8 MiB).  The maximum value
is 4GiB-2MiB on 32-bit systems and 16EiB-8MiB on 64-bit systems.
This maximum is very likely to be limited further by the operating system.

The status variable Innodb_buffer_pool_resize_status will reflect
the status of shrinking the buffer pool. When no shrinking is in
progress, the string will be empty.

Unlike before, the execution of SET GLOBAL innodb_buffer_pool_size
will block until the requested buffer pool size change has been
implemented, or the execution is interrupted by a KILL statement
a client disconnect, or server shutdown.  If the
buf_flush_page_cleaner() thread notices that we are running out of
memory, the operation may fail with ER_WRONG_USAGE.

SET GLOBAL innodb_buffer_pool_size will be refused
if the server was started with --large-pages (even if
no HugeTLB pages were successfully allocated). This functionality
is somewhat exercised by the test main.large_pages, which now runs
also on Microsoft Windows.  On Linux, explicit HugeTLB mappings are
apparently excluded from the reported Resident Set Size (RSS), and
apparently unshrinkable between mmap(2) and munmap(2).

The buffer pool will be mapped to a contiguous virtual memory area
that will be aligned and partitioned into extents of 8 MiB on
64-bit systems and 2 MiB on 32-bit systems.

Within an extent, the first few innodb_page_size blocks contain
buf_block_t objects that will cover the page frames in the rest
of the extent.  The number of such frames is precomputed in the
array first_page_in_extent[] for each innodb_page_size.
In this way, there is a trivial mapping between
page frames and block descriptors and we do not need any
lookup tables like buf_pool.zip_hash or buf_pool_t::chunk_t::map.

We will always allocate the same number of block descriptors for
an extent, even if we do not need all the buf_block_t in the last
extent in case the innodb_buffer_pool_size is not an integer multiple
of the of extents size.

The minimum innodb_buffer_pool_size is 256*5/4 pages.  At the default
innodb_page_size=16k this corresponds to 5 MiB.  However, now that the
innodb_buffer_pool_size includes the memory allocated for the block
descriptors, the minimum would be innodb_buffer_pool_size=6m.

my_virtual_mem_reserve(), my_virtual_mem_commit(),
my_virtual_mem_decommit(), my_virtual_mem_release():
New interface mostly by Vladislav Vaintroub, to separately
reserve and release virtual address space, as well as to
commit and decommit memory within it.

The function my_virtual_mem_reserve() is only defined for Microsoft Windows.
Other platforms should invoke my_large_virtual_alloc() instead.

my_large_virtual_alloc(): A new function, similar to my_large_malloc(),
for other platforms than Microsoft Windows.
For regular page size allocations, do not specify MAP_NORESERVE nor
MAP_POPULATE, to preserve compatibility with my_large_malloc().

After my_virtual_mem_decommit(), the virtual memory range will be
inaccessible.

opt_super_large_pages: Declare only on Solaris. Actually, this is
specific to the SPARC implementation of Solaris, but because we
lack access to a Solaris development environment, we will not revise
this for other MMU and ISA.

buf_pool_t::chunk_t::create(): Remove.

buf_pool_t::create(): Initialize all n_blocks of the buf_pool.free list.

buf_pool_t::allocate(): Renamed from buf_LRU_get_free_only().

buf_pool_t::LRU_warned: Changed to Atomic_relaxed<bool>,
only to be modified by the buf_flush_page_cleaner() thread.

buf_pool_t::shrink(): Attempt to shrink the buffer pool.
There are 3 possible outcomes: SHRINK_DONE (success),
SHRINK_IN_PROGRESS (the caller may keep trying),
and SHRINK_ABORT (we seem to be running out of buffer pool).
While traversing buf_pool.LRU, release the contended
buf_pool.mutex once in every 32 iterations in order to
reduce starvation. Use lru_scan_itr for efficient traversal,
similar to buf_LRU_free_from_common_LRU_list().
When relocating a buffer page, invalidate the page identifier
of the original page so that buf_pool_t::page_guess()
will not accidentally match it.

buf_pool_t::shrunk(): Update the reduced size of the buffer pool
in a way that is compatible with buf_pool_t::page_guess(),
and invoke my_virtual_mem_decommit().

buf_pool_t::resize(): Before invoking shrink(), run one batch of
buf_flush_page_cleaner() in order to prevent LRU_warn().
Abort if shrink() recommends it, or no blocks were withdrawn in
the past 15 seconds, or the execution of the statement
SET GLOBAL innodb_buffer_pool_size was interrupted.
After successfully shrinking the buffer pool, announce the success.
The size had already been updated in shrunk().  After failing to
shrink the buffer pool, re-enable the adaptive hash index
if it had been enabled before the resizing.

buf_pool_t::first_to_withdraw: The first block descriptor that is
out of the bounds of the shrunk buffer pool.

buf_pool_t::withdrawn: The list of withdrawn blocks.
If buf_pool_t::resize() is aborted before shrink() completes,
we must be able to resurrect the withdrawn blocks in the free list.

buf_pool_t::contains_zip(): Added a parameter for the
number of least significant pointer bits to disregard,
so that we can find any pointers to within a block
that is supposed to be free.

buf_pool_t::is_shrinking(): Return the total number or blocks that
were withdrawn or are to be withdrawn.

buf_pool_t::to_withdraw(): Return the number of blocks that will need to
be withdrawn.

buf_pool_t::usable_size(): Number of usable pages, considering possible
in-progress attempt at shrinking the buffer pool.

buf_pool_t::page_guess(): Try to buffer-fix a guessed block pointer.
Always check that the pointer is within the current buffer pool size
before dereferencing it.

buf_pool_t::get_info(): Replaces buf_stats_get_pool_info().

innodb_init_param(): Refactored. We must first compute
srv_page_size_shift and then determine the valid bounds of
innodb_buffer_pool_size.

buf_buddy_shrink(): Replaces buf_buddy_realloc().
Part of the work is deferred to buf_buddy_condense_free(),
which is being executed when we are not holding any
buf_pool.page_hash latch.

buf_buddy_condense_free(): Do not relocate blocks.

buf_buddy_free_low(): Do not care about buffer pool shrinking.
This will be handled by buf_buddy_shrink() and
buf_buddy_condense_free().

buf_buddy_alloc_zip(): Assert !buf_pool.contains_zip()
when we are allocating from the binary buddy system.
Previously we were asserting this on multiple recursion levels.

buf_buddy_block_free(), buf_buddy_free_low():
Assert !buf_pool.contains_zip().

buf_buddy_alloc_from(): Remove the redundant parameter j.

buf_flush_LRU_list_batch(): Add the parameter to_withdraw
to keep track of buf_pool.n_blocks_to_withdraw.
Keep evicting as long as the buffer pool is being shrunk,
for at most innodb_lru_scan_depth extra blocks.
Disregard the flush limit for pages that are marked as freed in files.

buf_flush_LRU_to_withdraw(): Update the to_withdraw target during
buf_flush_LRU_list_batch().

buf_pool_t::will_be_withdrawn(): Allow also ptr=nullptr (the condition
will not hold for it).

buf_flush_sync_for_checkpoint(): Wait for pending writes, in order
to guarantee progress even if the scheduler is unfair.

buf_do_LRU_batch(): Skip buf_free_from_unzip_LRU_list_batch()
if we are shrinking the buffer pool. In that case, we want
to minimize the page relocations and just finish as quickly
as possible.

buf_LRU_check_size_of_non_data_objects(): Avoid a crash when the
buffer pool is being shrunk.

trx_purge_attach_undo_recs(): Limit purge_sys.n_pages_handled()
in every iteration, in case the buffer pool is being shrunk
in the middle of a purge batch.

recv_sys_t::wait_for_pool(): Also wait for pending writes, so that
previously written blocks can be evicted and reused.

This ports the following changes from the 10.11 branch:
commit b6923420f326ac030e4f3ef89a2acddb45eccb30
commit 027d815546d45513ec597b490f2fa45b567802ba
commit 58a36773090223c97d814a07d57ab35ebf803cc5
commit 669f719cc21286020c95eec11f0d09b74f96639e (MDEV-36489)
commit a096f12ff75595ce51fedf879b71640576f70e52
commit f1a8b7fe95399ebe2a1c4a370e332d61dbf6891a (MDEV-36646)
commit 8fb09426b98583916ccfd4f8c49741adc115bac3 (MDEV-36759)
commit 56e0be34bc5d1e967ad610a9b8e24c3f5553bdd8 (MDEV-36780)
commit bb48d7bc812baf7cbd71c9e41b29fac6288cec97 (MDEV-36781)
commit 7b4b759f136f25336fdc12a5a705258a5846d224 (MDEV-36868)
commit cedfe8eca49506c6b4d2d6868f1014c72caaab36 (MDEV-37250)
commit 55e0c34f4f00ca70ad8d6f0522efa94bb81f74fb (MDEV-37263)
commit 21bb6a3e348f89c5cf23d4ee688c57f6078c7b02 (MDEV-37447)
commit 072c7dc774e7f31974eaa43ec1cbb3b742a1582e (MDEV-38671)
dbd82169ac4b...
dbd82169ac4b05695af800f119e4739d3c863fbf
bsrikanth-mariadb
MDEV-38805:Store optimizer_context into a IS table

Currently, optimizer context is written as JSON sub-element in the
Optimizer Trace.

In this task, we separate it out from the optimizer trace, and instead
store it in optimizer_context Information Schema table.

The structure of the context is changed to look like below: -
----------------------------------
SET var_name1=val1;

SET var_name2=val2;
.
.
.

CREATE TABLE t1 ( ... );
-- in case it is a constant table
REPLACE INTO t1 VALUES (...);

CREATE TABLE t2 ( ... );
...

set @context='{ JSON with all the captured calls }';
set @optimizer_context='context';

... the original query;
----------------------------------

The IS can be used to read the current stored context, as well as to
dump it to a sql file which can later be replayed in a different
environment.

It is done like below: -
--------------------------------------
set optimizer_record_context=ON;
set optimizer_trace=1

-- sample query

select into outfile '/tmp/captured-context.sql' context from information_schema.OPTIMIZER_CONTEXT;
---------------------------------------

All the existing tests are modified to query OPTIMIZER_CONTEXT IS table
d9664ddec78e...
d9664ddec78e23d517887df6311204bd00044240
Marko Mäkelä
fixup! daaf6768b9414aa6e57cda99ec1899690300738f
d6db8ba97e94...
d6db8ba97e944acd436339a5105370325d50726e
Sergei Petrunia
Testcase cleanup
d477356e5b33...
d477356e5b3365a18340ba17ea138eda14014687
Amodh Dhakal
MDEV-38349 Fix assert thd->abort_on_warning == 0 in mysql_insert

When prepare_for_replace() fails, the code returned early via
DBUG_RETURN(1), bypassing the abort label which resets
thd->abort_on_warning to 0. This left the THD in a dirty state,
triggering the dispatch_command assertion at end of command dispatch.

Signed-off-by: Amodh Dhakal <[email protected]>
cb447bc7fe28...
cb447bc7fe28de9b4876612803a8fce9f2a97bcd
Lawrin Novitsky
Fix of the build

Header inclusion was missing
c23f2bac3e1e...
c23f2bac3e1e63396b5e399b9d88ed7836bb981e
Marko Mäkelä
fixup! 99959b558e7089f6611d0cadf0b58b8a608a79b3
bce02070e82a...
bce02070e82a81a0bb5a44b2fd6170c9f926bfa3
Dave Gosselin
MDEV-38934: ICP, Reverse scan: range access will scan whole index for range
            w/o max endpoint

For some range condition x > N, a descending scan will walk off of the
left edge of the range and scan to the beginning of the index if we don't
set an endpoint on the storage engine.

In this patch, the logic to set such an endpoint is used in two places
now, so it's factored to a helper function.
b8cf396b2152...
b8cf396b2152c09563dcf09a3884a625a8e5f9ef
Rex Johnston
MDEV-35333 Performance Issue on TPC-H Query 18

Query 18 of the TPC-H benchmark looks like this

select
    c_name,
    c_custkey,
    o_orderkey,
    o_orderdate,
    o_totalprice,
    sum(l_quantity)
from
    ORDERS
    join CUSTOMER on c_custkey = o_custkey
    join LINEITEM on o_orderkey = l_orderkey
where
    o_orderkey in
    (
        select
            l_orderkey as ok
        from
            LINEITEM
        group by
            l_orderkey
        having
            sum(l_quantity) > 314
    )
group by
    c_name,
    c_custkey,
    o_orderkey,
    o_orderdate,
    o_totalprice
order by
    o_totalprice desc,
    o_orderdate
limit
    100;

Our derived table is converted into a semi-join, but the optimizer
doesn't know how many rows to expect before materialization, so
doesn't know where in the join order is best.

We need to be able to estimate the selectivity of
having sum(l_quantity) > 314 as well as the cardinality of l_orderkey
in the table LINEITEM.

Here we introduce a framework to calculate grouped selectivity, and perhaps
other ignored, non-grouping predicates in the future.

We add selectivity_estimate() to our Item class as the main entry point
for our calculations.  This method is implemented for
aggregate functions Item_func_{gt,ge,lt,le,eq,ne}.
we look for a ref item, indicating an aggregate function and a constant item.
If found, our ref item is asked for a likely distribution
of values.  Implemented ref items are Item_sum_{sum,min,max,count}
each returning something we can compare to our constant item.

The class Expected_distribution encapsulates both the collection of the
expected values from our aggregate function, as well as the comparison
operators (currently implemented using field->val_real()).
In order to calculate the probability of any aggregate function
comparison, we first calculate the required group size, then the
probability of this group size using a poisson distribution.
a5cc4d3bd16e...
a5cc4d3bd16ea470b214e6c9ab0b0d380ff2ae47
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
a458b035990f...
a458b035990fb153d804abd175c2005ed0ee372b
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
9e0f1f3132dc...
9e0f1f3132dc22adabe3bed601e54aafbf46b185
Rex Johnston
MDEV-35333 Performance Issue on TPC-H Query 18

Query 18 of the TPC-H benchmark looks like this

select
    c_name,
    c_custkey,
    o_orderkey,
    o_orderdate,
    o_totalprice,
    sum(l_quantity)
from
    ORDERS
    join CUSTOMER on c_custkey = o_custkey
    join LINEITEM on o_orderkey = l_orderkey
where
    o_orderkey in
    (
        select
            l_orderkey as ok
        from
            LINEITEM
        group by
            l_orderkey
        having
            sum(l_quantity) > 314
    )
group by
    c_name,
    c_custkey,
    o_orderkey,
    o_orderdate,
    o_totalprice
order by
    o_totalprice desc,
    o_orderdate
limit
    100;

Our derived table is converted into a semi-join, but the optimizer
doesn't know how many rows to expect before materialization, so
doesn't know where in the join order is best.

We need to be able to estimate the selectivity of
having sum(l_quantity) > 314 as well as the cardinality of l_orderkey
in the table LINEITEM.

Here we introduce a framework to calculate grouped selectivity, and perhaps
other ignored, non-grouping predicates in the future.

We add selectivity_estimate() to our Item class as the main entry point
for our calculations.  This method is implemented for
aggregate functions Item_func_{gt,ge,lt,le,eq,ne}.
we look for a ref item, indicating an aggregate function and a constant item.
If found, our ref item is asked for a likely distribution
of values.  Implemented ref items are Item_sum_{sum,min,max,count}
each returning something we can compare to our constant item.

The class Expected_distribution encapsulates both the collection of the
expected values from our aggregate function, as well as the comparison
operators (currently implemented using field->val_real()).
In order to calculate the probability of any aggregate function
comparison, we first calculate the required group size, then the
probability of this group size using a poisson distribution.
9a2552da1482...
9a2552da14829d68a77706686977520b656f5baf
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
9893cd2b96e9...
9893cd2b96e91444570ca9665463dffd0e65fb2d
Rucha Deodhar
MDEV-34723: NEW and OLD in a trigger as row variables

Implementation:
NEW and OLD represent the entire table row. So it can be thought of as
list of Item_trigger_field. When the old mode is appropriately set, we
are in a trigger and NEW or OLD is encountered, create Item_trigger_row
object with same constructor as Item_trigger_field, it will also be used
later while creating Item_trigger_field objects.
Populate the m_fields list while fixing fields. Create a corresponding
instruction sp_instr_set_trigger_row which will be used to set the values
9269d79e92b9...
9269d79e92b956a51b8bd06bfb5e1b86d0bd0669
Rucha Deodhar
MDEV-34723: NEW and OLD in a trigger as row variables

Implementation:
NEW and OLD represent the entire table row. So it can be thought of as
list of Item_trigger_field. When the old mode is appropriately set, we
are in a trigger and NEW or OLD is encountered, create Item_trigger_row
object with same constructor as Item_trigger_field, it will also be used
later while creating Item_trigger_field objects.
Populate the m_fields list while fixing fields. Create a corresponding
instruction sp_instr_set_trigger_row which will be used to set the values
8801d543fd12...
8801d543fd12b09e4f827f1f75f6b8d8e153a6c7
Dave Gosselin
MDEV-38934: ICP, Reverse scan: range access will scan whole index for range w/o max endpoint

Test to show the bug.
80aeedacaa8b...
80aeedacaa8b5f258a872d3afa75275cf204af86
Dave Gosselin
MDEV-38934: ICP, Reverse scan: range access will scan whole index for range
            w/o max endpoint

For some range condition x > N, a descending scan will walk off of the
left edge of the range and scan to the beginning of the index if we don't
set an endpoint on the storage engine.

In this patch, the logic to set such an endpoint is used in two places
now, so it's factored to a helper function.
7f023d784a43...
7f023d784a43240ad2f9009f353a8a560edab14b
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
689cf818ff3a...
689cf818ff3a446fceaf9e3d6422d3571a0010d4
Abhishek Bansal
MDEV-35211: Make VEC_FromText return VECTOR type instead of VARBINARY

VEC_FROMTEXT() was incorrectly inheriting its type handler from
Item_str_func, causing it to be identified as VARBINARY. This led
to incorrect column types (VARBINARY instead of VECTOR) when using
CREATE TABLE ... AS SELECT.

This change:
- Overrides type_handler() in Item_func_vec_fromtext to return
  type_handler_vector.
- Overrides create_field_for_create_select() to ensure the field is
  created correctly using the vector type handler during table
  creation.
- Adds a regression test in vector_funcs.test to verify that
  CREATE TABLE ... AS SELECT now correctly creates a column of
  type VECTOR.
5cbc904d4d12...
5cbc904d4d12199ecfac78e3d6243ee8babc8277
forkfun
MDEV-26112 STR_TO_DATE should work with lc_time_names
5c61c7c2bf2f...
5c61c7c2bf2fc9eb83d56a9d5c65e566a038a935
Raghunandan Bhat
MDEV-38864: Use mmap for MEMORY engine allocations

MEMORY engine blocks used to store data and indexes are allocated using
malloc/my_malloc. This created internal fragmentation within the system
allocator.

Switch to memory mapping for MEMORY engine's allocations.
4a2972117c13...
4a2972117c1335d4a9683d1868e7431b05e2bf42
Rex Johnston
MDEV-35333 Performance Issue on TPC-H Query 18

Query 18 of the TPC-H benchmark looks like this

select
    c_name,
    c_custkey,
    o_orderkey,
    o_orderdate,
    o_totalprice,
    sum(l_quantity)
from
    ORDERS
    join CUSTOMER on c_custkey = o_custkey
    join LINEITEM on o_orderkey = l_orderkey
where
    o_orderkey in
    (
        select
            l_orderkey as ok
        from
            LINEITEM
        group by
            l_orderkey
        having
            sum(l_quantity) > 314
    )
group by
    c_name,
    c_custkey,
    o_orderkey,
    o_orderdate,
    o_totalprice
order by
    o_totalprice desc,
    o_orderdate
limit
    100;

Our derived table is converted into a semi-join, but the optimizer
doesn't know how many rows to expect before materialization, so
doesn't know where in the join order is best.

We need to be able to estimate the selectivity of
having sum(l_quantity) > 314 as well as the cardinality of l_orderkey
in the table LINEITEM.

Here we introduce a framework to calculate grouped selectivity, and perhaps
other ignored, non-grouping predicates in the future.

We add selectivity_estimate() to our Item class as the main entry point
for our calculations.  This method is implemented for
aggregate functions Item_func_{gt,ge,lt,le,eq,ne}.
we look for a ref item, indicating an aggregate function and a constant item.
If found, our ref item is asked for a likely distribution
of values.  Implemented ref items are Item_sum_{sum,min,max,count}
each returning something we can compare to our constant item.

The class Expected_distribution encapsulates both the collection of the
expected values from our aggregate function, as well as the comparison
operators (currently implemented using field->val_real()).
In order to calculate the probability of any aggregate function
comparison, we first calculate the required group size, then the
probability of this group size using a poisson distribution.

We also remove a call to the old get_post_group_estimate() to a newer
estimate_post_group_cardinality() that includes estimates of
item list cardinality and provides an entry point to our new
selectivity estimate above.
47492eba69ca...
47492eba69ca9d02124529269264dc54bc0d205a
Marko Mäkelä
MDEV-38947 Reimplement SET GLOBAL innodb_buffer_pool_size

We deprecate and ignore the parameter innodb_buffer_pool_chunk_size
and let the buffer pool size to be changed in arbitrary 1-megabyte
increments.

innodb_buffer_pool_size_max: A new read-only startup parameter
that specifies the maximum innodb_buffer_pool_size. On 64-bit
systems other than IBM AIX the default is 8 TiB and the minimum
8 MiB. On other systems, the default and minimum are 0, and
the value 0 will be replaced with the initial innodb_buffer_pool_size
rounded up to the allocation unit (2 MiB or 8 MiB).  The maximum value
is 4GiB-2MiB on 32-bit systems and 16EiB-8MiB on 64-bit systems.
This maximum is very likely to be limited further by the operating system.

The status variable Innodb_buffer_pool_resize_status will reflect
the status of shrinking the buffer pool. When no shrinking is in
progress, the string will be empty.

Unlike before, the execution of SET GLOBAL innodb_buffer_pool_size
will block until the requested buffer pool size change has been
implemented, or the execution is interrupted by a KILL statement
a client disconnect, or server shutdown.  If the
buf_flush_page_cleaner() thread notices that we are running out of
memory, the operation may fail with ER_WRONG_USAGE.

SET GLOBAL innodb_buffer_pool_size will be refused
if the server was started with --large-pages (even if
no HugeTLB pages were successfully allocated). This functionality
is somewhat exercised by the test main.large_pages, which now runs
also on Microsoft Windows.  On Linux, explicit HugeTLB mappings are
apparently excluded from the reported Redident Set Size (RSS), and
apparently unshrinkable between mmap(2) and munmap(2).

The buffer pool will be mapped to a contiguous virtual memory area
that will be aligned and partitioned into extents of 8 MiB on
64-bit systems and 2 MiB on 32-bit systems.

Within an extent, the first few innodb_page_size blocks contain
buf_block_t objects that will cover the page frames in the rest
of the extent.  The number of such frames is precomputed in the
array first_page_in_extent[] for each innodb_page_size.
In this way, there is a trivial mapping between
page frames and block descriptors and we do not need any
lookup tables like buf_pool.zip_hash or buf_pool_t::chunk_t::map.

We will always allocate the same number of block descriptors for
an extent, even if we do not need all the buf_block_t in the last
extent in case the innodb_buffer_pool_size is not an integer multiple
of the of extents size.

The minimum innodb_buffer_pool_size is 256*5/4 pages.  At the default
innodb_page_size=16k this corresponds to 5 MiB.  However, now that the
innodb_buffer_pool_size includes the memory allocated for the block
descriptors, the minimum would be innodb_buffer_pool_size=6m.

my_large_virtual_alloc(): A new function, similar to my_large_malloc().

my_virtual_mem_reserve(), my_virtual_mem_commit(),
my_virtual_mem_decommit(), my_virtual_mem_release():
New interface mostly by Vladislav Vaintroub, to separately
reserve and release virtual address space, as well as to
commit and decommit memory within it.

The function my_virtual_mem_reserve() is only defined for Microsoft Windows.
Other platforms should invoke my_large_virtual_alloc() instead.

my_large_virtual_alloc(): Define only outside Microsoft Windows.
For regular page size allocations, do not specify MAP_NORESERVE nor
MAP_POPULATE, to preserve compatibility with my_large_malloc().

After my_virtual_mem_decommit(), the virtual memory range will be
inaccessible.

opt_super_large_pages: Declare only on Solaris. Actually, this is
specific to the SPARC implementation of Solaris, but because we
lack access to a Solaris development environment, we will not revise
this for other MMU and ISA.

buf_pool_t::chunk_t::create(): Remove.

buf_pool_t::create(): Initialize all n_blocks of the buf_pool.free list.

buf_pool_t::allocate(): Renamed from buf_LRU_get_free_only().

buf_pool_t::LRU_warned: Changed to Atomic_relaxed<bool>,
only to be modified by the buf_flush_page_cleaner() thread.

buf_pool_t::shrink(): Attempt to shrink the buffer pool.
There are 3 possible outcomes: SHRINK_DONE (success),
SHRINK_IN_PROGRESS (the caller may keep trying),
and SHRINK_ABORT (we seem to be running out of buffer pool).
While traversing buf_pool.LRU, release the contended
buf_pool.mutex once in every 32 iterations in order to
reduce starvation. Use lru_scan_itr for efficient traversal,
similar to buf_LRU_free_from_common_LRU_list().
When relocating a buffer page, invalidate the page identifier
of the original page so that buf_pool_t::page_guess()
will not accidentally match it.

buf_pool_t::shrunk(): Update the reduced size of the buffer pool
in a way that is compatible with buf_pool_t::page_guess(),
and invoke my_virtual_mem_decommit().

buf_pool_t::resize(): Before invoking shrink(), run one batch of
buf_flush_page_cleaner() in order to prevent LRU_warn().
Abort if shrink() recommends it, or no blocks were withdrawn in
the past 15 seconds, or the execution of the statement
SET GLOBAL innodb_buffer_pool_size was interrupted.
After successfully shrinking the buffer pool, announce the success.
The size had already been updated in shrunk().  After failing to
shrink the buffer pool, re-enable the adaptive hash index
if it had been enabled before the resizing.

buf_pool_t::first_to_withdraw: The first block descriptor that is
out of the bounds of the shrunk buffer pool.

buf_pool_t::withdrawn: The list of withdrawn blocks.
If buf_pool_t::resize() is aborted before shrink() completes,
we must be able to resurrect the withdrawn blocks in the free list.

buf_pool_t::contains_zip(): Added a parameter for the
number of least significant pointer bits to disregard,
so that we can find any pointers to within a block
that is supposed to be free.

buf_pool_t::is_shrinking(): Return the total number or blocks that
were withdrawn or are to be withdrawn.

buf_pool_t::to_withdraw(): Return the number of blocks that will need to
be withdrawn.

buf_pool_t::usable_size(): Number of usable pages, considering possible
in-progress attempt at shrinking the buffer pool.

buf_pool_t::page_guess(): Try to buffer-fix a guessed block pointer.
Always check that the pointer is within the current buffer pool size
before dereferencing it.

buf_pool_t::get_info(): Replaces buf_stats_get_pool_info().

innodb_init_param(): Refactored. We must first compute
srv_page_size_shift and then determine the valid bounds of
innodb_buffer_pool_size.

buf_buddy_shrink(): Replaces buf_buddy_realloc().
Part of the work is deferred to buf_buddy_condense_free(),
which is being executed when we are not holding any
buf_pool.page_hash latch.

buf_buddy_condense_free(): Do not relocate blocks.

buf_buddy_free_low(): Do not care about buffer pool shrinking.
This will be handled by buf_buddy_shrink() and
buf_buddy_condense_free().

buf_buddy_alloc_zip(): Assert !buf_pool.contains_zip()
when we are allocating from the binary buddy system.
Previously we were asserting this on multiple recursion levels.

buf_buddy_block_free(), buf_buddy_free_low():
Assert !buf_pool.contains_zip().

buf_buddy_alloc_from(): Remove the redundant parameter j.

buf_flush_LRU_list_batch(): Add the parameter to_withdraw
to keep track of buf_pool.n_blocks_to_withdraw.
Keep evicting as long as the buffer pool is being shrunk,
for at most innodb_lru_scan_depth extra blocks.
Disregard the flush limit for pages that are marked as freed in files.

buf_flush_LRU_to_withdraw(): Update the to_withdraw target during
buf_flush_LRU_list_batch().

buf_pool_t::will_be_withdrawn(): Allow also ptr=nullptr (the condition
will not hold for it).

buf_flush_sync_for_checkpoint(): Wait for pending writes, in order
to guarantee progress even if the scheduler is unfair.

buf_do_LRU_batch(): Skip buf_free_from_unzip_LRU_list_batch()
if we are shrinking the buffer pool. In that case, we want
to minimize the page relocations and just finish as quickly
as possible.

buf_LRU_check_size_of_non_data_objects(): Avoid a crash when the
buffer pool is being shrunk.

trx_purge_attach_undo_recs(): Limit purge_sys.n_pages_handled()
in every iteration, in case the buffer pool is being shrunk
in the middle of a purge batch.

recv_sys_t::wait_for_pool(): Also wait for pending writes, so that
previously written blocks can be evicted and reused.

This ports the following changes from the 10.11 branch:
commit b6923420f326ac030e4f3ef89a2acddb45eccb30
commit 027d815546d45513ec597b490f2fa45b567802ba
commit 58a36773090223c97d814a07d57ab35ebf803cc5
commit 669f719cc21286020c95eec11f0d09b74f96639e (MDEV-36489)
commit a096f12ff75595ce51fedf879b71640576f70e52
commit f1a8b7fe95399ebe2a1c4a370e332d61dbf6891a (MDEV-36646)
commit 8fb09426b98583916ccfd4f8c49741adc115bac3 (MDEV-36759)
commit 56e0be34bc5d1e967ad610a9b8e24c3f5553bdd8 (MDEV-36780)
commit bb48d7bc812baf7cbd71c9e41b29fac6288cec97 (MDEV-36781)
commit 7b4b759f136f25336fdc12a5a705258a5846d224 (MDEV-36868)
commit cedfe8eca49506c6b4d2d6868f1014c72caaab36 (MDEV-37250)
commit 55e0c34f4f00ca70ad8d6f0522efa94bb81f74fb (MDEV-37263)
commit 21bb6a3e348f89c5cf23d4ee688c57f6078c7b02 (MDEV-37447)
commit 072c7dc774e7f31974eaa43ec1cbb3b742a1582e (MDEV-38671)
44f45a8fb34c...
44f45a8fb34c0dd525bb546a4b7ae63df7ade3a4
Sergei Golubchik
MDEV-23507 Wrong duplicate key value printed in ER_DUP_ENTRY

repair_by_sort() does not use table->record[0]
but print_keydup_error() expects to see the conflicting row there.
2f921d7eb66e...
2f921d7eb66efac0adc9f502b5ba9010d783d5ad
Dave Gosselin
MDEV-38921: Wrong result for range w/o min endpoint, ORDER BY DESC and ICP

Test to show the bug.
29310b5c7a3d...
29310b5c7a3da77620a030b1a51c3e1ebbccc56e
Raghunandan Bhat
MDEV-38864: Use mmap for MEMORY engine allocations

MEMORY engine blocks used to store data and indexes are allocated using
malloc/my_malloc. This created internal fragmentation within the system
allocator.

Switch to memory mapping for MEMORY engine's allocations.
1855454d7749...
1855454d7749e3a38e11c4d58bec270ea768018a
Thirunarayanan Balathandayuthapani
Merge branch 10.6 into 10.11
14acea7e2f65...
14acea7e2f6517f838b540b09a34a7f06efbeda3
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
1317b930da7b...
1317b930da7b156b9951d5193ebd55a575850257
Dave Gosselin
MDEV-38921: Wrong result for range w/o min endpoint, ORDER BY DESC and ICP

In QUICK_SELECT_DESC::get_next(), when starting to scan a range with
no start endpoint, like "key < 10", clear the end_range on the storage engine.
We could have scanned another range before, like "key BETWEEN 20 and 30"
and after this the engine's end_range points to its left endpoint, the "20".

This is necessary especially when there are multiple ranges considered and
a later range indicates that there's no minimum endpoint (such as in the
attached test case), thus an earlier range endpoint must be removed/cleared.
0e229fe3c217...
0e229fe3c217dc714e7dee10949a55dd9ad66fb2
Rex Johnston
MDEV-35333 Performance Issue on TPC-H Query 18

A patch illustrating how estimating the selectivity of a
grouping derived table with a having clause at 0.5 can have
a positive effect on the execution time of TPC-H Query#18

We introduce selectivity_estimate() into the Item heirarchy.
This is called from estimate_post_group_cardinality().
estimate_post_group_cardinality() is substituted for
get_post_group_estimate() in Item_in_subselect::optimize()
for grouping subqueries.
0d693de5638f...
0d693de5638fa94df0a64eb8bc150ccfe182b8ba
bsrikanth-mariadb
MDEV-38805:Store optimizer_context into a IS table

Currently, optimizer context is written as JSON sub-element in the
Optimizer Trace.

In this task, we separate it out from the optimizer trace, and instead
store it in optimizer_context Information Schema table.

The structure of the context is changed to look like below: -
----------------------------------
SET var_name1=val1;

SET var_name2=val2;
.
.
.

CREATE TABLE t1 ( ... );
-- in case it is a constant table
REPLACE INTO t1 VALUES (...);

CREATE TABLE t2 ( ... );
...

set @context='{ JSON with all the captured calls }';
set @optimizer_context='context';

... the original query;
----------------------------------

The IS can be used to read the current stored context, as well as to
dump it to a sql file which can later be replayed in a different
environment.

It is done like below: -
--------------------------------------
set optimizer_record_context=ON;
set optimizer_trace=1

-- sample query

select into outfile '/tmp/captured-context.sql' context from information_schema.OPTIMIZER_CONTEXT;
---------------------------------------

All the existing tests are modified to query OPTIMIZER_CONTEXT IS table
0bfce6296a2f...
0bfce6296a2fa1194216b16e563a86e0d65fa082
Thirunarayanan Balathandayuthapani
Merge branch 10.6 into 10.11
044a3e856251...
044a3e85625112d1db286c8d1ba80b73146771ee
Vladislav Vaintroub
MDEV-14443 DENY statement

Implements DENY/REVOKE DENY and associated tasks.
0270068f7187...
0270068f7187c702e457ec00b35aa055b5a5e99e
Raghunandan Bhat
MDEV-38864: Use mmap for MEMORY engine allocations

MEMORY engine blocks used to store data and indexes are allocated using
malloc/my_malloc. This created internal fragmentation within the system
allocator.

Switch to memory mapping for MEMORY engine's allocations.
BuildBot (0.8.9) working for the MariaDB project.
Page built: Tue 03 Mar 2026 11:46:58 (UTC)