Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Vladislav Vaintroub
MDEV-33480 Windows MSI installer, convert from WIX v3 to supported versions

Custom actions : get rid of wcautil and dutil library dependencies
They are not an official part of WiX distribution anymore.

Replace the functionality with native MSI api, the Wix helpers
we used were thin wrappers anyway.

Also replace the VBScript FakeFailure custom action with a native
one, VBScript is deprecated in Windows and scheduled for removal.

Read MSI properties with a helper that handles any length and reports
errors. Values are no longer silently truncated to MAX_PATH, e.g a long
data directory could be cut to its parent, and then removed.

Failed custom actions return ERROR_INSTALL_FAILURE, some failures were
silently turned into success before. Use the W versions of MSI functions
only. Log() is a function now, so that it works with any preprocessor.
Vladislav Vaintroub
MDEV-33480 Windows MSI installer, convert from WIX v3 to supported versions

WiX v3 is out of support. Support legacy WiX v3.x (default) and modern
WiX (wix.exe, newer than v3). WXS sources stay in v3 format, modern WiX
(-DWITH_WIX3=OFF) builds them after on-the-fly 'wix convert'.

Two WXS edits make the converted sources valid: ALLUSERS property is
replaced by InstallScope='perMachine', and ProgramMenuFolder is moved
from under INSTALLDIR to TARGETDIR.

WIX_ACCEPT_EULA=ON accepts the EULA of WiX 7 and later, if WiX asks for it.

A failed WiX step is reported by name.
Vladislav Vaintroub
CA: ERROR_INSTALL_FAILURE on failure, Log() as a function, W functions only; CMake: name the failed WiX step
Thirunarayanan Balathandayuthapani
MDEV-41494  Server aborts on crash recovery after an interrupted instant ALTER TABLE reorders/drops columns

Problem:
========
  After an instant ALTER TABLE that reorders or drops columns,
innobase_instant_try() deletes/updates SYS_COLUMNS, SYS_VIRTUAL
and SYS_TABLES.N_COLS, and only afterwards writes the hidden
metadata record that describes the old column layout in the
clustered index. These are separate mini-transactions within
the same DDL transaction.

If the server is killed after the SYS_COLUMNS/SYS_TABLES updates
were made durable but before the metadata record was written,
and recovery does not roll back the transaction then
table's dictionary definition and its on-disk metadata record go out
of sync.

On the next load, btr_cur_instant_init_low() correctly
detects this as "missing instant ALTER metadata" or "contains
unrecognizable instant ALTER metadata" and sets table->corrupted,
but it did not set table->file_unreadable. Since
dict_table_t::is_readable() only tests file_unreadable, the table
was left in an inconsistent corrupted-but-readable state.

Solution:
=========
btr_cur_instant_init_low(): Also set index->table->file_unreadable
in the "missing instant ALTER metadata" and "incompatible instant
ALTER metadata" error paths, consistent with every other corruption
path in this function.
Oleksandr Byelkin
MDEV-32401 expression cache lead to crash if table of wrong type created and the cache switched off

1) take into account TMP_TABLE_ALL_COLUMNS when
  we are modifying agg_item->result_field
2) remove unused now "bool materialized_subquery;"

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Vladislav Vaintroub
CMake: drop EXTRA_WIX_BUILD_ARGS
Marko Mäkelä
fixup! 4898afa8c169aee679661e6848320a7f1c9d890a
Vladislav Vaintroub
MDEV-33480 Windows MSI installer, convert from WIX v3 to supported versions

Custom actions : get rid of wcautil and dutil library dependencies
They are not an official part of WiX distribution anymore.

Replace the functionality with native MSI api, the Wix helpers
we used were thin wrappers anyway.

Also replace the VBScript FakeFailure custom action with a native
one, VBScript is deprecated in Windows and scheduled for removal.

Read MSI properties with a helper that handles any length and reports
errors. Values are no longer silently truncated to MAX_PATH, e.g a long
data directory could be cut to its parent, and then removed.

A custom action that fails to read a property or to call a Windows API
returns ERROR_INSTALL_FAILURE, it was silently ignored before.
Oleg Smirnov
MDEV-32412 Pushdown from HAVING: Item_func is immutable while arguments are not

Fix-up for commit e97560eac03 (MDEV-28958), which made set_extraction_flag()
ignore basic constants so that the read-only Item_true/Item_false singletons
are never written to. Because of that, the callers that mark a subtree with
MARKER_IMMUTABLE before pushing a condition from HAVING into WHERE cannot
mark basic constants inside it.

Item::cleanup_excluding_immutables_processor() did not know about this
exception and cleaned such items up, unfixing them while their marked parents
stayed fixed. fix_fields() does not descend into fixed items, so the constant
was left unfixed and Item_direct_view_ref::used_tables() later dereferenced a
NULL null_ref_table. Skip basic constants there too.
Marko Mäkelä
fixup! 4898afa8c169aee679661e6848320a7f1c9d890a
Aleksey Midenkov
MDEV-25176 Shutdown hangs if event scheduler is started during shutdown

SET GLOBAL event_scheduler=1 executed by a connection during shutdown
could start the scheduler thread after close_connections() called
Events::stop(). Nothing stops this thread and close_connections()
waits for it forever.

Now it hangs instead of a crash since MDEV-30260 fix: it moved
deinit() from close_connections() to clean_up(), so the scheduler is
not freed under the running thread anymore.

The fix makes Event_scheduler::start() do nothing when abort_loop is
set. abort_loop is set before close_connections() and is checked under
LOCK_scheduler_state, which stop() also takes, so the scheduler is
either started before stop() and stopped by it, or not started at all.
No error needed: the server is shutting down.
Alexey (Holyfoot) Botchkov
MDEV-40394 XML schema fails on self-referencing type.

A schema type that refers to itself, directly or through other types or
groups, was validated with a single shared compositor state per type.
A nested use of the type reset the state of the outer one, so
XMLISVALID() gave wrong results for recursive schemas.

Now every recursion level of an XMLSchema_user_type gets its own copy
of the type's content (get_copy()/copy_tags()). Copies are kept on a
per-type stack while validating, and released copies are put in
m_c_free and reused, so memory doesn't grow with each validated row.
reset_types() unwinds any state left by a validation that aborted.

Other changes:
- XMLSchema_item::is_validate_done() is replaced with
  ::end_validation(st). It is no longer a pure check: it pops the
  compositor stack and must be paired with validate_prepare().
- validate_prepare() can now fail, and the failure is propagated.
- A recursion depth limit (MAX_RECURSION_DEPTH, 1024) is added, counted
  per type. When it is hit, or on OOM during validation, XMLISVALID()
  returns errors/warnings.
- XMLSchema_group_def now derives from XMLSchema_user_type, so groups
  get the same copy mechanism. As a side effect <xs:group final=...>
  is accepted.
- XMLSchema_group_reference now counts occurrences, so minOccurs and
  maxOccurs of a group reference are enforced.
- XMLSchema_group_def::check_type() checks for circular groups, and
  XMLSchema_attribute::check_type() is removed (named types and groups
  are checked globally).
- The circular-reference error names the construct:
  "complexType B has bad circular references" instead of
  "type B has ...". The affected .result lines are re-recorded.
- Hook chains (m_next_tag) are replaced with List<> because tags are
  now copied.
Vladislav Vaintroub
MDEV-33480 Windows MSI installer, convert from WIX v3 to supported versions

WiX v3 is out of support. Support legacy WiX v3.x (default) and modern
WiX (wix.exe, newer than v3). WXS sources stay in v3 format, modern WiX
(-DWITH_WIX3=OFF) builds them after on-the-fly 'wix convert'.

Two WXS edits make the converted sources valid: ALLUSERS property is
replaced by InstallScope='perMachine', and ProgramMenuFolder is moved
from under INSTALLDIR to TARGETDIR.

WIX_ACCEPT_EULA=ON accepts the EULA of WiX 7 and later, if WiX asks for it.

A failed WiX step is reported by name.
Vladislav Vaintroub
CA: review fixes, WiX version message
Khaled Riyad
MDEV-41408 MTR prepared statement protocol skipping statements starting with '('

With --ps-protocol, mysqltest runs a SELECT a second time and compares
the results. The regex ps2_re only matched a leading SELECT, so a
statement starting with '(' was executed only once.

Allow any number of '(', each optionally followed by spaces, before
SELECT in ps2_re.
Vladislav Vaintroub
MDEV-33480 Windows MSI installer, convert from WIX v3 to supported versions

Support legacy WiX v3.x (default) and modern WiX (wix.exe, v5 or later).
WXS sources stay in v3 format, modern WiX (-DWITH_WIX3=OFF) builds them
after on-the-fly 'wix convert'.

WIX_ACCEPT_EULA=ON accepts the EULA of WiX 7 and later, if WiX asks for it.
Rex Johnston
MDEV-41179 MAX_KEY_PARTS ranges in select causing SEGV

When a select containing 32 ranges is made on a table containing a
compound key with 32 parts, the range optimizer can run off the end
of a stack variable, invalidly overwriting subsequent stack variables.

In the struct st_sel_arg_range_seq, we have an array
  RANGE_SEQ_ENTRY stack[MAX_REF_PARTS];

MAX_REF_PARTS is 32.

check_quick_select
  / sel_arg_range_seq_init
    initialises stack[0] as NOT a key part
  / sel_arg_range_seq_next
    iterates through the key parts, adding key part n to stack[n+1]

key part #32 gets referenced by step_down_to(), setting seq->i off the
end of the array.

Fix: RANGE_SEQ_ENTRY stack[MAX_REF_PARTS+1];

The above change exposed an issue with key length calculation on
MS Windows.  Calling make_prev_keypart_map(32) caused the resultant
bitmap to be calculated as (1UL << 32) - 1.  Using the MSVC compiler
this resulted in an empty key length calculation during
handler::index_read_map, causing an assertion in ha_innobase::index_read().

As we only need 32 bits to represent our key map, we change the type thus
-typedef ulong key_part_map;
+typedef uint32 key_part_map;

We correct make_keypart_map() and make_prev_keypart_map() to call our
overflow safe my_set_bits().  We also correct bka_range_seq_next()
and bkah_range_seq_next() to use make_prev_keypart_map().

We also add some DBUG_ASSERTS in key_part_map processing elsewhere,
exposing some issues in our BNLH implementation.  We cap the number of
keyuse parts here, altering the explain output of 2 of our tests.

Numerous places needed bit shift operations altered to use
make*keymap_part and various format strings needed to be corrected.
Kristian Nielsen
MDEV-41217: Inefficient memory management by parallel slave

The rpl_group_info object is used to keep track of a lot of different state
in an event group/transaction across events. It is also used in parallel
replication for a small set of values that are enqueued by the SQL driver
thread for worker threads to handle. This wastes a lot of memory when a
large number of event groups are enqueued (eg. in case of slave lag), since
most of the rpl_group_info is not used in the queue.

This patch changes the queue to contain only the rgi_queued_part object (ie.
rgi->q), not the full rgi, saving the memory for the parts of rgi that are
not needed in the queue. With this patch, there is one rgi for every worker
thread independent of the amount of queued event groups (as well as one
serial_rgi in the SQL driver thread).

Signed-off-by: Kristian Nielsen <[email protected]>
Kristian Nielsen
MDEV-41217: Inefficient memory management by parallel slave

Remove from rpl_parallel_thread::queued_event the names of relaylog and
master binlog. Each one is an array of size FN_REFLEN, thus we save 1024
bytes of memory for each event queued for parallel replication worker
threads. This can be a very large saving when the slave is lagging behind
and many events are queued for worker threads.

Instead, reference these names from the inuse_relaylog object. This already
contains the relaylog name; this patch adds also the master binlog name in
inuse_relaylog. Since we rotate the slave relaylog whenever we see a master
binlog rotate, there can only ever be one master binlog file name associated
with events in one relaylog.

Signed-off-by: Kristian Nielsen <[email protected]>
Jan Lindström
Merge 10.11 to 11.4.
Kristian Nielsen
Fix inuse_relaylog atomics memory ordering

The inuse_relaylog is refcounted. The SQL driver thread first increments
queued_count until everything in the relaylog has been queued, then sets
completed=true. The worker threads increment dequeued_count, so that once
completed==true and queued_count==dequeued_count we know the inuse_relaylog
object can be safely de-allocated.

The accounting is done with atomic counters; however everything with
std::memory_order_relaxed ordering. This is not correct on weakly ordered
CPU architectures.

When we set completed==true, it most be done with release semantics so that
we know that all increments to queued_count will be visible to other
threads; this way another thread will not see completed==true with a too low
queued_count value. Similarly, when we increment dequeued_count it must be
done with release semantics so that previous accesses to inuse_relaylog will
have completed before the dequeue is visible to other threads. And the check
for completed==true must be done with matching acquire semantics.

Signed-off-by: Kristian Nielsen <[email protected]>
Kristian Nielsen
MDEV-41217: Inefficient memory management by parallel slave

Remove from rpl_parallel_thread::queued_event the names of relaylog and
master binlog. Each one is an array of size FN_REFLEN, thus we save 1024
bytes of memory for each event queued for parallel replication worker
threads. This can be a very large saving when the slave is lagging behind
and many events are queued for worker threads.

Instead, reference these names from the inuse_relaylog object. This already
contains the relaylog name; this patch adds also the master binlog name in
inuse_relaylog. Since we rotate the slave relaylog whenever we see a master
binlog rotate, there can only ever be one master binlog file name associated
with events in one relaylog.

Signed-off-by: Kristian Nielsen <[email protected]>
Vladislav Vaintroub
CMake: link wixca with shell32 and advapi32
Kristian Nielsen
Fix inuse_relaylog atomics memory ordering

The inuse_relaylog is refcounted. The SQL driver thread first increments
queued_count until everything in the relaylog has been queued, then sets
completed=true. The worker threads increment dequeued_count, so that once
completed==true and queued_count==dequeued_count we know the inuse_relaylog
object can be safely de-allocated.

The accounting is done with atomic counters; however everything with
std::memory_order_relaxed ordering. This is not correct on weakly ordered
CPU architectures.

When we set completed==true, it most be done with release semantics so that
we know that all increments to queued_count will be visible to other
threads; this way another thread will not see completed==true with a too low
queued_count value. Similarly, when we increment dequeued_count it must be
done with release semantics so that previous accesses to inuse_relaylog will
have completed before the dequeue is visible to other threads. And the check
for completed==true must be done with matching acquire semantics.

Signed-off-by: Kristian Nielsen <[email protected]>
Marko Mäkelä
squash! e0ae58b3f720814805bec494262f3a3ebc0467f6

InnoDB_backup::context::tracked: Correctly track the log files.

InnoDB_backup::archive_create(), innodb_backup_archive_create():
Hook for log_t::archive_create()
Rex Johnston
MDEV-41179 MAX_KEY_PARTS ranges in select causing SEGV

When a select containing 32 ranges is made on a table containing a
compound key with 32 parts, the range optimizer can run off the end
of a stack variable, invalidly overwriting subsequent stack variables.

In the struct st_sel_arg_range_seq, we have an array
  RANGE_SEQ_ENTRY stack[MAX_REF_PARTS];

MAX_REF_PARTS is 32.

check_quick_select
  / sel_arg_range_seq_init
    initialises stack[0] as NOT a key part
  / sel_arg_range_seq_next
    iterates through the key parts, adding key part n to stack[n+1]

key part #32 gets referenced by step_down_to(), setting seq->i off the
end of the array.

Fix: RANGE_SEQ_ENTRY stack[MAX_REF_PARTS+1];

The above change exposed an issue with key length calculation on
MS Windows.  Calling make_prev_keypart_map(32) caused the resultant
bitmap to be calculated as (1UL << 32) - 1.  Using the MSVC compiler
this resulted in an empty key length calculation during
handler::index_read_map, causing an assertion in ha_innobase::index_read().

As we only need 32 bits to represent our key map, we change the type thus
-typedef ulong key_part_map;
+typedef uint32 key_part_map;

We correct make_keypart_map() and make_prev_keypart_map() to call our
overflow safe my_set_bits().  We also correct bka_range_seq_next()
and bkah_range_seq_next() to use make_prev_keypart_map().

We also add some DBUG_ASSERTS in key_part_map processing elsewhere,
exposing some issues in our BNLH implementation.  We cap the number of
keyuse parts here, altering the explain output of 2 of our tests.

Numerous places needed bit shift operations altered to use
make*keymap_part and various format strings needed to be corrected.
Aleksey Midenkov
MDEV-40380 vcol.races sporadic timeout

After MDEV-39261 purge opens TABLE via open_purge_table() for any
undo record of a table with indexed virtual columns, including the
insert undo record. If purge does that after FLUSH TABLES, the TABLE
is put back into the table cache. Then DELETE from con1 reuses the
cached TABLE, skips open_table_from_share() and never hits the
TABLE_after_field_clone sync point. DELETE completes immediately,
wait_condition times out and SELECT returns an empty set.

The fix waits for purge to complete before FLUSH TABLES.
Khaled Riyad
MDEV-41408 MTR prepared statement protocol skipping statements starting with '('

With --ps-protocol, mysqltest runs a SELECT a second time and compares
the results. The regex ps2_re only matched a leading SELECT, so a
statement starting with '(' was executed only once.

Allow any number of '(', each optionally followed by spaces, before
SELECT in ps2_re.
Vladislav Vaintroub
MDEV-33480 Windows MSI installer, convert from WIX v3 to supported versions

WiX v3 is out of support. Support legacy WiX v3.x (default) and modern
WiX (wix.exe, v5 or later). WXS sources stay in v3 format, modern WiX
(-DWITH_WIX3=OFF) builds them after on-the-fly 'wix convert'.

WIX_ACCEPT_EULA=ON accepts the EULA of WiX 7 and later, if WiX asks for it.
Modern WiX takes extra arguments from the EXTRA_WIX_BUILD_ARGS environment
variable, instead of EXTRA_CANDLE_ARGS and EXTRA_LIGHT_ARGS.
A failed WiX step is reported by name.
Kristian Nielsen
MDEV-41217: Inefficient memory management by parallel slave

The rpl_group_info object is used to keep track of a lot of different state
in an event group/transaction across events. It is also used in parallel
replication for a small set of values that are enqueued by the SQL driver
thread for worker threads to handle. This wastes a lot of memory when a
large number of event groups are enqueued (eg. in case of slave lag), since
most of the rpl_group_info is not used in the queue.

This is a preparatory patch that splits out the part of rpl_group_info that
is needed for the enqueueing into a separate rgi_queued_part object, without
any functional or logic changes. This to separate a lot of mechanical
changes from the actual logic modification for easier review:

- Introduce struct rgi_queued_part and move a bunch of fields XXX into it
  from rpl_group_info.

- Introduce rpl_group_info::q pointing to rgi_queued_part and allocate it
  in the constructor.

- Change references like rgi->XXX into rgi->q->XXX.

This refactoring provides the basis for a follow-up patch where parallel
replication is changes so only the rgi_queued_part object is put in the
queue, not the whole rpl_group_info.

Signed-off-by: Kristian Nielsen <[email protected]>
Alessandro Vetere
MDEV-32286 Reuse remembered clustered leaves in secondary-index scans

Row_sel_get_clust_rec_for_mysql::operator() descends the clustered B-tree
from the root for every row whose clustered-index record a secondary-index
scan must read, although consecutive rows land on the same clustered leaf
page wherever the secondary order tracks the clustered one. A non-covering
scan needs one for every row, a locking read needs one whatever the
secondary index holds, because an exclusive select lock type makes
ha_innobase::build_template() build its template against the clustered
index, and a covering scan needs one for every row of a secondary leaf
whose PAGE_MAX_TRX_ID its read view cannot see. ANALYZE FORMAT=JSON
charges each descent its full height, and those descents are nearly the
whole cost: the secondary index is charged its own descent and one page
for each further leaf, and nothing per row, because the position that its
cursor holds between two rows is restored optimistically, which latches
the leaf again without counting an access. So pages_accessed is the row
count times the height of the clustered index, plus a handful: 1000 rows
over a 2-level clustered index cost 2006 and 750 rows over a 3-level one
cost 2291, where a full table scan of the same data costs 23 and 110.

Let a handle remember the clustered leaves that the lookups of one
statement reached, and let the next lookup try them before it descends
again. The reasoning behind each value and each rejection is in the
comments beside it.

row0mysql.h defines clust_leaf_hint_slot, which names one leaf: its page
number, copies of its first and last user record truncated to the key
fields, which bound the key range that the leaf held when it was
remembered, and the rec_get_offsets() of both. A slot of a
leaf that had no right sibling names no last key, because every key above
the last record of the rightmost leaf still belongs to it.
CLUST_LEAF_HINT_SLOTS (4) slots hang off the new
row_prebuilt_t::clust_leaf_hint, beside clust_leaf_hint_mru, the most
recently used order held as slot numbers, and clust_leaf_hint_n and
clust_leaf_hint_miss, the used-slot count and the miss counter.

row0sel.cc holds the policy. row_sel_clust_leaf_hint_covers() compares a
key against the remembered ranges, so a lookup that no slot can answer
costs no buffer pool access and no pages_accessed.
row_sel_clust_leaf_hint_search() probes the first slot that covers the key
and moves it to the front of the order.
row_sel_clust_leaf_hint_remember() records the leaf that a descent landed
on, and refreshes the slot of a leaf that is remembered already rather
than spend a second one on the same page. Two descents fill no slot: the
lookups of a statement before lookup CLUST_LEAF_HINT_MIN_LOOKUPS (4), and
a leaf that is the root. row_sel_clust_leaf_hint_armed() stands a scan
down once the slots stop paying for themselves: a miss adds
CLUST_LEAF_HINT_MISS_WEIGHT (2) to the miss counter and a hit takes one
away, so a scan gives the slots up where it answers too little of its
lookups to pay for them, CLUST_LEAF_HINT_MAX_MISSES (8) misses with no hit
between them still reach the threshold, and one lookup in
CLUST_LEAF_HINT_RETRY (1024) starts the count again, so a scan whose order
becomes correlated only later recovers. Both halves of the cost stop
there, the test of the slots and the copies that refresh them.
Row_sel_get_clust_rec_for_mysql::operator() calls all of this in place of
its btr_pcur_open_with_no_init(), and only where the adaptive hash index
is disabled. A guess of that index lands on the record with no page-local
search and no page access to charge, but neither of the two is faster for
every shape of index and scan, and a hint that runs first stops that index
from adding entries. The adaptive hash index is off by default, so the
hints are active in a default configuration.

btr0cur.h and btr0cur.cc add btr_cur_t::try_leaf_hint(), a PAGE_CUR_LE,
BTR_SEARCH_LEAF search on one named leaf, which starts from the record
where the last search of that leaf landed while that record is still valid,
and from a binary search of the page otherwise. It acquires the page with
buf_page_try_get(): a hint is never derived from a latched parent page,
so by the time it is tried it can precede the caller's already-latched
secondary-index leaf in the latching order, where a blocking wait can
deadlock. It then rejects the page unless the checks that it makes on the
latched frame put the match on it. Those checks are the sole authority on
the result, so a stale range costs a wasted probe or a needless descent,
never a wrong result, and the ranges need no invalidation protocol.

A slot also holds a btr_leaf_step, where the last search of its leaf
landed: the block, its buf_block_t::modify_clock and the record. A
record pointer stays valid while the block holds the same page and the
clock has not moved, because eviction, deletion and reorganization
advance the clock and an insertion moves no record. While the step is
valid, btr_cur_t::try_leaf_hint() starts from that record instead of a
binary search of the page. page0cur.cc adds page_cur_search_near(),
which compares the key with the record and with one neighbour, and
positions the cursor on the record, on the record after it or on the
record before it. It refuses any other answer and leaves it to
page_cur_search_with_match(). So a scan whose consecutive rows are
neighbours in the clustered index finds each row with two comparisons. A
scan that reads the clustered leaf in falling key order does the same: a
secondary key that falls as the primary key rises, or a secondary index
read backward. Where the secondary order tracks the clustered one with
gaps between the rows, the step fails, and btr_leaf_step::expect (below)
leaves the leaf to the binary search. The step refuses the metadata
pseudo-record, which compares below every key on 0 fields, and a record
before it that is the infimum. A debug build checks every successful
step against the binary search.

A failed step costs comparisons whose outcome no branch predictor can
guess, so btr_leaf_step::expect gates it. The next search tries the step
only where this one landed where a step would have: on the remembered
record, or right before or after it. The successor link of the landing
record tells the record right before, because finding a predecessor takes
a walk of the page directory. So a scan in random order over a few leaves,
which the slots answer in full, keeps the binary search and pays only the
tests that set expect. row_sel_clust_leaf_hint_remember() sets the step on
the record where a descent landed, with expect set.

ha_innodb.cc: ha_innobase::reset() zeroes the used-slot count and the miss
counter per statement, matching autoinc_last_value. row0mysql.cc:
row_prebuilt_free() frees the key buffers that the slots own.

innodb.clust_leaf_hint measures pages_accessed over key orders that differ
in how closely the secondary order tracks the clustered one, and further
tables check query results over the record formats and key shapes
that a clustered-index lookup has to read, down to the metadata
pseudo-record of instant ALTER TABLE, to leaves that split and merge while
a locking read walks them, and to a record that a remembered leaf supplies
for a scan that must then rebuild an older version of it. Two of the
tables scan a covering index, which reads a clustered record under an
exclusive select lock type and under a PAGE_MAX_TRX_ID that the read view
cannot see. clust_leaf_hint_off_debug runs the same body with the hints
turned off, through a debug switch that returns before a lookup tests or
refreshes the slots, so a diff of the two .result files is what the hints
save: 2006 to 1031 (2-level clustered index), 2291 to 1011 (3-level), 4006
to 2015 (two interleaved key ranges), 12016 to 9078 (locality in the
second half alone) and 20020 to 10045 for a covering scan that FOR UPDATE
makes non-covering, where the same scan without FOR UPDATE costs 20 in
both files. Two orders with too little locality to pay for the slots give
them up early and end within a hundred accesses of the unhinted count:
20020 to 19966 (decorrelated) and 20020 to 19999 (shuffled).

Two further tables in innodb.clust_leaf_hint cover the step back. t16
scans a secondary index whose keys fall as the primary key rises, and
reads an ascending secondary index backward: 20020 to 10047 for both. t17
runs the inverse scan on ROW_FORMAT=REDUNDANT with an instantly added
column. innodb.clust_leaf_hint_backward_debug reaches the two refusals of
the step back that no count shows, at the infimum and at the metadata
pseudo-record. A lookup reaches either only for a key that left the
clustered index while the slot still covers it. The rollback of an insert
over a delete-marked record removes the key while purge leaves the
secondary entry, and a PAGE_MAX_TRX_ID that the read view cannot see makes
the reader look that entry up. A debug switch writes each refusal to the
error log, and the case reads both back.

A slot stays valid when dict_index_t::clear_instant_alter() changes the
index under a reader that holds a shared metadata lock, between two
lookups or during one. That clear runs where purge or rollback removes
the last user record of a single-page table whose only instant change was
ADD COLUMN, and it raises dict_index_t::n_core_fields.
rec_copy_prefix_to_buf() writes the copies in the format of an index
without instant ADD COLUMN, their offsets are made with them, and the
comparisons depend on the key fields alone, which no instant change moves.
So the slots do not record n_core_fields, and the hints read it only under
a latch on a clustered leaf.

innodb.clust_leaf_hint_instant_alter makes a lookup compare against a slot
that a clear has outlived. The clear needs the clustered index to lose the
last user record of its root page, while no leaf that is the root fills a
slot, so the tree has to shrink between the two, which purge does there.
The reader therefore reads uncommitted rows at READ UNCOMMITTED and waits
in a stored function while a rollback and purge take them away, and one
row that arrives above the position it stopped at is the lookup that
tests the slots.

main.rowid_filter_innodb: 90 to 88, and its ahi combination unchanged.

A static_assert in Row_sel_get_clust_rec_for_mysql::operator() fails to
compile if btr_search.enabled stops being a Boolean that applies to every
index, because the test !btr_search.enabled that stands the hints down
would then have to become !btr_search.is_enabled(clust_index).
Rex Johnston
MDEV-41179 MAX_KEY_PARTS ranges in select causing SEGV

When a select containing 32 ranges is made on a table containing a
compound key with 32 parts, the range optimizer can run off the end
of a stack variable, invalidly overwriting subsequent stack variables.

In the struct st_sel_arg_range_seq, we have an array
  RANGE_SEQ_ENTRY stack[MAX_REF_PARTS];

MAX_REF_PARTS is 32.

check_quick_select
  / sel_arg_range_seq_init
    initialises stack[0] as NOT a key part
  / sel_arg_range_seq_next
    iterates through the key parts, adding key part n to stack[n+1]

key part #32 gets referenced by step_down_to(), setting seq->i off the
end of the array.

Fix: RANGE_SEQ_ENTRY stack[MAX_REF_PARTS+1];

The above change exposed an issue with key length calculation on
MS Windows.  Calling make_prev_keypart_map(32) caused the resultant
bitmap to be calculated as (1UL << 32) - 1.  Using the MSVC compiler
this resulted in an empty key length calculation during
handler::index_read_map, causing an assertion in ha_innobase::index_read().

As we only need 32 bits to represent our key map, we change the type thus
-typedef ulong key_part_map;
+typedef uint32 key_part_map;

We correct make_keypart_map() and make_prev_keypart_map() to call our
overflow safe my_set_bits().  We also correct bka_range_seq_next()
and bkah_range_seq_next() to use make_prev_keypart_map().

We also add some DBUG_ASSERTS in key_part_map processing elsewhere,
exposing some issues in our BNLH implementation.  We cap the number of
keyuse parts here, altering the explain output of 2 of our tests.

Numerous places needed bit shift operations altered to use
make*keymap_part and various format strings needed to be corrected.
Vladislav Vaintroub
MDEV-33480 Windows MSI installer, convert from WIX v3 to supported versions

Custom actions : get rid of wcautil and dutil library dependencies
They are not an official part of WiX distribution anymore.

Replace the functionality with native MSI api, the Wix helpers
we used were thin wrappers anyway.

Also replace the VBScript FakeFailure custom action with a native
one, VBScript is deprecated in Windows and scheduled for removal.

Read MSI properties with a helper that handles any length and reports
errors. Values are no longer silently truncated to MAX_PATH, e.g a long
data directory could be cut to its parent, and then removed.

A custom action that fails to read a property or to call a Windows API
returns ERROR_INSTALL_FAILURE, it was silently ignored before.
Aleksey Midenkov
MDEV-39306 Index corruption on UPDATE when using versioned InnoDB table (on UPDATE)

Duplicated by:

MDEV-39711 UPDATE on System-Versioned Tables uses the after-value
          in Row-format Binlog's before-image
MDEV-24646 ASAN heap-use-after-free in Field_blob::pack /
          THD::binlog_update_row upon DML on table with virtual column

vers_update_fields() recomputed virtual fields with VCOL_UPDATE_FOR_READ.
For BLOB virtual columns this mode swaps Field_blob::value and read_value
after the computation (swap_values). After the swap, Field_blob::value
holds the buffer that record[1] (the old row image) points to. The next
VCOL_UPDATE_FOR_WRITE computation, done by fill_record() right after
vers_update_fields(), stores the new value into that buffer and so
overwrites the old row image. As a result, InnoDB builds the old
secondary-index key from the new value (MDEV-39306), and the row-format
binlog writes the new value into the before-image (MDEV-39711), or a
freed buffer (MDEV-24646).

The fix removes the virtual fields computation from vers_update_fields().
Its callers now recompute virtual fields with VCOL_UPDATE_FOR_WRITE after
the system fields are set: fill_record() already did, while
update_generated_fields(), multi_update::do_updates() and the RBR
appliers do it explicitly. multi_update::do_updates() still computes
virtual fields twice per row: before BEFORE triggers and after
vers_update_fields(), since the system fields cannot be set before
compare_record().

Based on the MDEV-24646/MDEV-39711 fix by ParadoxV5 (0c2f3de1a2d),
which switched vers_update_fields() to VCOL_UPDATE_FOR_WRITE.
Vladislav Vaintroub
CMake: drop EXTRA_WIX_BUILD_ARGS
Aleksey Midenkov
MDEV-30441 ASAN heap-use-after-free in Field_blob::pack/pack_row

When InnoDB computes an indexed virtual column (e.g. the long unique
hash), TABLE::update_virtual_field() walked its expression with
update_vcol_processor() and recomputed every vcol it referenced,
including STORED ones. innobase_allocate_row_for_vcol() saves and
restores Field_blob::value only for non-stored blob vcols, so
recomputing a STORED blob vcol in Field_blob::store_field() did
val_str(&value) and freed the buffer that record[0] still pointed to.
The row-format binlog then packed record[0] from freed memory.

The buffer is allocated only when the value is converted, so the bug
needs a character set conversion (e.g. BLOB to utf8mb3 TEXT).

The fix skips STORED vcols in update_vcol_processor(). Their values are
already in the record: InnoDB copies them as base columns of the
virtual column, and the SQL layer computes them before writing.

The test cases detect the bug only under ASAN or Valgrind.
Vladislav Vaintroub
MDEV-33480 Windows MSI installer, convert from WIX v3 to supported versions

WiX v3 is out of support. Support legacy WiX v3.x (default) and modern
WiX (wix.exe, v5 or later). WXS sources stay in v3 format, modern WiX
(-DWITH_WIX3=OFF) builds them after on-the-fly 'wix convert'.

WIX_ACCEPT_EULA=ON accepts the EULA of WiX 7 and later, if WiX asks for it.

A failed WiX step is reported by name.
Vladislav Vaintroub
MDEV-33480 Windows MSI installer, convert from WIX v3 to supported versions

WiX v3 is out of support. Support legacy WiX v3.x (default) and modern
WiX (wix.exe, v5 or later). WXS sources stay in v3 format, modern WiX
(-DWITH_WIX3=OFF) builds them after on-the-fly 'wix convert'.

WIX_ACCEPT_EULA=ON accepts the EULA of WiX 7 and later, if WiX asks for it.

A failed WiX step is reported by name.
Khaled Riyad
MDEV-41408 MTR prepared statement protocol skipping statements starting with '('

With --ps-protocol, mysqltest runs a SELECT a second time and compares
the results. The regex ps2_re only matched a leading SELECT, so a
statement starting with '(' was executed only once.

Allow any number of '(', each optionally followed by spaces, before
SELECT in ps2_re.