Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
f82541289d43...
f82541289d43e9361a57e82622a9a9d2c45b917a
Rex Johnston
MDEV-39492 Parallel Query: start no more workers than the engine has chunks

parallel_worker_threads is a request, not a division of labour. InnoDB
divides the table at B-tree boundaries: Parallel_reader partitions at the root
page, so the number of chunks is that page's fanout, and create_contexts()
declines to subdivide further while the tree is shallower than
SPLIT_THRESHOLD. A 44MB table of a million rows yields four chunks, and it
yields four whether one worker was asked for or fifty.

Workers past the last chunk were created anyway. Such a worker is handed
HA_ERR_END_OF_FILE the first time it asks for work and exits without reading a
row, having already cost a THD, a table instance opened from the share for
every table in the join, the cloned conditions and select list, and a row
buffer. On that four-chunk table, asking for twelve workers started twelve and
found work for four. The query took the same time as with four, because the
other eight had nothing to contribute.

Ask the engine how many chunks it made and start no more workers than that.
handler::pscan_chunk_count() answers 0 for an engine that cannot say, and
InnoDB answers 0 as well while any chunk is still flagged for splitting: such
a chunk is replaced at run time by a variable number of finer ones, so the
count is not an upper bound yet and must not be used as one. That is what
keeps the large-table case, where the tail chunks do get split, from being
held down to the pre-split count.

Parallel_workers_started counts the workers really started, summed over the
queries Parallel_queries_executed counts, which is what makes the difference
visible from SQL. parallel_query_worker_count reads the two as a ratio,
because both are cumulative and --ps-protocol executes a statement more than
once, so workers-per-query is the figure that holds across protocols. Without
the fix a table of a single leaf page reports eight and sixteen workers where
it now reports one.

This makes nothing faster. It stops paying for threads that cannot be given
anything to read, which matters under concurrency, where they compete for the
same cores as everyone else.

This commit was prepared with Claude Code: it measured the chunk counts and
their row distribution by instrumenting create_contexts() and
pscan_get_next_row(), confirmed the failing forced-worker tests are unchanged
from HEAD, and wrote the test.
f292aa500bf3...
f292aa500bf3dfe0e21d21e9b5583c7fb681d47a
Rex Johnston
MDEV-39492 Parallel Query: workers must read the manager's snapshot

A parallel worker runs in its own THD, so it runs in its own transaction and
opened its own read view at its first read. Nothing tied that view to the one
the manager holds: the workers and the manager could each read a different
version of the table, and the chunk boundaries the engine partitioned under
the manager's view did not even belong to the snapshot the workers scanned.
Both directions were visible. Under REPEATABLE READ a worker returned rows
another session had committed after the manager's snapshot was taken, and rows
the manager's own transaction had written but not committed went missing,
because to the worker's view the manager was just another active transaction.

The snapshot is now shared through a new transaction_participant method,
clone_consistent_snapshot(thd, from_thd), reached from the SQL layer as
ha_clone_consistent_snapshot(). Each worker calls it once, on its own thread,
before it locks or reads anything; an engine that cannot share a snapshot
leaves the method NULL. Sharing at the transaction level rather than through
the parallel-scan interface is what makes this cover every table the worker
reads: the inner tables of the join are read at the same point in time as the
parallel-scanned driving table.

InnoDB implements it by copying the source transaction's read view.
ReadView::clone() installs the source's ReadViewBase state and opens the copy.
It inherits the source's m_creator_trx_id rather than keeping the worker's own,
so rows written by the sharing transaction itself stay visible through the copy
-- that is what changes_visible() uses the creator id for -- and it inherits
the source's isolation level, so a READ UNCOMMITTED source is still read
without consulting a view at all. The source view is read under its m_mutex,
the same protection the purge coordinator takes in ReadView::append_to(), and
from_thd's handlerton data under its LOCK_thd_data.

The manager's view is pinned in pscan_init_coordinator() before any worker is
created and stays open until the workers have been reaped in
quiesce_workers(), so there is always a snapshot to copy, and holding it is
also what stops purge from removing the row versions the workers still need.
A worker that finds no snapshot to adopt fails the query rather than reading a
different one.

parallel_query_snapshot: rows another session commits after START TRANSACTION
WITH CONSISTENT SNAPSHOT stay invisible, both a single row and 999 rows spread
over the whole clustered index, while the manager's own uncommitted row is
returned. Each case also asserts from the optimizer trace that the query
really did run in the workers. All three cases fail without this commit.

This commit was prepared with Claude Code: it found that the workers were
taking their own read views, then wrote the clone_consistent_snapshot
plumbing, ReadView::clone() and the test.
eefeb6b47ca9...
eefeb6b47ca9f278dea20c50c697e570ff2c2057
Rex Johnston
MDEV-39492 Parallel Query: check every table the worker will open

A worker opens its own copy of every non-const table of the join, not only the
one it scans in chunks, but the gate tested only the driving table. An internal
tmp table -- a materialized derived table or subquery sitting as an inner table
-- has a share built in memory rather than read from a .frm, so
open_table_from_share() walked off the end of it and the server died in
open_worker_tables(). One statement was enough,

  SET optimizer_switch='derived_merge=off';
  SELECT ta.a, d.n FROM ta, (SELECT a AS k, COUNT(*) AS n FROM tb GROUP BY a) d
    WHERE d.k = ta.a;

and it accounted for the largest group of crashes when the whole main suite was
run with parallel_worker_threads forced on: subselect_sj2_mat, subselect_sj2,
subselect_sj2_jcl6, derived_split_innodb, subselect-crash_15755 and
group_min_max_innodb.

Apply table_can_be_parallel_scanned() to every table in the join, and refuse a
join tab with no table at all. The function name reads oddly for a table the
worker only looks rows up in, but each condition it tests is one a worker-read
table needs: an internal tmp table cannot be copied, blob payloads live outside
the record buffer and do not survive the row transport whichever table they come
from, a partitioned table cannot be opened as a plain copy, and the engine flag
is also what tells us the engine can hand the worker the manager's snapshot.
That last one closes the other half of this hole, a join whose inner table is in
an engine that cannot share a snapshot and would have been read outside the
manager's.

The six tests above no longer crash. What is left in them is the plan and
EXPLAIN churn that the 1/N cost discount causes in forced mode, plus, in the
semijoin ones, a wrong result that this crash was hiding: the worker implements
none of the semijoin duplicate-elimination strategies. The following commit
refuses those plans.

parallel_query_excluded gains the derived-table case, which asserts that the
optimizer does not choose it and that the answer matches serial execution.

This commit was prepared with Claude Code: it reduced the crash to the statement
above, made the gate check every table, and wrote the test.
e0468dfabfe7...
e0468dfabfe7dfbf0a3fe6ce78c55fce45eed2c5
Rex Johnston
MDEV-39492 Parallel Query: give worker tables their place in the join

A worker's private table copies came out of open_table_from_share() with
TABLE::map still zero, because nothing assigns it outside the optimizer's
setup_tables(). Item_field::used_tables() reads TABLE::map, so every item
rebound onto a worker copy reported used_tables() == 0 -- it looked like a
constant. pwt_clone_rebind() re-fixes the clone, and Item_cond::fix_fields()
evaluates any argument that can_eval_in_optimize(), so cloning the condition of

  SELECT a, b, c FROM t1 WHERE a % 7 = 0 AND b > 100

evaluated "b > 100" on the manager thread, against a worker record buffer that
no row had been read into. Debug builds assert in Field_long::val_int() on
marked_for_read(); release builds read that unread buffer to compute a
null-rejection cache. Any WHERE with more than one predicate was affected --
that is, most of them.

Copy map and tablenr from the manager's table when the copy is opened, so a
rebound item computes the same used_tables() as the item it was cloned from and
nothing evaluates it at clone time. Mark all columns readable there too, before
any cloning touches those tables, rather than at the start of worker_run_query()
-- the read_set then holds for the whole life of the copy, and the per-table
loop in worker_run_query() goes away.

This was invisible because no test reached it. parallel_query_join and
parallel_query_worker_side compared parallel against serial by building the
same result twice with CREATE ... AS SELECT, and writing from a SELECT is a
locking read, which makes the engine decline the parallel scan: both sides ran
serially, so the multi-predicate WHERE never reached a worker clone.

Those comparisons are rewritten to compare something that really does run in
the workers. A result set that cannot be written to a table cannot be compared
afterwards either, so each query now goes inside a non-mergeable derived table
-- the inner select runs in the workers, the outer aggregate runs in the user
thread -- and is reduced to COUNT(*), SUM(CRC32(...)) and BIT_XOR(CRC32(...)).
That fingerprint is order-independent, which a parallel result needs since its
rows arrive in batch-completion order, and it compares the whole multiset at
full scale without printing it. The shared method lives in
include/parallel_query_fingerprint.inc.

Each comparison also asserts that the workers, not the user thread, did the
scanning: both runs read the same materialized rows, but only the serial one
also scans the driving table, so the serial run's Handler_read_rnd_next must be
the larger of the two. That is what stops these comparisons from silently going
hollow again -- with the parallel half forced back to 0 workers the check flips
from 1 to 0 and the tests fail.

Both tests were checked to be load-bearing by mutation: dropping one worker
result row in 500 moves every fingerprint (713 -> 712, 5000 -> 4990,
4967 -> 4958, 6000 -> 5988).

This commit was prepared with Claude Code: it root-caused the assertion to the
zero table map, wrote the fingerprint comparison and the include, and ran the
mutation checks.
df35884e82b2...
df35884e82b2e1de5cfa3829b834e1b7331fa99a
Rex Johnston
MDEV-39492 Parallel Query: give a worker its own JOIN

No functional change: nothing reads the worker's JOIN until its JOIN_TABs are
driven through sub_select(). Second step of replacing the worker's private nested
loop with the executor's.

A worker cannot share the manager's JOIN. sub_select() writes join->return_tab as
it descends -- the level to unwind to, set unconditionally on entry -- and the
guard that reads it compares against JOIN_TAB*, so with a JOIN_TAB array per
worker one worker's backtrack point would be compared against another's tabs:
pointers into different allocations, and a scan that can end early for no
visible reason. The stronger reason is join->thd, read at twelve sites in that
loop for the diagnostics area, the killed flag and the row counters. Sharing the
JOIN would send all of it to the manager's THD from a worker thread, which is the
bug fixed in "a worker's result table belongs to the worker", reintroduced once
per row and for every query shape. Unlike return_tab it cannot be gated away.

JOIN declares its copy constructor and assignment private and unimplemented, so
the worker's is built by JOIN's own constructor and given what it needs by name.
That constructor is JOIN::init(), a field-initialiser that allocates nothing,
plus a shallow copy of fields_list, so it is cheaper as well as better defined
than copying the bytes of a class whose author said not to. It also matches how
Join_plan_state already holds part of a JOIN: the convention here is a named list
of fields, not an object copy.

Only three fields need carrying, which is what makes a named list practical.
Walking sub_select() and evaluate_join_record() for the shapes the gate allows
leaves thd, return_tab and found_records. map2table is read only for
split_derived_to_update; join_tab_execution_startup() reads join only inside its
two semijoin-materialization branches; JOIN_TAB::preread_init() returns before
touching join->thd unless the table is a materialized derived. All three are
already asserted inert per table by pwt_assert_tab_inert().

result is left null deliberately. It is the manager's connection to the client,
and manager_collect_and_send() is the only thing that may send a row, so a worker
that reaches for it crashes rather than writing to a socket two threads share.

pwt_assert_join_inert() is the same idea as pwt_assert_tab_inert() one level up,
and none of its checks fires across the main suite with parallel_worker_threads
forced on. The sizeof(JOIN) tripwire is debug-only on purpose: JOIN carries
dbug_join_tab_array_size under #ifndef DBUG_OFF, so its size is not the same in
the two build types, and pinning both would leave two magic numbers of which only
one is ever checked by whoever changes the class.

This commit was prepared with Claude Code: it established the three-field surface
by walking the executor and the helpers it calls, found that JOIN is
deliberately non-copyable after having earlier reported the opposite, and
confirmed that main.xtradb_mrr appearing in one forced-worker sweep is a
load-dependent warning-count flake rather than a regression -- it passed six of
seven runs in isolation and a second sweep is back at the known eighteen.
dcf174feda15...
dcf174feda15e374ac9ee4b984987e7ab60f7eff
Rex Johnston
MDEV-39492 Parallel Query: bound the split, and use it where it is needed

A chunk came from a whole B-tree level, so a scan could be divided two ways and
no other. The root's fanout gave the coarse ranges, and Ctx::split() re-divided
one of those at the level above the leaves, producing a range per leaf page. On
a table of a million rows in 44MB that is a choice between 4 chunks and 2245,
where what the query wants is a few dozen. create_contexts() therefore declined
to split at all unless the tree was at least SPLIT_THRESHOLD deep, which for a
16K page means about 16GB, so in practice a table was left at its root fanout
however lopsided that was. Four chunks measured 36.0, 36.0, 18.0 and 10.0 per
cent of the rows: the largest alone held the scan open for a third of its
serial duration, and no number of workers changed that.

A range ends where the next one begins, so leaving out a start point folds that
sub-tree into the range before it and loses no rows. create_ranges() now takes
a bound on how many ranges a page may yield and strides its records to meet it,
skipping the descent into the sub-trees it folds -- not descending is the
saving. Ctx::split() asks for as many pieces as there are threads. The bound is
what makes splitting cheap enough to rely on: each piece costs a page traversal,
a context and a deep-copied tuple, built under the index S-latch, and at leaf
granularity that outweighed the balance it bought on any scan short enough to
notice.

So the depth test goes. What replaces it is structural rather than a heuristic:
a range is split unless there is nothing under the root to divide it by, which
is m_depth below 2, meaning the ranges already end at leaf pages. Those are the
trees of one or two levels, and they have at most about a thousand leaf pages,
which are their chunks already -- more than enough, and leaving them alone also
keeps the count final so init_parallel_workers() can still size its pool by it.

Measured on a release build, six cores pinned, medians of 27 samples with the
worker counts interleaved so no configuration gets a turbo window. A scan of a
million rows returning one, which is the case that suffers most from a coarse
chunk because it has no other work to hide behind, goes from 2.10 to 5.49 times
serial. The same scan joined to two dimension tables goes from 1.76 to 3.16.
Splitting the same table without the bound reached 1.93 and 2.61 and made the
scan-only case erratic, its inter-quartile range 27 per cent against 1 to 2 for
the others. A join over a thousand-row driving table is untouched, its tree
being two levels deep. COUNT, SUM and an order-independent CRC over both
million-row tables agree with the serial answer at 1, 3, 5, 12 and 50 workers,
so the folded ranges cover every row exactly once.

Parallel_scan_chunks reports what the table was divided into, which is what
makes the bound testable rather than only measurable: the new case in
parallel_query_worker_count is a table whose root holds two records and whose
tree has a level between root and leaves. It starts all eight workers asked for
where before it started two, and it divides into ten chunks. Removing the bound
divides the same table into 56, one per leaf page, and restoring the depth test
starts two workers again.

Not addressed: a split can still produce fewer pieces than there are threads,
if the page it strides has fewer records than that. The pool has already been
sized by then and the count is not knowable earlier, so a worker can still come
away with nothing. It costs a THD rather than a wrong answer.

This commit was prepared with Claude Code: it established that chunk boundaries
are B-tree level boundaries and measured the resulting chunk sizes by counting
rows per chunk in pscan_get_next_row(), wrote the striding, took the
measurements above, and confirmed by mutation that the test detects both the
loss of the bound and the loss of the split.

This is also the point at which the chunk figures settle, so it carries the tests
for the descent fix made earlier in the series -- they need
include/parallel_query_fingerprint.inc and a worker-count test to hang the figures
on, neither of which existed where that fix had to go.

A chunk is only re-split once the tree is at least three levels deep, so
demonstrating the descent needs a table larger than any other test here uses:
some 700000 rows of INTs. The new test reaches three levels in 5000 rows by
giving the table a wide PRIMARY KEY, which shrinks the node pointer fanout, and
it answers 2591 of those 5000 rows in the workers if the descent is reverted.

main.parallel_query_worker_count's check that every row is read exactly once was
written as a bare aggregate, a shape the gate refuses, so it ran serially and
compared the serial answer with itself -- and t4 is the only table there whose
tree is deep enough to be re-split, so that check is the one that would have
caught this. It now reduces the rows outside a non-merged derived table, so what
it measures is a scan the workers ran. t4's chunk count per query is printed
rather than only bounded, for the same reason.
dbb99e93d195...
dbb99e93d195932fa8baa523e689dcdb64c4d57c
Daniel Black
MDEV-40629 RCE via systemd environment (12.3)

Since 12.3.2 and 13.0.1, the MariaDB-server-galera package
provided a systemd drop-file that used EnvironmentFile=
as a mechanism to bootstrap and recover under Galera.

A malicious user with FILE privileges could manipulate
this file with LD_PRELOAD that would result in remote code
execution.
d30ab594ae1d...
d30ab594ae1d67e075e58ff7891cd14d67158ace
Rex Johnston
MDEV-39492 Parallel Query: workers evaluate with the session's variables

A worker runs in a background THD, which starts from the global variable values,
and init_parallel_workers() copied only the identity of the session: its
security context, database and command. So the workers evaluated the session's
own expressions with somebody else's settings, and the query quietly meant
something different there. With the session in one time zone and the server in
another,

  SET TIME_ZONE = "+03:00";
  SELECT HOUR(ts) FROM t1;

answered 3 in this thread and 12 in a worker, and the same difference inside a
WHERE dropped every row a condition on a TIMESTAMP column should have kept, which
is how function_defaults_innodb failed with parallel_worker_threads forced on.

Hand each worker the variables an expression reads while it is evaluated:
time_zone, sql_mode, default_week_format and old_behavior. One at a time rather
than as a whole struct, because system_variables owns per-THD allocations, the
dynamic variables and the session tracker among them, and it also carries
option_bits, which would tell a worker it is inside the session's
multi-statement transaction and change how it commits.

Which variables those are was settled by testing rather than by listing what
looked relevant. Variables read while an expression is *built* need no handover,
because the copies are built and fixed on the user's thread: DAYNAME(),
MONTHNAME() and DATE_FORMAT('%W') keep the lc_time_names they were fixed with,
and a division keeps the scale div_precincrement gave it, so neither variable is
copied. max_allowed_packet cannot be set per session, so a worker already has the
session's value. That leaves the four above, three of them because a test showed
each one changing an answer, and old_behavior because the date and time
conversions read it as they run, the same way sql_mode is read.

parallel_query_session_vars checks the five expressions in both places, projected
and in a WHERE. Without the handover the projection answers 12, 2 and 9 where it
should answer 3, 10 and 10, and the WHERE returns no rows where it should return
three.

This commit was prepared with Claude Code: it reduced the failure to HOUR() over
a TIMESTAMP, established which variables are read at evaluation time and which at
fix time, and wrote the test.
d2d441f077dd...
d2d441f077dd050be64d39e0bb625d5761a35ebf
Rex Johnston
MDEV-39492 Parallel Query: worker tables inherit the manager's column bitmaps

A worker's table copies were marked with use_all_columns(), which points both
read_set and write_set at the share's all_set. Correct, since it is a superset
of what the worker evaluates, but coarser than it needs to be in two ways.

InnoDB builds its row fetch template from read_set, so with every column marked
a worker converts every column of every row it scans to MySQL format, not just
the ones its cloned condition and select list reference -- work paid on the hot
path that parallel query exists to shorten, and it grows with the width of the
table rather than with the query. And an all-columns write_set on a table the
worker only ever reads gives up an assertion: a store into a source field no
longer trips marked_for_write().

Copy the manager table's read_set and write_set into the copy's own
def_read_set/def_write_set instead, and point the copy at those. The optimizer
has already marked exactly the columns this query reads, and
open_worker_tables() runs after it has finished, so those bitmaps are final; the
copy owns its bitmaps, so nothing is shared with the manager, and
column_bitmaps_set() signals the engine to rebuild its template.

This keeps the property the marking was added for: every column a cloned item
could touch is in the read_set before any cloning happens, because the optimizer
marked precisely that set.

Two cases in parallel_query_worker_side cover the fidelity of the copy, both
through the serial-vs-parallel fingerprint: a table with eighteen columns where
the query references two, and a virtual column whose base column is read but
never projected. Clearing a single bit of the copied read_set makes both of them
fail (Field_long::val_int() asserts on marked_for_read()), as does
parallel_query_join.

Index-only reads are unaffected by this: they need ha_start_keyread(), which the
worker-side path never issues, for any read_set.

This commit was prepared with Claude Code: it identified the cost of the blanket
marking, made the copy, and added the two coverage cases.
d2467dea0deb...
d2467dea0deb524fd298974aa0df9639c315badb
Rex Johnston
MDEV-39492 Parallel Query: give a worker real JOIN_TABs

No functional change. This is the first step of replacing the worker's private
nested loop with the executor's, and it replaces the structure that loop reads.

pwt_jointab described a table in five fields: its TABLE copy, the access type, a
ref, a condition and the sorted flag. sub_select() and evaluate_join_record()
want a JOIN_TAB, so every capability added to the private loop would have to be
added again when the split moves to the real one. A worker now holds an array of
JOIN_TAB, n_tables of them, indexed the way pwt_manager::mgr_tabs is so that [0]
is the parallel-scanned driving table -- the indexing sub_select() will be
started from. The driving table's condition moves onto join_tabs[0].select_cond
with every other table's, so worker_cond goes away.

The tabs are copied from the manager's and then rebound, rather than built from
nothing, so that a field this code does not know about holds the value the
optimizer chose instead of a zero that would look deliberate. Overwritten: the
TABLE, the condition, the ref, the tracker. Cleared: the places a condition can
hide (pre_idx_push_select_cond, cache_select) now that select_cond carries all of
it, and the join buffer, which a worker does not use. Nulled deliberately: the
JOIN, next_select, read_first_record and read_record, which the next two steps
fill in -- reaching them before then is a crash rather than a read of the
manager's execution state.

That last distinction is the point of pwt_assert_tab_inert(). A copy keeps the
manager's pointer in anything the copy does not overwrite, which is the trap
TABLE::map and TABLE::in_use were, and the fields that matter here are ones the
gate is supposed to have excluded rather than ones this code sets. So each is
asserted inert on the manager's tab before the copy: the outer-join chain, the
semijoin strategies and their weedout and firstmatch tables, emb_sj_nest, the
rowid filter, range access, split derived materialization, the DISTINCT
shortcut, HAVING, and the access type. Relax a gate without teaching the copy
about the field it lets through and a debug build stops here instead of
executing with another thread's state.

Writing those assertions is also what found the BNL scan filter fixed in "filter
by the condition the join buffer would apply": asking whether cache_select was
inert turned out to have the answer no, and a wrong result behind it. They are a
survey as much as a guard, so they are worth running before the code they
protect. Verified live on a debug build -- none of them fires across the main
suite with parallel_worker_threads forced on, 1418 tests, and a deliberately
false one aborts the server, so the negative result is a measurement rather than
an assumption.

main and innodb pass in full, 2103 tests, and the parallel tests pass under
--ps-protocol. The forced-worker sweep is at its 18 known failures, all EXPLAIN
and cost output.

This commit was prepared with Claude Code: it established which JOIN_TAB and
JOIN fields the real executor actually reads (4 of JOIN, some 29 of JOIN_TAB, of
which the gate makes most inert), wrote the copy and the assertions, and checked
the assertions execute rather than trusting that they had.
caaef2367fa2...
caaef2367fa2f416f1777814e26919dfb8f29a39
Rex Johnston
MDEV-39492 Parallel Query: fall back to serial when the engine declines

run_worker_side_join() returned 1 -- error -- when init_parallel_workers()
reported HA_ERR_UNSUPPORTED, although its own documented contract, and the
comment on that very branch, say -1 means "the engine declined, run the query
serially". No error was raised on the way out, because nothing had gone wrong:
the engine simply refused the scan. do_select() turned that 1 into
NESTED_LOOP_ERROR, so the statement failed with an empty diagnostics area and
Protocol::end_statement() hit its DA_EMPTY assertion. In a release build the
client gets a statement that neither succeeds nor reports an error.

The engine declines this way for any read that is not a consistent read:
pscan_init_coordinator()'s first check refuses the scan when
select_lock_type != LOCK_NONE. So SELECT ... FOR UPDATE,
SELECT ... LOCK IN SHARE MODE, CREATE ... AS SELECT and INSERT ... SELECT over
a parallel-scannable table all crashed a debug server as soon as
parallel_worker_threads was set -- the optimizer picks the table, then
execution has nowhere to go. Return -1 so do_select() takes its serial path,
which is what make_join_readinfo() left in place for exactly this case by
keeping the table's serial read_first_record.

parallel_query_fallback is the new test: FOR UPDATE, LOCK IN SHARE MODE,
CREATE ... AS SELECT and INSERT ... SELECT each return their rows with workers
enabled. All four reach the decline path (the DBUG_PRINT added here fires four
times), and the first of them already crashes the server without this commit.

This also un-breaks parallel_query_join and parallel_query_worker_side, which
have been failing on this branch since they were recorded: both open with a
CREATE ... AS SELECT that took the bad path. Their serial-vs-parallel
comparisons are built from CREATE ... AS SELECT pairs, so now that the fallback
works both sides run serially and those comparisons no longer say anything
about worker-side execution. The assertion in parallel_query_join that claimed
CTAS "runs worker-side" is corrected to state what the trace key actually
reports, which is the optimizer's choice. Restoring a like-for-like comparison
at scale needs a technique that does not write from a SELECT, and is left to a
follow-up.

This commit was prepared with Claude Code: it traced the DA_EMPTY assertion to
the wrong return code with a DBUG trace, reduced the repro from
CREATE ... AS SELECT to SELECT ... FOR UPDATE, and wrote the test.
bf76479ac09b...
bf76479ac09bb4b19e7004151b93300352553749
Rex Johnston
MDEV-39492 Parallel Query: a copy must share no node, not just no field

The gate refused a copy that reached one of the original's Item_field objects.
That is the condition under which rebinding damages the original, but it is not
the only way a shared node hurts. A node that is not a leaf carries evaluation
state, and several workers evaluating one shared object at once tear it.

  CREATE TABLE t6 (d DATE);
  SELECT * FROM t6 WHERE LEAST( UTC_TIME(), d );

wraps the constant argument in an Item_cache_time. Item_cache_int::deep_copy()
is a shallow copy, so each worker got its own cache object pointing at the
manager's UTC_TIME() item, and the workers evaluated that one item together.
Time::Time() asserted on a MYSQL_TIME left half written. The shared object is an
Item_func, not an Item_field, so the field-based test did not see it.

Ask the general question instead: does the copy reach any object the original
reaches. Item::find_item_processor() already answers that for one object, so the
missing half was a way to enumerate a tree's nodes, added here as
Item::collect_all_items_processor(). Together they replace both Field_enumerator
helpers, and the check now covers the shallow-copying classes not yet met rather
than the three now known.

type_temporal_innodb no longer crashes with parallel_worker_threads forced on. It
still differs there, in the row number a warning carries: the worker's THD does
not track the manager's current row, so a relayed warning says row 0. That
belongs with the other per-session counters the workers do not share.

parallel_query_clone gains the query above, answered serially and again with
workers. Each of that test's two cases fails on its own without this commit, the
semijoin one on the in_use assertion in Field::val_int() as before, the new one
on Time::Time().

This commit was prepared with Claude Code: it traced the assertion to the shared
cache argument, generalised the check, and wrote the test.
b9dc71dc39d3...
b9dc71dc39d38543c6b8076183e8fbcf1c8fc07f
Rex Johnston
MDEV-39492 Parallel Query: do not run in fewer than three workers

Measured on a release build, 4000000 rows, the scan bound by I/O, on a six-core
machine with the server pinned to those cores:

    workers    1      2      3      4      5      6
    speedup  0.36x  0.59x  1.42x  1.59x  1.74x  1.69x

One worker runs at little over a third of the serial speed and two at about
three fifths. Three is where it crosses over. So the parallel path is declined
below three workers, and the count tested is the one after the clamp to the chunk
count, because a table that divides only two ways gives two workers however many
were asked for.

The cause is not the scan, which is why more rows or a bigger table does not help.
Every row a worker reads is copied into a batch, handed over under a mutex and read
again by the manager, and the manager drains one worker at a time. With one or two
workers there is no slack in that exchange, so the manager and the workers
alternate instead of overlapping, and the row makes two trips where a serial scan
makes none. The floor is therefore a constant rather than something derived from
the machine: it is a property of the exchange. Deepening the channel so that two
workers can overlap with the manager is the fix that would move it, and that is
not this commit.

"decline by the chunk count, not the worker count" argued the opposite -- that one
worker was the user's to ask for and was not even reliably slower, because the
worker's scan overlaps the manager's sending. That reasoning was drawn from a debug
build, where the scan is inflated enough to hide the exchange entirely, and it is
wrong by a factor of nearly three. The comment saying so is corrected.

A cost term cannot do this instead. When a full table scan is the only access path
for a table, discounting it does not change which access is chosen, and the gate
then runs it in the workers whatever the cost model concluded. Declining at
execution, next to the existing decline for a table the engine cannot divide, is
the only place that can refuse.

main.parallel_query_oom asked for exactly one worker, so this stopped it reaching
the failure it injects -- it ran serially and quietly tested nothing, which is the
third time in this series that a test has kept passing while measuring nothing. It
now asks for four, and the comment says why the number matters. The output is
unchanged and was checked over six repeats, because the injected failure is
per-worker while the warning the manager surfaces is not.

main.parallel_query_worker_count asserted that a request for two workers was met in
full. It now asserts that two are declined and three are met, which is the
behaviour either side of the floor rather than one point on it.
b903f496623f...
b903f496623fd0626ad5e772326e7c69472fc4ba
Rex Johnston
MDEV-39492 Parallel Query: give ANALYZE the numbers the workers produced

ANALYZE reads a JOIN_TAB's counters from the tracker the optimizer left on the
Explain object, and the engine's counters from the handler it recorded there.
Both belong to the manager, and the manager never runs the driving table's read
loop, so a parallel query reported the table as untouched: r_loops 0, no r_rows,
no r_filtered, no r_engine_stats. For a feature whose whole purpose is to make a
scan faster, the tool for seeing where a scan spends its time said nothing about
it.

Each worker now counts what it did to each of its tables in the same terms
sub_select() and evaluate_join_record() use -- rows read, rows that passed the
table's condition, and one scan per probe of an inner table -- and copies the
engine's counters out of the tables while they are still open. The manager adds
all of it to the trackers and handlers ANALYZE reads, in quiesce_workers() after
every worker has been joined, so this thread is the only one touching either
side and no locking is needed. ha_handler_stats::add() already existed for the
partitioning case and does the engine half.

The driving table reports one scan, not one per worker: the chunks are one scan
of the table between them, which is what the serial plan reports and what keeps
the rows-per-scan figure comparable between the two.

  ANALYZE SELECT a FROM t1 WHERE a % 7 = 0;

now answers r_rows 5000.00 and r_filtered 14.28 whether it runs serially or in
the workers, differing only in the access type.

Not included: r_table_time_ms and r_other_time_ms, which come from the elapsed
time trackers rather than from counters, and which the workers measure in
parallel. Summing them would report more time against the table than the query
itself took, and reporting one worker's share would understate the work. That
needs a decision about what ANALYZE should mean for a parallel scan, so it is
left out rather than guessed at, and the two fields stay absent.

parallel_query_worker_side compares the tabular ANALYZE for the same query run
both ways, with the row estimate masked because it is an InnoDB approximation.
Without the handover the parallel run reports r_rows and r_filtered as NULL.

This commit was prepared with Claude Code.
adc49c281ccf...
adc49c281ccf3e316b3c6577aaaeaa8c560ecf78
Rex Johnston
MDEV-39492 Parallel Query: filter by the condition the join buffer would apply

An inner table read through a BNL join buffer keeps part of its condition
somewhere a worker never looked. JOIN_TAB::make_scan_filter() copies the
conjuncts that need only that table into cache_select->cond, for the buffer to
apply as it fills, and JOIN_TAB::remove_redundant_bnl_scan_conds() then removes
those same conjuncts from select_cond -- it calls set_cond(NULL) outright when
they were the whole of it. A worker uses no join buffer and cloned select_cond,
so the conjuncts were enforced in neither place.

  SELECT STRAIGHT_JOIN b2.v FROM b1, b2
  WHERE b2.x = b1.k AND b2.v > 15 AND b1.id <= 2;

answered two rows serially and ten with workers, the extra eight being every row
of b2 that b2.v > 15 excludes. The join condition still held -- ten rows, not
forty -- which is what pinned it on the single-table half: make_scan_filter()
extracts by table map, so the two-table equality stays in select_cond and only
the single-table predicate moves.

This is the third time the same shape has bitten. The optimizer moves a
predicate out of select_cond into a structure the worker does not replicate:
handler::pushed_idx_cond, then the semijoin strategies, now the join buffer's
scan filter. So the accessor that was introduced for the first case is now
pwt_table_conds(), reporting both halves, and one clone helper ANDs them. The
gate and both clone sites go through it, so the condition the gate approves
stays the condition a worker evaluates.

Only BNL is affected. The hashed and batched buffers give the table an access
type outside JT_EQ_REF/JT_REF/JT_ALL, which the gate already refuses, so they
never reach a worker -- verified across join_cache_level 2, 4, 6 and 8, where
only level 2 differed. BNL is the default.

The conjunction is built with quick_fix_field() over two already-fixed clones,
which is what remove_redundant_bnl_scan_conds() itself does when it rebuilds a
condition out of fixed conjuncts. A worker only evaluates the item, so the
fix-time caches Item_cond::fix_fields() would rebuild are not read.

parallel_query_join gains the query above, answered both ways, asserting that it
still ran in the workers and that b2 still carries a join buffer, so the case
cannot quietly stop being covered. Reverting the cache half of the accessor
brings the eight rows back.

Found while surveying JOIN_TAB for the executor-split work: the survey wanted to
assert that a cloned tab's join-cache fields are inert, which is what turned up
the fact that they are not.

This commit was prepared with Claude Code: it traced the predicate to
make_scan_filter() and remove_redundant_bnl_scan_conds(), established by
sweeping join_cache_level that only BNL reaches a worker, and wrote the test.
9f0810c0db1c...
9f0810c0db1c4a4e91f831795459846dff8a46c0
Rex Johnston
MDEV-39492 Parallel Query: cost a parallel scan by what it can actually do

scale_cost_for_parallel_scan() divided the row and copy cost by
parallel_worker_threads, with nothing else in the arithmetic. Three things were
missing and all three pointed the same way, so the optimizer was most
optimistic exactly where parallelism helps least.

The divisor was the request rather than what the table can be divided into. The
engine partitions at the root page, so the chunk count is that page's fanout,
and a worker beyond the last chunk gets nothing to read. With
parallel_worker_threads=50 on a table of a million rows in 44MB, the optimizer
believed the scan was fifty times cheaper. It is four chunks, so a little over
twice as cheap. An error of that size is enough to prefer a parallel full scan
over a perfectly good index, which is what the forced-worker runs of
greedy_optimizer, partition_explicit_prune, type_temporal_innodb, vector and
xtradb_mrr were showing.

Reading a row through a worker also costs more than reading it serially, some
1.16 times for a scan whose rows are cheap to evaluate, because every row is
copied into a batch buffer, handed over under a mutex and read again by the
manager. And each worker has to be built: a THD, a table instance per table in
the join opened from the share, the cloned items and a row buffer, some 22
microseconds. Both are measured on a release build.

So: clamp the divisor to handler::pscan_chunk_count_estimate(), multiply by the
per-row factor, and add the setup cost. The estimate comes from the clustered
index statistics rather than from a page read, since it is wanted at optimize
time: for a tree of three levels the root's fanout is every non-leaf page but
the root, and for a tree of two levels the root's records are the leaf pages.
It needs the statistics to have been gathered, and answers 1 without them, so
an un-analyzed table is costed as the serial scan it may well end up being.
That is the safe direction to be wrong in, because it never claims parallelism
the table cannot supply.

init_parallel_workers() now declines a table the engine cannot divide at all.
One chunk means one worker reading the whole table by itself, which is the
serial scan plus a copy, a handover and a re-read for every row of it. There is
nothing to parallelise and the serial path does it for less.

parallel_query_join's index-condition case needed two changes as a consequence,
and they are worth spelling out because the case had been passing for the wrong
reason. Its p2 held two rows in a single leaf page, so the engine cannot divide
it and the query now runs serially; p2 grows to more than one page. Its plan
also depended on the old cost model: a two-row table was only ever the driving
table because dividing its scan cost by four made it look cheaper than reading
p1 by its index. With the cost fixed the optimizer puts p1 first, where pk < c
cannot be pushed into an index at all, so the pushed-condition situation the
test exists to cover disappeared from the plan. STRAIGHT_JOIN now pins the
order the case needs. Reverting pwt_table_cond() under the new test answers
three thousand rows where one is correct, so the coverage is intact and rather
sharper than the six rows it used to produce.

The forced-worker sweep of the main suite goes from 25 failures to 20, the five
above, and adds none.

This commit was prepared with Claude Code: it measured the per-row factor and
the setup cost, derived the chunk-count estimate and checked it against
instrumented chunk counts on two tables, found that the index-condition case
had been relying on the mis-costing, and confirmed by mutation that both halves
of this change are what the tests detect.
9a4ac02738cc...
9a4ac02738cc260aa03ffe5c689d7e3aca37dec9
Daniel Black
MDEV-40629 RCE via systemd environment

The use of a systemd environment file to aid the bootstrap of galera and
its recovery can be abused to instigate remote code execution.

The systemd environment files where in place for:
* 11.6.0+ MDEV-19210
* 10.11.15+ MDEV-37726
* 10.6.25+ 4b5e6846066a

A user with FILE privileges could add/modify the systemd environment
files to contain a LD_PRELOAD environment variable. This would be loaded
and therefore executed for all Exec statements in the MariaDB service.

This was particularly impacting for Debian packages of the following
version that has ExecStartPost=!/etc/mysql/debian-start that is run as
root (MDEV-15502)
* 10.11.15+
* 11.4.9+
* 11.8.4+
* 12.1.2+

Since MariaDB-12.3.2+ these environment variable directives are only
in place for the MariaDB-server-galera package.

Resolved by clearing the wsrep-start-position wsrep-new-cluster files
in the /var/run directory as a ExecStopPost step which means after the
MariaDB process has any ability to modify them.
96404119faab...
96404119faabbf1e459f87ea3fb7a0e748d2589b
Rex Johnston
MDEV-39492 Parallel Query: give constant select-list items a result field

SELECT 42, a FROM t1 crashed the server whenever parallel_worker_threads was
set. create_tmp_table() does not give a constant item a field, which is right
for a query that materialises its result and can evaluate the constant once
outside the table, but the parallel result table is not that. Its layout has to
mirror the select list item for item, because a worker projects item i into
field i and ships the record image, and the manager sends one Item_field per
field to the client. With a constant in the list the table came out one field
short, and worker_emit_row() ran off the end of the field array into a NULL
Field pointer. The manager's send list was equally short, so even without the
crash the client would have been sent the wrong number of columns. Any constant
did it, a literal, a folded expression such as 1+1, or a session constant like
CONNECTION_ID().

Pass TMP_TABLE_ALL_COLUMNS when building the result table, so every item of the
select list gets a field, and assert the layout matches the list afterwards --
the transport is positional, so a mismatch from any other cause has to be
refused rather than walked over.

That exposed a second problem in the same place. create_tmp_table() overwrites
param->func_count with the number of items it actually has to copy, and
make_result_table() was called once per worker plus once for the manager from a
single TMP_TABLE_PARAM counted once by the caller. A constant needs a field but
no copy entry, so the count dropped to zero after the first table and every
later one allocated fewer fields than its layout, tripping the assertion in
Create_tmp_table::finalize(). Each result table now starts from a freshly
counted param, which is what N identical layouts from one param needs in any
case.

Session constants come out right rather than merely not crashing: the clone is
fixed on the manager's thread, so CONNECTION_ID(), USER() and DATABASE() carry
the user session's values, not the worker's. The test asserts that against
values captured outside the query.

parallel_query_worker_side covers a literal, a string literal, a folded
expression, an all-constant select list where no column of the scanned table
reaches the result, and the session constants, plus a serial-vs-parallel
fingerprint over a select list containing a constant so the case is known to run
in the workers. Every one of them crashes the server without this commit.

This commit was prepared with Claude Code: it found the crash while probing which
expressions the gate accepts, traced both causes from the core file, and wrote
the tests.
8c02de9e5c9f...
8c02de9e5c9fba5c93c809438a6d2e57262952f8
Rex Johnston
MDEV-39492 Parallel Query: cost by the leaf page count, not the root fanout

pscan_chunk_count_estimate() answered the root page's fanout, because that was
all a scan could be divided into. Ctx::split() now divides a coarse range
further, so the ceiling is no longer where the division starts but where it can
no longer go: a chunk cannot be smaller than a leaf page, there being nothing
under a leaf page to divide by. The estimate is therefore the clustered index's
leaf page count, which is also the right answer for a tree too shallow to be
split, where the ranges are the root's records and the root's records are the
leaf pages.

The old answer under-stated the ceiling on exactly the tables the split was
added for. A table of six thousand rows whose root holds two records was costed
as though two workers were the most it could occupy, so the optimizer stopped
believing in a third; the table has ninety-two leaf pages and the split reaches
them. Under-stating is the safe direction to be wrong in, which is why it was
left this way in the commit that added the estimate, but it is no longer
accurate, and it would have kept the optimizer from choosing plans the engine
can now execute several times faster.

The statistics are still what they are. InnoDB reports a thousand-row table as
one leaf page where its data length says four, so a table that small gets no
discount at all and is costed as the serial scan it very nearly is. That is the
same conservatism as before and it costs nothing real, the setup term dominating
anything that small either way.

The per-row factor stays at 1.16. It was measured before chunks were split and
re-measuring now gives a figure below 1, but only in the better of two regimes
the same query alternates between: at six workers the scan-only case runs at
either 19 or 31 milliseconds with nothing in between, and which one it picks
varies between runs of a few queries. Setting a constant from the good mode
would be claiming a per-row saving that is not reliably there, so the comment
now says which end of the range the number is rather than presenting it as a
measurement of the current code. The bimodality is worth its own investigation:
the fast mode implies a per-worker throughput above the serial reader's, which
would be the InnoDB prefetcher engaging, and it is a factor of 1.6.

parallel_query_worker_count now asserts on t4, whose root holds two records over
ninety-two leaf pages, that eight workers are costed below six. Six is what the
root fanout could pay for once the setup term is included, so improving past it
is the ceiling having risen. Reverting the estimate to the root fanout flips
that assertion. The t3 case, which used to carry the proof that a discount is
applied at all, now records the opposite and t4 carries the discount proof, so
neither assertion can pass vacuously.

The forced-worker sweep of the main suite is unchanged at 20 failures.

This commit was prepared with Claude Code: it derived the leaf page ceiling,
read the actual statistics out of mysql.innodb_index_stats when two of its own
assertions came out wrong, found that the setup term legitimately makes sixteen
workers dearer than four on a table this size, and settled on the pair of
assertions above after confirming by mutation that they detect the change.
8253bd708537...
8253bd70853780615ccaa89cb52a39371ff176f7
Rex Johnston
MDEV-39492 Parallel Query: refuse an expression copy that is not independent

A worker repoints the Item_field leaves of its copy of the WHERE condition, the
ref values and the select list at its own tables. That is only safe if the copy
owns those leaves. Several item classes implement deep_copy() as a shallow copy
while still holding child items, every Item_cache and Item_outer_ref among them,
so their copy keeps pointing at the original's children, and Item_cache::walk()
does visit those children. Rebinding one moved the manager's own Item_field onto
worker one's table, then worker two moved it again, so every worker and the
manager ended up sharing a field bound to some other thread's table. The
in_use assertion in Field::val_int() caught it in a debug build,

  SELECT STRAIGHT_JOIN count(*) FROM t1 JOIN t2 JOIN t3
    WHERE t1.f1 IN (SELECT f1 FROM t4) AND t2.f1 IN (SELECT f1 FROM t5);

crashing after semijoin conversion left a cached reference in the condition of a
select the workers ran. A release build reads another thread's record buffer,
and the manager's own plan is left rebound to a table that is closed when the
workers are reaped.

The gate now requires more than clonability, it requires the copy to share no
Item_field object with the item it came from. Testing it there costs nothing,
the gate already deep-copies every candidate item and throws the copy away, and
a query that fails the test declines to serial execution at optimize time.
pwt_clone_rebind() asserts the same property before it rebinds anything, so a
copy that slips through is caught at the point of damage rather than by whatever
reads a foreign table later.

Checking for a shared leaf rather than enumerating the classes that shallow-copy
means the check also covers the classes not yet met. There are more of them than
Item_cache: Item_outer_ref and Item_copy_string hold child items and copy
shallowly too.

The three tests that hit this with parallel_worker_threads forced on --
opt_hints_join_order, innodb_mrr_cpk and subselect_innodb -- no longer crash.
parallel_query_clone is the new test: the semijoin query above, plus a
correlated reference in a select list and one in a condition, each answered
serially and again with workers enabled. Without this commit the first of them
trips the assertion in pwt_clone_rebind(), and without that assertion too it
reaches the original in_use crash.

This commit was prepared with Claude Code: it traced the assertion from the core
file to the shallow deep_copy, wrote the shared-leaf check and the test.
817533135e40...
817533135e406671ac270dc09978e99b5e9a8069
Rex Johnston
MDEV-39492 Parallel Query: HAVING

The gate refused HAVING, and there is nothing left to do about that but stop
refusing it. end_send() applies join->having, the manager is where end_send()
runs, and the records the manager evaluates it against already hold the row: the
workers ship base-table columns and the drain copies them back before the
terminal is called.

HAVING binds to the select list through Item_ref, and an Item_ref resolves
through the query's reference array, which points at items reading base tables.
Those are exactly the tables the drain fills, so the references resolve to the
values a serial scan would have left in them. Nothing is cloned into a worker,
no reference array is swapped, and no worker evaluates HAVING.

That the references are what matter is not a guess about which shapes work: a
column reference in HAVING is always a reference to a select-list item, because a
column that is not in the select list is refused by name resolution
(ER_BAD_FIELD_ERROR). So every target of every reference is a select-list item,
and every select-list item reads the records the manager filled.

The test's alias case is what says so. SELECT a+b AS s ... HAVING s > 5985
reaches the select list only through a reference, so it is the shape that fails
first if the records are wrong, and it is checked against its serial answer
rather than against a row count -- an earlier attempt at HAVING answered 0 rows
where 5 were right, which a checksum of the result would not have caught.

This commit was prepared with Claude Code: it established that HAVING resolves
through the reference array rather than through base-table fields directly, which
is what decided that the records had to be right rather than the items
re-pointed, and wrote the alias case that distinguishes the two.
811cbbc42538...
811cbbc425384b774505609935f6eb65332a54e6
Rex Johnston
MDEV-39492 Parallel Query: run LIMIT and OFFSET in the workers

The gate refused a query with LIMIT or OFFSET. It no longer needs to: since the
manager's loop ends in end_send(), the limit is applied by the server, on the
manager's single thread, and select_result_sink::send_data_with_check() applies
the offset in the same place. Reaching the limit returns NESTED_LOOP_QUERY_LIMIT,
which manager_collect_and_send() turns into a stop for the producers -- the flag
the batch handoff protocol has carried since it was written and never used. So the
number of rows delivered is exact and the scan stops early rather than draining.

Which rows arrive is another matter, and that is the reason this was left for a
decision rather than committed with the refactor. A parallel scan delivers
whichever rows the workers finish first, not the ones a serial scan would reach.
For a query with no ORDER BY there is no order for a prefix to be taken of, so
both are correct answers to the question asked; the ordered case is refused by
the gate and runs serially, unchanged. Depending on the rows an unordered LIMIT
happens to return is an antipattern, and not one this server is going to keep
supporting on the strength of a test that encodes it.

In the event nothing in the suite encoded it. The main suite with
parallel_worker_threads forced on is at the same eighteen failures as before, all
EXPLAIN and cost output, and normal runs are untouched because
parallel_worker_threads is 0 by default. The reason is worth recording: nearly
every table in the suite is small enough that the engine cannot divide it, and
"decline by the chunk count, not the worker count" refuses those outright, so
their LIMIT results were never going to move.

parallel_query_limit compares how many rows six limit and offset shapes deliver
serially and in the workers, counted through a non-merged derived table so the
figure is the query's row count. FOUND_ROWS() is not used for this: it counts
rows the executor processed, so with OFFSET it includes the skipped ones -- 8 for
LIMIT 5 OFFSET 3 -- and serial does the same, so it is consistent but not the
number the test is about. Content is asserted only where a single row can
qualify, and the ORDER BY case asserts that it did not run in the workers, so the
gate's remaining refusal cannot lapse unnoticed.

This commit was prepared with Claude Code: it verified the limit path against a
temporarily opened gate before proposing the change, established that FOUND_ROWS()
counts processed rather than delivered rows by comparing serial with parallel
rather than assuming a bug, and confirmed no test in the suite depends on the
order of an unordered prefix.
6dc1f231e06c...
6dc1f231e06c630a0a8a008b3cc8295471e63f79
Rex Johnston
MDEV-39492 Parallel Query: GROUP BY

The manager hands each drained row to the plan's terminal function, and for a
GROUP BY that is sub_select_postjoin_aggr(): it writes the row into the server's
aggregation temp table keyed on the group, and at end of records reads the table
back and drives every stage after it. So the grouping, the aggregates, HAVING
over the groups and an ORDER BY of the groups are the server's own work, done on
the manager, and this commit adds none of them. What it does is stop refusing
them.

The gate no longer refuses group_list, and no longer refuses need_tmp. The shapes
a temp table is otherwise built for -- DISTINCT, ORDER BY, window functions, a
procedure -- are each refused on their own terms, which says what is excluded
instead of excluding a superset of it. need_tmp said little there in any case: the
gate runs from make_join_readinfo(), before make_aggr_tables_info() plans the
aggregation, so it still held its pre-planning value.

What a chunked scan cannot promise is row order, and two of the server's group
terminals need it. end_send_group and end_write_group find a new group by
comparing each row's group values with the previous row's, so given unordered rows
they would emit a group per run of equal neighbours. The optimizer picks one of
them when its plan delivers the rows already grouped, which EXPLAIN shows as
"Using filesort" without "Using temporary". end_update and end_unique_update look
each group up in the temp table by key and do not care what order the rows arrive
in, and that is the plan for a GROUP BY no index resolves -- the case worth
parallelising.

Which terminal was chosen is not known when the gate runs, for the same reason
need_tmp was not. So pwt_plan_needs_group_order() asks in run_worker_side_join(),
after the aggregation has been planned, and declines the parallel scan the way the
engine can when it will not do a consistent read. AGGR_OP gets a getter for
write_func so the question can be asked.

With no GROUP BY, end_send_group is the terminal and is safe: there is one
implicit group, group_fields is empty and no group change is ever reported. That
is the aggregate case, which continues to run in the workers.

An ORDER BY that reaches execution unrefused is sound only when there is an
aggregation temp table for the sort to be applied to, because then the order is
established on the manager after every row has arrived and the order they arrived
in does not matter. Asserted rather than assumed, since the gate cannot see such
an ORDER BY either.

Measured shapes are in the new test, including the two halves of the order
decision: a GROUP BY that gets the keyed temp table runs in the workers, and
COUNT(DISTINCT) per group and WITH ROLLUP get the sorted plan and are declined.
GROUP BY the primary key is refused earlier and for an unrelated reason -- the
plan reads the clustered index in order, and a driving table that is not a full
scan is not one the gate accepts.

This commit was prepared with Claude Code: it traced the terminal-function chain
to find that one call reaches every post-scan stage, established from
set_postjoin_aggr_write_func() which terminals depend on row order, wrote the
gate relaxation and the execution-time check, and measured each shape in the test
against its serial answer -- correcting two explanations it had first written,
about indexes forcing the ordered plan, that EXPLAIN then disproved.
6a124d65d641...
6a124d65d6415921cebd914ed58412520252c5de
Rex Johnston
MDEV-39492 Parallel Query: count the queries the workers ran

Nothing reported whether a query had actually been executed in parallel. The
optimizer trace records which table the optimizer picked, but the engine can
still decline afterwards and the query run serially, so the trace answers a
different question, as parallel_query_join's assertion about CREATE ... SELECT
used to claim wrongly. The tests that compare a parallel result against a serial
one had to infer execution from a side effect instead, that the manager's
Handler_read_rnd_next stays low because the workers do the reading. That
inference is about to stop holding, the workers' statistics belong in the session
and the next commit puts them there.

Add Parallel_queries_executed, a session and global status counter, incremented
in run_worker_side_join() once the workers are running, which is after the engine
has had its chance to decline. include/parallel_query_fingerprint.inc now asserts
on it: the serial run must show none and the parallel run must show some. That is
the question the tests were always asking.

This commit was prepared with Claude Code.
6492676a9aae...
6492676a9aae9f792069740cc114ac81c31549e1
Rex Johnston
MDEV-39492 Parallel Query: make the two cost factors session variables

PARALLEL_SCAN_ROW_COST_FACTOR and PARALLEL_WORKER_SETUP_COST were #defines, and
both are measured quantities rather than derivations -- the comment on the first
said as much, that 1.16 is the conservative end of a range the same query moves
around in. Measuring them again means sweeping them, and sweeping a #define means
a rebuild per value. They are now session variables:

  parallel_query_setup_cost      cost of starting one worker
  parallel_query_row_cost_ratio  what a row costs a parallel scan relative to a
                                serial one

This follows the pattern the engine-independent optimizer costs already use: the
default stays a #define in optimizer_defaults.h, the variable is a
Sys_var_optimizer_cost over a SESSION_VAR, and the provenance comment moves to
where the default now lives. The setup cost is a cost and takes COST_ADJUST(1000)
like its neighbours, so the user sees 22 for an internal 0.022. The ratio is
dimensionless and takes COST_ADJUST(1), like optimizer_disk_read_ratio.

Note for the next person to add one of these: a COST_ADJUST(1000) variable also
needs a line in mysqld.cc, next to the two that are already there --

  us_to_ms(global_system_variables.parallel_query_setup_cost);

SET goes through Sys_var_optimizer_cost::session_update(), which divides by
cost_adjust itself, but startup does not: it stores the option default in the
units the user gives it in and that hand-written list is what converts it. Without
the line the default is a thousand times too large, and here that meant a setup
cost of 22ms per worker instead of 0.022, which added 176ms to the cost of every
plan with eight workers and turned the parallel discount into a parallel penalty.
It was found by measuring the variable rather than by reading the class, and it
would not have been caught by the suite: the tests that assert on parallel costs
would have failed, but they only assert directions, and the direction was
consistently wrong rather than absent.

No behaviour change otherwise, and that is checked rather than asserted: the
per-table costs of a two-table join at zero and eight workers are the same
figures before and after, 3.353/17.968 and 0.662/2.246.
5dbe8cdf80a2...
5dbe8cdf80a22ea13bafbde06b3e75e8c3f5d897
Rex Johnston
MDEV-39492 Parallel Query: a worker runs the server's nested loop

The worker's hand-written join goes, and sub_select() takes its place. Third and
last step of the executor split on the worker side; the manager still collects
and sends.

worker_join_inner() was a nested loop that understood inner equi-joins and
nothing else, which is why can_run_query_in_workers() is a list of refusals: an
outer join, a semijoin strategy, a join buffer, LIMIT or an aggregate would each
have had to be written into it a second time. What replaces it is the loop the
serial plan runs. Each of the worker's JOIN_TABs points at the next through
next_select, the last one at pwt_end_send() where a serial plan would have
end_send(), and worker_run_query() drives it the way do_select() does -- once to
produce rows, once more to signal end of records. Every capability the gate
currently refuses becomes a question about what the manager does with the rows,
not about what the worker can join.

Two pieces of plumbing were needed. The driving table reads through the engine's
chunk reader rather than a table scan, so it gets a record source of its own;
every other table keeps the one make_join_readinfo() chose, because those
functions act on the JOIN_TAB they are handed and so already read this worker's
table through this worker's ref. And the executor reaches the worker through
function pointers whose signatures carry no worker, so a thread-local holds it
for the length of the join, the way mysqld.cc holds THR_THD.

Three things had to be got right that the copy in "give a worker real JOIN_TABs"
had left plausible but wrong, all found by the suite:

  - READ_RECORD mixes plan and per-scan state. make_join_readinfo() sets the
    row-fetching function and unlock_row at optimize time -- join_read_next_same
    for a ref table, join_no_more_records for eq_ref -- while the buffers are
    filled when the scan starts. Clearing the whole struct left null pointers for
    sub_select() to call, so the plan half is now restored by hand.

  - The worker's JOIN must describe the worker's array. JOIN_TAB::pfs_batch_update()
    finds the innermost table by join_tab + table_count - 1, and with the
    manager's table_count, which counts the const tables it resolved before the
    join, that lands past the end. table_count and top_join_tab_count are the
    worker's table count and const_tables is zero, and cached_pfs_batch_update is
    recomputed per tab rather than inherited, because sub_select() asserts it
    against the live answer.

  - TABLE::reginfo.join_tab is read at execution time, not just by the optimizer:
    join_read_next_same() finds the tab it is reading for through it. A worker's
    table copy has to point at the worker's tab. That is the third field of TABLE
    to need this after map and in_use, so the standing rule holds -- assume
    anything the optimizer sets on a TABLE is missing on a copy until checked.

read_first_record() means read the first record, which the pscan source at first
did not, leaving evaluate_join_record() to run once on a buffer nothing had
filled: one extra row per worker, and NULL, so a count changed while a checksum
over the same rows did not. Worth remembering as a shape -- an off-by-one in row
count that no fingerprint catches.

The trackers ANALYZE reads now come from the executor itself, which counts
r_scans, r_rows and r_rows_after_where into JOIN_TAB::tracker -- already this
worker's. So the handover in "give ANALYZE the numbers the workers produced"
keeps working unchanged, except that sub_select() counts one scan of the driving
table per worker where the report wants one between them, so that one is no
longer summed.

No test of its own: this is a refactor, and its test is that the suite still
passes. parallel_query_join and parallel_query_worker_side between them drive
eq_ref, ref with fan-out, a three-table chain, a non-indexed inner table, a
pushed index condition and a join buffer's scan filter, and all of it now goes
through sub_select() and evaluate_join_record(). main and innodb pass in full at
2103 tests, the parallel tests pass under --ps-protocol, and the forced-worker
sweep is at its known failures.

Not what was expected: the warning that reports "at row 0" was thought to fall
out of this, since sub_select() calls reset_current_row_for_warning(). It does
not. Warnings a worker raises already relayed with the same text as serial for
every class a SELECT can produce, so that defect is about a warning type reached
another way and stays open.

This commit was prepared with Claude Code: it established which JOIN_TAB and
READ_RECORD state is plan and which is per-scan, wrote the record source, the
terminal and the chaining, and diagnosed each of the four failures above from the
suite rather than from reading.
5551d2e9e59a...
5551d2e9e59ae043870059cbaa01193956879c4e
Rex Johnston
MDEV-39492 Parallel Query: divide the whole join's cost, not the split table's

scale_cost_for_parallel_scan() discounted the driving table's scan and nothing
else, so every table joined after it was costed at its full serial price. But a
worker does not only scan its chunk: it runs the whole join over that chunk, so the
work of each later table is divided between the workers exactly as the scan is.

Costing it the old way was wrong in two directions at once. It overstated what a
parallel plan costs, so parallelism was chosen less often than it should be. And
because the overstatement lands entirely on the tables after the split, it biased
the join order toward orders whose later tables are cheap, which is not the same
as the order that is cheapest to run in parallel.

POSITION gains parallel_workers, set on the driving table when the access finally
chosen for it was the scan that was costed as parallel, and left at 0 otherwise --
including on a driving table whose scan lost to an index, where nothing is
parallel. Each table joined after it divides its cost by that number.

Where the division is applied differs between the two cases, on purpose.

For the driving table it stays where it was, inside the costing of the candidate
access methods, because there it is the point: dividing the scan and not the index
is what can make a full scan worth more than an index, and that is a choice the
optimizer should make.

For the tables after it, the division is applied to the cost recorded for the plan,
after the access method has been chosen. Dividing every candidate for a table by
the same number cannot change which of them is cheapest, so applying it during the
choice would buy nothing, and it would risk changing a choice by scaling some cost
components and not others. Applying it to the recorded cost also fixes every
consumer at once: five places sum POSITION::read_time into a plan cost
(optimize_straight_join, best_extension_by_limited_search, greedy_search and two
re-costing loops), and each would otherwise have needed the same conditional.

Row counts are untouched. Parallelism divides the work, not the result.

Measured on a two-table straight join with eight workers: the driving table costs
3.353 serially and 0.662 in the workers, as before; the eq_ref table joined to it
costed 17.968 either way and now costs 2.246, which is 17.968/8. The plan costs
21.32 serially, would have been costed at 18.63, and is now costed at 2.91.

main and innodb pass at 2109 and the forced-worker sweep did not gain a failure,
but that is weaker evidence than it looks and is worth saying so: normal mode runs
with parallelism off, and the discount needs InnoDB's clustered-index statistics to
engage at all -- pscan_chunk_count_estimate() answers stat_n_leaf_pages, which is
zero until they are gathered, and zero means no discount for any table. So most of
the suite's freshly loaded tables do not exercise this. The new assertion in
main.parallel_query_worker_count is what pins it: it gathers statistics and checks
that the joined table's cost falls by exactly the worker count, and without this
change it records that the cost does not fall at all.

Left alone deliberately, as separate decisions: the per-row transport factor is
charged against the driving table's rows, where the rows that actually cross to the
manager are the join's output rows, and those differ once there is fan-out or a
selective join; there is still no notion of core count, so this divides by twelve
on a six-core machine; and there is no term for the contention of several workers
looking rows up in the same inner table.

This commit was prepared with Claude Code: it confirmed the reported gap by
extracting per-table costs from ANALYZE FORMAT=json rather than reading the code
alone, found that the discount does not engage without InnoDB statistics (the first
probe showed identical costs at zero and eight workers), chose the placement so
that no local access choice changes and all five cost consumers are fixed together,
and verified the new assertion fails when the division is removed.
523895cd16db...
523895cd16db8fe50962f96e766d5aa8d3857129
Rex Johnston
MDEV-40012 Parallel Query: execute the join in the worker threads

Each parallel worker now runs the join and the WHERE over its own chunk of
the driving table and ships the base-table columns of every row that
qualified. The manager copies those columns back into the fields they came
from in its own table instances, so that after a row is drained its records
hold what a serial scan would have left there, and then evaluates the select
list against them and sends the row. This replaces the model where workers
shipped raw source records and the manager ran the join.

The transport carries columns rather than projected select-list values
deliberately. Anything the manager does with a row beyond sending it -- a
condition, an aggregate, a temp table -- reads a record, and not all of those
reads go through Items that could be re-pointed at a shipped value:
create_tmp_table() builds Copy_field pairs holding raw Field pointers into
the base tables. Filling the records themselves is what makes every later
stage work without an indirection of its own.

make_join_readinfo()'s gate (can_run_query_in_workers) chooses the
worker-side path for an inner select-project[-join] with a parallel-
scannable driving table: no tmp table (group/distinct/order/window/
buffer), no LIMIT/SQL_CALC_FOUND_ROWS/procedure/aggregate, no outer join
or semijoin, and every non-driving table reached by eq_ref/ref/full
scan. do_select() then runs run_worker_side_join() instead of the nested
loop; anything ineligible runs serially.

Each worker opens a private copy of every non-const table, deep-clones
and field-rebinds the conditions and select list, and rebuilds each ref
(clone_table_ref, mirroring create_ref_for_key). It scans its driving
chunk, runs its own inner nested loop (cp_buffer_from_ref +
ha_index_read_map / ha_index_next_same for ref/eq_ref, rnd scan
otherwise), projects the read columns of each full match into a private
result table and ships the row image; the manager drains, copies the columns
into its own records and sends.

Writing into those records is not something a SELECT's write_set allows and
Field::store() asserts on it, so the drain marks the fields writable for its
duration; and Copy_field captures &table->null_row, so the manager's tables
have their reader flags cleared before it starts.

A killed worker's own ER_QUERY_INTERRUPTED is no longer treated as a
fatal evaluation error (PWT_error_handler guards with !thd->killed) so
kills keep propagating through kill_signal with the correct kill type.

Standalone helpers are attached to the object they operate on
(pwt_worker / pwt_manager members rather than file-static functions) and
the functions that carry real control flow have DBUG_ENTER tracing.

Tests: parallel_query_worker_side (single table) and parallel_query_join
(eq_ref, ref with fan-out, 3-table chain, full-scan inner table) compare
the parallel result set against serial. parallel_query / parallel_query_oom
moved to a plain SELECT (which now runs worker-side) and were re-recorded.

This commit was prepared with Claude Code: it wrote the worker-side join
execution (worker_join_inner / worker_emit_row, the per-worker table,
ref and expression cloning) and the two new tests; the naming, the
member-function layout and the DBUG tracing are the author's cleanup.
403915b2337f...
403915b2337fb75639144a110716f6d3a36a96a7
Rex Johnston
MDEV-39492 Parallel Query: filter by the condition from before the pushdown

When the optimizer pushes part of a condition into the engine,
push_index_cond() leaves tab->select_cond holding only the remainder and keeps
the original in tab->pre_idx_push_select_cond. The pushed half lives on from
there in handler::pushed_idx_cond, which belongs to the handler it was pushed
into -- the manager's. A worker reads through its own handler, opened by
open_table_from_share() with nothing pushed into it, and it cloned select_cond,
so the pushed half was enforced in neither place.

  SELECT pk, a, b FROM p1,p2,p3 WHERE b >= d AND pk < c AND b = '0';

answered one row serially and six with workers, `pk < c` having been applied
nowhere, and the unfiltered rows then multiplied against the third table.
Setting index_condition_pushdown=off made the parallel answer correct, which is
what pinned it on the pushdown rather than on the plan the cost model chose.

Clone the pre-pushdown condition where there is one. One accessor,
pwt_table_cond(), is used by the gate and by both clone sites, so the item the
gate approves is always the item a worker evaluates -- the two drifting apart is
what this bug was.

This gives up what the pushdown was for: the engine no longer rejects an index
entry before the row is read, so a worker does more clustered-index work per
match than the serial plan. The alternative was to refuse these plans at the
gate, which would have cost the parallel scan altogether on a common plan shape.
Correct and parallel beats correct and serial here, and pushing a clone onto the
worker's own handler would recover the difference -- that wants the worker to
hold a real JOIN_TAB to hang the key number off, so it belongs with the cloned
JOIN, not before it.

parallel_query_join gains the query above, answered serially and again with
workers, asserting that it still ran in the workers rather than falling back and
that p1 still carries a pushed index condition, so the case cannot quietly stop
being covered. Without the fix the parallel answer is six rows.

range_innodb, which is where this was found with parallel_worker_threads forced
on, now differs only in EXPLAIN output.

This commit was prepared with Claude Code: it traced the predicate to
push_index_cond() moving it out of select_cond, and wrote the test.
3b1d89688e70...
3b1d89688e703e0f11545197a575d7ed2c4a8b0b
Rex Johnston
MDEV-39492 Parallel Query: the workers' statistics belong to the session

A worker counts its reads, its lock calls and everything else in its own THD, and
~THD puts them straight into the global counters. Nothing ever reached the
session that asked for the work, so SHOW SESSION STATUS was short by whatever the
workers did: after a parallel scan of a 1000-row table Handler_read_rnd_next
reported around 500, the rows this thread read out of the materialised result,
rather than the 1502 the same query reports serially. Nine tests in the main
suite noticed, in Handler_read_%, COLUMN_DECOMPRESSIONS and
Optimizer_join_prefixes_check_calls.

Each worker now copies its status counters into its pwt_worker just before its
THD is destroyed, and quiesce_workers() adds them to the session's own after
joining every worker, so only one thread ever touches either side and no locking
is involved. The worker then clears its own counters, which stops ~THD adding the
same numbers to the global counters that the session will pass on later.

Only the counters move. Memory accounting stays with the worker's THD, because
more of that THD's memory is freed after the snapshot is taken and ~THD has to
reconcile all of it with the global counters. Clearing with the
clear_for_flush_status offset leaves those fields alone, and the snapshot drops
its copies of them. Suppressing ~THD's accounting entirely instead, which is the
obvious way to avoid counting the same numbers twice, loses that reconciliation:
the server then reports an internal memory accounting error of a couple of
hundred thousand bytes at shutdown.

Session and global counters were not equally wrong. Global was right all along,
because ~THD was adding to it, so this changes which session the work is
attributed to and not the total.

parallel_query_worker_side compares Handler_read_rnd_next for the same query run
serially and in parallel, and requires the parallel run to report at least as
much. Written as a comparison rather than an exact figure because a chunked scan
ends at each chunk, so it reads a few more than a single scan does. Without the
handover the parallel run reports about a third of the serial figure and the test
fails.

This commit was prepared with Claude Code.
3a62e0b58d55...
3a62e0b58d5553e267019a6c3b77b631f0ca0827
Rex Johnston
MDEV-39492 Parallel Query: refuse semijoin strategies the worker cannot run

The gate refused semijoin materialization, through bush_children, and LooseScan,
through loosescan_match_tab, but nothing else. FirstMatch and DuplicateWeedout
left the plan looking like an ordinary inner join in the flat table list, and
pwt_worker::worker_join_inner() is a plain nested loop that knows nothing of
either, so it emitted exactly the duplicates they exist to remove.

  SET join_cache_level=0;
  SET optimizer_switch='materialization=on,semijoin=on,firstmatch=on,loosescan=off';
  SELECT * FROM t1 JOIN t2 ON (t2.f4 = t1.f3)
    WHERE ( 8 ) IN (SELECT t3.f1 FROM t3, t4);

answers two rows serially. Run by the workers under a FirstMatch plan it
answered three, one of them repeated and the other lost, and the count varied
with how the chunks fell. It was only reachable after the previous commit stopped
this shape of query crashing.

That the plan, not the parallel execution, was at fault could be ruled out by
keeping parallel_worker_threads at 4, so the cost model still produced the
FirstMatch plan, and adding FOR UPDATE, so the engine declined the parallel scan
and the same plan ran serially. That answers two rows.

Refuse a join tab carrying any semijoin strategy state: sj_strategy,
do_firstmatch, check_weed_out_table, flush_weedout_table, first_weedout_table,
and loosescan_match_tab as before. Testing the strategy rather than the shape of
the join means the two strategies that hide in a flat table list are covered by
the same condition as the two that do not.

parallel_query_excluded gains the query above in its smallest form, asserting
that the optimizer does not choose it and that it answers one row per matching
outer row rather than one per inner match.

This commit was prepared with Claude Code: it found the wrong result while
checking what the previous commit had uncovered, isolated plan from execution,
and wrote the test.
2fb12d1572e0...
2fb12d1572e039b170e28a73719704c28e737fef
Oleg Smirnov
Enable release builds
2b7db2d561c8...
2b7db2d561c80e67cc8b20f1d55b102f8a3407ec
Rex Johnston
MDEV-39492 Parallel Query: a re-split chunk lost its first sub-tree

The commit before this one calls Ctx::split() on the contexts flagged for it, and
splitting re-partitions a chunk by calling Scan_ctx::partition(), which descends
from the index root again bounded by that chunk's start key. That is the first
caller in the engine to pass a start bound at all: while the SQL layer asked only
for whole-table scans, every partition began before the first record and the
bound was always null.

Scan_ctx::create_ranges() searched for that bound with PAGE_CUR_GE at every
level. Above the leaves a record is a node pointer keyed by the first key of the
child it points at, so the child that contains a start key is the last one whose
key is not greater than it. GE lands on the child after that one, so the sub-tree
holding the start key is never descended into, no range is created for it, and
every row in it is left out of the scan with nothing reported anywhere.
Scan_ctx::search() already makes the distinction -- PAGE_CUR_LE above the leaves,
PAGE_CUR_GE at them -- and the comment at the top of the file describes it.
create_ranges() now makes it too.

Measured before the fix, with re-split enabled: SELECT COUNT(*) over 1000000
narrow rows answered 820111 with four workers and 1000000 with two,
deterministically and with no error, because the loss needs a tree at least three
levels deep for a chunk to be re-split at all.

No test here, and it is placed where it is for that reason: the tests that
demonstrate it need a harness that does not exist yet at this point in the series
-- include/parallel_query_fingerprint.inc, and a worker-count test to hang the
chunk figures on. They arrive with "bound the split, and use it where it is
needed", which is also where the split becomes bounded and the chunk counts
settle. Putting the one-line descent fix here rather than there keeps every
commit in between from carrying a scan that silently drops rows.

This commit was prepared with Claude Code: the row loss turned up while it was
benchmarking an unrelated change, and it bisected the cause to the re-split path
by disabling re-split, then identified the GE/LE descent by comparing
create_ranges() with its sibling search().
276bba5cb407...
276bba5cb407bbc716231da2a43e15d809ecde5e
Rex Johnston
MDEV-39492 Parallel Query: decline by the chunk count, not the worker count

"start no more workers than the engine has chunks" clamped the worker count to
the chunk count and then declined the parallel scan when the result was below
two. Those are two different questions and only the first was meant. A table the
engine cannot divide is a table the serial path reads for less, which is worth
declining. One worker because parallel_worker_threads says one is the user
asking for one worker, on a table that may divide perfectly well, and it is not
even reliably slower -- the worker's scan overlaps the manager's sending of the
rows already produced, which measured as a gain rather than a loss on a join.

Test the chunk count, and clamp after.

Two tests had been exercising nothing since that commit. parallel_query_oom
injects a failure into a worker's error queue and expects the manager to surface
one ER_OUTOFMEMORY warning; it asks for one worker, so it was being declined and
no worker reached the injection. Its table also holds two rows, one leaf page and
therefore one chunk, so it needs both halves of this fix: the table grows to more
pages than one, and the decline no longer refuses its single worker.
parallel_query kills a worker mid-scan and looks for the thread group in the
processlist; its table was two rows as well, so three workers were asked for and
none started, and the test sat in wait_condition until it timed out.

Neither failure was visible because both tests require have_debug and the tree
was a release build, which skips them -- a suite that reports every test passing
while silently skipping the two that cover the thing being changed. The tables
now hold 2000 rows, enough leaf pages for the workers each test wants, and the
selects carry a WHERE so their output stays as small as it was while the whole
table is still scanned. parallel_query's ten-worker case now records the clamp
working rather than ten warnings.

This commit was prepared with Claude Code: it found both tests broken while
setting up a debug build to exercise new assertions, traced them to the decline
condition rather than to the tests, and distinguished the two questions the one
condition had been answering.
1c6ab486499b...
1c6ab486499be39a78d6d2f25c1d3d0dc78ae3fb
Rex Johnston
MDEV-39492 Parallel Query: the manager sends rows through the plan's terminal

The other half of the executor split, and the last of the two hand-written
executors. manager_collect_and_send() drained a worker's batch and called
select_result::send_data_with_check() itself, keeping its own count of rows sent,
rows accepted and duplicates -- a second copy of end_send() with none of what
end_send() also does.

It now stands on the real one, and does not choose it either. The row is handed
to the last real table's next_select, called with the tab after it -- exactly the
call sub_select() makes -- so whatever make_aggr_tables_info() decided this plan
should end in is what runs. For the shapes the gate accepts that is end_send().
The row accounting, HAVING, and the LIMIT checks including WITH TIES are the
server's from here on.

Nothing has to be pointed anywhere first, because the manager's records already
hold the row: the workers ship base-table columns and the drain copies them back
before the terminal is called.

end_send() loses its static. Its siblings end_send_group() and end_write_group()
are already declared in sql_select.h, and sub_select() already is too -- which is
what a worker was given to run in the commit before this one -- so this is the
odd one of the family rather than a new kind of exposure.

What it is for. The gate refuses LIMIT, HAVING, GROUP BY, ORDER BY and
aggregates, and every one of those is an operator that stands where end_send()
stands or just above it. With the manager's loop ending in the server's own
operator, each becomes a question of which operator to end in and what the
workers must produce for it, rather than another few dozen lines here. The
NESTED_LOOP_QUERY_LIMIT return is wired to the stop flag that the batch handoff
protocol has always had and never used, so a manager that has enough rows now
stops its producers instead of draining them.

LIMIT was checked by opening the gate temporarily: SELECT ... LIMIT 5 over a
20000-row table in four workers sends exactly five rows and terminates, and
LIMIT 999999 sends all twenty thousand. The gate is left closed, because LIMIT
without ORDER BY returns whichever rows finish first rather than the ones a
serial scan would reach, and while that is a legal answer to an unordered query
it is a different answer, so it wants a decision rather than a commit.

No test of its own: the rows the manager sends are the same rows, and the suite
is what says so. main and innodb pass at 2103, the parallel tests pass under
--ps-protocol, and the forced-worker sweep is at its known eighteen.

This commit was prepared with Claude Code: it traced the next_select chain to
establish that one call reaches whatever the plan ends in, which is what let the
manager stop choosing an operator for itself, and verified the LIMIT path against
a temporarily open gate rather than leaving the branch unexercised.
10cedd1fca8c...
10cedd1fca8c9fc8c13ab3b67f6c87dd3bfca3e7
Rex Johnston
MDEV-39492 Parallel Query: a worker's result table belongs to the worker

create_tmp_table() runs on the manager's thread, so every worker's result table
came out with the manager in TABLE::in_use, and nothing put the worker there
afterwards, unlike the table copies in open_worker_tables(). Field::get_thd()
hands out TABLE::in_use, so when worker_emit_row() projected an item into one of
those fields and the projection raised a warning, the warning was raised on the
manager's THD from the worker's thread.

  CREATE TABLE t1 (a TIME(6));
  INSERT INTO t1 VALUES ('838:59:59.999999');
  SELECT a, a + INTERVAL 2 YEAR FROM t1;

produces ER_DATETIME_FUNCTION_OVERFLOW per row from
Item::save_date_in_field(), which passes field->get_thd() down to the function
that raises it. Three things followed. The worker's own error handler never saw
the condition, because it is installed on the worker's THD and the condition was
raised on the manager's, so instead of being relayed through the message queue it
was stored directly in the manager's diagnostics area. That store came from a
worker thread while the other workers ran, unsynchronised, on a structure the
manager also uses. And the memory for it was charged to the worker's THD, since
thread-specific allocations are accounted to the running thread, while the block
itself belonged to the manager's Warning_info, so the worker's
status_var.local_memory_used was still 2040 at destruction and
destroy_background_thd() tripped the not-freed-memory assertion in ~THD.

Put the worker in TABLE::in_use once the table is built, next to where its other
tables get the same treatment. The warning is then raised on the worker's THD,
PWT_error_handler relays it, and finalize_parallel_workers() surfaces it on the
manager after the join, which is the path it was always meant to take.

parallel_query_worker_side runs the query above serially and again with workers,
and records both. The rows and the three warnings match, so the relay is checked
for what the user sees rather than only for the absence of a leak. Reverting the
one line makes the test abort in ~THD.

type_time_hires now passes with parallel_worker_threads forced on.
type_temporal_innodb still fails there, on Item_cache_time inside
Item_func_min_max: a shallow-copied cache shared between workers, the same family
as the copy that is not independent refused in 11481453439, but reached through a
cache rather than a field, which that commit's shared-leaf test does not see.

This commit was prepared with Claude Code: it traced the leaked block to the
diagnostics area with safemalloc's report, found the manager's THD reaching the
worker through Field::get_thd(), and wrote the test.
105003819d39...
105003819d394827f4cb9384bd78da08b0ba306c
Rex Johnston
MDEV-39492 Parallel Query: charge the manager's drain, per row, undivided

parallel_query_row_cost_ratio multiplied the driving scan's per-row cost by 1.16
and then divided the result by the worker count. That shape says the cost of
moving a row to the manager falls as workers are added, and it does not: the
manager is one thread, it drains one worker at a time, and every row it handles
costs it the same whether two workers or eight produced them. The ratio is
replaced by a cost that is added rather than divided.

  parallel_query_drain_row_cost  what the manager spends per row it drains

Measured on a release build at 29.6 ns per row, and the figure came out the same
for a scan bound by I/O and a scan resident in the buffer pool -- which is what
says the exchange is what was measured rather than the read. The default is
2.96e-05, in the milliseconds the optimizer's costs are in.

It is charged on the rows the join produces, at the point where the join order's
cost is complete, next to where a sort of the result is costed. Not on the rows
the driving table scans: a row the WHERE rejects never crosses to the manager, and
a join fans out or filters, so the two counts part company as soon as there is
more than one table. The consequence is that the driving table's own access
decision cannot see this term -- a parallel scan is weighed against an index for
that table without it -- which is the approximation the sort cost already makes,
and is smaller than charging the term against the wrong row count would be.

Checked arithmetically rather than by direction. For a 20000-row table with eight
workers the table's own cost is 0.59511, which is its serial 3.35292 over eight
plus eight setups, and the plan's cost is 1.18711: the difference is 0.592, which
is 20000 rows times 2.96e-05. For a two-table join over the same tables the joined
table costs 2.24603, its serial 17.96826 over eight, and the plan's cost exceeds
the sum of its tables by 0.592 again -- charged once, on the 20000 rows the join
produces, not twice.

What this does not fix, and is worth knowing before the numbers are trusted. The
model still makes a parallel plan look about 2.8 times cheaper than serial on that
table where measurement gives about 1.6, and the remaining optimism is not in this
term. It divides by the worker count with no notion of how many cores there are,
so it will divide by eight on a six-core machine where measurement plateaus at
five. And the optimizer's own estimate of that scan is about 167ns per row where
it measures at 45 to 79, so a correctly measured coefficient in real milliseconds
is being added to a scan estimate that is not in real milliseconds. Dividing an
overestimate overstates the saving, and this commit does not touch that.

main.parallel_query_worker_count pins both properties. Raising the variable by 10
over a 20000-row scan raises the plan's cost by exactly 200, which is the rows
times the rise; and over a join whose inner predicate leaves about a hundred of
those 20000 rows the same rise costs about 1 rather than 200, which is what says
the row count being charged is the join's output and not the driving table's scan.
Removing the two calls that add the term turns those into 0 and 0.

Two catalogue tests record the variable set and both needed re-recording:
main.mysqld--help and sys_vars.sysvars_server_notembedded. The second is not
picked up by --suite=sys_vars here and has to be named.

This commit was prepared with Claude Code: it measured the coefficient on a
release build in two regimes to establish that the exchange rather than the read
was what it had hold of, worked out that the ratio's shape was wrong rather than
its value, chose the charging point so that the row count is the join's output,
and checked the result by arithmetic and by removing the term rather than by
looking at whether costs moved in the right direction.
0d7f0918c482...
0d7f0918c4821a96b75479e7c8aca8ca92c0e122
Rex Johnston
MDEV-39492 Parallel Query: DISTINCT

The gate refused select_distinct. It no longer does, and the manager gains
nothing to do: DISTINCT is always carried out on a materialised temp table
downstream of the drain, and the two routines that carry it out are both
indifferent to the order the rows arrived in. remove_dup_with_hash_index()
obviously is. remove_dup_with_compare() is too, despite the name -- it compares
each row against all the rows after it rather than against its neighbour, so it
does not need its input sorted.

The honest measure of what this changes is narrower than removing the refusal
suggests, and worth stating. Most DISTINCT queries never reach the gate as
DISTINCT: the optimizer rewrites them to GROUP BY ("Change DISTINCT to GROUP BY",
sql_select.cc), which clears select_distinct before make_join_readinfo() runs, so
they have been arriving as the GROUP BY case and running in the workers already,
since the commit that stopped refusing GROUP BY. That was capability nobody asked
for and nothing tested. The first half of the new test covers it.

What removing the refusal actually adds is the shape the rewrite leaves alone.
The rewrite is conditional on there being no row limit, so a real DISTINCT is
DISTINCT with a LIMIT, and that is the path through remove_duplicates() described
above. Which rows a LIMIT of an unordered result keeps is whichever finish first,
exactly as for LIMIT without DISTINCT, so the test checks that the right number
come back, that they really are distinct, and that each is a value the table
holds.

One thing has to be declined rather than allowed. JOIN::optimize_distinct() marks
the trailing tables a DISTINCT does not select from, so the join stops looking for
further matches once a row has been produced -- every later match would only make
a duplicate for the temp table to remove. It runs from make_aggr_tables_info(),
after the gate, so this is asked in run_worker_side_join() as the group order
already is. A worker ignoring the flag would still answer correctly, since its
duplicates would be removed downstream, but it would read more than the plan asked
for, and doing that quietly is not something to decide by accident.

ORDER BY is not enabled by this commit. An ORDER BY of the scan itself is still
refused: that order comes from the driving table's own filesort, which cannot
survive being split into chunks finishing in an arbitrary order, and delivering it
would need the workers to sort their chunks and the manager to merge them. That
check now says so where it stands, because it is easy to mistake for a refusal of
every ORDER BY, and it is not one: where the rewrite absorbs the ORDER BY into the
GROUP BY it just made -- which it does when every field sorted on is a select-list
field it has grouped by -- join->order is null by the time the gate looks, the sort
runs on the manager over the finished temp table, and the query is ordered and
parallel. That covers the single-table DISTINCT ... ORDER BY shapes. Over a join it
does not hold, the gate sees the ORDER BY, and the query runs serially; both are in
the test, next to each other, because the difference is not obvious.

This commit was prepared with Claude Code: it established that DISTINCT was
already running in the workers through the GROUP BY rewrite before changing
anything, read both dedup routines to decide whether arrival order could matter
rather than assuming, measured which shapes the relaxation actually affects (the
answer was none of the ones first tried, which is why the LIMIT shapes are in the
test), and found the optimize_distinct() marking by asking what the existing
inertness assertions would catch.
0b75032ec026...
0b75032ec02669969fd193b3c5a823a562e7fd84
Rex Johnston
MDEV-39492 Parallel Query: aggregates with no GROUP BY

The gate refused any query with an aggregate. It no longer does, and there is
almost nothing behind that beyond the refusal itself: the manager hands each
drained row to the plan's terminal function, which for these queries is
end_send_group(), and the aggregation is then the server's own, done by the
query's own Item_sum objects over records that hold what a serial scan would
have left in them.

So no Item_sum is copied into a worker, no aggregate's argument is redirected
anywhere, no combine rule per aggregate kind is needed, and the empty result
answers COUNT 0 and the rest NULL through end_send_group()'s own end-of-records
path rather than a special case. The workers do not aggregate at all; they run
the join and the WHERE and ship columns, as they already did.

That is also why every kind of aggregate is accepted rather than a chosen few.
MIN and MAX read their argument through an Item_cache built when they were
fixed, AVG and STD keep their own counters, and the DISTINCT variants hold the
set in an Aggregator. None of that is the transport's business once the record
the aggregate reads is the right one.

GROUP_CONCAT is refused, and it is the only aggregate that is. Its value is the
order the rows arrived in, and workers finish chunks in an order that varies
between runs, so it would answer differently each time where serially it answers
the scan order. A query whose answer changes between identical runs is worse than
one that runs serially.

Two gate checks go with it. The select list is no longer probed for clonability:
nothing clones it, and probing was actively harmful, because
pwt_item_is_clonable() copies an item to test it and copying an
Item_sum_min_max crashes -- its copy constructor leaves cmp uninitialised while
its cleanup() deletes it, which looks like a defect worth raising separately.

What is still refused is a second aggregation stage, because the gate still
refuses need_tmp: an expression over an aggregate, HAVING over an aggregate, and
an ORDER BY of one all need a temp table and so still run serially. The test
records them at that, and they come in with the commit that stops refusing a
temp table.

This commit was prepared with Claude Code: it established by test which
aggregate kinds the manager can run rather than reasoning about them, found that
the gate's own clonability probe was what crashed on MIN, and checked each shape
in the test against its serial answer.
0af521a08baf...
0af521a08baf15cb7958923893b818bc70e42a59
Sergei Golubchik
BUG#39449066 Refactor performance schema OBJECT_INSTANCE_BEGIN columns

Fix for MariaDB 10.6
BuildBot (0.8.9) working for the MariaDB project.
Page built: Wed 05 Aug 2026 01:27:09 (UTC)