Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
f74936a3e098...
f74936a3e098e6d4fc423dfca6efb5ec8595e3f9
Rex Johnston
PQ: tidyup, answer some questions
f64ee3ac1a06...
f64ee3ac1a065769156c6bc9bb41725c54e92392
Rex Johnston
PQ WIP, transport tidy up
f3002155ed84...
f3002155ed84ef53fcf89aaecf1d762e5855ba4a
Daniel Black
MDEV-40949 Missing space after how-to-produce-a-full-stack-trace-for-mariadb link
f22bb7fe59d5...
f22bb7fe59d5c91dc6567e7c0666e950130823d1
Sergei Petrunia
Move pwt_worker_execution::stats into pwt_worker_base_with_stats
f14154b014e1...
f14154b014e1c4a6c9906fd630d194aad7460962
Rex Johnston
PQ: rebuild a heap temporary table to aria

A worker ships its rows through a temporary table, and when that table filled
the statement failed: nothing moved it to disk, so a result set larger than the
session's temporary-table limit could not be run in the workers at all.

pwt_tmp_table_sink::emit_row() now hands a full container to the server's own
create_internal_tmp_table_from_heap(), which rebuilds it with the on-disk engine
and writes the row that did not fit. Calling that from a worker took three
things, each a thread boundary the conversion never had to cross before.

Temporary-file space. temp_file_size_cb_func() charges whichever thread grows an
on-disk temp table and credits whichever thread shrinks or frees it, and for a
worker's container those are the worker and the manager. The worker now hands
its tmp_space_used over with its other counters -- add_to_status() already
carries the field, thread_func_end() was only zeroing it -- and zeroes its own,
so ~THD finds its books square and the later credit lands where the charge went.

Thread-specific memory. The rebuild allocates into the container, which the
manager frees. The worker measures what the rebuild cost it, takes that off its
own local_memory_used, and hands it to the manager in the sink's cleanup(),
which runs with every worker joined; adding to the manager's counter from the
worker would race with the drain that is running at the time.

The engine's own allocations. Aria allocates MY_THREAD_SPECIFIC unless it is
told the table is not confined to one thread, which is what replication's
HA_OPEN_GLOBAL_TMP_TABLE says. open_tmp_table()'s cross_thread argument now sets
that as well as dropping HA_OPEN_INTERNAL_TABLE, which is what the argument
always meant; only the parallel transport passes it.
create_internal_tmp_table_from_heap() takes it through to the re-open.

That leaves the stack-overflow guard. maria_open() caches a pointer into the
opening thread's my_thread_var for alloc_on_stack(), and a worker's is freed
when its THD is destroyed. handler::rebind_to_thread() is the missing sibling of
rebind_psi(): the same thing TABLE::in_use says to the SQL layer, said to the
engine. Aria overrides it to re-point stack_end_ptr, and the transport calls it
wherever it changes in_use. rebind_psi() cannot serve -- it is about
instrumentation, and is called only where a table leaves the table cache, which
a table handed straight from one thread to another never does.

main.parallel_query_spill covers both sizes and checks the content rather than
only the count: LENGTH(b) is evaluated by the manager, from bytes that came back
through a rebuilt container. main.parallel_query_transport had the old
limitation pinned in it, and is updated.

This commit was prepared with Claude Code: it took the three thread boundaries
one at a time from the assertion each produced, and established that the
manager's read path does not currently dereference stack_end_ptr -- Aria uses it
on key-search, write, delete and blob paths, and the manager scans a keyless
container sequentially -- so the rebind closes a latent hazard rather than a
live one.
e9561ab91330...
e9561ab91330accbc61d243d9063f9385e1fb3d1
Oleg Smirnov
Distribute parallel work more evenly

Reduce SPLIT_THRESHOLD and improve the chunk queue logic
e054cc5fb4f9...
e054cc5fb4f926c290e1733a7d38929706298db6
Rex Johnston
PQ: scan-only prototype, for testing the scan and the transport

Not for the shipping tree. Every hunk is bracketed START PROTOTYPE / END
PROTOTYPE so it can be lifted out again in one pass.

Behind debug_dbug='+d,pwt_scan_only' the workers do nothing but read their
chunk of the driving table and ship the rows. There are no worker JOIN_TABs, no
cloned conditions or select list, and no join in a worker. This thread runs the
plan it would have run serially, and the only difference is that the driving
table's JOIN_TAB reads its rows from the transport rather than from the
handler: do_select() stops diverting, start_scan_only() displaces
read_first_record, and the executor above it is untouched.

Why bother. The gate admits a few per cent of the queries in the test suite,
because almost every refusal it makes is about evaluating cloned Items in
another thread rather than about dividing a scan. Take the evaluation away and
nearly all of them go, so the chunked scan and the transport -- the two layers
underneath -- can be run over the whole suite instead of over that sliver.

How to run it. The scan-only flag on its own does nothing; the workers still
have to be asked for:

  ./mtr --parallel=16 --force --max-test-fail=0 --ignore-parallel-diff \
        --mysqld=--parallel-worker-threads=4 \
        --mysqld=--debug-dbug=+d,pwt_scan_only \
        --suite=main

--ignore-parallel-diff is what makes the run readable: without it a third of
the failures are nothing but the _parallel suffix EXPLAIN adds to the access
type. It does not cover FORMAT=JSON, where the value is quoted, so a handful of
json tests still differ on access_type alone. Never combine it with --record.

For one test, or from inside a test, the flag can be set on a live server:

  SET @sd=@@global.debug_dbug;
  SET GLOBAL debug_dbug='+d,pwt_scan_only';
  ...
  SET GLOBAL debug_dbug=@sd;

Reading the results. main/parallel_query_* fail by construction -- they measure
that the workers ran the join, and here they do not -- as do the environmental
five this tree always fails. What is worth reading is anything with a row
difference. As it stands the whole main suite gives 1358 passes and 39
failures, of which none is a wrong answer: 10 are trace, counter or JSON
EXPLAIN content, 7 are the parallel_query tests, 5 environmental, 2 EXPLAIN
only, 2 an unordered select whose rows arrive in another order, and one an
unordered LIMIT 2 that picks a different pair. No crashes.

What it cannot tell you, which is the more important half. Everything a worker
does with an Item is gone, and that is where the defects have actually been: a
worker THD that did not carry the session's time zone, a condition left only in
a Filesort, a reader that wanted a SQL_SELECT the worker copy did not have, a
materialized subquery re-opened per worker. None of those are reachable here;
most cannot even exist in this shape. Passing in this mode says nothing about
the path that ships. Its value is as a lower layer's test, and as a bisection
tool: a query that answers wrongly in both modes is wrong in the scan or the
transport, and one that answers wrongly only in the full mode is wrong in the
worker join.

Two things the mode needs that the full path gets for free. The displaced
reader is put back by finalize_parallel_workers(), because a JOIN_TAB outlives
one execution -- a prepared statement, a correlated subquery, a routine loop
all run the same plan again -- while the manager does not. And a tab whose
condition was partly pushed into the index is given the whole condition back
for the length of the scan: the pushed half lives in the manager handler's
pushed_idx_cond, which is no longer the handler producing the rows, so it would
otherwise be applied nowhere.

This commit was prepared with Claude Code: it wrote the mode, and found three
defects in it by running the suite -- a flush skipped because the scan loop
ends on HA_ERR_END_OF_FILE, the reader left installed across a re-execution,
and the index-condition pushdown above, which is the same trap the full path
hit from the opposite direction.
df939fc32a86...
df939fc32a862c82493a07c232ead718fc9ec1f3
Rex Johnston
MDEV-40012 Parallel Query: the manager applies the plan's ORDER BY

A plain ORDER BY was refused, so most sorted queries could not run in the
workers at all. The refusal was right at the time: make_aggr_tables_info()
sorts the driving table's own read for a plan that needs no temporary table,
and the workers take that read over, so the sort lost the scan it was attached
to. This shape has no aggregation table to move it to either -- the terminal
after the driving tab sends straight to the client.

The manager does it instead, over the rows it drains. A sort is the one
post-join step indifferent to the order its input arrives in, which is what
lets a plan that ends in one be handed out in chunks.

It sorts a container of the transport's own layout, not the base table, so the
plan's Filesort cannot be reused: its order names the manager's base-table
fields and the rows are in the container. pwt_row_layout remembers which
shipped column each ORDER element sorts on and builds the equivalent order over
a container's own fields, the same way the group key is rebuilt for the
pre-aggregation containers. The drain collects instead of sending, filesort()
runs, and each row read back takes the path a drained row would have taken:
copy_back_row() into the manager's base-table records, then out. The container
is rebuilt on disk if it fills, which needs none of the cross-thread accounting
a worker's does, this one being written, sorted and read in one thread.

pwt_manager_sort_order() decides whether a plan's sort is ours, and the gate and
the setup ask the same function. Every ORDER BY element has to name a column the
container holds -- a field of a scanned table that the query reads, and so ships.
An expression has no column there, and a sort that returns row ids, unpacks into
other fields or stops early is doing something for the plan beyond ordering; both
are left serial. A plan with an aggregation table is a different shape whose sort
AGGR_OP::end_send() already performs.

pwt_table_conds() had to change with it and the order of its three sources is
load-bearing. add_sorting_to_table() hands tab->select to the Filesort and nulls
tab->select_cond, so a sorted tab keeps its condition only in the filesort -- but
push_index_cond() ran long before and left that SQL_SELECT holding just the
remainder. pre_idx_push_select_cond therefore still wins; the filesort is asked
only when there was no pushdown. Taking them the other way round silently drops
the pushed half, which main.innodb_icp catches as rows the serial plan rejects.

main.parallel_query_sort reads named rows back with query_get_value, which takes
the Nth row as the client received it and so asserts the order rather than the
multiset, ascending and descending, and again once the container has spilled to
disk. main.parallel_query_trace used ORDER BY as its example of a declined query
shape and now uses LIMIT, which is still declined.

A capped SELECT must not run in the workers

With SET sql_select_limit=3, a query the gate accepted returned every row --
500 where the serial plan sends 3. The serial executor enforces a row cap in
end_send(), against unit->lim; the manager's drain sends every row a worker
ships and never consults it. So the gate must refuse any select whose unit
carries a cap, and it was testing the syntax instead of the cap:
limit_params.explicit_limit is only set by a LIMIT clause, while
sql_select_limit is installed by mysql_execute_command() as the default limit
of a top-level SELECT, with explicit_limit still unset.

The gate now tests join->unit->lim.is_unlimited(), which is the very value
end_send() would have enforced. This subsumes the explicit-LIMIT case, and it
is per-unit, so the session cap -- which applies only to the top-level select
-- does not cost a derived table's select its parallel scan.

Found reviewing the ORDER BY commit: its fs->limit != HA_ROWS_MAX check turned
out to be load-bearing for the same reason (the implicit cap becomes a filesort
limit), which raised the question of what protected the unsorted path. Nothing
did.

setup_worker_jointabs() builds each worker tab as a struct copy of the
manager's, and since the manager grew a sort stage the driving tab's copy has
carried the plan's Filesort pointer. Nothing reads it in a worker -- the
driving tab reads through the chunk reader, not join_init_read_record() -- and
worker tabs never run JOIN_TAB::cleanup(), so it is inert today. But
JOIN_TAB::cleanup() deletes the filesort and, through it, the SQL_SELECT
holding the plan's condition, so the copy was one future teardown call away
from a double free. The function's own policy is that manager-owned pointers
are cleared from the copy (select, cache_select, pre_idx_push_select_cond);
filesort and filesort_result now join the list, and pwt_assert_tab_inert()
pins the shape: a filesort is only ever the driving table's, and no sort has
run when workers are set up.

No test: the hazard is latent, reachable only by a teardown path that does not
exist yet. The asserts held over the whole main suite run with
parallel_worker_threads forced on.

This commit was prepared with Claude Code: it built the manager-side sort stage
on the existing container and layout machinery, and found the condition-order
defect above by running the whole main suite with parallel_worker_threads forced
on and separating the row differences from the EXPLAIN ones.

This commit was prepared with Claude Code: it probed the drain path with an
implicit sql_select_limit after noticing that only the sorted shape declined,
and confirmed the serial and parallel row counts diverged.

This commit was prepared with Claude Code, as a hardening it proposed while
reviewing the manager-side ORDER BY commit.
d6fe336c17d2...
d6fe336c17d2748a133d4bf3220d93614d4eaf8d
Rex Johnston
PQ:  allow workers to pre-aggregate SUM, COUNT, MIN, MAX for the manager
d64dab76b495...
d64dab76b4951041ae6bd79b5f8d29ab05b9328f
Rex Johnston
Parallel Query: Allow limit where it is the manager limiting (TPC-H Q3)

Claude code figured out it was safe, so lets see.
cfd9c22ff191...
cfd9c22ff19187f42678e2ceedfeb28e47db9ef2
Rex Johnston
PQ: changes to manage mtr output

1) undo cost discount -> revert mtr plan changes
2) worker thread env fixup -> wrong result during evalution of item
3) add --sorted-result in a few spots
4) add an include/not_parallel_execution.inc, mainly for
    main/innodb_mrr_cpk.test, so far.
5) add in missing parallel_query_index test
c4f4bee3df90...
c4f4bee3df9066912cedb0ead6823fa13636a68c
Sergei Petrunia
Cleanup in pwt_manager_base, pwt_worker_base.
be5c96191e66...
be5c96191e66f05852c86c68c9d795e9da51cfe4
Rex Johnston
PQ: allow secondary index scans
bbdeefd9814f...
bbdeefd9814fa53e3e0ca58217f801097733746f
Oleg Smirnov
PQ: add mtr --ignore-parallel-diff

Parallel query execution appends _parallel to the 'type' column of
tabular EXPLAIN/ANALYZE, so an existing test whose plan gets
parallelized fails on that one column alone:

  -17    DERIVED r1      ALL            NULL NULL NULL NULL 2
  +17    DERIVED r1      ALL_parallel    NULL NULL NULL NULL 2

Running the whole suite with --parallel-worker-threads > 0 then buries
the failures that are worth looking at.

Add --ignore-parallel-diff, which makes the result comparison ignore
that suffix: once the two files are known to differ, strip the
suffix from both and compare again.

The text has to be exactly one of ALL, range or index followed
by _parallel, and it has to fill the span between two field
delimiters ('\t', '\n', or end of file).
FORMAT=JSON does not match this criteria, so is not affected.
b99913fa6859...
b99913fa685948e48c48826f10eab54c902bb822
Sergei Petrunia
Make sql_parallel_thread module isolated.

This required adding
virtual pwt_worker_base::on_fatal_error()

which pwt_worker overrides.

pwt_manager_base doesn't track if fatal_error has occurred
Should it? fatal_error is in sql_parallel_workers...
b30d53b04e54...
b30d53b04e54832993767cb1badc060845d9c5c5
Sergei Petrunia
Move all thread creation/cleanup into sql_parallel_thread.*

The cleanup API is not yet very pretty.
b2b01eb98d22...
b2b01eb98d2247f11377b4b1349f919cbf26a727
Rex Johnston
Parallel Query: the worker's JOIN needs its unit

main.parallel_query_join and main.parallel_query_worker_side crashed in a
worker:

  Select_limit_counters::get_select_limit (this=0xa5a5a5a5a5a5ad85)
  join_init_read_record (sql/sql_select.cc:25937)
  sub_select
  pwt_worker::execute_and_handoff

0xa5a5.. is TRASH_ALLOC, so this is memory that was allocated and never
written, not memory that was freed.

setup_worker_join() builds the worker's JOIN by hand, because a worker joins
its own copies of the tables over its own chunk, and it sets the fields the
executor was known to read: the table counts, const_tables, select_lex, and
later join_tab and sum_funcs. join->unit was not among them, and until the
rebase onto current main nothing on the worker's path read it.
join_init_read_record() now does, asking the statement's row limit before it
opens a scan:

  init_table_full_scan_if_needed(tab->table,
                                tab->select ? tab->select->cond : NULL,
                                join->unit->lim.get_select_limit());

so the worker reached it through whatever the mem_root happened to hold.

It is given the manager's own unit rather than a copy. The unit is read and
never written here, and the manager holds it until every worker has been
joined; what it says is also what a worker needs to hear, because the gate
refuses a select that carries a row limit at all.

This commit was prepared with Claude Code: it read the poison pattern as
uninitialised rather than freed, which pointed at the JOIN the workers build
themselves rather than at a lifetime problem, and compared what that function
sets against what the executor now reads.
890c2cc60869...
890c2cc6086989daa40b7c89b3a4d48d8e7e0596
Sergei Petrunia
WIP: Separate a portion of pwt_manager/pwt_worker functionality.

into sql_parrallel_thread.* (name tentative).
No "unit test" like test yet.
Unfinished things:
pwt_manager::notify_fatal_error() should both update a flag
in pwt_manager_base and notify pwt_manager..
pwt_manager_base::{reaped, kill_signal} are not isolated yet.
85fdc7a9fba1...
85fdc7a9fba126b557bfd2e9ddc0b68732d266c2
Sergei Petrunia
WIP: Move "worker" part to pwt_manager_base2/pwt_worker_base2

"bool stop" and "reaped" are not moved yet.
83c6f03bf5cb...
83c6f03bf5cbfd001e03cdffc3dedd1cbc7a3af5
Rex Johnston
Parallel Query: AVG, and an aggregate inside an expression

TPC-H Q1 now runs in the workers. Two things were stopping it, and both are
here.

AVG
---
An average does not compose out of averages, so what a worker has for a group
is two numbers: the sum of its rows and how many there were. Nothing had to be
invented to carry them. Item_sum_avg::create_tmp_field() already packs a sum
and a count into one field, because that is how the serial grouped plan
accumulates an average, and a worker's grouped container was already
maintaining both.

What was missing was the merge. The manager primes its own aggregate with a
worker's partial through direct_add(), and Item_sum_sum::direct_add() carries
only the sum -- Item_sum_avg::count was left untouched, so every partial
counted as one row. direct_add() now takes the count alongside the sum, and
reset_field()/update_field() add it to the field's count instead of the 1 an
ordinary row contributes, in the same shape Item_sum_sum's already had.

Nothing changes for a serial plan: direct_added is false there, and the new
branches are not taken.

An aggregate inside an expression
---------------------------------
SUM(x)+1 was refused, and so was SUM(x)/COUNT(x) -- the obvious way to write an
average by hand. A select list item that is not itself an aggregate is
evaluated once per group from whatever row the terminal is holding, so a field
in it that GROUP BY does not name takes an arbitrary value; that is worth
refusing. But the check walked into the aggregates as well, found x under
SUM(x), and refused a query that was never in doubt.

It now steps over Item_sum subtrees, using with_sum_func() to answer an
aggregate-free subtree wholesale with the walk that was there before, and
decomposing only the node kinds that can hold an aggregate. Anything else
carrying one is refused rather than guessed at: being wrong in the permissive
direction here is wrong results. SUM(x)+x is still correctly refused, because
the bare x is walked and is not a group column.

The shipped row's shape
-----------------------
With more than one AVG the second onward came back wrong, which was not
arithmetic. Item_sum_avg is the one aggregate whose create_tmp_field() differs
between a keyed and an unkeyed temporary table -- it packs the count beside the
sum only for the keyed form -- and the two ends of the transport were built
differently:

  recv, group_container  make_container(..., group)  16-byte {sum, count}
  exec.result (shipped)  make_container(...)          plain 8-byte double

flush_groups() copies a group container record into the shipping container by
reclength, so it wrote past a narrower record and the manager read back a wider
one, drifting by one AVG's worth of bytes at a time. For SUM, COUNT, MIN and
MAX the two widths are identical, which is why this sat there unnoticed. The
shipping container is now built with the same key; a worker ships one row per
group, so there is nothing for the key to collapse.

main.parallel_query_aggregate gains an AVG section -- serial against parallel
over a DECIMAL average, and three AVGs at once, which is the case that exposed
the shape bug -- and AVG leaves its "what is not accepted" list. STDDEV,
BIT_XOR and COUNT(DISTINCT) stay there.

This commit was prepared with Claude Code: it bisected TPC-H Q1 down to the two
refusals, found the container shape mismatch from the pattern that only the
last AVG was wrong, and fixed a null Item passed to VDec that the aggregate
test caught as a crash.
77053622d9e5...
77053622d9e5dcfeeab3a6217b246e4611905d2d
Sergei Petrunia
Continue pwt_manager_base cleanup: introduce locked__process_manager_wakeup().

Note this one:
  //  TODO: the following was done when not holding LOCK_data. Does it
  // matter?
753f5103b4d4...
753f5103b4d455e5997e252567d8fe278ca507bd
Rex Johnston
PQ introduce transport API, batch transport, tmp table transport
744e63b170ad...
744e63b170ad82a39da9fcb22eb3691ba3f55e77
Sergei Petrunia
Make pwt_manager::workers a Dynamic_array<ptw_worker*>
6fee10db458a...
6fee10db458aa2778b978ef3ad71a80fe0948204
Sergei Petrunia
For &mgr->COND_data_avail, use mysql_cond_signal, not mysql_cond_broadcast.
65413b62a3a8...
65413b62a3a84bbf5061e6f3d9d4307170d30d88
Rex Johnston
Parallel Query: give the worker back optimizer_replay_context too

main.parallel_query_clone aborted in safemalloc while a worker's THD was being
destroyed:

  free_memory (mysys/safemalloc.c:275)  DBUG_ASSERT(irem->marker == MAGICSTART)
  my_free
  plugin_thdvar_cleanup (sql/sql_plugin.cc:3429)
  THD::free_connection -> ~THD -> destroy_background_thd
  pwt_thread::init_and_run_thread_func

A cleared marker is a second free of something already freed.

A worker THD takes the session's variables wholesale, so that an item
evaluated in a worker sees the session's time zone and the rest, and then puts
back the members a THD allocates and frees for itself. That list has to be
plugin_thdvar_cleanup()'s list, and it had fallen one behind it: the rebase
onto current main added

  my_free(thd->variables.optimizer_replay_context);

alongside the redirect_url and default_master_connection frees. It is a
SESSION_ONLY Sys_var_charptr, so it is allocated per THD -- but it was not
being restored, so every worker carried away the manager's pointer, the first
worker to exit freed it, and the next free of the same pointer found the marker
gone.

Restored with the others, and the invariant that was only implied is now
written down next to the list: whatever plugin_thdvar_cleanup() frees, this has
to give back. The failure is a long way from either end, so the next member
added there will not announce itself.

This commit was prepared with Claude Code: it read the abort as a double free
rather than a cross-thread free, and diffed what plugin_thdvar_cleanup() frees
against what the worker restores to find the member the rebase had added.
57704dab25df...
57704dab25df6edd2592185097c930f2204f76b3
Sergei Petrunia
Rename pwt_worker_base2->pwt_worker_base, pwt_manager_base2->pwt_manager_base
51e6c7707211...
51e6c7707211e7aa66f451b904c24331a428d75a
Rex Johnston
Parallel Query: count a worker in under the lock that counts it out

A query could hang for good, with the manager waiting on COND_data_avail for a
worker that had already finished and gone.

active_workers is decremented in report_worker_final_state() under LOCK_data,
and read in locked__process_manager_wakeup() with LOCK_data held, but
register_worker() was incrementing it under no lock at all. The two do run at
the same time: init_parallel_workers() builds the team one worker at a time and
starts each thread as it creates it, so worker 1 can be counting itself out
while worker 2 is being counted in.

  manager thread                      worker 1
  ------------------------------      ---------------------------------
  new pwt_worker  -> reads 2
                                      thread_func_end()
                                        LOCK_data, reads 2, writes 1
  writes 3

The decrement is lost and the count stands one too high. From then on
claim_next_result() waits for a worker that will never publish a result and
never signal again, because every worker there was has already exited.

Taking LOCK_data around the increment is the whole fix.

It wants a small table and enough load: a table the engine divides into one
chunk leaves the other workers with nothing to scan, so they reach
thread_func_end() almost at once, in the window while the rest of the team is
still being built. main.function_defaults_innodb is such a test, and this
reproduced it within six attempts:

  ./mtr --parallel=16 --mem --force --ps \
        --mysqld=--parallel-worker-threads=4 \
        main.function_defaults_innodb main.function_defaults_innodb \
        main.function_defaults_innodb main.function_defaults_innodb \
        main.function_defaults_innodb main.function_defaults_innodb

It now runs clean twelve times over. No test is added: a lost update needs a
race to be lost, and a test that only sometimes fails is worse than none.

This commit was prepared with Claude Code: it reproduced the hang, read the
manager's bookkeeping out of the stopped server with gdb -- active_workers 1
against four workers, all four with a destroyed THD and the statistics that
only thread_func_end() writes, so all four had decremented -- and reasoned back
from four decrements and a count of one to a lost increment.
4bbf7f6171f4...
4bbf7f6171f41903a43544917ad3b11655e08b5d
Rex Johnston
PQ WIP, make the engine declining to execute in parallel are rare thing

... by extending our checks to catch
locking read, ROW_FORMAT=REDUNDANT, a descending clustered key,
and a discarded tablespace during optimize/make_join_info.
Add optimizer trace bits and peices.
4632c67ae260...
4632c67ae26025950c305f8557d79c649ffd33bd
Sergei Petrunia
Rename pwt_manager_base->pwt_thread_manager, pwt_worker_base->pwt_thread
41348c305354...
41348c3053544d624bc6c178c9ba717e46e1197f
Sergei Petrunia
Make pwt_worker ctor accept pwt_manager *manager_arg argument
410241739d4a...
410241739d4a23fae4ecf52a5407162315bbbc79
Sergei Petrunia
Small cleanups: set thd->userstat_running in pwt_worker_base, comments.
3e5adedda2bc...
3e5adedda2bc93779ee7adaa41f9fdb2825c95ea
Rex Johnston
Parallel Query: the gate says why it turned a query away

"Why did this not run in parallel" was a question the reader had to answer
themselves, from a plan, in a debugger, some way from the query that provoked
it. It is now a question the server answers:

  SELECT JSON_EXTRACT(trace, '$**.parallel_scan_declined_because')
    FROM information_schema.optimizer_trace;

  ["grouped: the aggregation table is keyed by a unique constraint rather than
    by the group -- create_tmp_table() does that for a GROUP BY element too
    wide for a key part, and a hash of the row is not something a group can be
    looked up in",
  "the plan needs a temporary table and the workers cannot pre-aggregate into
    it"]

The first entry is the check that objected; the rest is that refusal working
its way back up. Every one of the gate's refusals now goes through
pwt_decline(), which keeps the DBUG_PRINT and adds the reason to the trace.
Sixteen of them had neither before -- a bare "return false" -- and those were
in the two functions a grouped query spends most of its time in.

Three things this needed beyond saying the words.

The gate is asked twice. Once from make_join_readinfo() while the plan is still
being built, and again from parallel_join_check() when it is finished. The
early call runs before make_aggr_tables_info() has built the aggregation table,
so it reported "the terminal is not end_update()" for every grouped query --
an answer about a plan that no longer exists by the time the query runs, and a
worse place to start than no answer at all. A trace flag threaded through the
gate means only the decision that stands speaks. Threaded rather than kept in a
static, because several connections optimize at once.

Which check fires first decides what you are told. Two reasons were being lost
to a coarser test reached earlier: ordered_index_usage now precedes the
access-method test, since "an index supplies the ORDER BY order" says more than
"this is not a scan the engine divides", and both refuse the same plans. And
the duplicated access-method test in table_can_be_parallel_scanned() is gone,
so is_parallel_scan_applicable() owns that question alone and can tell its
several answers apart.

A reason has to name the cause, not the symptom. A GROUP BY on a column too
wide for a key part is refused because create_tmp_table() keys the container by
a unique constraint, which makes the terminal end_unique_update(), which is why
pwt_plan_group_key() finds nothing -- and "the terminal is not end_update()" is
a true statement that sends the reader nowhere. The check that would have named
the width is never even reached. So when there is no group key, this asks
make_aggr_tables_info()'s own condition of the same table it asked it of, and
says which of the two it is.

main.parallel_query_why pins one refusal per shape that can reach it, so a
reason cannot quietly go missing or attach itself to the wrong query.

This commit was prepared with Claude Code, after a report that finding why a
wide GROUP BY column would not run in parallel took an afternoon in gdb. It
counted the refusal points first -- 22 of 38 traced, 16 silent, the one being
hunted among the silent -- and worked from there.
2ef40b48c129...
2ef40b48c1292587711e4ad4a464d199417c2c93
Sergei Petrunia
Make sql_parallel_thread self-contained

- Move 'reaped' and 'kill_signal' OUT to sql_parallel_workers
- Move 'workers' and 'nworkers' OUT to sql_parallel_workers
- Move server_threads update logic IN.
2b89d66f4cc0...
2b89d66f4cc027582d1060af6950a3a8ac9a1fe6
Sergei Petrunia
pwt_worker: cleanup the cleanup code.
291f7b7d1117...
291f7b7d111714a0f4c5069d47bd3d7fdf92fe42
Rex Johnston
PQ  remove accumulate_group(), use end_update() from sql_select.cc

Now we actually *have* to solve the aria table conversion, not
sidestep it.
1f989afce429...
1f989afce429070b5561c5871a05a58d1aeea02c
Rex Johnston
PQ: tidy up and many constraint updates
18381a4bc659...
18381a4bc65955a5b30a0ac5bb90e1271c2d9fcb
Rex Johnston
PQ  optional, TODO: simpler cleanup

... not actually simpler, but tidier.
0f0d7b24c1c7...
0f0d7b24c1c7f574334b38f4f88906953e54bc44
Rex Johnston
PQ  tidy up message queue processing, add in missing tests, fix...

in pwt_worker_base::init_worker_thd()
0d92c08ade4c...
0d92c08ade4cb9877a68226c7185e91cae4c1007
Sergei Petrunia
Add comments.
03d0068ea4af...
03d0068ea4afe3708dfcc93b194c8dc259824a69
Rex Johnston
MDEV-39492 PQ: abort_worker() could awake a THD the worker had destroyed

Separating pwt_worker_base out of pwt_worker dropped the assignment that
nulls the worker's thd on its way out. The exit path used to capture the
pointer and clear the member while holding LOCK_worker, then tear the THD
down from the local copy; after the split it kept a local for the detach but
left the member pointing at the THD it went on to destroy.

The hole survived because nothing reached abort_worker(). Its only caller is
the cleanup_old_workers path, which needs init_parallel_workers() to fail
after at least one worker thread is already running, and no test could
produce that. Add a DBUG injection that fails the last worker as if its
thread could not be created: the last one so that the maximum number are
running when the teardown aborts them, and before create_thread() rather
than after, because cleanup_thread_create closes that worker's tables and
destroys its THD, which is only safe while it has no thread of its own.

This commit was prepared with Claude Code: it found the dropped assignment
while evaluating the TODO tags added by the three preceding commits, traced
which readers depended on it, wrote the injection point and the test, and
confirmed by reverting the fix that the test reproduces the crash.
BuildBot (0.8.9) working for the MariaDB project.
Page built: Mon 07 Sep 2026 02:35:01 (UTC)