Console View
|
Categories: connectors experimental galera main |
|
| connectors | experimental | galera | main | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
send ssl_info_client to prove the authentication was not relayed A MitM can relay the authentication, let the client fail on ssl_info and keep using the authenticated connection. * MARIADB_CLIENT_SSL_INFO_CLIENT: set it whenever TLS is used * after the authentication OK packet send ssl_info_client, the server does not reply (but might disconnect in case of active MitM). |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42009: Set GLOBAL innodb_log_file_size=ON is not crash safe log_t::header_rewrite(): Before setting log_sys.archive=true, ensure that any garbage entries at the end of the file will be zeroed out so that they cannot be mistaken for newer records. On Linux and Microsoft Windows we will attempt a shortcut for writing zeroes, before falling back to log_file_t::write(). Reviewed by: Alessandro Vetere |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41251 Wrong result with split materialized query A query like SELECT * FROM t0 WHERE t0.b IN (SELECT t.n2 FROM t1, (SELECT n1, n2 FROM t1 GROUP BY n1, n2) t WHERE t.n1 = t1.n1 AND t1.c1 = 'a' UNION SELECT -1) returned only the row of t0 that was read first, when the derived table t was split-materialized. The IN->EXISTS transformation wraps t0.b in an Item_cache owned by the Item_in_optimizer, and injects t.n2 = <cache>(t0.b) into the subquery. The optimizer then pushes an equality on n2 into the derived table, so that the table only has to be filled with the matching rows. TABLE::add_splitting_info_for_key_field() built the right side of that equality as a deep copy of <cache>(t0.b). A copy of a cache is a second cache: it read t0.b for the first row of t0 and kept that value, because only the Item_in_optimizer stores new values, and it stores them into the original cache. Every later refill of the derived table was filtered on the first row's value. The right side is an expression of the outer select used inside the select that specifies the derived table. Add a copy hierarchy for that purpose. Item::new_select_copy_with_checks(thd, item_select) returns a copy for use in a select nested in item_select, the select the item belongs to. In the nested select the copy is an outer reference to item_select: - An Item_field is copied and its depended_from set to item_select. This is what add_splitting_info_for_key_field() did before with set_fields_as_dependent_processor, except that depended_from now names the select the column belongs to instead of the nested select. - An Item_direct_view_ref, a column of a merged view or derived table, is also copied like a column: the copy refers to a new_select_copy() of the item the view column stands for. A constant view column keeps its constant underlying item, which keeps a column of a constant NULL-complemented table constant in the pushed equality (the MDEV-30081 test in main.derived_cond_pushdown still shows ref "const" for its LATERAL DERIVED table). If the copy still depends on a table of item_select, which happens for a constant column of a view on the NULL-complemented side of an outer join, since used_tables() then names that table, the copy gets its depended_from set as well. - Any other item that depends on tables of item_select is not copied. The copy is an Item_direct_ref to the original item, with depended_from set to item_select. So the nested select reads the cache, aggregate or subquery that its owner keeps up to date, instead of a copy of it that nobody refreshes. Item_default_value, Item_insert_value and Item_trigger_field are treated this way too, as they are not plain columns. - An item that depends on no table of item_select is shared. This includes items whose deep_copy() returns nullptr, such as stored routine variables, which therefore no longer prevent splitting. The copy is fixed, and only refers to tables outside the select it is used in, which the wrapper asserts. add_splitting_info_for_key_field() now uses it for the right side of the pushed equality, and no longer walks the copy with set_fields_as_dependent_processor or calls update_used_tables() on it. A second test, taken from Dave Gosselin's MDEV-41251 commit, pushes a comparison, dt.a <=> (t1.b = 1), and its row form. The deep copy of the comparison was never fixed again. Its Arg_comparator still pointed at the original, which sets null_value of the original, and a copied row comparison had comparators set to NULL. The row of t1 with a NULL b was missing from the result, and the row form crashed in Arg_comparator::compare_row(). Now the comparison is not copied: the right side refers to the original. A third test pushes dt.a <=> v.one where v is a merged view with the constant column one, on the NULL-complemented side of an outer join. It covers the view column copy that depends on a table only through its NULL-complemented table. This commit was prepared with Claude Code (Opus 5), which designed and wrote the new_select_copy() hierarchy, switched the split equality to it, wrote the three-row test from the earlier rr analysis, added the comparison test, confirmed in a build of the parent commit that both tests fail without the change, and ran main and the SQL-layer suites in normal and --ps-protocol mode. Claude Code (Fable 5.1) then reviewed the commit, found and reproduced the NULL-complemented view column case, wrote its fix and test, and reran the derived and SQL-layer tests. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alexander Barkov
bar@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41527 MEDIUMTEXT gets truncated to TINYTEXT in assoc array RECORD field Adding 13.1 specific tests with package-wide data types. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Merge 13.1 into main | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39492 PQ: leave a condition the workers cannot evaluate to the manager A conjunct of a table's condition that no worker can evaluate kept the whole query serial. The common one is a correlated subquery: its JOIN belongs to the manager's thread, so there is no clone of it to hand a worker, and TPC-H Q21's NOT EXISTS over LINEITEM was enough to refuse a plan whose driving table the optimizer had already costed as a parallel scan. The gate now divides each table's condition instead of refusing it (pwt_split_table_cond). The conjuncts a worker can evaluate stay with the table, as the worker's select_cond, and the rest are ANDed into JOIN::parallel_residual_cond for the manager to apply to every row it drains, after copy_back_row() has put the shipped columns back into the manager's own records, where the residual's items already look. A correlated subquery among them runs on the manager's thread through the manager's own handlers exactly as it would have run serially. A conjunct of an inner join may be applied anywhere after its tables have been read, so the answer is the same; what changes is the number of rows it is asked of, which is the finished join's rather than this table's. For the residual to be answerable from a drained row, every field it reads from a join table has to be a shipped column, which pwt_row_layout::build() takes from the read_set. The split checks that, walking into subqueries for their outer references, and refuses the plan if a conjunct can be evaluated on neither side. The split is taken by the gate and taken again by init_parallel_workers() from the plan as it stands when the workers are built (pwt_split_plan_conditions), because it names the plan's own items. A plan with a residual ships joined rows rather than pre-aggregating: the filter has to run before any grouping, so pwt_row_layout::build() takes a ship_rows argument that keeps 'grouped' off while 'plan_aggregates' stays on, which is the fall-back shape the drain already handles, with the plan's own terminal grouping what the manager lets through. The non-aggregating drain applies the residual before sending a row, and before collecting it when the sort is the manager's. The optimizer trace records the manager's share under parallel_scan_manager_evaluates. On TPC-H SF1 Q21 the serial plan evaluates the NOT EXISTS once per late LINEITEM row of a finished order, 1.8M times, and the manager once per row of the finished join, 77k times, since the NATION filter has run by then. With a FirstMatch semijoin admitted to the workers in the next commit, Q21 goes from 2.9 s serially to 0.85 s with four workers and 0.68 s with eight, with the same result. main.parallel_query_residual pins the four shapes: whole-table aggregation, GROUP BY, a plain select, and a select whose sort the manager performs, each with a correlated subquery in the WHERE clause, each run in the workers and each giving the serial answer. This commit was prepared with Claude Code (Fable 5.1), which diagnosed the refusal from the optimizer trace, measured that applying the subqueries after the join was cheaper even serially, designed and wrote the split, the layout change and the drain-side filter, and wrote the test. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39492 PQ: run a FirstMatch semijoin in the workers The gate refused every semijoin duplicate-elimination strategy, on the grounds that a worker's nested loop implements none of them. For FirstMatch that was not true. FirstMatch is the nested loop backing out early: when the last inner table of the semijoin nest matches, the executor sets join->return_tab to the outer table the nest follows (JOIN_TAB::do_firstmatch) and sub_select() unwinds to it rather than looking for further matches that would only duplicate the row. A worker runs sub_select() over its own JOIN and its own JOIN_TABs, so it does the same -- provided its copy of do_firstmatch names a tab of its own array, since return_tab is only ever compared with the tabs of the array being walked. setup_worker_jointabs() now remaps do_firstmatch, first_sj_inner_tab and last_sj_inner_tab from the manager's array onto the worker's, and the gate admits SJ_OPT_FIRST_MATCH, refusing a nest whose return tab is a const table, which no worker holds. LooseScan and DuplicateWeedout keep state that no copy carries -- a match flag, a weed-out table -- and stay refused, as does materialization. This is the second half of what TPC-H Q21 needed: its EXISTS over LINEITEM is flattened to a FirstMatch nest after NATION, and with its NOT EXISTS left to the manager by the previous commit, the plan the optimizer had costed as a parallel scan of ORDERS now runs in the workers. main.parallel_query_firstmatch runs an EXISTS flattened to FirstMatch under whole-table aggregation, GROUP BY and a manager-side sort, with two matching inner rows per outer row so that a nested loop without the early return would double every count, and shows DuplicateWeedout still refused. This commit was prepared with Claude Code (Fable 5.1), which read what sub_select() and evaluate_join_record() consult for FirstMatch, wrote the remap and the gate change, and wrote the test. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Monty
monty@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Removed warning from cmake - DEPENDS cannot be used with ADD_CUSTOM_COMMAND. Removed it |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42087 Memory leak in sql_service when a statement fails after it sent its metadata When a statement that runs through mysql_real_connect_local() fails after it sent its metadata, the error is stored in the dataset of the result set. embedded_get_error() freed only the MYSQL_DATA, and not its MEM_ROOT, so 8192 bytes were lost, for example for a statement that is aborted by KILL QUERY. Free the dataset with free_rows(). Assisted-by: Claude:claude-sonnet-5-5 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42009: Set GLOBAL innodb_log_file_size=ON is not crash safe log_t::header_rewrite(): Before setting log_sys.archive=true, ensure that any garbage entries at the end of the file will be zeroed out so that they cannot be mistaken for newer records. On Linux and Microsoft Windows we will attempt a shortcut for writing zeroes, before falling back to log_file_t::write(). Reviewed by: Alessandro Vetere |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40530 Bind the client authentication response to the server certificate MDEV-31855 introduced ssl_info that let the client verify that it uses the correct certificate. But MitM could hijack the connection after the authentication - the client will detect it and disconnect but the server won't and will continue talking to the attacker. Let's introduce ssl_info_client packet that enables the server to verify that the client uses the correct certificate. When require_secure_transport=ON and the connection isn't local: * the server sets MARIADB_CLIENT_SSL_INFO_CLIENT in the initial handshake * the client sets it in the reply, if it understands it * when both sides have it, the client sends ssl_info_client right after the OK packet, acl_authenticate() reads it before the session is used (init_connect, audit, etc) * a connection is rejected unless ssl_info_client is valid, either for the server certificate fingerprint or for "CA" (the client verified the certificate itself) * an account without a password hash (plugin has no hash_password() or empty password) is rejected unless the account REQUIRE a client certificate Assisted-by: Claude:claude-opus-5-5 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38274: Unexpectedly large pages_read_count in main.analyze_stmt_prefetch_count The testcase checked the values of pages_prefetch_read_count and pages_read_count that InnoDB reported for the query. The numbers depend on how fast the prefetch thread is in comparison with the query thread. It can also happen that we request to prefetch a page but then still have to wait for it to be read, which is counted in page_read_count. Replace the checks in the file with ones that will hold in all circumstances. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42009: Set GLOBAL innodb_log_file_size=ON is not crash safe log_t::header_rewrite(): Before setting log_sys.archive=true, ensure that any garbage entries at the end of the file will be zeroed out so that they cannot be mistaken for newer records. On Linux and Microsoft Windows we will attempt a shortcut for writing zeroes, before falling back to log_file_t::write(). Reviewed by: Alessandro Vetere |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
fixup! MDEV-42087 Memory leak in sql_service when a statement fails after it sent its metadata SQL keywords in upper case in test_sql_service_abort Assisted-by: Claude:claude-sonnet-5-5 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39492 PQ: cost the parallel scan in the join-order search Until now the join order was chosen by the serial optimizer and the driving table it settled on was then handed to the workers whenever the gate admitted the query. On TPC-H that split REGION in Q5 and NATION in Q11: five- and twenty-five-row tables of which one row survives the WHERE, so the whole join under them ran on one worker and the query paid the setup of the workers for nothing. The plan that would actually divide -- ORDERS first in Q5, split four ways with the lookups below it spread evenly -- costs 1917 against the chosen plan's 873 when read serially, and the serial optimizer had no way to know that read by four workers it costs about 490. best_access_path() now prices the driving table's scan both ways and keeps the cheaper (parallel_scan_cost): serially, or as one worker's share of the scan plus optimizer_parallel_setup_cost per table in the join, with the rows surviving the table's own WHERE spread over min(N, rows_out) workers. The join-order search divides every later position's cost by that divisor (position_cost_in_plan), since each worker joins only its share of the driving rows and the plan's cost is one worker's. A large table can therefore win the driving position it loses serially, while a tiny driver gains nothing from the division and is charged the setup, so it stays serial. parallel_join_check() runs the workers only for a plan chosen that way. Three things are needed for the search to find such a plan honestly. The pruning heuristic drops a prefix that is worse on both rows and cost than the best at its depth, which is always true of a big-table prefix next to a five-row one; prefixes under a parallel driver are now compared only with each other. The tables under a parallel driver are costed as a worker runs them, a plain nested loop with no join buffer, hash join or rowid filter, and no covering-index or rowid-filtered scan is offered as a driver, because the gate refuses all of these and a plan chosen on their cost would run serially. And the refusals the gate can make without a plan -- limits, DISTINCT, HAVING, outer joins, window functions, aggregates whose partials do not merge, a GROUP BY on an expression, a select-list column GROUP BY does not name, an item a worker cannot evaluate -- are asked before the search as well (pwt_query_shape_allows_workers), so that a query the workers will refuse is costed as the serial plan it will run. Q7's GROUP BY on extract(year ...) is the case that showed why: chosen for LINEITEM first on the discount, refused after the plan, and run serially in that shape at five times its serial time. The setup cost is a session variable, optimizer_parallel_setup_cost, in microseconds, default 2000 per table. It was measured on this branch at about 2 ms for a single-table scan and about 8 ms for a three-table join, dominated by creating the worker threads with a THD each, opening every table once per worker, and building and draining the result containers. The parallel-query tests set it to 0, since their tables are far too small to repay it, and main.parallel_query_costing pins the three behaviours on a five-row dimension and a 20000-row fact table. Measured on TPC-H SF1 with histograms, four workers against serial: Q5 490 -> 164 ms, Q3 623 -> 188 ms, Q10 341 -> 139 ms, Q12 615 -> 191 ms, Q15 1225 -> 396 ms, Q1 3966 -> 1160 ms; Q11 and Q7 unchanged. This commit was prepared with Claude Code (Fable 5.1), which diagnosed the REGION/NATION splits and the Q7 regression on a scratch copy of the TPC-H data, implemented the costing, the two-family pruning, the worker-shaped costing of the parallel chain and the pre-plan gate, wrote the costing test, updated the existing tests, and ran the parallel-query tests in both protocols and the main suite. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38274: Unexpectedly large pages_read_count in main.analyze_stmt_prefetch_count The testcase checked the values of pages_prefetch_read_count and pages_read_count that InnoDB reported for the query. The numbers depend on how fast the prefetch thread is in comparison with the query thread. It can also happen that we request to prefetch a page but then still have to wait for it to be read, which is counted in pages_read_count. Replace the checks in the file with ones that will hold in all circumstances. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42009: Set GLOBAL innodb_log_file_size=ON is not crash safe log_t::header_rewrite(): Before setting log_sys.archive=true, ensure that any garbage entries at the end of the file will be zeroed out so that they cannot be mistaken for newer records. On Linux and Microsoft Windows we will attempt a shortcut for writing zeroes, before falling back to log_file_t::write(). log_t::set_archive(): Block concurrent writes to the log on every platform, because log_t::header_rewrite() now requires exclusive access to the log file. Reviewed by: Alessandro Vetere |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40012 PQ: merge AVG, MIN and MAX partials after the manager's table spills The manager merges a worker's partial for a group through the plan's own terminal. direct_add() hands the partial to each aggregate, and end_update() either opens the group with reset_field() or folds the partial into it with update_field(). Once the aggregation table spills from HEAP to disk, the terminal becomes end_unique_update(), which calls reset_field() for every row, before the write shows whether the group is new. For a group already in the table it then restores the stored row and calls update_field() with the same partial. reset_field() consumes the partial, so update_field() has to find it again. Item_sum_sum and Item_sum_count keep their direct values for that case and mark it with direct_reseted_field. Two aggregates lacked that: - Item_sum_avg::reset_field() zeroed direct_count, so update_field() added the partial's sum with a count of zero, and the average of such a group came out too high. - Item_sum_min_max::reset_field() cleared direct_added and had no direct_reseted_field, so update_field() compared the stored value with args[0], the manager's own argument over a base-table row the manager never fills, instead of with the partial. Item_sum_avg::reset_field() now leaves direct_count for update_field(), which already clears it. Item_sum_min_max gains direct_reseted_field, set by reset_field() and read and cleared by update_field(). A directly added item also bypasses the FILTER test in update_field(), as in Item_sum_sum, since the partial already reflects the filter. A serial plan never sets direct_added, so it never reaches the new branches. The grouped case already in main.parallel_query_spill has one row per group, so each group gets a single partial and the branch for a group already in the table never ran. The new case groups 20000 rows into 3000 groups, so each worker sends a partial for nearly every group, and the 64K heap limit makes the manager's table spill. Without either of the two changes, its parallel answer differs from the serial one. This commit was prepared with Claude Code (Opus 5.5), which found the wrong averages while letting the workers pre-aggregate on a wide GROUP BY column, traced them and the wrong MIN and MAX values to the reset_field() and update_field() pair under end_unique_update(), and showed with the new test that the AVG change and the MIN/MAX change each fix a failure of their own. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40012 PQ: let the workers pre-aggregate on a GROUP BY column too wide for an on-disk key The gate refused a grouped plan with a GROUP BY column too wide to be a key part, or with a key too long in total, on the grounds that create_tmp_table() would key the aggregation table by a unique constraint, a hash of the row with nothing to look a group up in. Since MDEV-38975 that holds only for a table that starts on disk: a HEAP table indexes a wide or blob key part, and Create_tmp_table::start() skips the fallback when it expects HEAP. Such a plan writes through end_update(), so pwt_plan_group_key() finds its key, and the gate then refused it on the per-column prediction alone. Both predictions are now out of the gate. A plan whose aggregation table starts on disk still has a unique constraint and no end_update() terminal, and the gate still refuses it, with the reason it already gave for that shape. Otherwise the containers are built the way the plan's own table was, and pwt_row_layout::build() checks the finished container for a group index, with plain rows shipped in place of partials if it has none. A worker container with a wide key fills sooner, and a worker that rebuilds it on disk inside end_update() exposed a fault in how memory moves between threads. The container's own MEM_ROOT was thread-specific, so whatever a worker allocated in it was charged to the worker and freed by the manager. pwt_end_send() moved the charge by bracketing put_record() and handing the whole change in the worker's local_memory_used to the manager. But that change also holds memory that stays with the worker: ha_maria::enable_indexes() allocates its HA_CHECK from thd->mem_root, the worker's own root, which ~THD frees later on the worker's thread. With that block counted against the manager, the worker's counter went negative in ~THD: Assertion `(longlong) thd->status_var.local_memory_used >= 0 || !debug_assert_on_not_freed_memory' failed. THD::~THD destroy_background_thd pwt_thread::init_and_run_thread_func A table one thread writes and another frees is not thread-specific memory, which is what HA_OPEN_GLOBAL_TMP_TABLE already says to its engines. Create_tmp_table::start() now gives a cross-thread table a root without MY_THREAD_SPECIFIC, and with nothing left to hand over, the hand-over code is removed: the bracket in pwt_end_send(), the same accounting in pwt_tmp_table_sink::emit_row() and cleanup(), and account_spilled_memory(). Tests: - main.parallel_query_why now asks for the on-disk shape (tmp_memory_table_size=0), which keeps the unique-constraint reason. At the default size the same query runs in the workers. - main.parallel_query_aggregate shows a VARCHAR(4000) GROUP BY that stays in HEAP and pre-aggregates in the workers, with fewer than 1000 tmp-table writes for 20000 rows and the serial answer. Its note on the no-group-index fallback no longer claims a gate refusal for wide keys. - main.parallel_query_spill shows the same kind of key at a 16M heap limit, where the workers rebuild their containers on disk part way through. Without the memory change it fails on the assertion above. A comment that still named accumulate_group(), long since replaced by end_update(), is corrected too. This commit was prepared with Claude Code (Opus 5.5), which traced the changed refusal reason to MDEV-38975, removed the two predictions, found the negative memory counter with the new spill case and located the HA_CHECK allocation with an rr recording, moved the container's root off thread-specific memory, and showed by reverting the memory change on its own that the new spill case fails on the assertion without it. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38274: Unexpectedly large pages_read_count in main.analyze_stmt_prefetch_count The testcase checked the values of pages_prefetch_read_count and pages_read_count that InnoDB reported for the query. The numbers depend on how fast the prefetch thread is in comparison with the query thread. It can also happen that we request to prefetch a page but then still have to wait for it to be read, which is counted in page_read_count. Replace the checks in the file with ones that will hold in all circumstances. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38274: Unexpectedly large pages_read_count in main.analyze_stmt_prefetch_count The testcase checked the values of pages_prefetch_read_count and pages_read_count that InnoDB reported for the query. The numbers depend on how fast the prefetch thread is in comparison with the query thread. It can also happen that we request to prefetch a page but then still have to wait for it to be read, which is counted in pages_read_count. Replace the checks in the file with ones that will hold in all circumstances. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
xnervwang
xnervwang@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41940 sys.ps_setup_reload_saved() leaves session sql_log_bin disabled after an error sys.ps_setup_reload_saved() saves @@sql_log_bin and sets sql_log_bin = 0, but restores it only at the end of the procedure. On an error, its EXIT HANDLER raises SQLSTATE 90001 without restoring sql_log_bin, leaving the calling session with sql_log_bin = 0. Keep the saved value in a local variable, as the fix for MDEV-42005 does, and restore it in the handler when sql_log_bin differs from it. A caller that may not set sql_log_bin fails at SET sql_log_bin = 0; the handler then leaves sql_log_bin alone, so that caller still gets error 90001. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
zeyad zahran
20912023100721@fci.zu.edu.eg |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40294 mariadb-binlog --start/stop-datetime incorrectly accepts over-max timestamps For a value outside of the TIMESTAMP range, my_system_gmt_sec() returns 0 and sets ER_WARN_DATA_OUT_OF_RANGE, but convert_str_to_timestamp() never checked it. So such values silently became 1970: an over-max --start-datetime printed every event and an over-max --stop-datetime printed none. Print a warning for out of range values and clamp the over-max ones to the max timestamp. Under-min values already became the min. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-25848 one range slot per ARRAY index Before, every MEMBER OF / JSON_OVERLAPS / JSON_CONTAINS point got its own cloned KEY_PART and key slot, and points were combined only through index_merge (OR) or by picking one slot (AND). Now all points on an ARRAY key part share the original slot: * SEL_ARG_ARRAY holds a list of make_key() points ("the array contains any of them") and the summed records_in_range() estimate and cost, taken once when the point is built. * key_or() of two ARRAY ranges takes the union of the points; key_and() keeps the operand with the lower estimate, the WHERE condition checks the rest. * KEY_PART cloning (get_claimed_key_part()) is now RTREE-only, so ARRAY-only queries size SEL_TREE::keys by param->keys again. * QUICK_RANGE_SELECT_ARRAY gets one QUICK_RANGE per point and passes all points to hlindex_read_first(); json_index reads them in turn and skips trefs it has already returned. * check_quick_select_array() uses the stored estimate, and rejects non-ARRAY SEL_ARGs (e.g. IS NULL) on the ARRAY key part. ORed MEMBER OF and JSON_OVERLAPS now give a single range scan instead of index_merge sort_union, with no MAX_KEY limit on the number of points. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38274: Unexpectedly large pages_read_count in main.analyze_stmt_prefetch_count The testcase checked the values of pages_prefetch_read_count and pages_read_count that InnoDB reported for the query. The numbers depend on how fast the prefetch thread is in comparison with the query thread. It can also happen that we request to prefetch a page but then still have to wait for it to be read, which is counted in pages_read_count. Replace the checks in the file with ones that will hold in all circumstances. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-25848 one range slot per ARRAY index Before, every MEMBER OF / JSON_OVERLAPS / JSON_CONTAINS point got its own cloned KEY_PART and key slot, and points were combined only through index_merge (OR) or by picking one slot (AND). Now all points on an ARRAY key part share the original slot: * SEL_ARG_ARRAY holds a list of make_key() points ("the array contains any of them") and the summed records_in_range() estimate and cost, taken once when the point is built. * key_or() of two ARRAY ranges takes the union of the points; key_and() keeps the operand with the lower estimate, the WHERE condition checks the rest. * KEY_PART cloning (get_claimed_key_part()) is now RTREE-only, so ARRAY-only queries size SEL_TREE::keys by param->keys again. * QUICK_RANGE_SELECT_ARRAY gets one QUICK_RANGE per point and passes all points to hlindex_read_first(); json_index reads them in turn and skips trefs it has already returned. * check_quick_select_array() uses the stored estimate, and rejects non-ARRAY SEL_ARGs (e.g. IS NULL) on the ARRAY key part. ORed MEMBER OF and JSON_OVERLAPS now give a single range scan instead of index_merge sort_union, with no MAX_KEY limit on the number of points. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40012 PQ: refuse a plan that reads a table through a join buffer A worker has no join buffer. setup_worker_jointabs() stripped the cache off every worker JOIN_TAB and joined a row at a time, and pwt_table_conds() handed the worker the buffer's scan filter (cache_select->cond) as well as select_cond so that it still rejected what the buffer would have. The result was correct, but it was not the plan the optimizer costed: BNL, BNLH, BKA and BKAH were chosen for scanning the inner table once per buffer-full of outer rows, and a worker scanned it once per driving row instead. For a full-scan inner table that can be orders of magnitude more work than the serial plan the parallel one is meant to beat. So the gate now refuses the plan, as the prototype did. The check sits in can_run_query_in_workers() beside the other per-table refusals, on tab->cache or tab->use_join_cache, both of which make_join_readinfo() has settled by the time parallel_join_check() runs in optimize_stage2(). The refusal names the table in the optimizer trace. The worker-side handling of a buffered table goes with it: - pwt_table_conds() becomes pwt_table_cond() and no longer reports cache_select->cond. remove_redundant_bnl_scan_conds() only moves conjuncts out of select_cond when the tab has a BNL/BNLH cache, so for every tab the gate now admits select_cond is already the whole condition. A tab can still carry a cache_select without a buffer, because make_join_select() builds the scan filter before check_join_cache_usage() decides, but its cond is then a copy of conjuncts select_cond still holds. The gate no longer asks whether that copy is worker-safe, since no worker evaluates it. - pwt_clone_table_conds(), which cloned both halves and ANDed them, is gone; setup_worker_jointabs() clones the one condition with pwt_clone_rebind(). - setup_worker_jointabs() no longer clears cache, use_join_cache and jbuf_tracker on the worker's copy; pwt_assert_tab_inert() asserts that the manager's tab has no cache instead. The copy's cache_select is still cleared, as it can be set on a tab without a buffer. parallel_query_join: the BNL case now expects the query to run serially, with the same rows. The full-scan inner table case (d3) was getting a BNL buffer under the default join_cache_level, so it runs with join_cache_level=0 to keep covering a worker-side full scan of an inner table. parallel_query_why pins the new refusal reason. This commit was prepared with Claude Code (Opus 5.5), which wrote the gate check, removed the worker-side join buffer handling it made unreachable, updated the tests, reconfigured and rebuilt the worktree, and ran the parallel_query tests in normal and --ps-protocol modes. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42009: Set GLOBAL innodb_log_file_size=ON is not crash safe log_t::header_rewrite(): Before setting log_sys.archive=true, ensure that any garbage entries at the end of the file will be zeroed out so that they cannot be mistaken for newer records. On Linux and Microsoft Windows we will attempt a shortcut for writing zeroes, before falling back to log_file_t::write(). Reviewed by: Alessandro Vetere |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thirunarayanan Balathandayuthapani
thiru@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41336 CHECK TABLE EXTENDED crashes with innodb_force_recovery=3 Problem: ========= After a restart where purge has not run (When innodb_force_recovery=3), purge_sys.head and purge_sys.tail are both unset (trx_no=0). trx_lists_init_at_db_start() calls purge_sys.clone_oldest_view<true>(), which clamped end_view to that 0, so end_view.m_low_limit_id became 0 and changes_visible() returned false for every transaction id. end_view therefore claimed that no history had been purged, although undo log records purged before the restart may already have been freed and reused. CHECK TABLE EXTENDED relies on end_view to decide whether an old version's undo log record is still available. It followed the roll pointer of a purged transaction into a reused undo page and trx_undo_rec_get_pars() hit the assertion Solution: ======== In purge_sys_t::clone_oldest_view<true>(), clamp end_view only if purge has a known position (head.trx_no/tail.trx_no is nonzero). Otherwise keep the oldest view as is, so that the undo log records that purge is allowed to remove are reported as DB_MISSING_HISTORY rather than being read. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38274: Unexpectedly large pages_read_count in main.analyze_stmt_prefetch_count The testcase checked the values of pages_prefetch_read_count and pages_read_count that InnoDB reported for the query. The numbers depend on how fast the prefetch thread is in comparison with the query thread. It can also happen that we request to prefetch a page but then still have to wait for it to be read, which is counted in pages_read_count. Replace the checks in the file with ones that will hold in all circumstances. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42009: Set GLOBAL innodb_log_file_size=ON is not crash safe log_t::header_rewrite(): Before setting log_sys.archive=true, ensure that any garbage entries at the end of the file will be zeroed out so that they cannot be mistaken for newer records. On Linux and Microsoft Windows we will attempt a shortcut for writing zeroes, before falling back to log_file_t::write(). log_t::set_archive(): Block concurrent writes to the log on every platform, because log_t::header_rewrite() now requires exclusive access to the log file. Reviewed by: Alessandro Vetere |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42009: Set GLOBAL innodb_log_file_size=ON is not crash safe log_t::header_rewrite(): Before setting log_sys.archive=true, ensure that any garbage entries at the end of the file will be zeroed out so that they cannot be mistaken for newer records. On Linux and Microsoft Windows we will attempt a shortcut for writing zeroes, before falling back to log_file_t::write(). Reviewed by: Alessandro Vetere |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Monty
monty@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Preparation for MDEV-41866 layout-dependent my.cnf location - Allow on to change location of /etc/mysql in Debian/Ubunty by setting DEFAULT_SYSCONFDIR=/etc/mariadb - Search for MariaDB configuration files also in /etc/mariadb. In 14.0 we plan to remove searching in /etc/mysql by default. - Fixed mariadb.logrotate to use DEFAULT_SYSCONFDIR if used. - mariadbd_multi now also uses /etc/mariadb |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
cleanup: st_mariadb_extension in the server's client library Like in Connector/C, MYSQL::extension is allocated in mysql_init() and holds auto_local_infile, tls_self_signed_error and the MariaDB extended capabilities of the server and the client. The MYSQL layout does not change. Assisted-by: Claude:claude-opus-5-5 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42009: Set GLOBAL innodb_log_archive=ON is not crash safe log_t::header_rewrite(): Before setting log_sys.archive=true, ensure that any garbage entries at the end of the file will be zeroed out so that they cannot be mistaken for newer records. On Linux and Microsoft Windows we will attempt a shortcut for writing zeroes, before falling back to log_file_t::write(). log_t::set_archive(): Block concurrent writes to the log on every platform, because log_t::header_rewrite() now requires exclusive access to the log file. Reviewed by: Alessandro Vetere |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40530 Bind the client authentication response to the server certificate MDEV-31855 introduced ssl_info that let the client verify that it uses the correct certificate. But MitM could hijack the connection after the authentication - the client will detect it and disconnect but the server won't and will continue talking to the attacker. Let's introduce ssl_info_client packet that enables the server to verify that the client uses the correct certificate. When require_secure_transport=ON and the connection isn't local: * the server sets MARIADB_CLIENT_SSL_INFO_CLIENT in the initial handshake * the client sets it in the reply, if it understands it * when both sides have it, the client sends ssl_info_client right after the OK packet, acl_authenticate() reads it before the session is used (init_connect, audit, etc) * a connection is rejected unless ssl_info_client is valid, either for the server certificate fingerprint or for "CA" (the client verified the certificate itself) * an account without a password hash (plugin has no hash_password() or empty password) is rejected unless the account REQUIRE a client certificate Assisted-by: Claude:claude-opus-5-5 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39492 PQ: a LIKE is clonable, so a worker can evaluate it The gate asks whether every expression a worker would evaluate can be deep-copied without the copy sharing a node with the original, because the copy's Item_field leaves are repointed at the worker's own tables and a shared node would move the manager's item with them. Item_func_like failed that test: its escape is a child it keeps outside args, and Item_func_or_sum::deep_copy() clones args only, so the copy still pointed at the original's escape item. Every LIKE was therefore left to the manager as a residual condition, which for a LIKE on the driving table ships every row of the scan to be filtered serially -- TPC-H Q14's CASE WHEN p_type LIKE 'PROMO%' was declined outright, as an aggregate argument a worker cannot evaluate. Item_func_like::deep_copy() now clones the escape item as well. The TurboBM shift tables and the pattern are read-only, allocated on the statement's mem_root by fix_fields() and never freed by cleanup(), so the copy keeps pointing at them. Q14 on the SF1 scratch data: 566 ms serially, 180 ms with four workers and 113 ms with eight, with the same result. This commit was prepared with Claude Code (Fable 5.1), which found the LIKE in the gate's trace as the manager-evaluated share of a Q14 probe, wrote the deep_copy() override and the test, and measured the query. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41632 UPPER(key_col) IN (constant subquery) does not use the index Item_func_in::varchar_upper_cmp_transformer() rewrites UPPER(key_col) IN (const, ...) into key_col IN (const, ...). It needs a copy of the IN predicate to put the new first argument into, it used deep_copy_with_checks(). A deep copy copies every node of the tree and refuses as a whole if any node cannot be copied, so an IN list holding an uncorrelated scalar subquery, which is constant but has no copy, silently lost the rewrite: UPPER(a) IN ((SELECT MAX(c) FROM t2),'A-2') read the whole table instead of using a range on the index on a. The rows returned were correct; the symptom is the plan. After copying, the rewrite walked the copy with cleanup_excluding_const_fields_processor and re-fixed it, which would also have cleaned up a subquery had one been copied. A copy for use in the same SELECT_LEX does not need to copy what can be shared there. We add a separate copy hierarchy for that purpose. Item::same_select_copy_with_checks() calls the protected virtual Item::same_select_copy(), and cleans up every node it actually copied, so that the caller only has to call fix_fields() on the result. fix_fields() re-fixes the copied nodes and skips the shared ones, which are still fixed. It does not use deep_copy() at any level; the only other copy function it calls is shallow_copy() on the node being copied. It returns nullptr on out of memory, and when a copied node's shallow_copy() returns nullptr; every Item_func subclass whose shallow_copy() returns nullptr therefore overrides same_select_copy() to share the item, and a future class in that position must do the same or callers silently lose their rewrite. Item default is to share the item. Item_func, Item_cond and Item_row copy their node and recurse into their arguments. Item_func copies an argument that occurs more than once in args[] only once. Item_bool_rowready_func2, Item_func_case_simple and Item_func_in detach the comparators and value array they would otherwise share with the original, as their deep_copy() overrides do. Item_func_interval points its row member at the copied args[0], which it aliases. Fields, references, constants, parameters, caches, subqueries, aggregate and window functions are shared because they don't derive from Item_func. Some Item_func subclasses cannot safely be copied, so they override same_select_copy() to share the original instead: - Functions that own memory or handles, which the copy would free a second time in cleanup() or its destructor: the regex functions, UDFs, stored functions, the JSON functions that build hashes, VEC_FromText, and the geometry functions with Gcalc state. - Functions that their SELECT_LEX keeps a pointer to, so a copy would not be the item the SELECT_LEX uses: MATCH, ROWNUM and Item_equal. - Functions whose copy would behave differently: RAND would restart its random sequence, and SET @var would assign the variable twice. - TO_NUMBER, which refuses to be copied at all. - Functions whose arguments point at each other: NULLIF uses args[0] a second time as args[2] and hides it from arg_count until cleanup(), and Item_in_optimizer's cache belongs to its subquery. The rewrite now uses same_select_copy_with_checks() and no longer walks the copy with a cleanup processor. TABLE::add_splitting_info_for_key_field() uses it too, for the left side of the equality it pushes into a split-materialized derived table: the item of the derived table's select list, used in the WHERE condition of the same select. Its real item is always a column, so the copy is the select list item itself, where the deep copy made a second Item_field for it. Sharing one item between the select list and the WHERE clause of a select is intended here: state that condition processing sets on the item, such as Item_field::item_equal, describes the same column in the same select. The equality is fixed when it is injected. This commit was prepared with Claude Code (Opus 5), which found the subquery case that lost the rewrite and confirmed that the new test fails without the change, designed and wrote the same_select_copy() hierarchy and the list of shared Item_func classes, switched the casefold rewrite to it, and ran main and the SQL-layer suites in normal and --ps-protocol mode. Claude Code (Fable 5.1) then reviewed the commit, corrected the commit message, kept Item_func_case_simple::shallow_copy() protected, added the Item_func_interval override and the nullptr note, and re-ran the tests. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||