Console View
|
Categories: connectors experimental galera main |
|
| connectors | experimental | galera | main | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Teemu Ollakka
teemu.ollakka@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
fix galera_bf_abort_orphan_lock: use transaction_isolation tx_isolation is deprecated and now emits a warning, which broke the test's expected result. Use the non-deprecated transaction_isolation instead. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
squash! e16b1b4e2739be7bc8e91643b0a71e1f04a1b02f Ensure the minimum file size |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
fixup! e16b1b4e2739be7bc8e91643b0a71e1f04a1b02f Try harder to fix a hang of mariabackup.huge_lsn,SERVER,strict_full_crc32 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40629 environment injection via wsrep bootstrap in the service file * don't create mariadb-wsrep-new-cluster in the mariadbd-writable path, the server should not be able to poison the environment with OUTFILE. Create it in /run * As in /run it must be deleted by root, let galera_new_cluster delete it, not the service * wsrep-start-position cannot be created by root, so avoid a file for it at all Assisted-By: Claude:claude-5-opus |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| fixup! bfed32bb60c003fbe974f60f925c8da20ef35adf | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Brandon Nesterenko
brandon.nesterenko@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MDEV-40643 (Regression): Corrupt Heartbeat Log Event can Crash Slave | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Brandon Nesterenko
brandon.nesterenko@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40643: Corrupt Heartbeat Log Event can Crash Slave A slave can crash when its master sends an event shorter than the event's own header. A truncated heartbeat makes debug builds fail an assertion in Binary_string::realloc_raw(). On 64-bit release builds it stops the slave IO thread with an error message that omits the log file name. If the master also declares an oversized common header in its format description event, the slave allocates and fills nearly 4GB instead. On 32-bit release builds it writes that error message past the end of a stack buffer. An event under four bytes crashes every build, because the slave checksums roughly 16 EiB and reads far past the end of the packet. Neither queue_event() nor the Heartbeat_log_event constructor bounded the length the master sent. queue_event() handed that length to event_checksum_test(), which subtracted the checksum length from it, and the result wrapped on an event under four bytes. The constructor computed the log file name length as event_len minus the header lengths, and on a truncated heartbeat that subtraction wrapped to a value near 4GB. queue_event() rejected the heartbeat as invalid, so its error path appended the log file name to the error message using the wrapped length. Add the missing bound in both places. queue_event() now rejects an event shorter than the common header before anything reads that header, and the slave IO thread stops with an error. The Heartbeat_log_event constructor now compares event_len against the combined header lengths before the subtraction. A heartbeat that clears the first check but still stops inside its headers keeps ident_len at 0 and log_ident at NULL, so the error path appends nothing. That constructor check also precedes the read of the extended log position, which previously ran on a short event before any validation. A short event from the master now stops the slave IO thread with an error instead of crashing the server. Reviewed-by: TODO Signed-off-by: Brandon Nesterenko <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Clarify NULL handling comment in next_min() | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| fixup! a8fcc6617fe752014462a8b57e79c4102144dd14 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Teemu Ollakka
teemu.ollakka@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
crash in wsrep_provider_options_init() with wsrep provider plugin wsrep_provider_plugin_init() marked wsrep_provider_options read only by mutating the sys_var's flags directly, instead of going through the sys_var layer. Since MDEV-40341 move_allocated_sysvars_to_root() moves the value of every read only global string sysvar into the read only memroot and clears the ALLOCATED flag. wsrep_provider_options_init() then called my_free() on memroot memory, which aborted on the garbage allocation header. Remove the flag mutation instead of working around its effects. wsrep_provider_options stays a normal read-write variable and is kept in sync with the provider whenever a plugin sysvar changes. SET on wsrep_provider_options is still rejected while the wsrep-provider plugin is loaded, via the existing wsrep_provider_options_check() function. Its error message now states the real reason instead of "is a read only variable", which was misleading: the variable is not statically read only, and information_schema correctly reports READ_ONLY=NO for it. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Clarify NULL handling comment in next_min() | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39492 Parallel Query: let the workers aggregate their own chunk An aggregate over a whole scan shipped every qualifying row to the manager, which aggregated them on its single thread. That per-row work is the part no number of workers reduces, and for an aggregate query it is most of the query: measured on a release build, the drain alone is 29.6 microseconds per thousand rows of strictly serial work, and TPC-H Q1 -- 5.9 million rows into four groups -- reached 1.15x at best, with 4.2 of its 4.8 seconds spent on the manager. Each worker now aggregates its own chunk and ships one row holding a partial value per aggregate. The manager folds each partial into the query's own aggregates with Item_sum::direct_add() and lets the plan's own terminal send the result. What crosses the transport becomes one row per worker instead of one row per qualifying row. direct_add() is what makes this small. Merging a partial is not adding a row -- COUNT has to add a count rather than increment, and MIN and MAX have to reach the Item_cache their add() reads rather than args[0] -- and Item_sum has provided exactly that all along, on Item_sum_count, Item_sum_sum with a decimal and a real overload, and Item_sum_min_max. It had no caller; 518f083d4d4 established what it does. So the accepted aggregates are COUNT, SUM, MIN and MAX, without DISTINCT, over a query with no GROUP BY whose select list holds nothing but those and constants. pwt_preagg_supported() asks at execution time, because which terminal the optimizer chose is not known when the gate runs. Anything it declines still runs in the workers, shipping its rows exactly as before, and the reasons are each worth stating: - A GROUP BY needs a grouping structure per worker and a merge per group. Nothing here is in the way of it; it is not built. - AVG, STD and VARIANCE cannot merge a value: their state would have to be shipped, which their temp-table field already holds. - BIT_AND/OR/XOR have no direct_add. They do not need one, being self-composing, but folding them in is a second mechanism for a rare case. - The DISTINCT variants need the whole set before it can be counted, which the server agrees with: every merge path asserts the aggregator is not a DISTINCT one. - An aggregate whose argument cannot be copied, asked before the copy is attempted so that such a query ships rows rather than failing. Three things needed care. result_table holds one column per aggregate rather than the base-table columns, and create_tmp_table() gives an Item_sum no field at all unless save_sum_fields is set -- "SELECT COUNT(*)" built a table of no columns. Nothing is copied back; what replaces copy_back is an Item_field per column, which is what direct_add(Item *) takes for MIN and MAX. A worker's aggregate is cloned, but its argument is cloned and rebound separately and grafted in, because Item_sum::fix_fields() registers the aggregate with the select_lex the manager is also using. setup_caches() then rebuilds what the shell derived from the old argument. The manager does not call the terminal per row, since that would add the partial row itself on top of the partial it carries. It calls it once at end of records, which is the call that sends an aggregate's single row -- and sets join->first_record first, because end_send_group() reads that to decide whether a row was ever seen and otherwise clears the aggregates and reports an empty result. When no partial arrived at all, leaving it alone is right: the empty-result answer is then the one a serial run gives. main.parallel_query_preaggregate compares sixteen shapes against their serial answers -- every accepted aggregate, an all-NULL column, no qualifying row at all, rows in some chunks and not others, constants, HAVING, a join, and an expression as the argument -- and checks the refusals still run in the workers without pre-aggregating. Parallel_partial_aggregations counts the scans that took the path, reported as a ratio against Parallel_queries_executed because both are cumulative and --ps-protocol executes a statement more than once. SUM over a floating-point column is compared rounded, and that is not this feature's doing: a parallel scan hands the manager its rows in whatever order the chunks finish, so the same query already gave different low bits between two runs before the workers aggregated anything -- verified against a shape that pre-aggregation declines. Pre-aggregation changes how far the low bits move, not whether they move. SUM over a DECIMAL is exact and compared as such. This commit was prepared with Claude Code: it read the transport and the terminal to find where a partial could be folded in, built the worker and manager halves, and established by measurement that the float non-determinism it exposes was already there. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39492 Parallel Query: a GROUP BY over constants "SELECT COUNT(*) FROM t1 GROUP BY 'x'" crashed a debug server in the workers, on the assertion that JOIN::group_optimized_away is not set. The name reads, in this file, as though the optimizer had removed a grouping the workers were counting on. It means the reverse. A GROUP BY whose every expression is constant has one group by construction, so JOIN::optimize_stage2() empties group_list and sets the flag to say so; make_aggr_tables_info() then reads it as implicit_grouping and plans an ordinary aggregate over the whole scan, terminal end_send_group. That is the plainest shape the manager runs, and the one every aggregate query with no GROUP BY at all already takes. So the assertion was simply wrong, and nothing else was: with it removed the query answers as it does serially. Both shapes of the flag are covered -- a bare constant, and an expression over constants. This commit was prepared with Claude Code: it found the crash while writing tests for grouped pre-aggregation, established that the shape reaches execution as an implicit-group aggregate, and wrote the test. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Brandon Nesterenko
brandon.nesterenko@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MDEV-40492 (Regression): Oversized Fake Rotate Event can Crash Slave | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40383:innodb_gis.point_basic fails on replay There are 2 problems: - 1. The REPLACE statement that is recorded doesn't store the value of geometry type field correctly. 2. The table definition that got recorded has fields with non-null constraint, and no default value is specified. Also, the "REPLACE INTO" statement that gets stored in the context, doesn't have any value specified for these non-null fields. Solution is to: - 1. When using REPLACE INTO statement, store all the non-numeric values in HEX, whenever conversion from field's charset to output's charset is lossy. 2. Instead of storing only the column values that were projected in the query, store all the non-virtual column values into the recorded REPLACE INTO statement. Implementation details: - 1. Introduce a new method is_charset_conversion_lossless() in filesort.cc, to check if the output charset to which field's data is being written to, results in a lossless conversion. If so, non-numeric values being witten using REPLACE INTO statement are stored in string representation, else they are converted to HEX. 2. Modify join_read_const(), and join_read_system() methods in sql_select.cc, and opt_sum_query() method in opt_sum.cc the following way: - a. Extend the read_set to make sure, we read all the non-virtual column using Optimizer_context_recorder::prepare_captured_row_read(). This method also saves the original read_set. b. Read the row. c. Dump the row into the context when no error is noticed while reading. Irrespective of the error, restore back the read_set state to the original using Optimizer_context_recorder::finish_captured_row_read() |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
cleanup: encryption.filekeys_encfile_badfile combine all tests for wrong FILE: values into one test add a test for a wrong key. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
ODBC-498 Reading SQL_ATTR_METADATA_ID caused stack corruption The fix and the trestcase. The priver would write additionally 4 bytes past the given buffer. Plus the value written into the buffer would be wrong. The break in the switch was missing. The test is also covering stmt attribute |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Brandon Nesterenko
brandon.nesterenko@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40492: Oversized Fake Rotate Event can Crash Slave A slave can crash when the master's binlog_checksum setting differs from the checksum the slave's relay log carries. On the first Rotate event of such a connection, the slave IO thread copies the event onto a stack buffer sized for the longest binary log file name a master can send. A master that names a longer file makes the copy run off the end of that buffer, and what it writes past the end is the file name itself. A name that overruns the buffer by a few hundred bytes reaches only the other buffers of the same stack frame, which the slave does not read on this path, so it keeps running. A name that overruns the whole frame takes the return address with it, and the slave crashes. Builds with AddressSanitizer stop at the copy and report a stack-buffer-overflow. The fake Rotate event that opens a connection is rewritten when the master's checksum policy and the relay log's disagree. One case adds a checksum to the event, and the other strips one. Both copy the event into rot_buf using the length the master sent, and neither compared that length against the size of rot_buf. The Rotate_log_event constructed just above bounds its own copy of the file name at FN_REFLEN-1, which leaves the length the master sent untouched. rot_buf was itself one checksum shorter than the event the first case produces from the longest file name. Bound the event before either case copies it. queue_event() now compares the event's length against the longest Rotate a master can send: the two headers, a file name of at most FN_REFLEN bytes, and the checksum the master's own policy puts on the event. rot_buf now reserves that checksum length as well. A Rotate event naming a longer file stops the slave IO thread with an error reporting the length the master sent, instead of running off the end of the buffer. Reviewed-by: TODO Signed-off-by: Brandon Nesterenko <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40383:innodb_gis.point_basic fails on replay There are 2 problems: - 1. The REPLACE statement that is recorded doesn't store the value of geometry type field correctly. 2. The table definition that got recorded has fields with non-null constraint, and no default value is specified. Also, the "REPLACE INTO" statement that gets stored in the context, doesn't have any value specified for these non-null fields. Solution is to: - 1. When using REPLACE INTO statement, store all the non-numeric values in HEX, whenever conversion from field's charset to output's charset is lossy. 2. Instead of storing only the column values that were projected in the query, store all the non-virtual column values into the recorded REPLACE INTO statement. Implementation details: - 1. Introduce a new method is_charset_conversion_lossless() in filesort.cc, to check if the output charset to which field's data is being written to, results in a lossless conversion. If so, non-numeric values being witten using REPLACE INTO statement are stored in string representation, else they are converted to HEX. 2. Modify join_read_const(), and join_read_system() methods in sql_select.cc, and opt_sum_query() method in opt_sum.cc the following way: - a. Extend the read_set to make sure, we read all the non-virtual column using Optimizer_context_recorder::prepare_captured_row_read(). This method also saves the original read_set. b. Read the row. c. Dump the row into the context when no error is noticed while reading. Irrespective of the error, restore back the read_set state to the original using Optimizer_context_recorder::finish_captured_row_read() |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39492 Parallel Query: correct a comment about copying aggregates The note beside can_run_query_in_workers()'s select-list check said that copying an Item_sum_min_max crashes because its copy constructor leaves cmp uninitialised. Neither half is true any more. The diagnosis was wrong -- the implicit copy constructor that get_item_copy() uses left cmp aliasing the original's, to be deleted twice, rather than uninitialised -- and it was fixed along with the aggregator and orig_args in 4938029fe4c. Verified at execution time, in clone_worker_exprs(), which is where an earlier attempt at this reported the fix not holding: Item_sum_count, _sum, _min, _max, _avg, _std, _variance, _and, _or and _xor all clone, and none of the clones shares a node with the item it came from. No behaviour changes, so no test accompanies this; main.parallel_query_clone covers the copies themselves. This commit was prepared with Claude Code. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MDEV-40658 file_key_management crash on empty FILE: file | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39492 Parallel Query: cover TPC-H Q1 The query the last few commits were for, run end to end: a filtered scan of one large table into four SUMs, three AVGs and a COUNT, grouped on two columns. It is the shape that showed pre-aggregation was worth building -- measured, it spent 4.2 of its 4.8 seconds on the manager, which is work no number of workers reduced. Now every worker's whole chunk leaves it as six rows. Compared against the serial answer as it stands rather than as a fingerprint, because its own ORDER BY is applied to the finished groups. This commit was prepared with Claude Code. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-25964: Unexpected bypass of lock When an uncommitted transaction inserts rows into a table and another statement locks rows in the same table (SELECT ... FOR UPDATE) while computing a MIN or MAX, then: 1. In a Debug build, the server aborts on an assertion 2. In a Release build, the server returns wrong results These errors occur because, while reading a group of rows for computing a MAX, the transaction timeout error was swallowed. Under the scenario described above and captured in the new test at this commit, QUICK_GROUP_MIN_MAX_SELECT::next_max() emits a lock timeout error during QUICK_GROUP_MIN_MAX_SELECT::get_next() but the error was suppressed if we computed a MIN. The InnoDB storage engine has an unwritten convention that after it has returned a fatal error (which is any error except HA_ERR_END_OF_FILE or HA_ERR_KEY_NOT_FOUND), then the SQL layer should not try to make any further reads. This is because InnoDB might have rolled back the current transaction already. So in the case of an error, return immediately from QUICK_GROUP_MIN_MAX_SELECT::get_next(). |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39492 Parallel Query: let the workers aggregate their own groups A GROUP BY shipped every qualifying row to the manager, which grouped them on its single thread. That per-row work is what no number of workers reduces. Now each worker groups its own chunk and ships one row per group, so what crosses the channel is the number of groups a worker saw rather than the number of rows it read. How a partial per group merges The manager keeps calling the plan's own terminal per row, which for these plans is end_update(): it builds the group key from the row, finds the group in the server's aggregation table and folds the row in with update_field(), or starts the group with reset_field(). Both of those consume an Item_sum direct value in place of the row when one is pending, so the manager sets one per aggregate before each call and end_update() does the rest. There is no merging code here. For that to work the row has to mean to end_update() what a joined row would, so a worker ships the base columns as well as the partials: the base columns of whichever row of the group reached it first. The manager copies them back into its own records exactly as it does for the row transport, and end_update()'s copy_fields() reads them from there. What a worker does with its chunk It keeps a grouping table of its own, whose columns are the shipped layout and whose key is the GROUP BY, and runs the same algorithm end_update() does: fill the columns from the row, build the key, look it up, extend that group's partial or start a new one. Making the table's rows the rows to ship means the flush at end of records is a scan and a memcpy per row, with no projection and no field mapping. When the table fills it ships what it has and starts again empty, rather than moving to disk the way end_update() would. A partial is mergeable, so a group split across two flushes costs one extra row and nothing else; and it keeps the heap-to-disk conversion, which reads thd->lex, off a worker thread. A grouping table that would not be an in-memory one to begin with is refused for the same reason: the space an on-disk temp table accounts for is charged to the thread that writes it and released by the thread that frees it, and those are the worker and the manager. What is accepted The same four aggregates as before -- COUNT, SUM, MIN, MAX -- because the same four are the ones whose reset_field() and update_field() honour a direct value. Plus, of the GROUP BY itself: every group expression a plain, non-constant base-table column, so the key is over columns the worker already ships and create_tmp_table() will build a real index for it; and every non-aggregate item the terminal evaluates reading only group columns, since a group's other columns are those of an arbitrary row of it. Whether the terminal is end_update() at all is asked of the plan, not guessed: pwt_plan_group_key() takes the key from the aggregation table the manager will feed, and returns nothing for any other write function. Everything else still runs in the workers on the row transport, as it did. Two things that needed care The group key entries carry the key field and its place in the key buffer, so they belong to one table: every worker builds its table from a private copy of the list, and the manager's copy is built with the key too -- not because it groups anything, but because asking for a key changes the record layout and both sides must agree on it byte for byte. What those entries name has to change after the table is built. While create_tmp_table() runs they must name the column definitions, which is how it finds the column each key part covers. From then on they must name something that reads the row the key is being built for, and a column definition does not: it is a clone of an Item_field over the manager's copy of the base table, which no worker fills. Left that way every row keys on NULL and a whole chunk becomes one group. They are repointed at the grouping table's own columns, which the worker has just filled. Verified as serial-vs-parallel over many GROUP BY shapes: few groups and many, a NULL group, two group columns, MIN/MAX over a column the group does not determine, a grouping table small enough that it flushes constantly, a join, an empty result, HAVING and ORDER BY above the grouping. Each refusal is checked too, so the boundary is a stated one. This commit was prepared with Claude Code: it wrote the worker-side grouping and the manager-side priming, found the repointing bug above by dumping the field a group key was reading, found and fixed the on-disk grouping table by tracing an assertion about a worker's temp space, and wrote the tests. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40656 Bypass REVOKE DENY ... FROM PUBLIC privilege check. DENY ... TO PUBLIC denies everyone, including whoever tries to revoke it, via the "deny wins" merge at every scope (global, db, table, column, routine). Allow REVOKE DENY ... FROM PUBLIC when the revoker can UPDATE mysql.global_priv (same as hand-editing). Assisted-by: Claude:claude-5-sonnet |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
ParadoxV5
paradox.ver5@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40365/MDEV-40366 test fixes * The crafted invalid FDEs were omitted their checksums, even though the base code (still) adds the checksum length to the event length. This commit fixes this discrepancy by not skipping the event footer step (write the checksum, and finish up encryption if active), so the fault injections are more self-contained. This discrepancy did not matter in practice because * The event loading simply assumes the first few bytes of the next event as the unused checksum. * The fix to `get_checksum_alg()` is detecting invalidity before the code reaches the fixed parser-contructor. This is rather an implementation detail, though, as the constructor fix would come to effect if we refactor `get_checksum_alg()` away. * This commit also disables echoing `SHOW BINLOG EVENTS IN` to the results in case the `$binlog_file` is not consistent. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39492 Parallel Query: cover Item_sum::direct_add() direct_add() is the primitive worker-side pre-aggregation would merge partials with: a worker aggregates its own chunk and the manager folds each worker's partial into the group's running value. That is a different operation from adding a row -- for COUNT it adds a count rather than incrementing, and for MIN and MAX it has to reach the Item_cache that add() reads, which an args[0] redirect cannot -- and Item_sum provides it already, on Item_sum_sum with separate decimal and real overloads, Item_sum_count and Item_sum_min_max. It has no caller anywhere in the server. So nothing exercises it, no SQL reaches it, and the behaviour a partials scheme would rest on is unverified. This adds the call, under the pwt_verify_direct_add DBUG keyword, and pins the arithmetic. The check runs once per parallel query from init_parallel_workers(), on the manager's thread so its notes reach the user's diagnostics area rather than a worker's, over clones of the query's own aggregates so the query's answer is untouched. Each note reports what direct_add computed and what it should have, so a regression is legible in the diff rather than merely a changed number. Feeding a NULL partial is checked as carefully as feeding a value, because a worker whose chunk held no qualifying row for a group ships one and it must be skipped rather than read as a zero or taken as the minimum. The two SUM overloads say NULL differently and it is easy to get backwards: the real one takes a flag, the decimal one a null pointer, where a pointer to decimal zero means the value 0. Writing the check the wrong way round was the one failure of the session, and it was the check that was wrong. What has no direct_add is reported too, since that is where the boundary of a partials scheme sits. BIT_AND, BIT_OR and BIT_XOR need none, being self-composing: a partial in args[0] and an ordinary add merges. AVG, STD and VARIANCE do need one, a partial average not being averageable, and would have to ship the state their temp-table field already holds. The test also reports the worker count, because the check only happens when the workers really run: were the query to fall back to serial execution there would be no notes at all and the absence would read as a pass. This commit was prepared with Claude Code: it found that direct_add existed and was dormant, drove all four classes by hand to establish what they do, and then turned that into this test. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
fixup! 23ecceccc6201b2122ee946a05e5860f8a2fadb6 btr_search_drop_page_hash_index() is being invoked on a non-file page (state < FREED). Everywhere else, the more readable !is_read_fixed() or !is_io_fixed() assertions are safe to use. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
PranavKTiwari
pranav.tiwari@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Fixed macor | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
ODBC-499 bookmark buffer overflow in case of array fetch. If bookmarks were used together with array row fetch, the address of the bookmark buffer is calculated incorrectly, and with column-wise binding it is guaranteed that the driver writes past the allocated buffer. With row-based it's less probable, but address was calculated incorrectly and would overwrite column data. Also, the bookmark themselves would be written for the 1st row in the block cursor, and that is incorrect. The patch adds thorough test of the bookmarks use. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39492 Parallel Query: MIN and MAX merged the row, not the partial A grouped pre-aggregation answered a MIN() or MAX() wrongly whenever the manager's aggregation table outgrew memory: with tmp_table_size squeezed so that 300 groups do not fit, the same query gave one answer serially and another in the workers. end_update() folds a partial into a group with update_field(), which takes the pending direct value. When the table fills, the server moves it to disk and hands the rest of the rows to end_unique_update() instead, which folds them a different way round: it calls reset_field() first, before it knows whether the group is new, and only calls update_field() if the write turns out to be a duplicate. Item_sum_sum and Item_sum_count are built for that. Their reset_field() records in direct_reseted_field that the direct value went into the field rather than being added, and their update_field() honours the flag, so the value is merged once either way. Item_sum_min_max cleared direct_added in reset_field() and had no such flag, so the update_field() that followed compared args[0] -- the base-table column, holding the row the worker shipped -- against the group. What that row carries is one row of the group, not the group's extreme, so the answer was the extreme over one row per worker per flush. Give Item_sum_min_max the flag its siblings have. Nothing else in the server reaches this: direct_add() has no other caller. The test makes the difference visible rather than incidental. The group's shipped row is its smallest pk, so a MAX() that merged the row instead of the partial reports a smaller number, while the MIN() beside it looks right either way. This commit was prepared with Claude Code: it reasoned that the shape had to be wrong, could not make it fail until it arranged for the manager's table to overflow while the workers' tables did not, and wrote the fix and the test. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| fixup! cb5a92348ebb37ab690338c0391b717ae66c6fb8 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-25964: Unexpected bypass of lock When an uncommitted transaction inserts rows into a table and another statement locks rows in the same table (SELECT ... FOR UPDATE) while computing a MIN or MAX, then: 1. In a Debug build, the server aborts on an assertion 2. In a Release build, the server returns wrong results These errors occur because, while reading a group of rows for computing a MAX, the transaction timeout error was swallowed. Under the scenario described above and captured in the new test at this commit, QUICK_GROUP_MIN_MAX_SELECT::next_max() emits a lock timeout error during QUICK_GROUP_MIN_MAX_SELECT::get_next() but the error was suppressed if we computed a MIN. The InnoDB storage engine has an unwritten convention that after it has returned a fatal error (which is any error except HA_ERR_END_OF_FILE or HA_ERR_KEY_NOT_FOUND), then the SQL layer should not try to make any further reads. This is because InnoDB might have rolled back the current transaction already. So in the case of an error, return immediately from QUICK_GROUP_MIN_MAX_SELECT::get_next(). |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
ParadoxV5
paradox.ver5@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
[temp] Test MDEV-40647 It does not typically fail, but should trip MSAN. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39492 Parallel Query: merge a partial AVG per group AVG was refused on the grounds that an average cannot be averaged, and that is true of the value but not of what a worker actually has. Its temp-table field holds the pair the average is computed from -- the sum and the count of values that went into it -- and a pair like that merges by adding both halves. A worker accumulating a group into its own table therefore already has, in the shipped row, exactly what the manager needs. - Item_sum_avg::direct_add() takes that pair. The sum half rides on Item_sum_sum's own direct_* members, so a count of zero is told to it the way it tells itself a NULL sum: a null pointer for the decimal overload, a flag for the real one, and nothing is merged. - reset_field() and update_field() honour it, one starting a group's pair from the partial and the other adding the partial into it. Both also record direct_reseted_field, so that a switch to end_unique_update() -- which resets before it knows whether the group is new -- merges the pair once rather than twice or not at all. - pwt_manager::direct_add_partials() reads the shipped column the way the aggregate reads its own field: the sum packed at the front and the count in the eight bytes after it, at the sizes the aggregate was laid out with. Only with a GROUP BY, and for the shape of the column rather than the arithmetic. Item_sum_avg::create_tmp_field() lays the pair out only for a table it is asked to key; for any other it gives a plain column holding the average, which nothing can be merged into. A worker's grouping table is keyed, so a grouped query has the pair to ship and an ungrouped one does not. Verified against the serial answer over a decimal average, a real one, a few groups and many, and a group whose averaged column is entirely NULL -- where the partial is a count of zero and the answer has to stay NULL rather than become 0. This commit was prepared with Claude Code: it worked out that the pair was already in the field, wrote the overloads and the unpacking, and wrote the tests. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40656 Bypass REVOKE DENY ... FROM PUBLIC privilege check. DENY ... TO PUBLIC denies everyone, including whoever tries to revoke it, via the "deny wins" merge at every scope (global, db, table, column, routine). Allow REVOKE DENY ... FROM PUBLIC when the revoker can UPDATE mysql.global_priv (same as hand-editing). Assisted-by: Claude:claude-5-sonnet |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40632 Use my_fprintf in trx_print_low To fix for windows %p in `fprintf(f, "TRANSACTION (%p)", trx);` is not prefixed with 0x |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39499 Updates to derived-with-keys, window functions determining: Unfinished review input. The biggest change is making handle_single_part_rownumber() reuse estimate_post_group_cardinality(). fails most of MDEV-39499's testcases in main/derived_opt.test, not sure why. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-37013 crash in applying FK cascade with virtual column Added a simplified version of the Stefan Frye's test scenario |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||