Console View
|
Categories: connectors experimental galera main |
|
| connectors | experimental | galera | main | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Fix ubsan failure with not a valid value for enum_mysql_show_type It produced errors like: runtime error: load of value 19, which is not a valid value for type 'enum enum_mysql_show_type' In order to get a correct server-side definition of enum_mysql_show_type, one must include sql_plugin.h (see #define SHOW_always_last there) before including include/mysql/plugin.h, either directly or indirectly. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Do not dump stats and const rows for read only engines' tables. Stats for tables from engines such as Archive, S3, PerfSchema, and Sequence shouldn't be recorded in the context. Similarly, const row records should also not be stored in the context. Added few tests for Sequence's engine tables like seq_1_to_5. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40728 Recovery wrongly fails if FILE_CREATE is followed by FILE_RENAME fil_name_process(): Simplify the logic. If no matching tablespace is found but file_name_t::create_lsn had been set in response to parsing a FILE_CREATE record, try to apply FILE_RENAME to deferred_spaces. deferred_spaces.deferred_dblwr(): Skip newly created tablespaces to avoid a bogus invocation of fil_space_free(). |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
KhaledR57
khaled57.dev@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40495 KEY_OP_SHIFT redo moves data past the page buffer The KEY_OP_SHIFT branch of _ma_apply_redo_index() took the shift length straight from the redo record and used it to form a bmove() source and size, and to update page_length. The only guards were DBUG_ASSERTs, which are compiled out when DBUG_OFF is set. A corrupt record with a negative length could therefore move data from outside the page and wrap page_length. Turn both asserts into runtime checks. The page offset must be set and inside the used page, the resulting page length must still fit the page, and for a negative shift the source must stay inside the used page too. The conditions are the ones the asserts already tested, so debug builds keep the same behaviour. This also fixes MDEV-40496, which covers the page_length side of the same branch. The first check bounds it. The test forges the logged shift length with two new debug keywords, corrupt_shift_down and corrupt_shift_up, then crashes the server so recovery has to replay the record. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40608 MariaDB-devel is incomplete for plugins * create and install mariadb-plugin-config.cmake * deb: move all headers that plugins need to libmariadb-dev, together with libmysqlservices.a. At least until we'll create mariadb-plugin-dev.Nobody should need huge libmariadbd-dev to develop a plugin * rpm: all in MariaDB-devel already, no changes here * adjust plugin.cmake to work for external plugins * move server-internal part to top-level CMakeLists.txt * remove WITH_WSREP from my_config.h (it upsets external plugins) * remove double-defined macros from unireg.h (the guard doesn't help if unireg.h is included first) ColumnStore, until fixed, needs a backward-compatibility workaround |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40387: perfschema.misc fails on replay Disable the testfile, as we don't capture context for performance schema tables. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Daniel Black
daniel@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sql_test: mallinfo2 msan exclusion no longer needed MSAN interceptor was added in clang-18.1. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Daniel Black
daniel@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-17846 Wrong result with grouping select (fix) Prevent unused variable 'ref_type' warnings on non-debug builds. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40486 Length check for vector fields in CREATE TABLE ... SELECT The changes of MDEV-39558 2b6529426a7e7c65d286e093d84138be9dcc34a3 added length check assertion in Field_varstring constructors, and length check in type inference for SELECT set operations, to emit errors before reaching the assertions. That change caused an error to turn into an assertion failure in a separate path, when the length limit violation is not detected before tripping the assertion. So in this patch we fix it by adding an earlier length check in that path. The reason that we place this check inside Item_func_vec_fromtext::fix_length_and_dec rather than say `create_field_for_create_select is for consistency: If create table t1 as select vec_fromtext(concat('[',group_concat(1),']')) as c1 from seq_1_to_64; fails due to length limit violation, then so should create table t1 (v vector(64) not null); insert into t1 select vec_fromtext(concat('[',group_concat(1),']')) from seq_1_to_64; Also use max_char_length() instead of max_length. This is a more accurate length of characters. And add handling of empty string edge case. Added testcases accordingly. The change that uses max_char_length() causes side effects where creating a table using a VEC_FROMTEXT(CHAR(1)) would result in a 0-dimensional vector field. This is accurate but 0-dim vector table fields should not be allowed. So we make cases like this result in a one-dimensional field. Also fixed the underflow in (args[0]->max_length - 1) * 2 when the arg's max length is 0. Previously this underflow would cause create table t1 select vec_fromtext(NULL) to fail with ER_TOO_BIG_FIELDLENGTH. Now it will be a VECTOR(1) field |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39360: set statement optimizer_record_context for query fails Move the initialization of context recorder, and replay after run_set_statement_if_requested() is invoked in the mysql_execute_command() in sql_parse.cc |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40383:innodb_gis.point_basic fails on replay There are 2 problems: - 1. The REPLACE statement that is recorded doesn't store the value of geometry type field correctly. 2. The table definition that got recorded has fields with non-null constraint, and no default value is specified. Also, the "REPLACE INTO" statement that gets stored in the context, doesn't have any value specified for these non-null fields. Solution is to: - 1. When using REPLACE INTO statement, store all the non-numeric values in HEX, whenever conversion from field's charset to output's charset is lossy. 2. Instead of storing only the column values that were projected in the query, store all the non-virtual column values into the recorded REPLACE INTO statement. Implementation details: - 1. Introduce a new method is_charset_conversion_lossless() in filesort.cc, to check if the output charset to which field's data is being written to, results in a lossless conversion. If so, non-numeric values being witten using REPLACE INTO statement are stored in string representation, else they are converted to HEX. 2. Modify join_read_const(), and join_read_system() methods in sql_select.cc, and opt_sum_query() method in opt_sum.cc the following way: - a. Extend the read_set to make sure, we read all the non-virtual column using Optimizer_context_recorder::prepare_captured_row_read(). This method also saves the original read_set. b. Read the row. c. Dump the row into the context when no error is noticed while reading. Irrespective of the error, restore back the read_set state to the original using Optimizer_context_recorder::finish_captured_row_read() |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Code cleanup in JSON array-of-object reading, add unit tests. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38701: Optimizer Context Replay: merge into 13.1 tree Optimizer Context Replay feature allows one to record and replay a query's Optimizer Context. Optimizer Context includes everything that one needs to replicate how the Query Optimizer processed the query. It can be replayed on another to host to debug how the Query Optimizer processed the query, run what-if scenarios, etc. == Example recording == set optimizer_record_context=1; < Run the query of interest. Typically it's EXPLAIN ...>; select context [into dumpfile '/tmp/context.sql'] from information_schema.optimizer_context; == Example replay == -- On another machine, just source the script source context.sql This will * Set relevant system variables to match the recording side; * Create the dataases, tables and views the query needs; * Load EITS statistics for the tables; * Provide the optimizer with other context data * Finally re-run the query. If it was an EXPLAIN, one should get the same output as on the recording side. Approved-by: Sergei Petrunia ([email protected]) |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40384: innodb_gis.geometry fails on replay The test had innodb_strict_mode turned OFF, when running the test. But, in the replay, it was enabled, which caused the creation of tables with KEY_BLOCK_SIZE=16 fail. Solution is to record the innodb_strict_mode variable in the context, so that it gets used during the replay. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| attempt #2 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40388: sequence.simple fails on replay The problem is that, when recording is enabled for the query such as, explain select * from seq_1_to_10; it recorded the table context having a DDL definition as: - CREATE TABLE `seq_1_to_10` ( -> `seq` bigint(20) unsigned NOT NULL, -> PRIMARY KEY (`seq`) -> ) ENGINE=SEQUENCE DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_uca1400_ai_ci; Now, when that context is replayed, the DDL statement is executed. But, we cannot create such a table, and instead it errors out saying ERROR 1050 (42S01): Table 'seq_1_to_10' already exists. Solution is to use: - CREATE TABLE IF NOT EXISTS seq_1_to_10 ...; ===== Also, there is a different way to use sequences as: - Create sequence s1; Explain select * from s1; Here, we should be recording the DDL statement, but no need to store the stats for it. However, we didn't record the DDL statement earlier. Moreover, sequence's next value should be the same in the replay environment. Solution here is to record the DDL for such a sequence as CREATE TABLE IF NOT EXISTS s1 ...; and also set its start value as the recorded environment's previous value using SELECT SETVAL(s1, prev_value); |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| attempt #3 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40742: Optimizer Context Replay: wrong table names on MacOS show_create_table_ex() will use table's alias when lower_case_table_names=2. Specify that table names must be used like the table is stored on disk. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Daniel Black
daniel@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40243 Fix MEMORY_LEAK_C leaks in mariadb-dump (rockdb tests) With memory leaks fixed in the rocksdb.mysqldump/mysqldump2 no longer need to run with leak detection disabled. There tests are still disabled as there's no --rockdb arg to mariadb-dump, but removing so there's no precidence to ignoring leaks. ref: 2217477fb8fc82f4921d9d13afcd24b1d86b34d6 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Georg Richter
georg@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Cleanup for test: drop my_vector table | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40389: type_test.type_test_int8 fails on replay plugins are not yet supported in replay mode. So, disabling tests type_test.type_test_int8, and type_test.type_test_double to be run in replay-server mode |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Daniel Black
daniel@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
RocksDB: compile fix std::replace requires algorithm header Otherwise it compile fails. Found in clang-24. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| fixup! 38d0d83910a9d9a8db2921bdce2a101065e9f68e | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MDEV-39226: Add multi-table update, delete feature | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alexander Barkov
bar@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-28498 Incorrect information in file: './test/t0.frm' on CREATE TABLE Applying HEX encoding write writting an ENUM/SET TYPELIB to FRM if the TYPELIB has 0x00 bytes in the value. This HEX encoding was earlier used only to write UCS2/UTF16/UTF32 TYPELIBs. A new flag FIELDFLAG_FRM_HEX_ENCODED_TYPELIB was added to indicate that the TYPELIB is hex encoded. It's used only inside FRM. Note, it's mangled with FIELDFLAG_TREAT_BIT_AS_CHAR. This should not be harmful: - BIT and ENUM/SET columns are handled by two separate code branches when opening an FRM - The flag is unset immediately after decoding TYPELIB, so the rest of the code does not se an unexpected flag combination. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
KhaledR57
khaled57.dev@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40495 KEY_OP_SHIFT redo moves data past the page buffer The KEY_OP_SHIFT branch of _ma_apply_redo_index() took the shift length straight from the redo record and used it to form a bmove() source and size, and to update page_length. The only guards were DBUG_ASSERTs, which are compiled out when DBUG_OFF is set. A corrupt record with a negative length could therefore move data from outside the page and wrap page_length. Turn both asserts into runtime checks. The page offset must be set and inside the used page, the resulting page length must still fit the page, and for a negative shift the source must stay inside the used page too. The conditions are the ones the asserts already tested, so debug buildskeep the same behaviour. This also fixes MDEV-40496, which covers the page_length side of the same branch. The first check bounds it. The test forges the logged shift length with corrupt_shift_length debug, then crashes the server so recovery has to replay the record. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39492 PQ workers must read the manager's transaction snapshot A parallel worker runs in its own THD, so it runs in its own transaction and opened its own read view at its first read. Nothing tied that view to the one the manager holds: the workers and the manager could each read a different version of the table, and the chunk boundaries the engine partitioned under the manager's view did not even belong to the snapshot the workers scanned. Both directions were visible. Under REPEATABLE READ a worker returned rows another session had committed after the manager's snapshot was taken, and rows the manager's own transaction had written but not committed went missing, because to the worker's view the manager was just another active transaction. InnoDB implements it by copying the source transaction's read view. ReadView::clone() installs the source's ReadViewBase state and opens the copy. It inherits the source's m_creator_trx_id rather than keeping the worker's own, so rows written by the sharing transaction itself stay visible through the copy -- that is what changes_visible() uses the creator id for -- and it inherits the source's isolation level, so a READ UNCOMMITTED source is still read without consulting a view at all. The source view is read under its m_mutex, the same protection the purge coordinator takes in ReadView::append_to(), and from_thd's handlerton data under its LOCK_thd_data. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40728 Recovery wrongly fails if FILE_CREATE is followed by FILE_RENAME fil_name_process(): Simplify the logic. If no matching tablespace is found but file_name_t::create_lsn had been set in response to parsing a FILE_CREATE record, try to apply FILE_RENAME to deferred_spaces. deferred_spaces.deferred_dblwr(): Skip newly created tablespaces to avoid a bogus invocation of fil_space_free(). fil_delete_apply(): A wrapper for fil_space_free(). When recovering a log in innodb_log_archive=ON format, we must apply FILE_DELETE records in order to avoid a future clash with FILE_CREATE or FILE_RENAME. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Remove incorrectly added sql/opt_sum.cc.orig | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40012 Parallel Query: execute the join in the worker threads Each parallel worker now runs the join and the WHERE over its own chunk of the driving table and ships the base-table columns of every row that qualified. The manager copies those columns back into the fields they came from in its own table instances, so that after a row is drained its records hold what a serial scan would have left there, and then evaluates the select list against them and sends the row. This replaces the model where workers shipped raw source records and the manager ran the join. The transport carries columns rather than projected select-list values deliberately. Anything the manager does with a row beyond sending it -- a condition, an aggregate, a temp table -- reads a record, and not all of those reads go through Items that could be re-pointed at a shipped value: create_tmp_table() builds Copy_field pairs holding raw Field pointers into the base tables. Filling the records themselves is what makes every later stage work without an indirection of its own. make_join_readinfo()'s gate (can_run_query_in_workers) chooses the worker-side path for an inner select-project[-join] with a parallel- scannable driving table: no tmp table (group/distinct/order/window/ buffer), no LIMIT/SQL_CALC_FOUND_ROWS/procedure/aggregate, no outer join or semijoin, and every non-driving table reached by eq_ref/ref/full scan. do_select() then runs run_worker_side_join() instead of the nested loop; anything ineligible runs serially. Each worker opens a private copy of every non-const table, deep-clones and field-rebinds the conditions and select list, and rebuilds each ref (clone_table_ref, mirroring create_ref_for_key). It scans its driving chunk, runs its own inner nested loop (cp_buffer_from_ref + ha_index_read_map / ha_index_next_same for ref/eq_ref, rnd scan otherwise), projects the read columns of each full match into a private result table and ships the row image; the manager drains, copies the columns into its own records and sends. Writing into those records is not something a SELECT's write_set allows and Field::store() asserts on it, so the drain marks the fields writable for its duration; and Copy_field captures &table->null_row, so the manager's tables have their reader flags cleared before it starts. A killed worker's own ER_QUERY_INTERRUPTED is no longer treated as a fatal evaluation error (PWT_error_handler guards with !thd->killed) so kills keep propagating through kill_signal with the correct kill type. Standalone helpers are attached to the object they operate on (pwt_worker / pwt_manager members rather than file-static functions) and the functions that carry real control flow have DBUG_ENTER tracing. Tests: parallel_query_worker_side (single table) and parallel_query_join (eq_ref, ref with fan-out, 3-table chain, full-scan inner table) compare the parallel result set against serial. parallel_query / parallel_query_oom moved to a plain SELECT (which now runs worker-side) and were re-recorded. This commit was prepared with Claude Code: it wrote the worker-side join execution (worker_join_inner / worker_emit_row, the per-worker table, ref and expression cloning) and the two new tests; the naming, the member-function layout and the DBUG tracing are the author's cleanup. run_worker_side_join() returned 1 -- error -- when init_parallel_workers() reported HA_ERR_UNSUPPORTED, although its own documented contract, and the comment on that very branch, say -1 means "the engine declined, run the query serially". No error was raised on the way out, because nothing had gone wrong: the engine simply refused the scan. do_select() turned that 1 into NESTED_LOOP_ERROR, so the statement failed with an empty diagnostics area and Protocol::end_statement() hit its DA_EMPTY assertion. In a release build the client gets a statement that neither succeeds nor reports an error. The engine declines this way for any read that is not a consistent read: pscan_init_coordinator()'s first check refuses the scan when select_lock_type != LOCK_NONE. So SELECT ... FOR UPDATE, SELECT ... LOCK IN SHARE MODE, CREATE ... AS SELECT and INSERT ... SELECT over a parallel-scannable table all crashed a debug server as soon as parallel_worker_threads was set -- the optimizer picks the table, then execution has nowhere to go. Return -1 so do_select() takes its serial path, which is what make_join_readinfo() left in place for exactly this case by keeping the table's serial read_first_record. parallel_query_fallback is the new test: FOR UPDATE, LOCK IN SHARE MODE, CREATE ... AS SELECT and INSERT ... SELECT each return their rows with workers enabled. All four reach the decline path (the DBUG_PRINT added here fires four times), and the first of them already crashes the server without this commit. This also un-breaks parallel_query_join and parallel_query_worker_side, which have been failing on this branch since they were recorded: both open with a CREATE ... AS SELECT that took the bad path. Their serial-vs-parallel comparisons are built from CREATE ... AS SELECT pairs, so now that the fallback works both sides run serially and those comparisons no longer say anything about worker-side execution. The assertion in parallel_query_join that claimed CTAS "runs worker-side" is corrected to state what the trace key actually reports, which is the optimizer's choice. Restoring a like-for-like comparison at scale needs a technique that does not write from a SELECT, and is left to a follow-up. This commit was prepared with Claude Code: it traced the DA_EMPTY assertion to the wrong return code with a DBUG trace, reduced the repro from CREATE ... AS SELECT to SELECT ... FOR UPDATE, and wrote the test. MDEV-39492 Parallel Query: give worker tables their place in the join A worker's private table copies came out of open_table_from_share() with TABLE::map still zero, because nothing assigns it outside the optimizer's setup_tables(). Item_field::used_tables() reads TABLE::map, so every item rebound onto a worker copy reported used_tables() == 0 -- it looked like a constant. pwt_clone_rebind() re-fixes the clone, and Item_cond::fix_fields() evaluates any argument that can_eval_in_optimize(), so cloning the condition of SELECT a, b, c FROM t1 WHERE a % 7 = 0 AND b > 100 evaluated "b > 100" on the manager thread, against a worker record buffer that no row had been read into. Debug builds assert in Field_long::val_int() on marked_for_read(); release builds read that unread buffer to compute a null-rejection cache. Any WHERE with more than one predicate was affected -- that is, most of them. Copy map and tablenr from the manager's table when the copy is opened, so a rebound item computes the same used_tables() as the item it was cloned from and nothing evaluates it at clone time. Mark all columns readable there too, before any cloning touches those tables, rather than at the start of worker_run_query() -- the read_set then holds for the whole life of the copy, and the per-table loop in worker_run_query() goes away. This was invisible because no test reached it. parallel_query_join and parallel_query_worker_side compared parallel against serial by building the same result twice with CREATE ... AS SELECT, and writing from a SELECT is a locking read, which makes the engine decline the parallel scan: both sides ran serially, so the multi-predicate WHERE never reached a worker clone. Those comparisons are rewritten to compare something that really does run in the workers. A result set that cannot be written to a table cannot be compared afterwards either, so each query now goes inside a non-mergeable derived table -- the inner select runs in the workers, the outer aggregate runs in the user thread -- and is reduced to COUNT(*), SUM(CRC32(...)) and BIT_XOR(CRC32(...)). That fingerprint is order-independent, which a parallel result needs since its rows arrive in batch-completion order, and it compares the whole multiset at full scale without printing it. The shared method lives in include/parallel_query_fingerprint.inc. Each comparison also asserts that the workers, not the user thread, did the scanning: both runs read the same materialized rows, but only the serial one also scans the driving table, so the serial run's Handler_read_rnd_next must be the larger of the two. That is what stops these comparisons from silently going hollow again -- with the parallel half forced back to 0 workers the check flips from 1 to 0 and the tests fail. Both tests were checked to be load-bearing by mutation: dropping one worker result row in 500 moves every fingerprint (713 -> 712, 5000 -> 4990, 4967 -> 4958, 6000 -> 5988). This commit was prepared with Claude Code: it root-caused the assertion to the zero table map, wrote the fingerprint comparison and the include, and ran the mutation checks. MDEV-39492 Parallel Query: worker tables inherit the manager's column bitmaps A worker's table copies were marked with use_all_columns(), which points both read_set and write_set at the share's all_set. Correct, since it is a superset of what the worker evaluates, but coarser than it needs to be in two ways. InnoDB builds its row fetch template from read_set, so with every column marked a worker converts every column of every row it scans to MySQL format, not just the ones its cloned condition and select list reference -- work paid on the hot path that parallel query exists to shorten, and it grows with the width of the table rather than with the query. And an all-columns write_set on a table the worker only ever reads gives up an assertion: a store into a source field no longer trips marked_for_write(). Copy the manager table's read_set and write_set into the copy's own def_read_set/def_write_set instead, and point the copy at those. The optimizer has already marked exactly the columns this query reads, and open_worker_tables() runs after it has finished, so those bitmaps are final; the copy owns its bitmaps, so nothing is shared with the manager, and column_bitmaps_set() signals the engine to rebuild its template. This keeps the property the marking was added for: every column a cloned item could touch is in the read_set before any cloning happens, because the optimizer marked precisely that set. Two cases in parallel_query_worker_side cover the fidelity of the copy, both through the serial-vs-parallel fingerprint: a table with eighteen columns where the query references two, and a virtual column whose base column is read but never projected. Clearing a single bit of the copied read_set makes both of them fail (Field_long::val_int() asserts on marked_for_read()), as does parallel_query_join. Index-only reads are unaffected by this: they need ha_start_keyread(), which the worker-side path never issues, for any read_set. This commit was prepared with Claude Code: it identified the cost of the blanket marking, made the copy, and added the two coverage cases. MDEV-39492 Parallel Query: give constant select-list items a result field SELECT 42, a FROM t1 crashed the server whenever parallel_worker_threads was set. create_tmp_table() does not give a constant item a field, which is right for a query that materialises its result and can evaluate the constant once outside the table, but the parallel result table is not that. Its layout has to mirror the select list item for item, because a worker projects item i into field i and ships the record image, and the manager sends one Item_field per field to the client. With a constant in the list the table came out one field short, and worker_emit_row() ran off the end of the field array into a NULL Field pointer. The manager's send list was equally short, so even without the crash the client would have been sent the wrong number of columns. Any constant did it, a literal, a folded expression such as 1+1, or a session constant like CONNECTION_ID(). Pass TMP_TABLE_ALL_COLUMNS when building the result table, so every item of the select list gets a field, and assert the layout matches the list afterwards -- the transport is positional, so a mismatch from any other cause has to be refused rather than walked over. That exposed a second problem in the same place. create_tmp_table() overwrites param->func_count with the number of items it actually has to copy, and make_result_table() was called once per worker plus once for the manager from a single TMP_TABLE_PARAM counted once by the caller. A constant needs a field but no copy entry, so the count dropped to zero after the first table and every later one allocated fewer fields than its layout, tripping the assertion in Create_tmp_table::finalize(). Each result table now starts from a freshly counted param, which is what N identical layouts from one param needs in any case. Session constants come out right rather than merely not crashing: the clone is fixed on the manager's thread, so CONNECTION_ID(), USER() and DATABASE() carry the user session's values, not the worker's. The test asserts that against values captured outside the query. parallel_query_worker_side covers a literal, a string literal, a folded expression, an all-constant select list where no column of the scanned table reaches the result, and the session constants, plus a serial-vs-parallel fingerprint over a select list containing a constant so the case is known to run in the workers. Every one of them crashes the server without this commit. This commit was prepared with Claude Code: it found the crash while probing which expressions the gate accepts, traced both causes from the core file, and wrote the tests. MDEV-39492 Parallel Query: refuse an expression copy that is not independent A worker repoints the Item_field leaves of its copy of the WHERE condition, the ref values and the select list at its own tables. That is only safe if the copy owns those leaves. Several item classes implement deep_copy() as a shallow copy while still holding child items, every Item_cache and Item_outer_ref among them, so their copy keeps pointing at the original's children, and Item_cache::walk() does visit those children. Rebinding one moved the manager's own Item_field onto worker one's table, then worker two moved it again, so every worker and the manager ended up sharing a field bound to some other thread's table. The in_use assertion in Field::val_int() caught it in a debug build, SELECT STRAIGHT_JOIN count(*) FROM t1 JOIN t2 JOIN t3 WHERE t1.f1 IN (SELECT f1 FROM t4) AND t2.f1 IN (SELECT f1 FROM t5); crashing after semijoin conversion left a cached reference in the condition of a select the workers ran. A release build reads another thread's record buffer, and the manager's own plan is left rebound to a table that is closed when the workers are reaped. The gate now requires more than clonability, it requires the copy to share no Item_field object with the item it came from. Testing it there costs nothing, the gate already deep-copies every candidate item and throws the copy away, and a query that fails the test declines to serial execution at optimize time. pwt_clone_rebind() asserts the same property before it rebinds anything, so a copy that slips through is caught at the point of damage rather than by whatever reads a foreign table later. Checking for a shared leaf rather than enumerating the classes that shallow-copy means the check also covers the classes not yet met. There are more of them than Item_cache: Item_outer_ref and Item_copy_string hold child items and copy shallowly too. The three tests that hit this with parallel_worker_threads forced on -- opt_hints_join_order, innodb_mrr_cpk and subselect_innodb -- no longer crash. parallel_query_clone is the new test: the semijoin query above, plus a correlated reference in a select list and one in a condition, each answered serially and again with workers enabled. Without this commit the first of them trips the assertion in pwt_clone_rebind(), and without that assertion too it reaches the original in_use crash. This commit was prepared with Claude Code: it traced the assertion from the core file to the shallow deep_copy, wrote the shared-leaf check and the test. MDEV-39492 Parallel Query: the workers' statistics belong to the session A worker counts its reads, its lock calls and everything else in its own THD, and ~THD puts them straight into the global counters. Nothing ever reached the session that asked for the work, so SHOW SESSION STATUS was short by whatever the workers did: after a parallel scan of a 1000-row table Handler_read_rnd_next reported around 500, the rows this thread read out of the materialised result, rather than the 1502 the same query reports serially. Nine tests in the main suite noticed, in Handler_read_%, COLUMN_DECOMPRESSIONS and Optimizer_join_prefixes_check_calls. Each worker now copies its status counters into its pwt_worker just before its THD is destroyed, and quiesce_workers() adds them to the session's own after joining every worker, so only one thread ever touches either side and no locking is involved. The worker then clears its own counters, which stops ~THD adding the same numbers to the global counters that the session will pass on later. Only the counters move. Memory accounting stays with the worker's THD, because more of that THD's memory is freed after the snapshot is taken and ~THD has to reconcile all of it with the global counters. Clearing with the clear_for_flush_status offset leaves those fields alone, and the snapshot drops its copies of them. Suppressing ~THD's accounting entirely instead, which is the obvious way to avoid counting the same numbers twice, loses that reconciliation: the server then reports an internal memory accounting error of a couple of hundred thousand bytes at shutdown. Session and global counters were not equally wrong. Global was right all along, because ~THD was adding to it, so this changes which session the work is attributed to and not the total. parallel_query_worker_side compares Handler_read_rnd_next for the same query run serially and in parallel, and requires the parallel run to report at least as much. Written as a comparison rather than an exact figure because a chunked scan ends at each chunk, so it reads a few more than a single scan does. Without the handover the parallel run reports about a third of the serial figure and the test fails. This commit was prepared with Claude Code. MDEV-39492 Parallel Query: filter by the condition from before the pushdown When the optimizer pushes part of a condition into the engine, push_index_cond() leaves tab->select_cond holding only the remainder and keeps the original in tab->pre_idx_push_select_cond. The pushed half lives on from there in handler::pushed_idx_cond, which belongs to the handler it was pushed into -- the manager's. A worker reads through its own handler, opened by open_table_from_share() with nothing pushed into it, and it cloned select_cond, so the pushed half was enforced in neither place. SELECT pk, a, b FROM p1,p2,p3 WHERE b >= d AND pk < c AND b = '0'; answered one row serially and six with workers, `pk < c` having been applied nowhere, and the unfiltered rows then multiplied against the third table. Setting index_condition_pushdown=off made the parallel answer correct, which is what pinned it on the pushdown rather than on the plan the cost model chose. Clone the pre-pushdown condition where there is one. One accessor, pwt_table_cond(), is used by the gate and by both clone sites, so the item the gate approves is always the item a worker evaluates -- the two drifting apart is what this bug was. This gives up what the pushdown was for: the engine no longer rejects an index entry before the row is read, so a worker does more clustered-index work per match than the serial plan. The alternative was to refuse these plans at the gate, which would have cost the parallel scan altogether on a common plan shape. Correct and parallel beats correct and serial here, and pushing a clone onto the worker's own handler would recover the difference -- that wants the worker to hold a real JOIN_TAB to hang the key number off, so it belongs with the cloned JOIN, not before it. parallel_query_join gains the query above, answered serially and again with workers, asserting that it still ran in the workers rather than falling back and that p1 still carries a pushed index condition, so the case cannot quietly stop being covered. Without the fix the parallel answer is six rows. range_innodb, which is where this was found with parallel_worker_threads forced on, now differs only in EXPLAIN output. This commit was prepared with Claude Code: it traced the predicate to push_index_cond() moving it out of select_cond, and wrote the test. MDEV-39492 Parallel Query: give ANALYZE the numbers the workers produced ANALYZE reads a JOIN_TAB's counters from the tracker the optimizer left on the Explain object, and the engine's counters from the handler it recorded there. Both belong to the manager, and the manager never runs the driving table's read loop, so a parallel query reported the table as untouched: r_loops 0, no r_rows, no r_filtered, no r_engine_stats. For a feature whose whole purpose is to make a scan faster, the tool for seeing where a scan spends its time said nothing about it. Each worker now counts what it did to each of its tables in the same terms sub_select() and evaluate_join_record() use -- rows read, rows that passed the table's condition, and one scan per probe of an inner table -- and copies the engine's counters out of the tables while they are still open. The manager adds all of it to the trackers and handlers ANALYZE reads, in quiesce_workers() after every worker has been joined, so this thread is the only one touching either side and no locking is needed. ha_handler_stats::add() already existed for the partitioning case and does the engine half. The driving table reports one scan, not one per worker: the chunks are one scan of the table between them, which is what the serial plan reports and what keeps the rows-per-scan figure comparable between the two. ANALYZE SELECT a FROM t1 WHERE a % 7 = 0; now answers r_rows 5000.00 and r_filtered 14.28 whether it runs serially or in the workers, differing only in the access type. Not included: r_table_time_ms and r_other_time_ms, which come from the elapsed time trackers rather than from counters, and which the workers measure in parallel. Summing them would report more time against the table than the query itself took, and reporting one worker's share would understate the work. That needs a decision about what ANALYZE should mean for a parallel scan, so it is left out rather than guessed at, and the two fields stay absent. parallel_query_worker_side compares the tabular ANALYZE for the same query run both ways, with the row estimate masked because it is an InnoDB approximation. Without the handover the parallel run reports r_rows and r_filtered as NULL. This commit was prepared with Claude Code. MDEV-39492 Parallel Query: filter by the condition the join buffer would apply An inner table read through a BNL join buffer keeps part of its condition somewhere a worker never looked. JOIN_TAB::make_scan_filter() copies the conjuncts that need only that table into cache_select->cond, for the buffer to apply as it fills, and JOIN_TAB::remove_redundant_bnl_scan_conds() then removes those same conjuncts from select_cond -- it calls set_cond(NULL) outright when they were the whole of it. A worker uses no join buffer and cloned select_cond, so the conjuncts were enforced in neither place. SELECT STRAIGHT_JOIN b2.v FROM b1, b2 WHERE b2.x = b1.k AND b2.v > 15 AND b1.id <= 2; answered two rows serially and ten with workers, the extra eight being every row of b2 that b2.v > 15 excludes. The join condition still held -- ten rows, not forty -- which is what pinned it on the single-table half: make_scan_filter() extracts by table map, so the two-table equality stays in select_cond and only the single-table predicate moves. This is the third time the same shape has bitten. The optimizer moves a predicate out of select_cond into a structure the worker does not replicate: handler::pushed_idx_cond, then the semijoin strategies, now the join buffer's scan filter. So the accessor that was introduced for the first case is now pwt_table_conds(), reporting both halves, and one clone helper ANDs them. The gate and both clone sites go through it, so the condition the gate approves stays the condition a worker evaluates. Only BNL is affected. The hashed and batched buffers give the table an access type outside JT_EQ_REF/JT_REF/JT_ALL, which the gate already refuses, so they never reach a worker -- verified across join_cache_level 2, 4, 6 and 8, where only level 2 differed. BNL is the default. The conjunction is built with quick_fix_field() over two already-fixed clones, which is what remove_redundant_bnl_scan_conds() itself does when it rebuilds a condition out of fixed conjuncts. A worker only evaluates the item, so the fix-time caches Item_cond::fix_fields() would rebuild are not read. parallel_query_join gains the query above, answered both ways, asserting that it still ran in the workers and that b2 still carries a join buffer, so the case cannot quietly stop being covered. Reverting the cache half of the accessor brings the eight rows back. Found while surveying JOIN_TAB for the executor-split work: the survey wanted to assert that a cloned tab's join-cache fields are inert, which is what turned up the fact that they are not. This commit was prepared with Claude Code: it traced the predicate to make_scan_filter() and remove_redundant_bnl_scan_conds(), established by sweeping join_cache_level that only BNL reaches a worker, and wrote the test. MDEV-39492 Parallel Query: give a worker real JOIN_TABs No functional change. This is the first step of replacing the worker's private nested loop with the executor's, and it replaces the structure that loop reads. pwt_jointab described a table in five fields: its TABLE copy, the access type, a ref, a condition and the sorted flag. sub_select() and evaluate_join_record() want a JOIN_TAB, so every capability added to the private loop would have to be added again when the split moves to the real one. A worker now holds an array of JOIN_TAB, n_tables of them, indexed the way pwt_manager::mgr_tabs is so that [0] is the parallel-scanned driving table -- the indexing sub_select() will be started from. The driving table's condition moves onto join_tabs[0].select_cond with every other table's, so worker_cond goes away. The tabs are copied from the manager's and then rebound, rather than built from nothing, so that a field this code does not know about holds the value the optimizer chose instead of a zero that would look deliberate. Overwritten: the TABLE, the condition, the ref, the tracker. Cleared: the places a condition can hide (pre_idx_push_select_cond, cache_select) now that select_cond carries all of it, and the join buffer, which a worker does not use. Nulled deliberately: the JOIN, next_select, read_first_record and read_record, which the next two steps fill in -- reaching them before then is a crash rather than a read of the manager's execution state. That last distinction is the point of pwt_assert_tab_inert(). A copy keeps the manager's pointer in anything the copy does not overwrite, which is the trap TABLE::map and TABLE::in_use were, and the fields that matter here are ones the gate is supposed to have excluded rather than ones this code sets. So each is asserted inert on the manager's tab before the copy: the outer-join chain, the semijoin strategies and their weedout and firstmatch tables, emb_sj_nest, the rowid filter, range access, split derived materialization, the DISTINCT shortcut, HAVING, and the access type. Relax a gate without teaching the copy about the field it lets through and a debug build stops here instead of executing with another thread's state. Writing those assertions is also what found the BNL scan filter fixed in "filter by the condition the join buffer would apply": asking whether cache_select was inert turned out to have the answer no, and a wrong result behind it. They are a survey as much as a guard, so they are worth running before the code they protect. Verified live on a debug build -- none of them fires across the main suite with parallel_worker_threads forced on, 1418 tests, and a deliberately false one aborts the server, so the negative result is a measurement rather than an assumption. main and innodb pass in full, 2103 tests, and the parallel tests pass under --ps-protocol. The forced-worker sweep is at its 18 known failures, all EXPLAIN and cost output. This commit was prepared with Claude Code: it established which JOIN_TAB and JOIN fields the real executor actually reads (4 of JOIN, some 29 of JOIN_TAB, of which the gate makes most inert), wrote the copy and the assertions, and checked the assertions execute rather than trusting that they had. MDEV-39492 Parallel Query: give a worker its own JOIN No functional change: nothing reads the worker's JOIN until its JOIN_TABs are driven through sub_select(). Second step of replacing the worker's private nested loop with the executor's. A worker cannot share the manager's JOIN. sub_select() writes join->return_tab as it descends -- the level to unwind to, set unconditionally on entry -- and the guard that reads it compares against JOIN_TAB*, so with a JOIN_TAB array per worker one worker's backtrack point would be compared against another's tabs: pointers into different allocations, and a scan that can end early for no visible reason. The stronger reason is join->thd, read at twelve sites in that loop for the diagnostics area, the killed flag and the row counters. Sharing the JOIN would send all of it to the manager's THD from a worker thread, which is the bug fixed in "a worker's result table belongs to the worker", reintroduced once per row and for every query shape. Unlike return_tab it cannot be gated away. JOIN declares its copy constructor and assignment private and unimplemented, so the worker's is built by JOIN's own constructor and given what it needs by name. That constructor is JOIN::init(), a field-initialiser that allocates nothing, plus a shallow copy of fields_list, so it is cheaper as well as better defined than copying the bytes of a class whose author said not to. It also matches how Join_plan_state already holds part of a JOIN: the convention here is a named list of fields, not an object copy. Only three fields need carrying, which is what makes a named list practical. Walking sub_select() and evaluate_join_record() for the shapes the gate allows leaves thd, return_tab and found_records. map2table is read only for split_derived_to_update; join_tab_execution_startup() reads join only inside its two semijoin-materialization branches; JOIN_TAB::preread_init() returns before touching join->thd unless the table is a materialized derived. All three are already asserted inert per table by pwt_assert_tab_inert(). result is left null deliberately. It is the manager's connection to the client, and manager_collect_and_send() is the only thing that may send a row, so a worker that reaches for it crashes rather than writing to a socket two threads share. pwt_assert_join_inert() is the same idea as pwt_assert_tab_inert() one level up, and none of its checks fires across the main suite with parallel_worker_threads forced on. The sizeof(JOIN) tripwire is debug-only on purpose: JOIN carries dbug_join_tab_array_size under #ifndef DBUG_OFF, so its size is not the same in the two build types, and pinning both would leave two magic numbers of which only one is ever checked by whoever changes the class. This commit was prepared with Claude Code: it established the three-field surface by walking the executor and the helpers it calls, found that JOIN is deliberately non-copyable after having earlier reported the opposite, and confirmed that main.xtradb_mrr appearing in one forced-worker sweep is a load-dependent warning-count flake rather than a regression -- it passed six of seven runs in isolation and a second sweep is back at the known eighteen. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39492 PQ framework for manager and worker threads Introduces parallel_worker_threads variable to control the number of worker threads created by a parallel execution query. 2 new files, sql_parallel_workers.h sql_parallel_workers.cc which contain structures for the creation, management and deletion of parallel worker threads (pwt_ in the name). Main management class created in the stack in JOIN::exec, implemented for the top level select. Threads are registed in server_threads, so are visible in information_schema.processlist and the show processlist command. We check that a kill query on a parallel worker is passed onto it's manager and the query is properly aborted, and that a kill connection is handled properly in parallel_worker.test. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Daniel Black
daniel@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38405 Assertion `tbl->trn == 0' failed in _ma_set_trn_for_table Aria bulk insert operations disables share->now_transaction meaning a concurrent open of the stable table will have trn == &dummy_transaction_object for its MARIA_HA object during opening. As the _ma_set_trn_for_table is setting the trn, its harmless if the current trn is the dummy_transaction_object. Relax the assert to allow for this state. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40518: add both drop table and view stmts The context only stored "DROP TABLE IF EXISTS t1" before adding a "CREATE TABLE t1" statement. However, there can be a view named t1 already existing in the database. When the context was replayed, the CREATE statement failed stating t1 already exists. Solution is to add both "DROP TABLE IF EXISTS t1", and "DROP VIEW IF EXISTS t1" before adding a "CREATE TABLE t1" statement into the context. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40220: Add server version to optimizer context Additionaly, include Version_source_revision as well. These are read only informative sya variables. So, they are included as comments instead of SET commands. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| fix windows | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| attempt 4 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40390: compat/oracle.sp-package fails on replay UDFs are not yet supported in replay mode. So, disabling test compat/oracle.sp-package |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Daniel Black
daniel@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-34482 main.events_processlist test fix As the test result is dependent of SHOW PROCESSLIST output, adjust the wait condition to ensure the state is in sleeping rather than "init" or another state. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40740: Optimizer Context Replay: innodb.xa_unlock_unmodified fails assert Optimizer Context code changed format_and_store_row() to check both table->read_set and table->write_set (when required). It used to use one of those depending on the lock level. But we don't set the table->read_set bit so we can get an assertion failure when dumping the column value. This is fairly rare as Optimizer Context now reads all columns, and the only other user is statements like "DBUG_PRINT("dml", dbug_format_row(..." |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||