Console View
|
Categories: connectors experimental galera main |
|
| connectors | experimental | galera | main | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40168 Make an over-long key a prefix key instead of no key A key image that does not fit in a fulltext token was rejected, which costs the index for that value entirely: no key in the document, and an OVERLAPS that mentions the value gives up on the index altogether. Cut the image down to MVI_KEY_IMAGE_MAX_LEN instead. Two values that agree on that many bytes then share a key, which costs false positives and nothing else, since the predicate is rechecked on every row the index produces. This is what the non-binary path has always done -- strnxfrm() is asked for exactly that many bytes of weights and cannot return more -- so it makes the binary path, which is the one a JSON column takes, behave the same. No wildcard is needed in the fulltext query for this. Both sides of the index cut at the same point, so the key a search builds for a long value is the same string as the token the document has for it, and an exact term match finds it. A trailing '*' would only widen the term to keys that are longer than the one searched for, and after the cut there are none. Co-Authored-By: Claude Opus 5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41012 Galera appliers hang with foreign key of types UUID, INET4, INET6 Bumped application protocol version to level 5 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41021 SET GLOBAL innodb_log_archive ignores read-only mode log_t::set_archive(): Prevent a crash in log_t::header_rewrite() by refusing the operation if the log is read-only. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41012 Galera appliers hang with foreign key of types UUID, INET4, INET6 Added a deterministic test for reproducing the issue |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40168 Make an over-long key a prefix key instead of no key A key image that does not fit in a fulltext token was rejected, which costs the index for that value entirely: no key in the document, and an OVERLAPS that mentions the value gives up on the index altogether. Cut the image down to MVI_KEY_IMAGE_MAX_LEN instead. Two values that agree on that many bytes then share a key, which costs false positives and nothing else, since the predicate is rechecked on every row the index produces. This is what the non-binary path has always done -- strnxfrm() is asked for exactly that many bytes of weights and cannot return more -- so it makes the binary path, which is the one a JSON column takes, behave the same. No wildcard is needed in the fulltext query for this. Both sides of the index cut at the same point, so the key a search builds for a long value is the same string as the token the document has for it, and an exact term match finds it. A trailing '*' would only widen the term to keys that are longer than the one searched for, and after the cut there are none. Co-Authored-By: Claude Opus 5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Kristian Nielsen
knielsen@knielsen-hq.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40745: Validate field parsing of optional_metadata Protect accesses into the m_optional_metadata buffer to not access outside of the buffer in case of corrupt/malicious event data. Protect against buffer overflow of the m_column_name array in case of excessive column names in the event data. Signed-off-by: Kristian Nielsen <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Coding style fixes, remove out-of-date comments | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Merge branch '13.0' into 13.1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40168 Factor out the MVI array walk Item_func_mvi_encode::val_str_ascii() and collect_mvi_keys() walked a JSON array the same way: descend into a nested array, skip an object, encode everything else with encode_mvi_key(). The two copies had to agree for the index to answer correctly, and they had already drifted once. Move the walk into walk_mvi_json_array() and give the callers a visitor each: Mvi_key_appender, which joins the keys with a space into the fulltext document of a row, and Mvi_key_collector, which turns them into the keys of an Mvi_access and decides when a missing key means we have to give up on the index. The walk reports why it stopped, so the encoding side can still tell a malformed document from one that is not an array, and the query side can pick the scalar literal out of MVI_WALK_NOT_ARRAY. No functional changes. Co-Authored-By: Claude Opus 5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40168 Factor out the MVI array walk WIP NOTE (ycp): this is a squash of two commits. The first one factors the walk out using two new classes. A bit too heavy handed imo. The commit can be found at 68ae51b89bc850a4093e1b91c7da80ee74fa61bf. I reviewed that commit and it looks ok. The second commit redid the refactoring to an iterator. I have not yet reviewed this new implementation yet. MDEV-40168 Factor out the MVI array walk Item_func_mvi_encode::val_str_ascii() and collect_mvi_keys() walked a JSON array the same way: descend into a nested array, skip an object, encode everything else with encode_mvi_key(). The two copies had to agree for the index to answer correctly, and they had already drifted once. Move the walk into walk_mvi_json_array() and give the callers a visitor each: Mvi_key_appender, which joins the keys with a space into the fulltext document of a row, and Mvi_key_collector, which turns them into the keys of an Mvi_access and decides when a missing key means we have to give up on the index. The walk reports why it stopped, so the encoding side can still tell a malformed document from one that is not an array, and the query side can pick the scalar literal out of MVI_WALK_NOT_ARRAY. No functional changes. Co-Authored-By: Claude Opus 5 <[email protected]> MDEV-40168 Turn the MVI array walk into an iterator The walk was a visitor: five virtual callbacks, each returning a bool to stop the walk, and a key_buffer() the walk asked for a buffer with. Two callers do not need that much interface, and neither of them wanted its control flow inverted -- both had to encode "give up" as a bool that the walk then turned back into a result code. Pull instead of push. Mvi_array_iterator::start() and next() return the next thing the walk found -- a key, an element with no key, a nested array opened or closed, or one of the ways the walk ends -- and the caller loops over them with its own control flow: MVI_ENCODE goes back to its gotos, collect_mvi_keys() to plain returns, and the state it kept in a visitor is local variables again. The buffer to encode into is a constructor argument, so MVI_ENCODE still gets its keys written straight into the document it is building. Mvi_json_array_visitor, Mvi_key_appender and Mvi_key_collector go away. What made the walk worth sharing is unchanged: the two sides of the index cannot disagree about what the keys of a document are, because this is the only place that makes them. No functional changes. Co-Authored-By: Claude Opus 5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-14992/MDEV-38362/MDEV-39092 BACKUP SERVER The following SQL statements will be introduced: BACKUP SERVER TO '/path/to/directory' [ 1 CONCURRENT ]; BACKUP SERVER WITH [ 1 CONCURRENT ] 'command'; In place of the 1, any positive number of threads may be specified. For the first variant, '/path/to' must exist and '/path/to/directory' must not exist; that is where the backup will be written to. For BACKUP SERVER WITH, the string mariadb-backup- will be prepended to the specified command and quoted. Path separators or quotes are not allowed in command. We expect the mariadb-backup-command to be in the search PATH of the mariadbd process. For now, we distribute no such script or program; the user has to write one, something like this: exec zstdmt|ssh [email protected] "exec cat > $1.tar.zstd" The standard input of that command will be in a format compatible with GNU tar --format=oldgnu (and also BSD tar variants that are also part of Microsoft Windows and Apple macOS). The command is expected to optionally compress and encrypt the stream and redirect it to a file on a local or a remote server. The BACKUP SERVER WITH will append an additional argument, a positive base-ten number in ASCII, starting with 1, to identify the current thread. In this way, there can be multiple concurrent streams. The InnoDB write-ahead log will be streamed near the end, because the stream format requires the file sizes to be declared upfront. TODO: implement the following: In BACKUP SERVER TO ... 2 CONCURRENT or more, one thread will be responsible for copying the InnoDB write-ahead log from the beginning of the backup. Other files will be copied by other threads. Note: In single-threaded BACKUP SERVER TO, the parameter innodb_log_recovery_start that is written out to backup.cnf is STRICTLY NECESSARY TO AVOID CORRUPTION during recovery! By default, InnoDB crash recovery starts from the latest available log checkpoint. However, for restoring a backup, recovery must start from the checkpoint that was the latest when the backup was started. Starting recovery from a possible later checkpoint will result in a corrupted database! The backup or the first stream will contain a file backup.cnf, which includes parameters needed for restoring the backup. Currently, these are innodb_log_recovery_start and innodb_log_recovery_target. If innodb_log_recovery_target>0, InnoDB will be in read-only mode, not allowing any writes to persistent files other than via the log application. To restore a streaming backup made with BACKUP SERVER WITH, an empty directory needs to be created and all streams be extracted there using the standard tar utility of the operating system, optionally after undoing any encryption or compression that had been added by the backup command. Then, the backup is prepared or MariaDB server started up on the extracted directory, similar to as if the BACKUP SERVER TO statement had been used. The following will be implemented separately: MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER MDEV-40163 Partial backup and restore MDEV-39091 Back up ENGINE=RocksDB MDEV-40333 Concurrent DDL for Aria tables in BACKUP SERVER The implementation introduces a basic multi-threaded driver Sql_cmd_backup, storage engine interfaces, and basic copying of the storage engines InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV. backup_target: A structured data type to represent a target directory. On Microsoft Windows, we must use directory paths because there is no variant of CopyFileEx() that would work on file handles. backup_sink: Wraps a per-thread output stream as well as storage engine specific context. handlerton::backup_start(), handlerton::backup_end(): Invoked at the start or end of a backup phase, in the thread that executes a BACKUP SERVER statement. handlerton::backup_step(): A backup step that can be invoked from multiple threads concurrently, between the execution of the corresponding handlerton::backup_start() and handlerton::backup_end() of the same phase. copy_entire_file(): A file copying service for POSIX systems. copy_mmap(): A zero-copy alternative to backup::copy(), to copy from a memory-mapped buffer. copy_file_range_try(): A wrapper for Linux copy_file_range(2), which may fail with EOPNOTSUPP or EXDEV and thus require a fallback to copy_mmap() or backup::copy(). backup::copy(): A partial or sparse file-copying service. On other platforms than FreeBSD or Microsoft Windows, there are shortcut alternatives to this. Note: On Linux we never invoke sendfile(2) for copying between files, because can be much slower than the alternatives. backup_stream_append_plain(): A wrapper of backup::append(), which is the streaming equivalent of backup::copy(). backup_stream_zeropad(): Zero-pad the last tar block if needed. backup_stream_append_async(): A variant of backup_stream_append_plain() where the source file region is guaranteed to be immutable after the call returns. Zero-copy mmap(2) or Linux sendfile(2) are inherently risky for copying data files that may be modified in place, because it could introduce a race condition between a page write that runs concurrently with a child process that is reading the data from the pipe. backup::append(): On systems where we can determine the size of the pipe buffer, invoke backup_stream_append_async() for the initial write, and pread_write() for the last part, to guarantee that the data written by the zero-copy shortcut will have been consumed before the call returns and the caller is able to resume writes to the source region. pread_write(): On 64-bit systems, allocate a buffer of up to 1 MiB. This is the "slow path" of copying or streaming files. struct Aria_backup: Context for multi-threaded backup, comprising directory handles, a mutex and enum Aria_backup_status. aria_backup_start(): Prepare the context for aria_backup_step(). Most files are copied in BACKUP_PHASE_NO_DDL after flush_tables(thd, FLUSH_NON_TRANS_TABLES) has been invoked. All ENGINE=Aria files (including TRANSACTIONAL=0) are copied in BACKUP_PHASE_NO_COMMIT. Thanks to Andrzej Jarząbek for writing test cases and suggesting this logic. aria_backup_step(): Copy one non-ACID file. Acquires Aria_backup::mutex, traverses directories to construct one file name, releases the mutex, and copies the file if one was found. aria_backup_data(): Copy one data file. On Microsoft Windows, this assumes that the current directory is the datadir. This assumption would not hold in the Embedded Server library, which is not supported on Microsoft Windows. aria_backup_log(): Copy one ENGINE=Aria log file. aria_backup_end(): Finish a copying phase and clean up the context. InnoDB_backup::init(): Wait for a possible previous BACKUP SERVER operation to reach the very end of InnoDB_backup::context::cleanup() so that the context can be safely reused. InnoDB_backup::queue: Collection of tablespace IDs and payload sizes at the start of the backup, and the log_sys.first_lsn of log files that have to be included in the backup. If any data file is created or extended while the backup is executing, we must have the corresponding write-ahead-log entries that we are copying since the latest checkpoint that was completed when the backup started. If any tablespaces are deleted during the backup, we may or may not copy them, and the application of a FILE_DELETE record will remove them. Similarly, applying FILE_RENAME or FILE_CREATE records will rename or create files during recovery as needed. log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this is to make BACKUP SERVER prevent the concurrent execution of SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size when innodb_log_archive=OFF. log_sys.archived_checkpoint: Keep track of the earliest available checkpoint, corresponding to log_sys.archived_lsn. This reflects SET GLOBAL innodb_log_recovery_start (which is settable now), for incremental backup. fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed to be known. To speed up startup, InnoDB does not normally open all tablespace files. fil_space_t::create_lsn: Change to Atomic_relaxed and use this to indicate tablespace creation LSN, in addition to indicate undo tablespace rebuild LSN. fil_space_t::backup_end: The first page number that is not being backed up (by default 0, to indicate that no backup is in progress). fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be covered by fil_space_t::backup_end. This is the unit of "page range locking" during InnoDB backup. buf_page_t::write_fix_try(), buf_page_t::write_unfix_try(): Try to set or unset a fake "write fix" on a page, to prevent concurrent flush() during a backup batch. The atomic operations may run concurrently with set_reinit() and set_freed(). The fake "write fix" does not prevent any concurrent read or write of the page data in the buffer pool; it only blocks writes to the underlying data file. buf_page_t::flush(): Atomically test and set write fix, and skip the operation if the fake "write fix" was set. buf_page_t::set_freed(), buf_page_t::set_reinit(): Employ a compare-and-exchange loop to accommodate for the "write fix". innodb_backup_batch_wait(): Look up any pages that we are about to back up. For any dirty pages, invoke buf_page_t::write_fix_try() to try to set a fake "write fix" lock-free. If the page is currently write-fixed between buf_page_t::flush() and buf_page_t::write_complete(), acquire and release a page U-latch to wait for the conflicting write to complete. InnoDB_backup::backup_batch_start(), InnoDB_backup::backup_batch_stop(): Adjust fil_space_t::backup_end and fake "write fix" of dirty pages to protect the copying of a range of pages from the underlying file. InnoDB_backup::commit(): Enqueue the remaining log to be copied. InnoDB_backup::checkpoint_complete(): If backup is running and commit() has not been called, add each completed innodb_archive_log=ON file to the queue. Else, skip or delete, as appropriate. log_t::backup_start(): If we were running with innodb_log_archive=ON, ensure that the latest file is a valid recovery starting point. That is, wait for the latest log checkpoint to be within the file. buf_flush_list_space(): Check for concurrent backup before writing each page. This is inefficient, but this function may be invoked from multiple threads concurrently, and it cannot be changed easily, especially for fil_crypt_thread(). fil_ibd_create(): Set fil_space_t::create_lsn after the file has been created. dict_load_tablespaces(): Determine the size of each file if upgrade==true. Backup depends on that. buf_dblwr_t::begin(), buf_dblwr_t::end(), buf_dblwr_t::size(): Accessors to allow BACKUP SERVER to skip the contents of the doublewrite buffer in the system tablespace. It is only useful for crash recovery in case a data page had been incompletely written by the time the server was killed. If the server is killed during a backup, the backup will be incomplete and unusable anyway. Furthermore, the page range locking makes page writes and backup mutually exclusive. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Merge 13.1 into main | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
KhaledR57
khaled57.dev@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-37167 Nested BEGINs (4600+) cause a segmentation fault Each nested BEGIN adds one sp_pcontext. Two walks over the finished tree recursed once per nesting level and overran the thread stack. ~sp_pcontext() freed its children recursively. sp_head now threads a single linked list through the contexts and frees them iteratively, so teardown depth is constant and a context no longer frees its children. retrieve_field_definitions() descends the children to build the run-time frame. It emits them in run-time offset order, so it stays recursive, but it now checks the stack and returns an error instead of crashing. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Georgi (Joro) Kodinov
joro@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Right now test moxygen docs generation triggers on every pull request regardless of what is changed. There's no need for that. It's enough to run it only when something in include/mysql changes. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41072 add SBOM author/tool metadata Set SBOM author to "MariaDB Plc" and add metadata.tools.components describing the generator (generate_sbom.cmake) per CISA 2026 minimum elements guidance. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39868 Wrong result with a window fn over merged derived table column Problem: ======== A query with a window function over a column of a merged derived table returns an empty set when another table is joined on a condition over the same column and is accessed with "Range checked for each record": SELECT AVG(subq.c2) OVER (), t2.c1 FROM t1 LEFT JOIN (SELECT * FROM t3) AS subq ON t1.c1 = subq.c1 STRAIGHT_JOIN t2 ON subq.c2 > t2.c1; With derived_merge=on all the references to subq.c2 are Item_direct_view_ref objects sharing one underlying Item_field, because their ref pointers all point into the derived table's field_translation. Item::split_sum_func2() calls real_item() and puts that shared Item_field into the list of the window function's temporary table fields, so create_tmp_field_from_item_field() sets its result_field to a column of the temporary table. Item_field::val_int() reads field, but Item_field::save_in_field() reads result_field, so the two now return different values. The join condition is evaluated through the same Item_field, and the runtime range analysis in Field::get_mm_leaf_int() uses save_in_field_no_warnings(). It reads the still empty temporary table column instead of the value of t3.c2, treats the value as NULL, and builds a SEL_TREE::IMPOSSIBLE. Table t2 then produces no rows. Solution: ========= Do not unwrap Item_direct_view_ref in Item::split_sum_func2(). The wrapper is created per reference and is not shared, so the temporary table field is attached to the wrapper alone and the conditions that refer to the same view column keep reading the base table field. Item_ref::create_tmp_field_ex() already creates the same temporary table field for a view ref over a column, and change_to_use_tmp_fields() already handles REF_ITEM, so no other change is needed. Ref access was never affected: get_store_key() takes real_item()->field explicitly. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Jan Lindström
jan.lindstrom@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41028 : Galera appliers deadlock on a foreign key referencing a CHAR column in a multi-byte character set The write set key of a row is built from the MySQL record by wsrep_store_key_val_for_row(), and the key of a foreign key parent row from the InnoDB record by wsrep_rec_get_foreign_key(). A CHAR is not padded the same way in the two formats: the MySQL record pads it to n_chars * mbmaxlen bytes, while InnoDB strips that padding down to, but not below, n_chars bytes. That compares a byte count with a character count, so a value holding a multi byte character and shorter than the column was left with a different number of characters on the two paths, and the keys differed. A child INSERT then had no dependency on its parent row and the appliers ran it in parallel with a change of that very row. The two paths did not agree on the strnxfrm buffer length either, 3072 on one and 3500 on the other. Both now go through wsrep_store_string_key_val(), which brings a CHAR to exactly the number of characters the column holds and always normalizes with WSREP_MAX_SUPPORTED_KEY_LENGTH, so that the key of a column does not depend on how much room the columns before it happened to leave. Only the copy into the caller's buffer is bounded by the space that is left, which also stops wsrep_rec_get_foreign_key() from writing past its key buffer. This changes the write set keys, so it is done from protocol version 5 on and the old encoding is kept below that. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Georgi (Joro) Kodinov
joro@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Right now test moxygen docs generation triggers on every pull request regardless of what is changed. There's no need for that. It's enough to run it only when something in include/mysql changes. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
In Parallel_coordinator, remove partition_id() and m_partition_id. MariaDB's innodb doesn't have those. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41012 Galera appliers hang with foreign key of types UUID, INET4, INET6 Added second test for testing key collisions from transactions modifying separate rows |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40168 Factor out the MVI array walk Item_func_mvi_encode::val_str_ascii() and collect_mvi_keys() walked a JSON array the same way: descend into a nested array, skip an object, encode everything else with encode_mvi_key(). The two copies had to agree for the index to answer correctly, and they had already drifted once. Move the walk into walk_mvi_json_array() and give the callers a visitor each: Mvi_key_appender, which joins the keys with a space into the fulltext document of a row, and Mvi_key_collector, which turns them into the keys of an Mvi_access and decides when a missing key means we have to give up on the index. The walk reports why it stopped, so the encoding side can still tell a malformed document from one that is not an array, and the query side can pick the scalar literal out of MVI_WALK_NOT_ARRAY. No functional changes. Co-Authored-By: Claude Opus 5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Remove Parallel_coordinator::Exec_ctx::m_id, it is not used anywhere. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| new columnstore (df4f261f1f622adb2c266527e3fd3c8c4ddaa636 25.10.7) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Arcadiy Ivanov
arcadiy@ivanov.biz |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Fix the `HA_NO_KEY_READ` blob key guard `HA_NO_KEY_READ` marks a key whose blob segment `heap_prepare_hp_create_info()` converted from the VARTEXT2 form, so that `heap_rkey()` refuses an index read on it. It never worked, because the mark was written to the wrong structure member. `heap_rkey()` tests `HP_KEYDEF::flag`, which is where the flag belongs: `HA_NO_KEY_READ` is declared among the key flags, not the key-seg flags. The assignment instead targeted `HA_KEYSEG::flag`, which no reader consults for this flag, so the guard could never fire. That member is also `uint16`, so bit 20 was discarded on assignment as well; `-Wall -Wextra` does not warn, only `-Wconversion` does, and it is not enabled. Write the flag to `keydef[key].flag` instead. `HP_KEYDEF::flag` is `uint` and holds bit 20, and `heap_create()` copies it into the share that `heap_rkey()` reads. `hp_test_key_setup-t` covers the marking, the unmarked case, that `heap_create()` does not lose the flag while folding its own bits into `keydef->flag`, and that `heap_rkey()` refuses a marked key while accepting an unmarked one. The last pair clears `my_assert` so the guard reports instead of aborting, the same way the server's `--debug-assert=0` does, and skips on builds without `DBUG_ASSERT`. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40168 Make an over-long key a prefix key instead of no key A key image that does not fit in a fulltext token was rejected, which costs the index for that value entirely: no key in the document, and an OVERLAPS that mentions the value gives up on the index altogether. Cut the image down to MVI_KEY_IMAGE_MAX_LEN instead. Two values that agree on that many bytes then share a key, which costs false positives and nothing else, since the predicate is rechecked on every row the index produces. This is what the non-binary path has always done -- strnxfrm() is asked for exactly that many bytes of weights and cannot return more -- so it makes the binary path, which is the one a JSON column takes, behave the same. No wildcard is needed in the fulltext query for this. Both sides of the index cut at the same point, so the key a search builds for a long value is the same string as the token the document has for it, and an exact term match finds it. A trailing '*' would only widen the term to keys that are longer than the one searched for, and after the cut there are none. Co-Authored-By: Claude Opus 5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40756 Incorrect multi-batch recovery of file size file_name_t::page0_lsn: Keep track of the last applied recv_sys_t::parse_page0() so that a multi-batch recovery will not reset the file to a smaller size. Reviewed by: Thirunarayanan Balathandayuthapani (cherry picked from commit 8f00e6caca633c783140db86d3a48a96de67cf38) |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39868 Wrong result with a window fn over merged derived table column Problem: ======== A query with a window function over a column of a merged derived table returns an empty set when another table is joined on a condition over the same column and is accessed with "Range checked for each record": SELECT AVG(subq.c2) OVER (), t2.c1 FROM t1 LEFT JOIN (SELECT * FROM t3) AS subq ON t1.c1 = subq.c1 STRAIGHT_JOIN t2 ON subq.c2 > t2.c1; With derived_merge=on all the references to subq.c2 are Item_direct_view_ref objects sharing one underlying Item_field, because their ref pointers all point into the derived table's field_translation. Item::split_sum_func2() calls real_item() and puts that shared Item_field into the list of the window function's temporary table fields, so create_tmp_field_from_item_field() sets its result_field to a column of the temporary table. Item_field::val_int() reads field, but Item_field::save_in_field() reads result_field, so the two now return different values. The join condition is evaluated through the same Item_field, and the runtime range analysis in Field::get_mm_leaf_int() uses save_in_field_no_warnings(). It reads the still empty temporary table column instead of the value of t3.c2, treats the value as NULL, and builds a SEL_TREE::IMPOSSIBLE. Table t2 then produces no rows. Solution: ========= Do not unwrap Item_direct_view_ref in Item::split_sum_func2(). The wrapper is created per reference and is not shared, so the temporary table field is attached to the wrapper alone and the conditions that refer to the same view column keep reading the base table field. Item_ref::create_tmp_field_ex() already creates the same temporary table field for a view ref over a column, and change_to_use_tmp_fields() already handles REF_ITEM, so no other change is needed. Ref access was never affected: get_store_key() takes real_item()->field explicitly. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40168 Make an over-long key a prefix key instead of no key A key image that does not fit in a fulltext token was rejected, which costs the index for that value entirely: no key in the document, and an OVERLAPS that mentions the value gives up on the index altogether. Cut the image down to MVI_KEY_IMAGE_MAX_LEN instead. Two values that agree on that many bytes then share a key, which costs false positives and nothing else, since the predicate is rechecked on every row the index produces. This is what the non-binary path has always done -- strnxfrm() is asked for exactly that many bytes of weights and cannot return more -- so it makes the binary path, which is the one a JSON column takes, behave the same. No wildcard is needed in the fulltext query for this. Both sides of the index cut at the same point, so the key a search builds for a long value is the same string as the token the document has for it, and an exact term match finds it. A trailing '*' would only widen the term to keys that are longer than the one searched for, and after the cut there are none. Co-Authored-By: Claude Opus 5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Remove Parallel_coordinator::Exec_ctx::m_id, it is not used anywhere. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Merge branch '11.4' into 11.8 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alessandro Vetere
iminelink@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-32286 Reuse remembered clustered leaves in secondary-index scans Row_sel_get_clust_rec_for_mysql::operator() descends the clustered B-tree from the root for every row whose clustered-index record a secondary-index scan must read, although consecutive rows land on the same clustered leaf page wherever the secondary order tracks the clustered one. A non-covering scan needs one for every row, a locking read needs one whatever the secondary index holds, because an exclusive select lock type makes ha_innobase::build_template() build its template against the clustered index, and a covering scan needs one for every row of a secondary leaf whose PAGE_MAX_TRX_ID its read view cannot see. ANALYZE FORMAT=JSON charges each descent its full height, and those descents are nearly the whole cost: the secondary index is charged its own descent and one page for each further leaf, and nothing per row, because the position that its cursor holds between two rows is restored optimistically, which latches the leaf again without counting an access. So pages_accessed is the row count times the height of the clustered index, plus a handful: 1000 rows over a 2-level clustered index cost 2006 and 750 rows over a 3-level one cost 2291, where a full table scan of the same data costs 23 and 110. Let a handle remember the clustered leaves that the lookups of one statement reached, and let the next lookup try them before it descends again. The reasoning behind each value and each rejection is in the comments beside it. row0mysql.h defines clust_leaf_hint_slot, which names one leaf: its page number, copies of its first and last user record truncated to the key fields, which bound the key range that the leaf held when it was remembered, the rec_get_offsets() of both, and the dict_index_t::n_core_fields that the copies were made under. A slot of a leaf that had no right sibling names no last key, because every key above the last record of the rightmost leaf still belongs to it. CLUST_LEAF_HINT_SLOTS (4) slots hang off the new row_prebuilt_t::clust_leaf_hint, beside clust_leaf_hint_mru, the most recently used order held as slot numbers, and clust_leaf_hint_n and clust_leaf_hint_miss, the used-slot count and the miss counter. row0sel.cc holds the policy. row_sel_clust_leaf_hint_covers() compares a key against the remembered ranges, so a lookup that no slot can answer costs no buffer pool access and no pages_accessed. row_sel_clust_leaf_hint_search() probes the first slot that covers the key and moves it to the front of the order. row_sel_clust_leaf_hint_remember() records the leaf that a descent landed on, and refreshes the slot of a leaf that is remembered already rather than spend a second one on the same page. Two descents fill no slot: the first CLUST_LEAF_HINT_MIN_LOOKUPS (4) lookups of a statement, and a leaf that is the root. row_sel_clust_leaf_hint_armed() stands a scan down once the slots stop paying for themselves: a miss adds CLUST_LEAF_HINT_MISS_WEIGHT (2) to the miss counter and a hit takes one away, so a scan gives the slots up where it answers too little of its lookups to pay for them, CLUST_LEAF_HINT_MAX_MISSES (8) misses with no hit between them still reach the threshold, and one lookup in CLUST_LEAF_HINT_RETRY (1024) starts the count again, so a scan whose order becomes correlated only later recovers. Both halves of the cost stop there, the test of the slots and the copies that refresh them. Row_sel_get_clust_rec_for_mysql::operator() calls all of this in place of its btr_pcur_open_with_no_init(), and only where the adaptive hash index is disabled, whose guess solves the same problem better: it lands on the record with no page-local search and no page access to charge. That index is off by default, so the hints are active in a default configuration. btr0cur.h and btr0cur.cc add btr_cur_t::try_leaf_hint(), a PAGE_CUR_LE, BTR_SEARCH_LEAF search on one named leaf. It acquires the page with buf_page_try_get(): a hint is never derived from a latched parent page, so by the time it is tried it can precede the caller's already-latched secondary-index leaf in the latching order, where a blocking wait can deadlock. It then rejects the page unless the checks that it makes on the latched frame put the match on it. Those checks are the sole authority on the result, so a stale range costs a wasted probe or a needless descent, never a wrong result, and the ranges need no invalidation protocol. ha_innodb.cc: ha_innobase::reset() zeroes the used-slot count and the miss counter per statement, matching autoinc_last_value. row0mysql.cc: row_prebuilt_free() frees the key buffers that the slots own. innodb.clust_leaf_hint measures pages_accessed over key orders that differ in how closely the secondary order tracks the clustered one, and eight further tables check query results over the record formats and key shapes that a clustered-index lookup has to read, down to the metadata pseudo-record of instant ALTER TABLE, to leaves that split and merge while a locking read walks them, and to a record that a remembered leaf supplies for a scan that must then rebuild an older version of it. Two of the tables scan a covering index, which reads a clustered record under an exclusive select lock type and under a PAGE_MAX_TRX_ID that the read view cannot see. clust_leaf_hint_off_debug runs the same body with the hints turned off, through a debug switch that returns before a lookup tests or refreshes the slots, so a diff of the two .result files is what the hints save: 2006 to 1031 (2-level clustered index), 2291 to 1011 (3-level), 4006 to 2015 (two interleaved key ranges), 12016 to 9078 (locality in the second half alone) and 20020 to 10045 for a covering scan that FOR UPDATE makes non-covering, where the same scan without FOR UPDATE costs 20 in both files. Two orders with too little locality to pay for the slots give them up early and end within a hundred accesses of the unhinted count: 20020 to 19966 (decorrelated) and 20020 to 19999 (shuffled). innodb.clust_leaf_hint_instant_alter covers the one rejection that no count reaches, of a slot whose keys were copied under another dict_index_t::n_core_fields than the index reports. dict_index_t::clear_instant_alter() is the only writer of that value that a shared metadata lock allows, and it needs the clustered index to lose the last user record of its root page, while no leaf that is the root fills a slot, so the tree has to shrink between the two, which purge does there. The reader therefore reads uncommitted rows at READ UNCOMMITTED and waits in a stored function while a rollback and purge take them away, and one row that arrives above the position it stopped at is the lookup that tests the slots. The rejection leaves nothing that a query can read, so that branch writes the two counts to the error log under a debug switch and the case reads them back with search_pattern_in_file.inc. They are printed and not named in the pattern, so that a run which reaches the branch with other counts, or in the direction where the clear lowers them, is a difference to look at and not a pass. main.rowid_filter_innodb: 90 to 88, and its ahi combination unchanged. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41012 Galera appliers hang with foreign key of types UUID, INET4, INET6 wsrep_store_key_val_for_row() built the certification key of a row by collating the column value whenever the field reports MYSQL_TYPE_STRING or MYSQL_TYPE_VAR_STRING, taking the collation from Field::charset(). The data types implemented on Field_fbt - UUID, INET6 and INET4 report MYSQL_TYPE_STRING, and their charset() is my_charset_numeric, which is latin1. Their values are however plain binary and accordingly get_innobase_type_from_mysql_type() maps them to DATA_FIXBINARY. Their keys were therefore run through latin1_swedish_ci, which folds them. That corrupts the key in two ways: 1. A key mismatch for one and the same row. The reference key that wsrep_rec_get_foreign_key() appends for the parent of a child INSERT is built from the InnoDB record and is not collated, so it no longer matched the primary key carried by the parent row's own writeset. Certification saw no dependency between a child INSERT and a concurrent parent UPDATE, and two appliers could apply them in parallel causing a hang or crash. 2. A key collision between distinct rows. The folding is many to one, so different values collapse onto one key, Certification compares keys byte for byte, so unrelated rows were treated as the same row. Concurrent transactions on them certified as a conflict and one was aborted with ER_LOCK_DEADLOCK. Fix is for wsrep_store_key_val_for_row() to skip the collation for fields that InnoDB stores as binary, using the same condition as get_innobase_type_from_mysql_type(). This is a no-op for the types that worked before. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alessandro Vetere
iminelink@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-32286 Reuse remembered clustered leaves in secondary-index scans Row_sel_get_clust_rec_for_mysql::operator() descends the clustered B-tree from the root for every row whose clustered-index record a secondary-index scan must read, although consecutive rows land on the same clustered leaf page wherever the secondary order tracks the clustered one. A non-covering scan needs one for every row, a locking read needs one whatever the secondary index holds, because an exclusive select lock type makes ha_innobase::build_template() build its template against the clustered index, and a covering scan needs one for every row of a secondary leaf whose PAGE_MAX_TRX_ID its read view cannot see. ANALYZE FORMAT=JSON charges each descent its full height, and those descents are nearly the whole cost: the secondary index is charged its own descent and one page for each further leaf, and nothing per row, because the position that its cursor holds between two rows is restored optimistically, which latches the leaf again without counting an access. So pages_accessed is the row count times the height of the clustered index, plus a handful: 1000 rows over a 2-level clustered index cost 2006 and 750 rows over a 3-level one cost 2291, where a full table scan of the same data costs 23 and 110. Let a handle remember the clustered leaves that the lookups of one statement reached, and let the next lookup try them before it descends again. The reasoning behind each value and each rejection is in the comments beside it. row0mysql.h defines clust_leaf_hint_slot, which names one leaf: its page number, copies of its first and last user record truncated to the key fields, which bound the key range that the leaf held when it was remembered, the rec_get_offsets() of both, and the dict_index_t::n_core_fields that the copies were made under. A slot of a leaf that had no right sibling names no last key, because every key above the last record of the rightmost leaf still belongs to it. CLUST_LEAF_HINT_SLOTS (4) slots hang off the new row_prebuilt_t::clust_leaf_hint, beside clust_leaf_hint_mru, the most recently used order held as slot numbers, and clust_leaf_hint_n and clust_leaf_hint_miss, the used-slot count and the miss counter. row0sel.cc holds the policy. row_sel_clust_leaf_hint_covers() compares a key against the remembered ranges, so a lookup that no slot can answer costs no buffer pool access and no pages_accessed. row_sel_clust_leaf_hint_search() probes the first slot that covers the key and moves it to the front of the order. row_sel_clust_leaf_hint_remember() records the leaf that a descent landed on, and refreshes the slot of a leaf that is remembered already rather than spend a second one on the same page. Two descents fill no slot: the first CLUST_LEAF_HINT_MIN_LOOKUPS (4) lookups of a statement, and a leaf that is the root. row_sel_clust_leaf_hint_armed() stands a scan down once the slots stop paying for themselves: a miss adds CLUST_LEAF_HINT_MISS_WEIGHT (2) to the miss counter and a hit takes one away, so a scan gives the slots up where it answers too little of its lookups to pay for them, CLUST_LEAF_HINT_MAX_MISSES (8) misses with no hit between them still reach the threshold, and one lookup in CLUST_LEAF_HINT_RETRY (1024) starts the count again, so a scan whose order becomes correlated only later recovers. Both halves of the cost stop there, the test of the slots and the copies that refresh them. Row_sel_get_clust_rec_for_mysql::operator() calls all of this in place of its btr_pcur_open_with_no_init(), and only where the adaptive hash index is disabled, whose guess solves the same problem better: it lands on the record with no page-local search and no page access to charge. That index is off by default, so the hints are active in a default configuration. btr0cur.h and btr0cur.cc add btr_cur_t::try_leaf_hint(), a PAGE_CUR_LE, BTR_SEARCH_LEAF search on one named leaf. It acquires the page with buf_page_try_get(): a hint is never derived from a latched parent page, so by the time it is tried it can precede the caller's already-latched secondary-index leaf in the latching order, where a blocking wait can deadlock. It then rejects the page unless the checks that it makes on the latched frame put the match on it. Those checks are the sole authority on the result, so a stale range costs a wasted probe or a needless descent, never a wrong result, and the ranges need no invalidation protocol. ha_innodb.cc: ha_innobase::reset() zeroes the used-slot count and the miss counter per statement, matching autoinc_last_value. row0mysql.cc: row_prebuilt_free() frees the key buffers that the slots own. innodb.clust_leaf_hint measures pages_accessed over key orders that differ in how closely the secondary order tracks the clustered one, and eight further tables check query results over the record formats and key shapes that a clustered-index lookup has to read, down to the metadata pseudo-record of instant ALTER TABLE, to leaves that split and merge while a locking read walks them, and to a record that a remembered leaf supplies for a scan that must then rebuild an older version of it. Two of the tables scan a covering index, which reads a clustered record under an exclusive select lock type and under a PAGE_MAX_TRX_ID that the read view cannot see. clust_leaf_hint_off_debug runs the same body with the hints turned off, through a debug switch that returns before a lookup tests or refreshes the slots, so a diff of the two .result files is what the hints save: 2006 to 1031 (2-level clustered index), 2291 to 1011 (3-level), 4006 to 2015 (two interleaved key ranges), 12016 to 9078 (locality in the second half alone) and 20020 to 10045 for a covering scan that FOR UPDATE makes non-covering, where the same scan without FOR UPDATE costs 20 in both files. Two orders with too little locality to pay for the slots give them up early and end within a hundred accesses of the unhinted count: 20020 to 19966 (decorrelated) and 20020 to 19999 (shuffled). innodb.clust_leaf_hint_instant_alter covers the one rejection that no count reaches, of a slot whose keys were copied under another dict_index_t::n_core_fields than the index reports. dict_index_t::clear_instant_alter() is the only writer of that value that a shared metadata lock allows, and it needs the clustered index to lose the last user record of its root page, while no leaf that is the root fills a slot, so the tree has to shrink between the two, which purge does there. The reader therefore reads uncommitted rows at READ UNCOMMITTED and waits in a stored function while a rollback and purge take them away, and one row that arrives above the position it stopped at is the lookup that tests the slots. The rejection leaves nothing that a query can read, so that branch writes the two counts to the error log under a debug switch and the case reads them back with search_pattern_in_file.inc. They are printed and not named in the pattern, so that a run which reaches the branch with other counts, or in the direction where the clear lowers them, is a difference to look at and not a pass. main.rowid_filter_innodb: 90 to 88, and its ahi combination unchanged. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
KhaledR57
khaled57.dev@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-37167 Nested BEGINs (4600+) cause a segmentation fault Each nested BEGIN adds one sp_pcontext. Two walks over the finished tree recursed once per nesting level and overran the thread stack. ~sp_pcontext() freed its children recursively. sp_head now threads a single linked list through the contexts and frees them iteratively, so teardown depth is constant and a context no longer frees its children. retrieve_field_definitions() descends the children to build the run-time frame. It emits them in run-time offset order, so it stays recursive, but it now checks the stack and returns an error instead of crashing. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||