Console View
|
Categories: connectors experimental galera main |
|
| connectors | experimental | galera | main | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41012 Galera appliers hang with foreign key of types UUID, INET4, INET6 Bumped application protocol version to level 5 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41021 SET GLOBAL innodb_log_archive ignores read-only mode log_t::set_archive(): Prevent a crash in log_t::header_rewrite() by refusing the operation if the log is read-only. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41012 Galera appliers hang with foreign key of types UUID, INET4, INET6 Added a deterministic test for reproducing the issue |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Kristian Nielsen
knielsen@knielsen-hq.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40745: Validate field parsing of optional_metadata Protect accesses into the m_optional_metadata buffer to not access outside of the buffer in case of corrupt/malicious event data. Protect against buffer overflow of the m_column_name array in case of excessive column names in the event data. Signed-off-by: Kristian Nielsen <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Coding style fixes, remove out-of-date comments | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Merge branch '13.0' into 13.1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-14992/MDEV-38362/MDEV-39092 BACKUP SERVER The following SQL statements will be introduced: BACKUP SERVER TO '/path/to/directory' [ 1 CONCURRENT ]; BACKUP SERVER WITH [ 1 CONCURRENT ] 'command'; In place of the 1, any positive number of threads may be specified. For the first variant, '/path/to' must exist and '/path/to/directory' must not exist; that is where the backup will be written to. For BACKUP SERVER WITH, the string mariadb-backup- will be prepended to the specified command and quoted. Path separators or quotes are not allowed in command. We expect the mariadb-backup-command to be in the search PATH of the mariadbd process. For now, we distribute no such script or program; the user has to write one, something like this: exec zstdmt|ssh [email protected] "exec cat > $1.tar.zstd" The standard input of that command will be in a format compatible with GNU tar --format=oldgnu (and also BSD tar variants that are also part of Microsoft Windows and Apple macOS). The command is expected to optionally compress and encrypt the stream and redirect it to a file on a local or a remote server. The BACKUP SERVER WITH will append an additional argument, a positive base-ten number in ASCII, starting with 1, to identify the current thread. In this way, there can be multiple concurrent streams. The InnoDB write-ahead log will be streamed near the end, because the stream format requires the file sizes to be declared upfront. TODO: implement the following: In BACKUP SERVER TO ... 2 CONCURRENT or more, one thread will be responsible for copying the InnoDB write-ahead log from the beginning of the backup. Other files will be copied by other threads. Note: In single-threaded BACKUP SERVER TO, the parameter innodb_log_recovery_start that is written out to backup.cnf is STRICTLY NECESSARY TO AVOID CORRUPTION during recovery! By default, InnoDB crash recovery starts from the latest available log checkpoint. However, for restoring a backup, recovery must start from the checkpoint that was the latest when the backup was started. Starting recovery from a possible later checkpoint will result in a corrupted database! The backup or the first stream will contain a file backup.cnf, which includes parameters needed for restoring the backup. Currently, these are innodb_log_recovery_start and innodb_log_recovery_target. If innodb_log_recovery_target>0, InnoDB will be in read-only mode, not allowing any writes to persistent files other than via the log application. To restore a streaming backup made with BACKUP SERVER WITH, an empty directory needs to be created and all streams be extracted there using the standard tar utility of the operating system, optionally after undoing any encryption or compression that had been added by the backup command. Then, the backup is prepared or MariaDB server started up on the extracted directory, similar to as if the BACKUP SERVER TO statement had been used. The following will be implemented separately: MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER MDEV-40163 Partial backup and restore MDEV-39091 Back up ENGINE=RocksDB MDEV-40333 Concurrent DDL for Aria tables in BACKUP SERVER The implementation introduces a basic multi-threaded driver Sql_cmd_backup, storage engine interfaces, and basic copying of the storage engines InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV. backup_target: A structured data type to represent a target directory. On Microsoft Windows, we must use directory paths because there is no variant of CopyFileEx() that would work on file handles. backup_sink: Wraps a per-thread output stream as well as storage engine specific context. handlerton::backup_start(), handlerton::backup_end(): Invoked at the start or end of a backup phase, in the thread that executes a BACKUP SERVER statement. handlerton::backup_step(): A backup step that can be invoked from multiple threads concurrently, between the execution of the corresponding handlerton::backup_start() and handlerton::backup_end() of the same phase. copy_entire_file(): A file copying service for POSIX systems. copy_mmap(): A zero-copy alternative to backup::copy(), to copy from a memory-mapped buffer. copy_file_range_try(): A wrapper for Linux copy_file_range(2), which may fail with EOPNOTSUPP or EXDEV and thus require a fallback to copy_mmap() or backup::copy(). backup::copy(): A partial or sparse file-copying service. On other platforms than FreeBSD or Microsoft Windows, there are shortcut alternatives to this. Note: On Linux we never invoke sendfile(2) for copying between files, because can be much slower than the alternatives. backup_stream_append_plain(): A wrapper of backup::append(), which is the streaming equivalent of backup::copy(). backup_stream_zeropad(): Zero-pad the last tar block if needed. backup_stream_append_async(): A variant of backup_stream_append_plain() where the source file region is guaranteed to be immutable after the call returns. Zero-copy mmap(2) or Linux sendfile(2) are inherently risky for copying data files that may be modified in place, because it could introduce a race condition between a page write that runs concurrently with a child process that is reading the data from the pipe. backup::append(): On systems where we can determine the size of the pipe buffer, invoke backup_stream_append_async() for the initial write, and pread_write() for the last part, to guarantee that the data written by the zero-copy shortcut will have been consumed before the call returns and the caller is able to resume writes to the source region. pread_write(): On 64-bit systems, allocate a buffer of up to 1 MiB. This is the "slow path" of copying or streaming files. struct Aria_backup: Context for multi-threaded backup, comprising directory handles, a mutex and enum Aria_backup_status. aria_backup_start(): Prepare the context for aria_backup_step(). Most files are copied in BACKUP_PHASE_NO_DDL after flush_tables(thd, FLUSH_NON_TRANS_TABLES) has been invoked. All ENGINE=Aria files (including TRANSACTIONAL=0) are copied in BACKUP_PHASE_NO_COMMIT. Thanks to Andrzej Jarząbek for writing test cases and suggesting this logic. aria_backup_step(): Copy one non-ACID file. Acquires Aria_backup::mutex, traverses directories to construct one file name, releases the mutex, and copies the file if one was found. aria_backup_data(): Copy one data file. On Microsoft Windows, this assumes that the current directory is the datadir. This assumption would not hold in the Embedded Server library, which is not supported on Microsoft Windows. aria_backup_log(): Copy one ENGINE=Aria log file. aria_backup_end(): Finish a copying phase and clean up the context. InnoDB_backup::init(): Wait for a possible previous BACKUP SERVER operation to reach the very end of InnoDB_backup::context::cleanup() so that the context can be safely reused. InnoDB_backup::queue: Collection of tablespace IDs and payload sizes at the start of the backup, and the log_sys.first_lsn of log files that have to be included in the backup. If any data file is created or extended while the backup is executing, we must have the corresponding write-ahead-log entries that we are copying since the latest checkpoint that was completed when the backup started. If any tablespaces are deleted during the backup, we may or may not copy them, and the application of a FILE_DELETE record will remove them. Similarly, applying FILE_RENAME or FILE_CREATE records will rename or create files during recovery as needed. log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this is to make BACKUP SERVER prevent the concurrent execution of SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size when innodb_log_archive=OFF. log_sys.archived_checkpoint: Keep track of the earliest available checkpoint, corresponding to log_sys.archived_lsn. This reflects SET GLOBAL innodb_log_recovery_start (which is settable now), for incremental backup. fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed to be known. To speed up startup, InnoDB does not normally open all tablespace files. fil_space_t::create_lsn: Change to Atomic_relaxed and use this to indicate tablespace creation LSN, in addition to indicate undo tablespace rebuild LSN. fil_space_t::backup_end: The first page number that is not being backed up (by default 0, to indicate that no backup is in progress). fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be covered by fil_space_t::backup_end. This is the unit of "page range locking" during InnoDB backup. buf_page_t::write_fix_try(), buf_page_t::write_unfix_try(): Try to set or unset a fake "write fix" on a page, to prevent concurrent flush() during a backup batch. The atomic operations may run concurrently with set_reinit() and set_freed(). The fake "write fix" does not prevent any concurrent read or write of the page data in the buffer pool; it only blocks writes to the underlying data file. buf_page_t::flush(): Atomically test and set write fix, and skip the operation if the fake "write fix" was set. buf_page_t::set_freed(), buf_page_t::set_reinit(): Employ a compare-and-exchange loop to accommodate for the "write fix". innodb_backup_batch_wait(): Look up any pages that we are about to back up. For any dirty pages, invoke buf_page_t::write_fix_try() to try to set a fake "write fix" lock-free. If the page is currently write-fixed between buf_page_t::flush() and buf_page_t::write_complete(), acquire and release a page U-latch to wait for the conflicting write to complete. InnoDB_backup::backup_batch_start(), InnoDB_backup::backup_batch_stop(): Adjust fil_space_t::backup_end and fake "write fix" of dirty pages to protect the copying of a range of pages from the underlying file. InnoDB_backup::commit(): Enqueue the remaining log to be copied. InnoDB_backup::checkpoint_complete(): If backup is running and commit() has not been called, add each completed innodb_archive_log=ON file to the queue. Else, skip or delete, as appropriate. log_t::backup_start(): If we were running with innodb_log_archive=ON, ensure that the latest file is a valid recovery starting point. That is, wait for the latest log checkpoint to be within the file. buf_flush_list_space(): Check for concurrent backup before writing each page. This is inefficient, but this function may be invoked from multiple threads concurrently, and it cannot be changed easily, especially for fil_crypt_thread(). fil_ibd_create(): Set fil_space_t::create_lsn after the file has been created. dict_load_tablespaces(): Determine the size of each file if upgrade==true. Backup depends on that. buf_dblwr_t::begin(), buf_dblwr_t::end(), buf_dblwr_t::size(): Accessors to allow BACKUP SERVER to skip the contents of the doublewrite buffer in the system tablespace. It is only useful for crash recovery in case a data page had been incompletely written by the time the server was killed. If the server is killed during a backup, the backup will be incomplete and unusable anyway. Furthermore, the page range locking makes page writes and backup mutually exclusive. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Merge 13.1 into main | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
KhaledR57
khaled57.dev@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-37167 Nested BEGINs (4600+) cause a segmentation fault Each nested BEGIN adds one sp_pcontext. Two walks over the finished tree recursed once per nesting level and overran the thread stack. ~sp_pcontext() freed its children recursively. sp_head now threads a single linked list through the contexts and frees them iteratively, so teardown depth is constant and a context no longer frees its children. retrieve_field_definitions() descends the children to build the run-time frame. It emits them in run-time offset order, so it stays recursive, but it now checks the stack and returns an error instead of crashing. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alessandro Vetere
iminelink@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-32286 Reuse remembered clustered leaves in secondary-index scans Row_sel_get_clust_rec_for_mysql::operator() descends the clustered B-tree from the root for every row whose clustered-index record a secondary-index scan must read, although consecutive rows land on the same clustered leaf page wherever the secondary order tracks the clustered one. A non-covering scan needs one for every row, a locking read needs one whatever the secondary index holds, because an exclusive select lock type makes ha_innobase::build_template() build its template against the clustered index, and a covering scan needs one for every row of a secondary leaf whose PAGE_MAX_TRX_ID its read view cannot see. ANALYZE FORMAT=JSON charges each descent its full height, and those descents are nearly the whole cost: the secondary index is charged its own descent and one page for each further leaf, and nothing per row, because the position that its cursor holds between two rows is restored optimistically, which latches the leaf again without counting an access. So pages_accessed is the row count times the height of the clustered index, plus a handful: 1000 rows over a 2-level clustered index cost 2006 and 750 rows over a 3-level one cost 2291, where a full table scan of the same data costs 23 and 110. Let a handle remember the clustered leaves that the lookups of one statement reached, and let the next lookup try them before it descends again. The reasoning behind each value and each rejection is in the comments beside it. row0mysql.h defines clust_leaf_hint_slot, which names one leaf: its page number, copies of its first and last user record truncated to the key fields, which bound the key range that the leaf held when it was remembered, the rec_get_offsets() of both, and the dict_index_t::n_core_fields that the copies were made under. A slot of a leaf that had no right sibling names no last key, because every key above the last record of the rightmost leaf still belongs to it. CLUST_LEAF_HINT_SLOTS (4) slots hang off the new row_prebuilt_t::clust_leaf_hint, beside clust_leaf_hint_mru, the most recently used order held as slot numbers, and clust_leaf_hint_n and clust_leaf_hint_miss, the used-slot count and the miss counter. row0sel.cc holds the policy. row_sel_clust_leaf_hint_covers() compares a key against the remembered ranges, so a lookup that no slot can answer costs no buffer pool access and no pages_accessed. row_sel_clust_leaf_hint_search() probes the first slot that covers the key and moves it to the front of the order. row_sel_clust_leaf_hint_remember() records the leaf that a descent landed on, and refreshes the slot of a leaf that is remembered already rather than spend a second one on the same page. Two descents fill no slot: the first CLUST_LEAF_HINT_MIN_LOOKUPS (4) lookups of a statement, and a leaf that is the root. row_sel_clust_leaf_hint_armed() stands a scan down once the slots stop paying for themselves: a miss adds CLUST_LEAF_HINT_MISS_WEIGHT (2) to the miss counter and a hit takes one away, so a scan gives the slots up where it answers too little of its lookups to pay for them, CLUST_LEAF_HINT_MAX_MISSES (8) misses with no hit between them still reach the threshold, and one lookup in CLUST_LEAF_HINT_RETRY (1024) starts the count again, so a scan whose order becomes correlated only later recovers. Both halves of the cost stop there, the test of the slots and the copies that refresh them. Row_sel_get_clust_rec_for_mysql::operator() calls all of this in place of its btr_pcur_open_with_no_init(), and only where the adaptive hash index is disabled, whose guess solves the same problem better: it lands on the record with no page-local search and no page access to charge. That index is off by default, so the hints are active in a default configuration. btr0cur.h and btr0cur.cc add btr_cur_t::try_leaf_hint(), a PAGE_CUR_LE, BTR_SEARCH_LEAF search on one named leaf. It acquires the page with buf_page_try_get(): a hint is never derived from a latched parent page, so by the time it is tried it can precede the caller's already-latched secondary-index leaf in the latching order, where a blocking wait can deadlock. It then rejects the page unless the checks that it makes on the latched frame put the match on it. Those checks are the sole authority on the result, so a stale range costs a wasted probe or a needless descent, never a wrong result, and the ranges need no invalidation protocol. ha_innodb.cc: ha_innobase::reset() zeroes the used-slot count and the miss counter per statement, matching autoinc_last_value. row0mysql.cc: row_prebuilt_free() frees the key buffers that the slots own. innodb.clust_leaf_hint measures pages_accessed over key orders that differ in how closely the secondary order tracks the clustered one, and eight further tables check query results over the record formats and key shapes that a clustered-index lookup has to read, down to the metadata pseudo-record of instant ALTER TABLE, to leaves that split and merge while a locking read walks them, and to a record that a remembered leaf supplies for a scan that must then rebuild an older version of it. Two of the tables scan a covering index, which reads a clustered record under an exclusive select lock type and under a PAGE_MAX_TRX_ID that the read view cannot see. clust_leaf_hint_off_debug runs the same body with the hints turned off, through a debug switch that returns before a lookup tests or refreshes the slots, so a diff of the two .result files is what the hints save: 2006 to 1031 (2-level clustered index), 2291 to 1011 (3-level), 4006 to 2015 (two interleaved key ranges), 12016 to 9078 (locality in the second half alone) and 20020 to 10045 for a covering scan that FOR UPDATE makes non-covering, where the same scan without FOR UPDATE costs 20 in both files. Two orders with too little locality to pay for the slots give them up early and end within a hundred accesses of the unhinted count: 20020 to 19966 (decorrelated) and 20020 to 19999 (shuffled). innodb.clust_leaf_hint_instant_alter covers the one rejection that no count reaches, of a slot whose keys were copied under another dict_index_t::n_core_fields than the index reports. dict_index_t::clear_instant_alter() is the only writer of that value that a shared metadata lock allows, and it needs the clustered index to lose the last user record of its root page, while no leaf that is the root fills a slot, so the tree has to shrink between the two, which purge does there. The reader therefore reads uncommitted rows at READ UNCOMMITTED and waits in a stored function while a rollback and purge take them away, and one row that arrives above the position it stopped at is the lookup that tests the slots. The rejection leaves nothing that a query can read, so that branch writes the two counts to the error log under a debug switch and the case reads them back with search_pattern_in_file.inc. They are printed and not named in the pattern, so that a run which reaches the branch with other counts, or in the direction where the clear lowers them, is a difference to look at and not a pass. main.rowid_filter_innodb: 90 to 88, and its ahi combination unchanged. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| fixup! 69dad73b01a4706718bf7e078f3c861a341110fe | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Georgi (Joro) Kodinov
joro@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Right now test moxygen docs generation triggers on every pull request regardless of what is changed. There's no need for that. It's enough to run it only when something in include/mysql changes. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleg Smirnov
olernov@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| InnoDB: extract pscan_chunk_clamp_t as a separate struct from row_prebuilt_t | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Khaled Riyad
khaled57.dev@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40551 Copy/Paste friendly output format for MariaDB Command Line Client Copy/paste friendly output was only reachable by starting the client with --silent --skip-column-names, which cannot be done from a running interactive session. Add \S, a statement terminator which prints the result of one statement in the tab separated format without column names. com_silent() sets output_plain, opt_silent and column_names around com_go(), then restores them, the same way com_ego() handles vertical. output_plain selects print_tab_data() ahead of the vertical and table branches, so \S gives the same output whether the session was started plainly or with --table, --vertical or --silent. --html and --xml still win, matching \G. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41072 add SBOM author/tool metadata Set SBOM author to "MariaDB Plc" and add metadata.tools.components describing the generator (generate_sbom.cmake) per CISA 2026 minimum elements guidance. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39868 Wrong result with a window fn over merged derived table column Problem: ======== A query with a window function over a column of a merged derived table returns an empty set when another table is joined on a condition over the same column and is accessed with "Range checked for each record": SELECT AVG(subq.c2) OVER (), t2.c1 FROM t1 LEFT JOIN (SELECT * FROM t3) AS subq ON t1.c1 = subq.c1 STRAIGHT_JOIN t2 ON subq.c2 > t2.c1; With derived_merge=on all the references to subq.c2 are Item_direct_view_ref objects sharing one underlying Item_field, because their ref pointers all point into the derived table's field_translation. Item::split_sum_func2() calls real_item() and puts that shared Item_field into the list of the window function's temporary table fields, so create_tmp_field_from_item_field() sets its result_field to a column of the temporary table. Item_field::val_int() reads field, but Item_field::save_in_field() reads result_field, so the two now return different values. The join condition is evaluated through the same Item_field, and the runtime range analysis in Field::get_mm_leaf_int() uses save_in_field_no_warnings(). It reads the still empty temporary table column instead of the value of t3.c2, treats the value as NULL, and builds a SEL_TREE::IMPOSSIBLE. Table t2 then produces no rows. Solution: ========= Do not unwrap Item_direct_view_ref in Item::split_sum_func2(). The wrapper is created per reference and is not shared, so the temporary table field is attached to the wrapper alone and the conditions that refer to the same view column keep reading the base table field. Item_ref::create_tmp_field_ex() already creates the same temporary table field for a view ref over a column, and change_to_use_tmp_fields() already handles REF_ITEM, so no other change is needed. Ref access was never affected: get_store_key() takes real_item()->field explicitly. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Jan Lindström
jan.lindstrom@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41028 : Galera appliers deadlock on a foreign key referencing a CHAR column in a multi-byte character set The write set key of a row is built from the MySQL record by wsrep_store_key_val_for_row(), and the key of a foreign key parent row from the InnoDB record by wsrep_rec_get_foreign_key(). A CHAR is not padded the same way in the two formats: the MySQL record pads it to n_chars * mbmaxlen bytes, while InnoDB strips that padding down to, but not below, n_chars bytes. That compares a byte count with a character count, so a value holding a multi byte character and shorter than the column was left with a different number of characters on the two paths, and the keys differed. A child INSERT then had no dependency on its parent row and the appliers ran it in parallel with a change of that very row. The two paths did not agree on the strnxfrm buffer length either, 3072 on one and 3500 on the other. Both now go through wsrep_store_string_key_val(), which brings a CHAR to exactly the number of characters the column holds and always normalizes with WSREP_MAX_SUPPORTED_KEY_LENGTH, so that the key of a column does not depend on how much room the columns before it happened to leave. Only the copy into the caller's buffer is bounded by the space that is left, which also stops wsrep_rec_get_foreign_key() from writing past its key buffer. This changes the write set keys, so it is done from protocol version 5 on and the old encoding is kept below that. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Georgi (Joro) Kodinov
joro@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Right now test moxygen docs generation triggers on every pull request regardless of what is changed. There's no need for that. It's enough to run it only when something in include/mysql changes. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
In Parallel_coordinator, remove partition_id() and m_partition_id. MariaDB's innodb doesn't have those. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41012 Galera appliers hang with foreign key of types UUID, INET4, INET6 Added second test for testing key collisions from transactions modifying separate rows |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
squash! b387a4a6f9f9b3194a29c1a80c39c983d5dc4fd5 handlerton::backup_file: Check if a file should be included in the backup. Implemented for ENGINE=Aria in maria_backup_file(). aria_backup_start(): Copy the Aria log files (FIXME: currently, single-threaded) backup::copy_or_stream(): Copy or stream a file. backup_context: Process-wide BACKUP SERVER context. Handles the directory traversal and copying of files for built-in storage engines that do not implement this backup interface. backup_target_phase. Wrap backup_context. backup_target_phase::step(), backup_context::step(): Process a file from a directory scan, or by invoking handlerton::backup_step(). |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Remove Parallel_coordinator::Exec_ctx::m_id, it is not used anywhere. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| new columnstore (df4f261f1f622adb2c266527e3fd3c8c4ddaa636 25.10.7) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Arcadiy Ivanov
arcadiy@ivanov.biz |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Fix the `HA_NO_KEY_READ` blob key guard `HA_NO_KEY_READ` marks a key whose blob segment `heap_prepare_hp_create_info()` converted from the VARTEXT2 form, so that `heap_rkey()` refuses an index read on it. It never worked, because the mark was written to the wrong structure member. `heap_rkey()` tests `HP_KEYDEF::flag`, which is where the flag belongs: `HA_NO_KEY_READ` is declared among the key flags, not the key-seg flags. The assignment instead targeted `HA_KEYSEG::flag`, which no reader consults for this flag, so the guard could never fire. That member is also `uint16`, so bit 20 was discarded on assignment as well; `-Wall -Wextra` does not warn, only `-Wconversion` does, and it is not enabled. Write the flag to `keydef[key].flag` instead. `HP_KEYDEF::flag` is `uint` and holds bit 20, and `heap_create()` copies it into the share that `heap_rkey()` reads. `hp_test_key_setup-t` covers the marking, the unmarked case, that `heap_create()` does not lose the flag while folding its own bits into `keydef->flag`, and that `heap_rkey()` refuses a marked key while accepting an unmarked one. The last pair clears `my_assert` so the guard reports instead of aborting, the same way the server's `--debug-assert=0` does, and skips on builds without `DBUG_ASSERT`. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-36166: Accept bracketed points inside MULTIPOINT ST_GEOMFROMTEXT('MULTIPOINT((0 0),(1 1))') returned NULL while ST_GEOMFROMTEXT('MULTIPOINT(0 0,1 1)') returned the geometry. The bracketed spelling is the one the OGC WKT grammar defines. In 06-103r4 section 7.2.2 a <multipoint text> is a list of <point text>, and a <point text> has its own parentheses, the same way a <multilinestring text> is a list of <linestring text>. The bare spelling matches no production in that grammar, so the text MariaDB rejected was the conformant one. The first point now determines which of the two bracketing forms the remaining list elements will use. A mixed list such as MULTIPOINT((0 0),1 1) is an error. The bare form stays accepted because existing data and applications use it. Geometry::create_from_wkt is the single entry into the WKT reader, so ST_MPOINTFROMTEXT and a MULTIPOINT nested in a GEOMETRYCOLLECTION are covered by the same change. Co-Authored-By: Claude Opus 5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-36166: support for notation with brackets inside MULTIPOINT ST_GEOMFROMTEXT('MULTIPOINT((0 0),(1 1))') returned NULL while ST_GEOMFROMTEXT('MULTIPOINT(0 0,1 1)') returned the geometry. The bracketed spelling is the one the OGC WKT grammar defines. In 06-103r4 section 7.2.2 a <multipoint text> is a list of <point text>, and a <point text> has its own parentheses, the same way a <multilinestring text> is a list of <linestring text>. The bare spelling matches no production in that grammar, so the text MariaDB rejected was the conformant one. The first point now fixes which of the two bracketing forms the whole list uses, requiring subsequent points to use the same bracketing. A mixed list such as MULTIPOINT((0 0),1 1) is an error. The bare form stays accepted because existing data and applications use it. Geometry::create_from_wkt is the single entry into the WKT reader, so ST_MPOINTFROMTEXT and a MULTIPOINT nested in a GEOMETRYCOLLECTION are covered by the same change. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40756 Incorrect multi-batch recovery of file size file_name_t::page0_lsn: Keep track of the last applied recv_sys_t::parse_page0() so that a multi-batch recovery will not reset the file to a smaller size. Reviewed by: Thirunarayanan Balathandayuthapani (cherry picked from commit 8f00e6caca633c783140db86d3a48a96de67cf38) |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
ParadoxV5
paradox.ver5@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40996 Support `--sync_with_master 0, $variable` in mysqltest `--sync_with_master` uses `get_string()`, which has `$variable` support, but it only uses the read buffer, which is written with the unexpanded string and not the variable value. Reviewed-by: KhaledR57 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39868 Wrong result with a window fn over merged derived table column Problem: ======== A query with a window function over a column of a merged derived table returns an empty set when another table is joined on a condition over the same column and is accessed with "Range checked for each record": SELECT AVG(subq.c2) OVER (), t2.c1 FROM t1 LEFT JOIN (SELECT * FROM t3) AS subq ON t1.c1 = subq.c1 STRAIGHT_JOIN t2 ON subq.c2 > t2.c1; With derived_merge=on all the references to subq.c2 are Item_direct_view_ref objects sharing one underlying Item_field, because their ref pointers all point into the derived table's field_translation. Item::split_sum_func2() calls real_item() and puts that shared Item_field into the list of the window function's temporary table fields, so create_tmp_field_from_item_field() sets its result_field to a column of the temporary table. Item_field::val_int() reads field, but Item_field::save_in_field() reads result_field, so the two now return different values. The join condition is evaluated through the same Item_field, and the runtime range analysis in Field::get_mm_leaf_int() uses save_in_field_no_warnings(). It reads the still empty temporary table column instead of the value of t3.c2, treats the value as NULL, and builds a SEL_TREE::IMPOSSIBLE. Table t2 then produces no rows. Solution: ========= Do not unwrap Item_direct_view_ref in Item::split_sum_func2(). The wrapper is created per reference and is not shared, so the temporary table field is attached to the wrapper alone and the conditions that refer to the same view column keep reading the base table field. Item_ref::create_tmp_field_ex() already creates the same temporary table field for a view ref over a column, and change_to_use_tmp_fields() already handles REF_ITEM, so no other change is needed. Ref access was never affected: get_store_key() takes real_item()->field explicitly. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Remove Parallel_coordinator::Exec_ctx::m_id, it is not used anywhere. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alessandro Vetere
iminelink@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-32286 Reuse remembered clustered leaves in secondary-index scans Row_sel_get_clust_rec_for_mysql::operator() descends the clustered B-tree from the root for every row whose clustered-index record a secondary-index scan must read, although consecutive rows land on the same clustered leaf page wherever the secondary order tracks the clustered one. A non-covering scan needs one for every row, a locking read needs one whatever the secondary index holds, because an exclusive select lock type makes ha_innobase::build_template() build its template against the clustered index, and a covering scan needs one for every row of a secondary leaf whose PAGE_MAX_TRX_ID its read view cannot see. ANALYZE FORMAT=JSON charges each descent its full height, and those descents are nearly the whole cost: the secondary index is charged its own descent and one page for each further leaf, and nothing per row, because the position that its cursor holds between two rows is restored optimistically, which latches the leaf again without counting an access. So pages_accessed is the row count times the height of the clustered index, plus a handful: 1000 rows over a 2-level clustered index cost 2006 and 750 rows over a 3-level one cost 2291, where a full table scan of the same data costs 23 and 110. Let a handle remember the clustered leaves that the lookups of one statement reached, and let the next lookup try them before it descends again. The reasoning behind each value and each rejection is in the comments beside it. row0mysql.h defines clust_leaf_hint_slot, which names one leaf: its page number, copies of its first and last user record truncated to the key fields, which bound the key range that the leaf held when it was remembered, the rec_get_offsets() of both, and the dict_index_t::n_core_fields that the copies were made under. A slot of a leaf that had no right sibling names no last key, because every key above the last record of the rightmost leaf still belongs to it. CLUST_LEAF_HINT_SLOTS (4) slots hang off the new row_prebuilt_t::clust_leaf_hint, beside clust_leaf_hint_mru, the most recently used order held as slot numbers, and clust_leaf_hint_n and clust_leaf_hint_miss, the used-slot count and the miss counter. row0sel.cc holds the policy. row_sel_clust_leaf_hint_covers() compares a key against the remembered ranges, so a lookup that no slot can answer costs no buffer pool access and no pages_accessed. row_sel_clust_leaf_hint_search() probes the first slot that covers the key and moves it to the front of the order. row_sel_clust_leaf_hint_remember() records the leaf that a descent landed on, and refreshes the slot of a leaf that is remembered already rather than spend a second one on the same page. Two descents fill no slot: the first CLUST_LEAF_HINT_MIN_LOOKUPS (4) lookups of a statement, and a leaf that is the root. row_sel_clust_leaf_hint_armed() stands a scan down once the slots stop paying for themselves: a miss adds CLUST_LEAF_HINT_MISS_WEIGHT (2) to the miss counter and a hit takes one away, so a scan gives the slots up where it answers too little of its lookups to pay for them, CLUST_LEAF_HINT_MAX_MISSES (8) misses with no hit between them still reach the threshold, and one lookup in CLUST_LEAF_HINT_RETRY (1024) starts the count again, so a scan whose order becomes correlated only later recovers. Both halves of the cost stop there, the test of the slots and the copies that refresh them. Row_sel_get_clust_rec_for_mysql::operator() calls all of this in place of its btr_pcur_open_with_no_init(), and only where the adaptive hash index is disabled, whose guess solves the same problem better: it lands on the record with no page-local search and no page access to charge. That index is off by default, so the hints are active in a default configuration. btr0cur.h and btr0cur.cc add btr_cur_t::try_leaf_hint(), a PAGE_CUR_LE, BTR_SEARCH_LEAF search on one named leaf. It acquires the page with buf_page_try_get(): a hint is never derived from a latched parent page, so by the time it is tried it can precede the caller's already-latched secondary-index leaf in the latching order, where a blocking wait can deadlock. It then rejects the page unless the checks that it makes on the latched frame put the match on it. Those checks are the sole authority on the result, so a stale range costs a wasted probe or a needless descent, never a wrong result, and the ranges need no invalidation protocol. ha_innodb.cc: ha_innobase::reset() zeroes the used-slot count and the miss counter per statement, matching autoinc_last_value. row0mysql.cc: row_prebuilt_free() frees the key buffers that the slots own. innodb.clust_leaf_hint measures pages_accessed over key orders that differ in how closely the secondary order tracks the clustered one, and eight further tables check query results over the record formats and key shapes that a clustered-index lookup has to read, down to the metadata pseudo-record of instant ALTER TABLE, to leaves that split and merge while a locking read walks them, and to a record that a remembered leaf supplies for a scan that must then rebuild an older version of it. Two of the tables scan a covering index, which reads a clustered record under an exclusive select lock type and under a PAGE_MAX_TRX_ID that the read view cannot see. clust_leaf_hint_off_debug runs the same body with the hints turned off, through a debug switch that returns before a lookup tests or refreshes the slots, so a diff of the two .result files is what the hints save: 2006 to 1031 (2-level clustered index), 2291 to 1011 (3-level), 4006 to 2015 (two interleaved key ranges), 12016 to 9078 (locality in the second half alone) and 20020 to 10045 for a covering scan that FOR UPDATE makes non-covering, where the same scan without FOR UPDATE costs 20 in both files. Two orders with too little locality to pay for the slots give them up early and end within a hundred accesses of the unhinted count: 20020 to 19966 (decorrelated) and 20020 to 19999 (shuffled). innodb.clust_leaf_hint_instant_alter covers the one rejection that no count reaches, of a slot whose keys were copied under another dict_index_t::n_core_fields than the index reports. dict_index_t::clear_instant_alter() is the only writer of that value that a shared metadata lock allows, and it needs the clustered index to lose the last user record of its root page, while no leaf that is the root fills a slot, so the tree has to shrink between the two, which purge does there. The reader therefore reads uncommitted rows at READ UNCOMMITTED and waits in a stored function while a rollback and purge take them away, and one row that arrives above the position it stopped at is the lookup that tests the slots. The rejection leaves nothing that a query can read, so that branch writes the two counts to the error log under a debug switch and the case reads them back with search_pattern_in_file.inc. They are printed and not named in the pattern, so that a run which reaches the branch with other counts, or in the direction where the clear lowers them, is a difference to look at and not a pass. main.rowid_filter_innodb: 90 to 88, and its ahi combination unchanged. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41012 Galera appliers hang with foreign key of types UUID, INET4, INET6 wsrep_store_key_val_for_row() built the certification key of a row by collating the column value whenever the field reports MYSQL_TYPE_STRING or MYSQL_TYPE_VAR_STRING, taking the collation from Field::charset(). The data types implemented on Field_fbt - UUID, INET6 and INET4 report MYSQL_TYPE_STRING, and their charset() is my_charset_numeric, which is latin1. Their values are however plain binary and accordingly get_innobase_type_from_mysql_type() maps them to DATA_FIXBINARY. Their keys were therefore run through latin1_swedish_ci, which folds them. That corrupts the key in two ways: 1. A key mismatch for one and the same row. The reference key that wsrep_rec_get_foreign_key() appends for the parent of a child INSERT is built from the InnoDB record and is not collated, so it no longer matched the primary key carried by the parent row's own writeset. Certification saw no dependency between a child INSERT and a concurrent parent UPDATE, and two appliers could apply them in parallel causing a hang or crash. 2. A key collision between distinct rows. The folding is many to one, so different values collapse onto one key, Certification compares keys byte for byte, so unrelated rows were treated as the same row. Concurrent transactions on them certified as a conflict and one was aborted with ER_LOCK_DEADLOCK. Fix is for wsrep_store_key_val_for_row() to skip the collation for fields that InnoDB stores as binary, using the same condition as get_innobase_type_from_mysql_type(). This is a no-op for the types that worked before. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alessandro Vetere
iminelink@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-32286 Reuse remembered clustered leaves in secondary-index scans Row_sel_get_clust_rec_for_mysql::operator() descends the clustered B-tree from the root for every row whose clustered-index record a secondary-index scan must read, although consecutive rows land on the same clustered leaf page wherever the secondary order tracks the clustered one. A non-covering scan needs one for every row, a locking read needs one whatever the secondary index holds, because an exclusive select lock type makes ha_innobase::build_template() build its template against the clustered index, and a covering scan needs one for every row of a secondary leaf whose PAGE_MAX_TRX_ID its read view cannot see. ANALYZE FORMAT=JSON charges each descent its full height, and those descents are nearly the whole cost: the secondary index is charged its own descent and one page for each further leaf, and nothing per row, because the position that its cursor holds between two rows is restored optimistically, which latches the leaf again without counting an access. So pages_accessed is the row count times the height of the clustered index, plus a handful: 1000 rows over a 2-level clustered index cost 2006 and 750 rows over a 3-level one cost 2291, where a full table scan of the same data costs 23 and 110. Let a handle remember the clustered leaves that the lookups of one statement reached, and let the next lookup try them before it descends again. The reasoning behind each value and each rejection is in the comments beside it. row0mysql.h defines clust_leaf_hint_slot, which names one leaf: its page number, copies of its first and last user record truncated to the key fields, which bound the key range that the leaf held when it was remembered, the rec_get_offsets() of both, and the dict_index_t::n_core_fields that the copies were made under. A slot of a leaf that had no right sibling names no last key, because every key above the last record of the rightmost leaf still belongs to it. CLUST_LEAF_HINT_SLOTS (4) slots hang off the new row_prebuilt_t::clust_leaf_hint, beside clust_leaf_hint_mru, the most recently used order held as slot numbers, and clust_leaf_hint_n and clust_leaf_hint_miss, the used-slot count and the miss counter. row0sel.cc holds the policy. row_sel_clust_leaf_hint_covers() compares a key against the remembered ranges, so a lookup that no slot can answer costs no buffer pool access and no pages_accessed. row_sel_clust_leaf_hint_search() probes the first slot that covers the key and moves it to the front of the order. row_sel_clust_leaf_hint_remember() records the leaf that a descent landed on, and refreshes the slot of a leaf that is remembered already rather than spend a second one on the same page. Two descents fill no slot: the first CLUST_LEAF_HINT_MIN_LOOKUPS (4) lookups of a statement, and a leaf that is the root. row_sel_clust_leaf_hint_armed() stands a scan down once the slots stop paying for themselves: a miss adds CLUST_LEAF_HINT_MISS_WEIGHT (2) to the miss counter and a hit takes one away, so a scan gives the slots up where it answers too little of its lookups to pay for them, CLUST_LEAF_HINT_MAX_MISSES (8) misses with no hit between them still reach the threshold, and one lookup in CLUST_LEAF_HINT_RETRY (1024) starts the count again, so a scan whose order becomes correlated only later recovers. Both halves of the cost stop there, the test of the slots and the copies that refresh them. Row_sel_get_clust_rec_for_mysql::operator() calls all of this in place of its btr_pcur_open_with_no_init(), and only where the adaptive hash index is disabled, whose guess solves the same problem better: it lands on the record with no page-local search and no page access to charge. That index is off by default, so the hints are active in a default configuration. btr0cur.h and btr0cur.cc add btr_cur_t::try_leaf_hint(), a PAGE_CUR_LE, BTR_SEARCH_LEAF search on one named leaf. It acquires the page with buf_page_try_get(): a hint is never derived from a latched parent page, so by the time it is tried it can precede the caller's already-latched secondary-index leaf in the latching order, where a blocking wait can deadlock. It then rejects the page unless the checks that it makes on the latched frame put the match on it. Those checks are the sole authority on the result, so a stale range costs a wasted probe or a needless descent, never a wrong result, and the ranges need no invalidation protocol. ha_innodb.cc: ha_innobase::reset() zeroes the used-slot count and the miss counter per statement, matching autoinc_last_value. row0mysql.cc: row_prebuilt_free() frees the key buffers that the slots own. innodb.clust_leaf_hint measures pages_accessed over key orders that differ in how closely the secondary order tracks the clustered one, and eight further tables check query results over the record formats and key shapes that a clustered-index lookup has to read, down to the metadata pseudo-record of instant ALTER TABLE, to leaves that split and merge while a locking read walks them, and to a record that a remembered leaf supplies for a scan that must then rebuild an older version of it. Two of the tables scan a covering index, which reads a clustered record under an exclusive select lock type and under a PAGE_MAX_TRX_ID that the read view cannot see. clust_leaf_hint_off_debug runs the same body with the hints turned off, through a debug switch that returns before a lookup tests or refreshes the slots, so a diff of the two .result files is what the hints save: 2006 to 1031 (2-level clustered index), 2291 to 1011 (3-level), 4006 to 2015 (two interleaved key ranges), 12016 to 9078 (locality in the second half alone) and 20020 to 10045 for a covering scan that FOR UPDATE makes non-covering, where the same scan without FOR UPDATE costs 20 in both files. Two orders with too little locality to pay for the slots give them up early and end within a hundred accesses of the unhinted count: 20020 to 19966 (decorrelated) and 20020 to 19999 (shuffled). innodb.clust_leaf_hint_instant_alter covers the one rejection that no count reaches, of a slot whose keys were copied under another dict_index_t::n_core_fields than the index reports. dict_index_t::clear_instant_alter() is the only writer of that value that a shared metadata lock allows, and it needs the clustered index to lose the last user record of its root page, while no leaf that is the root fills a slot, so the tree has to shrink between the two, which purge does there. The reader therefore reads uncommitted rows at READ UNCOMMITTED and waits in a stored function while a rollback and purge take them away, and one row that arrives above the position it stopped at is the lookup that tests the slots. The rejection leaves nothing that a query can read, so that branch writes the two counts to the error log under a debug switch and the case reads them back with search_pattern_in_file.inc. They are printed and not named in the pattern, so that a run which reaches the branch with other counts, or in the direction where the clear lowers them, is a difference to look at and not a pass. main.rowid_filter_innodb: 90 to 88, and its ahi combination unchanged. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Brandon Nesterenko
brandon.nesterenko@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40906: rpl.rpl_gtid_thread_id assert_grep.inc failed rpl.rpl_gtid_thread_id could fail sporadically due to a non-deterministic slave state during an assert. The test asserted that a certain number of transaction's exist in the slave's binary log file; however, there was no sync between the master and slave after the last transaction executed on the master. This means the slave's binary log could be checked before the transaction ever was sent to/committed on the slave. The fix is to simply sync the master and slave before checking the slave's binary log. Signed-off-by: Brandon Nesterenko <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-36166: Accept bracketed points inside MULTIPOINT ST_GEOMFROMTEXT('MULTIPOINT((0 0),(1 1))') returned NULL while ST_GEOMFROMTEXT('MULTIPOINT(0 0,1 1)') returned the geometry. The bracketed spelling is the one the OGC WKT grammar defines. In 06-103r4 section 7.2.2 a <multipoint text> is a list of <point text>, and a <point text> has its own parentheses, the same way a <multilinestring text> is a list of <linestring text>. The bare spelling matches no production in that grammar, so the text MariaDB rejected was the conformant one. The first point now determines which of the two bracketing forms the remaining list elements will use. A mixed list such as MULTIPOINT((0 0),1 1) is an error. The bare form stays accepted because existing data and applications use it. Geometry::create_from_wkt is the single entry into the WKT reader, so ST_MPOINTFROMTEXT and a MULTIPOINT nested in a GEOMETRYCOLLECTION are covered by the same change. Co-Authored-By: Claude Opus 5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| fixup! a77b37c49cf47284580e507303d538a5daa2ea79 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
KhaledR57
khaled57.dev@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-37167 Nested BEGINs (4600+) cause a segmentation fault Each nested BEGIN adds one sp_pcontext. Two walks over the finished tree recursed once per nesting level and overran the thread stack. ~sp_pcontext() freed its children recursively. sp_head now threads a single linked list through the contexts and frees them iteratively, so teardown depth is constant and a context no longer frees its children. retrieve_field_definitions() descends the children to build the run-time frame. It emits them in run-time offset order, so it stays recursive, but it now checks the stack and returns an error instead of crashing. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||