Console View
|
Categories: connectors experimental galera main |
|
| connectors | experimental | galera | main | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alessandro Vetere
iminelink@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-32286 Reuse remembered clustered leaves in secondary-index scans Row_sel_get_clust_rec_for_mysql::operator() descends the clustered B-tree from the root for every row whose clustered-index record a secondary-index scan must read, although consecutive rows land on the same clustered leaf page wherever the secondary order tracks the clustered one. A non-covering scan needs one for every row, a locking read needs one whatever the secondary index holds, because an exclusive select lock type makes ha_innobase::build_template() build its template against the clustered index, and a covering scan needs one for every row of a secondary leaf whose PAGE_MAX_TRX_ID its read view cannot see. ANALYZE FORMAT=JSON charges each descent its full height, and those descents are nearly the whole cost: the secondary index is charged its own descent and one page for each further leaf, and nothing per row, because the position that its cursor holds between two rows is restored optimistically, which latches the leaf again without counting an access. So pages_accessed is the row count times the height of the clustered index, plus a handful: 1000 rows over a 2-level clustered index cost 2006 and 750 rows over a 3-level one cost 2291, where a full table scan of the same data costs 23 and 110. Let a handle remember the clustered leaves that the lookups of one statement reached, and let the next lookup try them before it descends again. The reasoning behind each value and each rejection is in the comments beside it. row0mysql.h defines clust_leaf_hint_slot, which names one leaf: its page number, copies of its first and last user record truncated to the key fields, which bound the key range that the leaf held when it was remembered, the rec_get_offsets() of both, and the dict_index_t::n_core_fields that the copies were made under. A slot of a leaf that had no right sibling names no last key, because every key above the last record of the rightmost leaf still belongs to it. CLUST_LEAF_HINT_SLOTS (4) slots hang off the new row_prebuilt_t::clust_leaf_hint, beside clust_leaf_hint_mru, the most recently used order held as slot numbers, and clust_leaf_hint_n and clust_leaf_hint_miss, the used-slot count and the miss counter. row0sel.cc holds the policy. row_sel_clust_leaf_hint_covers() compares a key against the remembered ranges, so a lookup that no slot can answer costs no buffer pool access and no pages_accessed. row_sel_clust_leaf_hint_search() probes the first slot that covers the key and moves it to the front of the order. row_sel_clust_leaf_hint_remember() records the leaf that a descent landed on, and refreshes the slot of a leaf that is remembered already rather than spend a second one on the same page. Two descents fill no slot: the first CLUST_LEAF_HINT_MIN_LOOKUPS (4) lookups of a statement, and a leaf that is the root. row_sel_clust_leaf_hint_armed() stands a scan down once the slots stop paying for themselves: a miss adds CLUST_LEAF_HINT_MISS_WEIGHT (2) to the miss counter and a hit takes one away, so a scan gives the slots up where it answers too little of its lookups to pay for them, CLUST_LEAF_HINT_MAX_MISSES (8) misses with no hit between them still reach the threshold, and one lookup in CLUST_LEAF_HINT_RETRY (1024) starts the count again, so a scan whose order becomes correlated only later recovers. Both halves of the cost stop there, the test of the slots and the copies that refresh them. Row_sel_get_clust_rec_for_mysql::operator() calls all of this in place of its btr_pcur_open_with_no_init(), and only where the adaptive hash index is disabled, whose guess solves the same problem better: it lands on the record with no page-local search and no page access to charge. That index is off by default, so the hints are active in a default configuration. btr0cur.h and btr0cur.cc add btr_cur_t::try_leaf_hint(), a PAGE_CUR_LE, BTR_SEARCH_LEAF search on one named leaf. It acquires the page with buf_page_try_get(): a hint is never derived from a latched parent page, so by the time it is tried it can precede the caller's already-latched secondary-index leaf in the latching order, where a blocking wait can deadlock. It then rejects the page unless the checks that it makes on the latched frame put the match on it. Those checks are the sole authority on the result, so a stale range costs a wasted probe or a needless descent, never a wrong result, and the ranges need no invalidation protocol. ha_innodb.cc: ha_innobase::reset() zeroes the used-slot count and the miss counter per statement, matching autoinc_last_value. row0mysql.cc: row_prebuilt_free() frees the key buffers that the slots own. innodb.clust_leaf_hint measures pages_accessed over key orders that differ in how closely the secondary order tracks the clustered one, and eight further tables check query results over the record formats and key shapes that a clustered-index lookup has to read, down to the metadata pseudo-record of instant ALTER TABLE, to leaves that split and merge while a locking read walks them, and to a record that a remembered leaf supplies for a scan that must then rebuild an older version of it. Two of the tables scan a covering index, which reads a clustered record under an exclusive select lock type and under a PAGE_MAX_TRX_ID that the read view cannot see. clust_leaf_hint_off_debug runs the same body with the hints turned off, through a debug switch that returns before a lookup tests or refreshes the slots, so a diff of the two .result files is what the hints save: 2006 to 1031 (2-level clustered index), 2291 to 1011 (3-level), 4006 to 2015 (two interleaved key ranges), 12016 to 9078 (locality in the second half alone) and 20020 to 10045 for a covering scan that FOR UPDATE makes non-covering, where the same scan without FOR UPDATE costs 20 in both files. Two orders with too little locality to pay for the slots give them up early and end within a hundred accesses of the unhinted count: 20020 to 19966 (decorrelated) and 20020 to 19999 (shuffled). innodb.clust_leaf_hint_instant_alter covers the one rejection that no count reaches, of a slot whose keys were copied under another dict_index_t::n_core_fields than the index reports. dict_index_t::clear_instant_alter() is the only writer of that value that a shared metadata lock allows, and it needs the clustered index to lose the last user record of its root page, while no leaf that is the root fills a slot, so the tree has to shrink between the two, which purge does there. The reader therefore reads uncommitted rows at READ UNCOMMITTED and waits in a stored function while a rollback and purge take them away, and one row that arrives above the position it stopped at is the lookup that tests the slots. The rejection leaves nothing that a query can read, so that branch writes the two counts to the error log under a debug switch and the case reads them back with search_pattern_in_file.inc. They are printed and not named in the pattern, so that a run which reaches the branch with other counts, or in the direction where the clear lowers them, is a difference to look at and not a pass. main.rowid_filter_innodb: 90 to 88, and its ahi combination unchanged. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41094 KDF() aliases large iteration/width to weak 32-bit values Better check parameters of KDF (i.e. iteration count should be positive 32 bit, key length 16 bit unsigned). |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-33616: Match the macOS dlopen error in plugins.multiauth The client reports why it could not load client_ed25519, and macOS names every path that dlopen() tried. Two expressions are added, one for the chunk that holds the start of that message and one for the chunk that holds the rest of it. The line runs to 563 bytes, 52 of prefix and the 511 that the client error buffer holds, while do_exec() reads the output with fgets() into a 512 byte buffer and runs the replacements on each chunk on its own. A long enough vardir therefore splits the line, because the path appears four times in the dlopen text. The second chunk is the tail of a path and carries no colon, where the first chunk keeps the colons of the mysqltest prefix. That chunk also holds the only line terminator the error line gets, so the expression captures the newline and the replacement puts it back. A replacement is inserted as written, so a \n spelled there would reach the output as a backslash and an n. Both expressions stop at a newline. reg_replace compiles with REG_DOTALL, so an unrestricted .* runs past the line terminator whenever the whole message reaches the replacement in one chunk, and the error line then joins the line after it. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Andrzej Jarzabek
andrzej.jarzabek@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-37521 Attempt to compare iterators from different sequences in range_set::add_range fil_space_t::freed_ranges is a range_set guarded by freed_range_mutex, and every mutator takes that lock except one: the branch of mtr_t::commit() that frees pages for an unlogged mini-transaction on the temporary tablespace (the common case, since temp-space pages never set m_modifications). That path called fil_space_t::add_free_range() directly, racing with any other locked mutator of the same std::set - most notably fil_space_t::flush_freed(), which the page cleaner invokes for every tablespace while resizing the buffer pool. The race corrupts the range_set's underlying tree and crashes the server. Take freed_range_mutex around the loop, matching every other caller (process_freed_pages(), flush_freed(), clear_freed_ranges()). Added innodb.temp_truncate_freed_race, which reproduces the crash with a configurable probability by racing concurrent temporary-table churn against a continuously resizing buffer pool across several rounds against the same server. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-35168 subselects with outer references to derived tables may be incorrectly evaluated as constant Subselects with outer references to derived tables may be incorrectly evaluated as having no table references. This can lead to these subselects being marked as constant, leading to an incorrect result. During the calculation of the tables used in a subselect, we construct a table map of outer references in our (not necessarily new) "new_parent" select. This is currently done purely by finding Item_fields in our tree and using the attached table to update our bitmap. It can be that a reference to a derived table also needs to have it's table added to this map. If the derived table can be null, this is the case. We add a new processor to our item walk system, enumerate_table_refs_processor which is defined at this stage only for Item_direct_view_ref items. This called alongside enumerate_field_refs_processor in Item_subselect::recalc_used_tables(). Coverage of the merged_into() function is provided at the end of the tests in subselect4.test. Be aware that they are not a regression test, we could not find anything that provided an incorrect output. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Do the same thing as the parent commit to quick_mode_{1,3} | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
design document update + updates for eligibility checking for cascaded tables |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thirunarayanan Balathandayuthapani
thiru@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41207 Acquire metadata locks for recovered transaction Problem: ======= A transaction being rolled back during recovery holds LOCK_IX on the table(not the metadata locks), and the rollback thread holds a reference on it. An online ALTER TABLE on that table falls back to acquiring LOCK_S, which conflicts and fails. prepare_inplace_alter_table_dict() asserted that the reference count is 1 before checking whether the table lock was acquired, so the reference still held by the rollback thread makes the assertion fail. Solution: ======== trx_recovery_thd: A background connection that owns the metadata locks of the recovered transactions. One connection is shared by all of recovering transaction. trx_lists_init_at_db_start(): Create trx_recovery_thd, before any metadata lock can be acquired. trx_resurrect_table_locks(): Acquire a shared metadata lock on each table that the recovered transaction had modified, and remember the locks in trx_recovery_mdl. Recovered XA PREPARED transactions are excluded, because they are completed by a user connection. trx_recovery_mdl: The metadata locks of the recovered transactions, by transaction. They are kept outside trx_t. Only a recovered transaction ever has an entry. trx_recovery_mdl_exists: Whether trx_recovery_mdl is not empty. It is read for every transaction in trx_t::free(), so that the map will only be consulted while some recovered transaction still holds metadata locks. It becomes false as soon as the rollback of the recovered transactions has been completed, long before trx_recovery_thd is destroyed. trx_t::free(): Release the metadata locks of a recovered transaction, once its rollback has been completed. This is the only place that releases them, so that a transaction which was rolled back by trx_rollback_recovered(false) will not keep its locks until shutdown. trx_recovery_thd_destroy(): Destroy trx_recovery_thd. It is invoked by innodb_shutdown() only, after trx_sys.close() has freed any recovered transaction that was left. row_undo_mod(): Add the debug injection rollback_wait |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39525 [wip][to-squash] fix vcol_type_both_bounded Reverted changes to Field_longstr::is_supertype Also improve test coverage of the domain check TODO: check results |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dmitry Shulga
dmitry.shulga@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-30645: CREATE TRIGGER FOR { STARTUP | SHUTDOWN } Follow-up patch to fix missing call to my_error() in case not all mandatory columns present in the table mysql.event |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Mohammad Tafzeel Shams
tafzeel.shams@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41242 : Fix resource leaks on InnoDB/mariabackup error paths found by Infer Several error-handling paths returned without releasing a resource already acquired earlier in the function, or checked the wrong handle entirely, risking use of an unopened handle. Changes: - SysTablespace::read_lsn_and_check_flags(): close the datafile handle on header-validation failure. - xb_process_datadir(): check the freshly opened `dir` handle instead of the stale `dbdir`, fixing a handle leak and a possible use of an unopened directory handle. - wsrep.cc / xb_load_list_file(): close file handles before die(), and null-check fopen() results in wsrep.cc. - datadir_iter_new(): free datadir_path and destroy the mutex on the os_file_opendir() failure path. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-33616: Normalize the strerror text in innodb_fts.index_table The injected deadlock reaches the client as ER_GET_ERRNO carrying errno 11, and the text comes from my_strerror(). 11 is EAGAIN on Linux and EDEADLK on macOS, so the message reads "Resource temporarily unavailable" on one and "Resource deadlock avoided" on the other. Replace the quoted text so the test does not depend on it. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-33616: Skip the redo log upgrade tests without sparse file support innodb.log_upgrade and innodb.log_upgrade_101_flags build 8GB redo log files by seeking past the end of an empty file and writing a single byte. That needs a filesystem which leaves the skipped range unallocated. HFS on macOS allocates every block of it instead, so the write fails with ENOSPC and the test reports a perl failure. include/have_sparse_files.inc probes a directory the caller names, writing one byte 64MB into an empty file there and comparing the allocated block count against that offset. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40837: Sequences used only in a column DEFAULT missing from optimizer context Problem: ======== A sequence referenced only in a column's DEFAULT expression (e.g. "a INT DEFAULT NEXTVAL(s1)") is opened only when a statement actually evaluates DEFAULT values (INSERT, LOAD DATA, etc). A plain SELECT on the table never opens it, so the sequence never appears in thd->lex->query_tables and dump_sql_script() had no way to see it. The dependent table's definition then got captured without the sequence it needs, making the captured context unusable Solution: ========= TABLE::internal_tables already holds the sequence tables DEFAULT expressions depend on, populated whenever the table is opened regardless of statement type. For each table bein walk this list, open any sequence not already open, and dump its CREATE SEQUENCE and current value (via SETVAL) before th own CREATE TABLE statement, so replay can recreate both in the correct order. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40967 PROXY protocol host check sent in clear text mid-SSL handshake Defer the host-privileged/host-blocked check for a PROXY-header-derived address until after the client's SSL handshake completes, instead of sending it immediately in clear text. Co-Authored-By: Claude Sonnet 5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-33616: Make two tests independent of lower_case_table_names macOS puts the data directory on a case insensitive file system, so lower_case_table_names is 2 and both tests recorded an answer that only holds for 0. period.i_s_notembedded looked up I_S.PERIODS and I_S.KEY_PERIOD_USAGE by the schema name TEST. That comparison follows the table name comparison, so it finds the table under 1 and 2 and finds nothing under 0. Those four queries move to the new test period.i_s_case_sensitive, which requires lower_case_table_names=0. The win rdiff of period.i_s_notembedded covered the same difference and is no longer needed. atomic.drop_db_long_names generated table and view names in upper case and compared the DROP statements that DDL recovery writes to the binary log. Under 2 the names come back from the directory in lower case. Generating them in lower case to begin with gives the same names on every setting. Lower case also changes where the view name sorts against its table name for the letters after v, which moves one view between two of the recorded DROP VIEW statements. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-33616: Exclude innodb_log_file_mmap from sys_vars.sysvars_innodb Its default value depends on the operating system, ON where the log can be memory mapped and OFF elsewhere, so the recorded row only holds on some platforms. The other variables whose default depends on the operating system are already excluded the same way. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-33616: Allocate the recovery buffer from the heap recv_sys.tmp_buf comes from malloc() rather than from the large page allocator. main.large_pages fails on macOS with "Warning: Memory not freed: 16375" at shutdown. recv_sys_t::find_checkpoint() asks for 1048585 bytes, my_large_malloc() rounds that up to 1064960 and charges the rounded figure to the server memory accounting, and recv_sys_t::tmp_free() credits back the 1048585 that was requested. ut_malloc_dontdump() takes the size by value, so it has nowhere to report what my_large_malloc() wrote back. The rounding happens whenever my_next_large_page_size() finds a reported large page size at or below the request. macOS has no huge page interface for my_get_large_page_sizes() to consult, so its fallback branch reports the ordinary page size, 16384 on Apple silicon, and the request is always rounded. Linux reads the sizes from /sys/kernel/mm/hugepages, where the smallest entry is usually 2 MiB, and a 1 MiB request then gets no large page and no rounding. The buffer has no alignment requirement. recv_sys_t::parse() copies a mini-transaction into it when the record is encrypted in the FORMAT_ENC_11 log, where it is then decrypted in place, or when the record wraps around the end of the log file, and reads it back as a byte sequence. tmp_free() calls std::free() because the member function recv_sys_t::free() hides the one from <cstdlib>. log_sys.buf and log_sys.flush_buf keep the large page allocator. They round the same way, so a server started with --large-pages --innodb-log-buffer-size=2101248 still reports 24576 on macOS. The core dump exclusion that recv_sys.tmp_buf gives up applies only where MADV_DONTDUMP exists, so nothing changes on macOS, while a release build on Linux would now include the buffer in a dump. tmp_free() overwrites the redo log records that innodb_encrypt_log decrypted before releasing the memory, through a volatile function pointer because GCC removes a plain memset() that is followed by free(). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Mohammad Tafzeel Shams
tafzeel.shams@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-35154 : dict_sys_t::load_table() is holding exclusive dict_sys.latch for unnecessarily long time Issue: dict_load_table_one() was invoked with the exclusive dict_sys.latch held and never released it. The whole load ran inside that one critical section: reading the SYS_TABLES record, opening the .ibd file, and reading SYS_COLUMNS, SYS_VIRTUAL, SYS_INDEXES, SYS_FIELDS, the clustered index root page, and SYS_FOREIGN, as well as loading every table related by FOREIGN KEY constraints. Almost all of that is I/O. Because dict_sys.latch is the single latch that guards every dictionary lookup, one cold table open blocked every other session from opening any table, including tables that were already cached. A slow enough load could trigger the fatal semaphore wait check. Fix: Split the load into three phases. A short latched phase creates the table object and publishes it as an incomplete "stub"; the I/O runs with no latch held; a final latched phase links the FOREIGN KEY constraints. The stub's progress is held in the top two bits of dict_table_t:: n_ref_count (LOADING_DEF, LOADING_FK, LOAD_FAILED), analogous to how buf_page_t::state() combines a small lifecycle state with the buffer-fix count in one atomic. A thread that finds a loading table waits by pinning it (dict_table_t::try_pin_for_wait(), so it cannot be freed while unlatched) and acquiring dict_table_t::lock_latch in shared mode, the same latch the loader holds in exclusive mode for the duration of the load. A failed load can be torn down safely even though other threads may have already taken such a pin to wait on it: try_pin_for_wait() refuses once LOAD_FAILED is set, and the teardown path drains any pins taken earlier before freeing the stub. lock_latch is otherwise used for record-lock bookkeeping and statistics on a table; the only place the loader itself needs it for that unrelated purpose while still holding it for the load is dict_get_and_save_data_dir_path(), which skips re-acquiring the latch whenever the table is loading(), since only the loading thread can reach the table at that point. A table remains hidden (LOADING_FK) until every table related to it by FOREIGN KEY constraints has been loaded as well, and dict_sys_t:: load_table() makes them all visible in one step. The intermediate state is visible to constraint linking via dict_sys_t::find_table_fk(), so that two threads loading tables that reference each other can still link the constraint between them. Tables that InnoDB internal SQL can refer to are loaded without ever releasing the latch. The internal SQL parser is not reentrant and is serialized only by the exclusive dict_sys.latch, and it opens tables in the middle of parsing; releasing the latch there let another thread run the parser concurrently and corrupt its state. Changes: - dict_table_t : hold the progress of loading in the top bits of n_ref_count (std::atomic<uint32_t>): LOADING_DEF, LOADING_FK, LOAD_FAILED, loading(). Add the loader-only helpers start_loading(), advance_to_loading_fk(), finish_loading(), mark_load_failed(), and try_pin_for_wait() for waiters, all built on the existing lock_latch. acquire()/release() are unconditional. Add debug load_thread and is_loader(). - dict_load_table_one() : publish the table as a LOADING_DEF stub in table_non_LRU and release the latch before loading the tablespace, the columns and the indexes; reacquire it, move the table to table_LRU and advance it to LOADING_FK before loading the foreign key constraints. Failures go through dict_load_table_one_discard(). Add the hold_latch parameter and the dict_load_table_one_no_latch debug sync point. - dict_sys_t::load_table() : wait for a concurrent load of the same table via wait_for_load(); allocate the foreign key table names on a local heap, because the latch is released while draining them; in the drain loop, wait for a table whose definition is still being loaded by another thread and skip one that is only waiting for its own related tables; make all tables loaded by this invocation visible in one step, releasing each one's lock_latch. - dict_sys_t::wait_for_load() : pins the observed table via try_pin_for_wait() (returning immediately if the load already failed), releases the exclusive dict_sys.latch, blocks on the table's own lock_latch, unpins, and reacquires the latch. - dict_load_table_one_discard() : used wherever a stub must be torn down (retry after DB_SUCCESS_LOCKED_REC, column or virtual-column load failure, a corrupted index or missing FK index). Marks the stub LOAD_FAILED, releases lock_latch to wake any already-pinned waiters, drains the reference count to zero, then removes the stub. - dict_sys_t::add() : take lock_latch in exclusive mode on the loader's behalf before a loading stub becomes reachable via find_table_any(). - dict_get_and_save_data_dir_path() : skip re-acquiring lock_latch when the table is loading(), because the loading thread already holds it in exclusive mode and is the only thread that can reach the table at that point. - dict_load_hold_latch() : whether a table may be referenced by InnoDB internal SQL, and therefore must be loaded without releasing the latch: InnoDB system tables, FULLTEXT INDEX auxiliary tables and the persistent statistics tables. - dict_load_table_on_id() : copy the table name and release the SYS_TABLES page latch before calling load_table(), which may now block. Restore the cursor position only if the scan has to continue. - dict_load_foreign(), dict_load_foreigns() : add the fk_heap parameter and allocate the names appended to fk_tables on it. - dict_sys_t::find_table_any() : the previous body of find_table(), returning tables that are being loaded. Only the name is read. - dict_sys_t::find_table() : hide tables that are being loaded, both in the by-name and the by-id variant. In the by-id variant, loading is checked before any bit-field, to avoid a torn read. - dict_sys_t::find_table_fk() : like find_table(), but LOADING_FK tables are returned, so that constraints can be linked into them while holding the exclusive latch. - dict_table_can_be_evicted() : a table that is being loaded may only be freed by the thread that is loading it. - dict_foreign_add_to_cache() : resolve both sides with find_table_fk(). - btr_search_disable() : skip tables that are being loaded. Walking their indexes would race with the loading thread, which appends to that list without holding the latch. Such tables cannot have any adaptive hash index references. - create_table_info_t::create_foreign_keys() : take a temporary reference on a referenced table as soon as it is resolved, released on every exit path by a scope guard; call dict_sys.prevent_eviction() only once the constraint is actually committed to the dictionary cache, replacing the temporary reference. - create_table_info_t::create_table() : acquire a reference to the created table around the foreign key handling, and use a local heap for the names of the foreign key related tables. - row_rename_table_for_mysql() : use one local heap for both the dropped-constraint names and the foreign key table names. - assertion fix : the load path may now run without dict_sys.latch, but only in the thread that is loading the table, and a cached table is no longer necessarily fully loaded. dict_sys.locked() is relaxed to "dict_sys.locked() || table->is_loader()" in dict_load_columns(), dict_load_virtual_col(), dict_load_fields(), dict_load_indexes(), dict_index_add_to_cache(), dict_index_find_cols(), dict_index_build_internal_clust(), dict_index_build_internal_non_clust() and dict_index_build_internal_fts(). Checks of dict_table_t::cached are relaxed and reordered after the atomic loading in dict_table_add_system_columns(), dict_sys_t::add(), dict_table_rename_in_cache() and hash_insert(). - innodb.dict_load_concurrent A load is parked at dict_load_table_one_no_latch while holding no latch; another table can be loaded meanwhile, and a second opener of the same table waits. A second case checks that a table is not made visible while a table related to it by a FOREIGN KEY constraint is still being loaded by another thread. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-19817 Fix crash in Multiupdate_prelocking_strategy::handle_end Multiupdate tables in case of reopening (in case of deadlock for example do not clean up views/derived (with trigggers). So SET list stay "fixed" and do not mark write bits which cause problems. There is still problem in case of concurent flush table which replace TABLE object, but it should be covered by MDEV-21630. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dmitry Shulga
dmitry.shulga@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40951: System triggers give no visibility into slow/failed execution and misreport server readiness Added output into a log information about start/finish execution of startup and shutdown triggers. In case error happens on running any of startup/shutdown triggers, the error number and message is output into error log. Additionally, minor refactoring was done to avoid source code duplication. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40967 PROXY protocol host check sent in clear text mid-SSL handshake Defer the host-privileged/host-blocked check for a PROXY-header-derived address until after the client's SSL handshake completes, instead of sending it immediately in clear text. Co-Authored-By: Claude Sonnet 5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| fixup! 7e375803bb8a7dcb46c96f4a5727620ff0a6d0bb | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
yuchen.pei@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Allow a covering index scan for a locking SELECT under a full-scan table lock Follow-up to MDEV-24813 (innodb_table_lock_on_full_scan) and MDEV-40805. ha_innobase::build_template() retrieves the whole clustered index record whenever select_lock_type is LOCK_X, and row_search_with_covering_prefix() refuses the covering-index optimisation for the same reason. As a result a covering secondary index scan such as SELECT sec, id FROM t1 FOR UPDATE visits the clustered index once per scanned row, while the same scan with LOCK IN SHARE MODE stays inside the secondary index. On a one million row table that is a million extra clustered index lookups. Only part of that work is inherent to LOCK_X. UPDATE and DELETE do need the clustered index record, because they are going to write it. A plain locking SELECT does not: it visits the clustered index only to place the per-row exclusive lock that gives SELECT ... FOR UPDATE its row level exclusivity. When innodb_table_lock_on_full_scan made us take a table level LOCK_X for the whole scan, that per-row lock is redundant, because the table lock is already mutually exclusive with any other transaction's LOCK_IX, and hence with any record lock or implicit exclusive lock in the table. Introduce row_prebuilt_t::full_scan_covering_read, set in ha_innobase::extra_opt() next to full_table_scan and only when the statement is a plain SELECT outside the HANDLER interface, and skip the LOCK_X restriction in both places when it is set. The flag is never set unless full_table_scan is set, so row level locking behaviour is unchanged when innodb_table_lock_on_full_scan is off. Whether a column outside the scanned index is needed is still decided by the existing logic in build_template(), so SELECT pad ... FOR UPDATE continues to read the clustered index. Co-Authored-By: Claude Opus 5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-33616: Take the read lock many times in perfschema.func_mutex The wait timer can have a granularity coarser than the time an uncontended read lock is held, so the recorded duration of one lock can be zero, which reads back as NULL. This can cause the test to fail with a false negative. Take the lock twenty more times at each measurement point, with the extra statements silent so the recorded result does not change. The mutex part of the test already works this way, since one SELECT produces ten THR_LOCK::mutex events. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39275 XA COMMIT / XA ROLLBACK / XA RECOVER don't require any privileges Require a new XA RECOVER ADMIN privilege for XA RECOVER, matching MySQL's XA_RECOVER_ADMIN. XA COMMIT/ROLLBACK are left unchanged. Co-Authored-By: Claude Sonnet 5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Fix remaining .result drift from the new privilege bit Co-Authored-By: Claude Sonnet 5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39525 [wip][to-squash] fix vcol_type_both_bounded Also improve test coverage of the domain check TODO: check results |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41212: multi_source.status_vars fails on MacOS platform Replace the two recorded reads of Slave_received_heartbeats with an assertion that the counter is nonzero. The counter advances once per heartbeat period for as long as the connection is running. The test waited for it to reach 2 and then read it again in a separate query, so a heartbeat arriving between those two queries recorded an unexpected value. The same wait timed out when the counter was already past 2 at the first poll, so it now accepts any value at or above the target. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-33616: Only one of two routines named in a statement is found With lower_case_table_names 0 the server can have databases Db1 and db1, each with a function f1. A single statement naming both databases, like SELECT Db1.f1(), db1.f1(), reported that db1.f1 does not exist. The set of routines a statement uses compared its entries without regard to case. Only one routine was loaded but the reference to the other found nothing. The set now compares its entries exactly, as the routine cache and the lock manager already do. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-33616: MTR flag to mark tests as incompatible with macOS Introduces a new MTR include, not_mac.inc, which when included at the top of a test, prevents that test from running on macOS. sys_vars.sysvars_readonly_debug is the first user. It expects the server to fault when a read only sysvar is written behind the sysvar interface. That protection needs the ro_after_init section, which a linker script places and ld64 has no option to take, so HAVE_RO_AFTER_INIT stays undefined on macOS. Without it no variable is moved into the read only root either, so neither of the two assignments is refused. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alexey (Holyfoot) Botchkov
holyfoot@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-36261 XMLTYPE: methods - step 1 Method functions added to the XMLTYPE. functions added here are: XMLTYPE.isFragment() XMLTYPE.getSchemaURL() XMLTYPE.getNumberVal() XMLTYPE.getStringVal() XMLTYPE.getRootElement() |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41228 Test deep clones of Item_cache items Deep clones of Item items are created in very few places, so most of the clone code has no test coverage at all, although Parallel Query relies on it heavily. One of the places where clones are created is the generation of the key parts for a lookup into a split materialized table, in TABLE::add_splitting_info_for_key_field(). An Item_cache reaches that place through the IN->EXISTS transformation. Item_in_optimizer::fix_left() wraps the left expression of the predicate in an Item_cache, and the equality injected into the subquery refers to it, so the key field value being cloned is an Item_direct_ref over that cache. Two more things are needed for the clone to happen: the grouping field the equality matches has to be the first component of some index of the underlying table, otherwise it is not among spl_opt_info->spl_fields and the function returns before cloning, and the subquery must not be converted to a semi-join. The latter is achieved by the shape of the query, a UNION in the subquery, rather than by turning optimizer switches off, so that the plan is the one a user gets with a default optimizer_switch. Add Item::check_deep_copy(), which validates a clone against its original in a debug build. It walks both item trees and reports, as notes, whether they have the same shape with the same Item class at every node, and whether the clone shares an Item object with the original. Sharing is what distinguishes a shallow copy from a deep one: an item that is shallow by design, Item_field for instance, still produces a separate object and only shares a Field, which is not an Item. Call it from TABLE::add_splitting_info_for_key_field() under the "split_materialized_clones" debug flag, which additionally makes the optimizer use a clone as the value of the generated key part. The clone then has to work both in the condition pushed into the materialized table and as the value looked up in the filled table. Note that the clone built for the pushed condition cannot be reused for this, as it has already been made dependent on the select that specifies the materialized table. With the check in place, the clone of the cache turned out not to be a deep one: every Item_cache_* class implemented deep_copy() as a plain shallow copy. That is wrong beyond sharing the example item. A cache is filled by the store()/cache_value() calls of the item that owns it, Item_in_optimizer here, and nobody does that for a clone, so a clone that inherited the cached value of the original kept returning that value for the rest of the query. Implement Item_cache::deep_copy() once for all cache classes instead. The clone is given an empty cache, so that it computes the value itself out of the item the value is read from, and a copy of that item. The exception is an example containing an aggregate or a window function: those are not clonable yet, as a copy of one shares the per-execution data of the original and both would free it, so such an example is shared and the check reports the clone as not fully deep. Item_cache_row is not clonable at all now, as a copy of it shared the values[] array of element caches with the original. The test uses a single row in the outer table, because the answer to the same query with more rows is wrong for an unrelated reason, MDEV-41251: a split materialized table in a dependently executed subquery is never refilled when the outer row changes. This commit was prepared with Claude Code (Opus 5), which located the query shape that makes the split key part generation clone an Item_cache by instrumenting TABLE::add_splitting_info_for_key_field() and searching for a shape that reaches it, wrote Item::check_deep_copy() and the debug hook, implemented Item_cache::deep_copy(), bisected the crash in Item_sum::cleanup() that cloning the example item first caused to the aggr and cmp pointers shared by an Item_sum copy, and separated the remaining wrong result into MDEV-41251 by showing that it survives the clone fix and does not depend on Item_cache. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Do the same thing as the parent commit to quick_mode_{1,3} | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thirunarayanan Balathandayuthapani
thiru@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39061 mariadb-backup compatible wrapper for BACKUP SERVER This adds a shell script that lets users keep using their existing mariadb-backup commands while the real work is done by the new server-side BACKUP SERVER command. The goal is "drop-in": users should not have to change their backup scripts. mariadb-backup-server.sh : Understands the usual mariadb-backup modes and translates each one. mbstream-server.sh : lets streamed backups be unpacked by pipelines that expect the mbstream CLI mariadb-backup-pipe.sh is the stream sink that BACKUP SERVER itself runs. All three are documented in README.md. --backup ======== Connects with the mariadb client and runs "BACKUP SERVER TO '<dir>'". Connection options (--user, --host, --port, --socket, --defaults-file, ssl, ...) are passed through to the client. --parallel=N becomes "<N+1> CONCURRENT": mariadb-backup runs a dedicated log_copying_thread() besides its N data-copy threads, while CONCURRENT counts every BACKUP SERVER worker. The result is clamped to the 1..256 the parser accepts, and to a minimum of 2. After the backup it writes backup-prepare.cnf into the backup directory, recording what --prepare needs later: - where mariadbd lives - InnoDB parameters (page size, data file path, undo tablespaces, checksum algorithm, log file size) - If the server is encrypted then how to reload the encryption key plugin (the file_key_management variables), so an encrypted backup can be prepared without extra input. --backup --stream ================= Runs "BACKUP SERVER WITH 'pipe'". The server only accepts a bare command name, which it resolves n the PATH of the mariadbd process after prepending mariadb-backup-, so uses the installed mariadb-backup-pipe instead. mariadb-backup-pipe writes the tar into .mariadb-backup-pipe-<stream>.tar, a named pipe the wrapper created in the server @@datadir and is already draining to its own stdout. The backup therefore reaches the consumer as it is produced and never lands on local disk. The wrapper appends backup-prepare.cnf as a final tar afterwards; the server's tar carries no end-of-archive marker, so that trailing archive supplies the only one and the whole stream extracts with a plain "tar -x". --parallel is ignored here, with a warning. Four properties follow from how BACKUP SERVER streams, all differing from mariadb-backup: - local: the stream command runs inside the server, so the wrapper must share its filesystem; - fifo lives in @@datadir, because mariadb-backup-pipe resolves it relative to the working directory of mariadbd, so --stream needs write access there; - tar only: any --stream=<format> yields tar; - single-threaded: one worker, so no parallel read either. --target-dir is optional in stream mode; mbstream-server.sh maps the mbstream CLI onto a plain "tar -x"/"tar -c", so existing "mbstream -x"/"-c" pipelines keep working on the wrapper's stream. mbstream-only flags (-p/--parallel, ...) are accepted and ignored; any other unknown option is rejected. Environment overrides: MARIADB (client), MARIADBD (the --prepare bootstrap server) and TAR (the tar implementation, e.g. TAR=bsdtar) can each be overridden. To run the bootstrap under rr, put it in MARIADBD and let rr's own _RR_TRACE_DIR choose the trace location, e.g. _RR_TRACE_DIR=/dev/shm/rr MARIADBD='rr record mariadbd' --prepare ========= Starts "mariadbd --bootstrap" on the backup directory using backup-prepare.cnf as its defaults file, replays the archived redo log between the start and target LSN read from backup.cnf, then builds a fresh ib_logfile0 so a normal server can start on the directory. mariadbd is taken from the path recorded in backup-prepare.cnf if that binary exists, otherwise by searching /libexec, /sbin, /bin and the configured install directories. PATH is not searched; set MARIADBD to point elsewhere. User --defaults-file/-extra-file and encryption options are layered onto the bootstrap. --copy-back / --move-back ========================= Copy or move a prepared backup into the datadir. The datadir is created if missing, a non-empty datadir is refused unless --force-non-empty-directories is given, and a chown reminder is printed. If --aria-log-dir-path is given, the Aria logs (aria_log_control, aria_log.*) are relocated into that directory. Packaging ========= The wrapper is not installed by default and never replaces the real mariadb-backup / mbstream binaries. 1. cmake -DWITH_MARIABACKUP_WRAPPER=ON (default OFF) controls it. 2. When ON, the scripts install as /usr/bin/mariadb-backup-server, /usr/bin/mbstream-server and /usr/bin/mariadb-backup-pipe, tagged COMPONENT Backup so they ship in the mariadb-backup package. mariadb-backup-pipe must end up in the PATH of the mariadbd process, not merely in the PATH of whoever runs the wrapper. 3. RPM: nothing extra to do. the component handles it. 4. DEB: not wired. debian/rules uses --fail-missing and does not enable the option, so the -server binaries are not listed. To ship via DEB, make a paired change: add -DWITH_MARIABACKUP_WRAPPER=ON in debian/rules and list all three of usr/bin/mariadb-backup-server, usr/bin/mbstream-server and usr/bin/mariadb-backup-pipe in debian/mariadb-backup.install together. 5. The real mariadb-backup/mbstream binaries and the mariabackup symlink are left untouched; opt in via an alias or a symlink early in PATH. Limitations (not supported yet) =============================== 1) Incremental backup & prepare (--incremental-basedir, --incremental-dir, --apply-log-only) 2) --rollback-xa 3) Partial backup (--databases, --tables, --tables-file) 4) Output compression and encryption (--compress, --encrypt) 5) --export is accepted but only warns and runs a plain recovery 6) --extra-lsndir is ignored 7) --parallel is ignored with --stream 8) Windows: POSIX sh only, not installed on Windows Behaviour differences from native mariadb-backup ================================================ - The wrapper needs the mariadb client on PATH for --backup; --prepare needs mariadbd recorded in backup-prepare.cnf, in a standard install directory, or named by MARIADBD - BACKUP SERVER refuses an already-existing target directory - BACKUP SERVER does copy the data file as raw pages without checksum validation, so a corrupted table is not detected at backup time - --prepare only works on a wrapper-made backup: it needs backup-prepare.cnf - --stream is tar, not xbstream, local-only and single-threaded, and needs write access to @@datadir for the fifo Tests ===== include/have_mariabackup_wrapper.inc redirects $XTRABACKUP to mariadb-backup-server.sh and $XBSTREAM to mbstream-server.sh, skipping when a wrapper or the mariadb client is unavailable. include/have_mariabackup_combination.inc runs a test under both the [CLIENT] mariadb-backup binary and the [SERVER] wrapper. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41094 KDF() aliases large iteration/width to weak 32-bit values Item_func_kdf::val_str() read the PBKDF2 iteration count as a signed 64-bit longlong but only rejected values <= 0 before narrowing it to the 32-bit int expected by PKCS5_PBKDF2_HMAC(). Iteration counts that differ by 2^32 therefore aliased to the same 32-bit value and derived identical keys: e.g. 4294968296 silently did the work of 1000. Since the value is reproducible as (iter mod 2^32), any key derived this way was already only as strong as the aliased low iteration count, so no previously-derived ciphertext is orphaned by rejecting the alias now; it simply reports the weak request instead of silently honouring it. Item_func_kdf::fix_length_and_dec() had the identical bug for the key width argument: `key_length= (uint)args[4]->val_int()` narrows before the range check, and because the result is cached as a constant, the runtime guard in val_str() (which uses a wider type and is otherwise safe) is never reached for a constant width argument. This let a width like 4294967552 silently alias to 256, and let negative widths alias to a plausible positive one, both without warning. Both call sites now validate the argument's true 64-bit value before narrowing, reusing the existing invalid_argument_error() and NULL result already used for other invalid KDF() arguments. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-33616: Detect select() on macOS macOS declares select() in sys/select.h, which the HAVE_SELECT probe did not include. clang rejects a call to an undeclared function, so the probe failed and HAVE_SELECT was left undefined. my_sleep() then took its last fallback, a busy loop on time() that rounds the requested interval up to a whole second. Every sub-second sleep in the server became a one second spin on a CPU, which is what made rpl.rpl_perfschema_applier_status_by_worker, rpl.rpl_shutdown_sighup and rpl.rpl_semi_sync_shutdown_await_ack fail. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39525 Add supertype checks to vcol index substitution in WHERE TODO: fix commit message - check if vcol field is a supertype to vcol expr - check if both the vcol expr and vcol field are supertypes to the RHS If either check returns true, then the substitution is safe. Skip the checks for IS NULL / IS NOT NULL. TODO: Does not yet work for the JSON_EXTRACT/JSON_VALUE vcol examples in vcol_sargable because capacity_limit_is_in_characters returns different values for varchar and blob/text MDEV-39525 [to-squash] fix longstr supertype check The string supertype check is too strict: it bails when src (resp. dst) is length-limited in chars (i.e. CHAR/VARCHAR) and dst (resp. src) is length-limited in octets (i.e. TEXT/BLOB). We change this to conversion of src length limit to that of dst, i.e. - if src limit is declared in octets and dst limit in chars, convert src limit to be in chars, and compare - if src limit is declared in chars and dst limit in octets, convert src limit to be in octets, and compare This restores the JSON_EXTRACT cases in vcol.vcol_sargable MDEV-39525 [to-squash] Move collation match check outside of longstr is_supertype This allows WHERE json_unquote(json_extract(...))=<const_string> to use compare_collation instead. MDEV-39525 [to-squash] Check the constant is strictly inside the field domain The "both bounded" check in vcol_type_both_bounded() used Field::is_supertype() for the constant as well. That is imprecise in two ways: - It compares declared data types, and Item_int always reports INT or BIGINT, so for a TINYINT/SMALLINT/MEDIUMINT vcol the check never passes, not even for `tinyint_vcol_expr = 2`. - Being representable in the field's data type is not enough. The check should be open interval rather the closed interval For `a tinyint, va tinyint as (a+1)` and a row with a=127, the expression is 128 while va is clamped to 127, so both `a+1 > 127` and `a+1 = 127` give different results before and after the substitution. Also expand eligible Item_bool_rowready_func2 to include "<", "<=", ">" and ">=". This restores explain results with int in vcol.vcol_sargable cases. Added testcases accordingly. Note, Field::is_supertype() is still needed for the vcol field against the vcol expression: it covers the case when the conversion is not lossy at all, including a constant sitting exactly on the boundary of the domain. TODO: check results |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-33616: Widen the block count filter in the buffer pool resize test The test replaces the number of buffer pool blocks with a fixed value so that the message is stable. The pattern only accepted 5.., and macOS builds without a futex use SUX_LOCK_GENERIC, which enlarges buf_block_t enough to bring the count down into 4... |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40837: Crash while recording context when character_set_results is NULL The value of character_set_results can be set to null unlike character_set_client, or collation_connection. When recording context for a query in opt_context_store_replay.cc, character_set_results->cs_name was accessed, without checking if character_set_results value was null or not. Hence the crash. The solution is to add a null check for the variable character_set_results, before accessing cs_name, inside Optimizer_context_recorder::dump_sql_script(). |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||