Console View
|
Categories: connectors experimental galera main |
|
| connectors | experimental | galera | main | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Inline the mvi_key variable into its only use mysql_prepare_alter_table() computed it at the top of the key loop and read it ~290 lines below, at the one place that wants it. Nothing in between can change the answer, and the early call also ran for keys that never reach the assignment. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
GnuTLS: actually load CRLs from ssl_crlpath directory GnuTLS has no gnutls_certificate_set_x509_crl_dir(), so ssl_crlpath was silently ignored, letting revoked server certificates through. Enumerate the directory and load each regular file as a CRL instead. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Mohammad Tafzeel Shams
tafzeel.shams@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-37467: InnoDB Instant ALTER TABLE is not crash safe The hidden metadata record of instant ALTER TABLE was not written crash-safely, in two independent ways. First, the metadata record may include externally stored BLOB metadata. The existing BLOB storage path in btr_store_big_rec_extern_fields() writes the clustered index record first, with zero BLOB pointers, and only fills in the BLOB pointers afterwards. If the server is killed after the mini-transaction that wrote the (incomplete) metadata record was durably committed, but before the BLOB pointers were written, the table could become inaccessible on recovery. Make metadata BLOB storage crash-safe by writing the BLOB pages and computing their pointers before the metadata record itself is inserted or updated, so that the record is always written with complete BLOB pointers. If the server is killed before the metadata record is written, the already-written BLOB pages are merely orphaned, which is safe. Second, trx_undo_report_row_operation() writes the undo log record in a mini-transaction of its own, which is committed before the mini-transaction that writes the metadata record. Because innobase_instant_try() had already updated SYS_COLUMNS and SYS_TABLES in earlier mini-transactions, a kill in between left a durable undo log record for the table while the metadata record was unchanged. On recovery, trx_resurrect_table_locks() would then load the table definition before the incomplete transaction was rolled back. The data dictionary described the table as it would be after the operation, while the metadata record still described it as it was before, and btr_cur_instant_init() failed on that disagreement. Write the undo log record of the metadata record in the same mini-transaction that inserts or updates the record, so that the two cannot be separated by a crash: until that mini-transaction is committed, neither of them is durable. An undo log record is never split between pages. If the DEFAULT values of the columns being added are large enough that the undo log record for updating the metadata record would not fit on one page, innobase_instant_try() would fail. Determine this before the operation starts, so that it can be performed by another algorithm instead. - btr_store_big_rec_metadata(): New function to store the off-page columns of a metadata record ahead of time. Each BLOB page is allocated and linked in its own mini-transaction, and the resulting BLOB pointers are written directly into the (heap-resident) index entry. On failure, it frees any pages it already allocated and resets the pointers to zero. - btr_free_big_rec_metadata(): New helper to free the BLOB pages written by btr_store_big_rec_metadata() and reset the entry's BLOB pointers to zero, used both on failure inside that function and by its callers when the metadata record ends up not being written. - row_ins_clust_index_entry_low(): For a metadata entry that needs external storage, convert it to a big record and call btr_store_big_rec_metadata() (with log_free_check() allowed, since no latches are held yet) before inserting the record. On failure, free the metadata BLOBs and convert the entry back. - btr_cur_pessimistic_update(): When updating a metadata record that requires external storage, call btr_store_big_rec_metadata() (without log_free_check(), since index and page latches are held) before modifying the record, and free the temporary big_rec vector via btr_free_big_rec_metadata() or dtuple_big_rec_free() on the various failure/success paths. - btr_cur_optimistic_insert(): Remove the special-cased jump to convert_big_rec for metadata entries, since their BLOBs are now always stored ahead of time by the caller; assert that a metadata entry never needs external storage at this point. - innobase_instant_try(): Since btr_cur_pessimistic_update() now stores metadata BLOBs before updating the record, big_rec is always NULL here; assert this instead of calling btr_store_big_rec_extern_fields(). - trx_undo_report_row_operation(): New parameter caller_mtr. If it is specified, the undo log record is written in that mini-transaction, which is never committed or restarted here. An undo log page is added within the same mini-transaction if the record does not fit on the current one. A temporary table never uses the caller's mini-transaction, because that would require changing its logging mode. All other callers pass NULL and are unaffected. - btr_cur_ins_lock_and_undo(), btr_cur_upd_lock_and_undo(): For an instant ALTER TABLE metadata record, pass the mini-transaction that is going to insert or modify the record. - trx_undo_max_rec_size(): New function to determine the maximum size of an undo log record, that is, the space available on an empty undo log page. - ha_innobase::check_if_supported_inplace_alter(): Refuse ALGORITHM=INSTANT if the metadata record already exists and the undo log record for updating it would exceed trx_undo_max_rec_size(). trx_undo_page_report_modify() stores the DEFAULT value of each column that is being added in that record in full, inline. No such limit applies when the metadata record is being inserted, because trx_undo_page_report_insert() writes TRX_UNDO_INSERT_METADATA and no field data. - Added test in innodb.instant_alter and innodb.instant_alter_crash to test normal working of INSTANT ALTER, crash safety and full table. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40382 Support --ssl-crl on server builds with WolfSSL --ssl-crl was silently discarded on WolfSSL: sslopt-case.h nulled opt_ssl_crl after option parsing, and new_VioSSLFd() skipped CRL loading for HAVE_WOLFSSL. A revoked certificate could still authenticate a REQUIRE X509 / REQUIRE SUBJECT account. WolfSSL's OpenSSL-compat layer already implements X509_STORE_load_locations/X509_STORE_set_flags with real CRL enforcement, so remove both restrictions. Also fix X509_STORE_set_flags()'s success check: unlike OpenSSL, WolfSSL can return negative error codes on failure, which == 0 missed. Existing CRL tests were fixed to run with WolfSSL as well. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40382 Support --ssl-crl on server builds with WolfSSL --ssl-crl was silently discarded on WolfSSL: sslopt-case.h nulled opt_ssl_crl after option parsing, and new_VioSSLFd() skipped CRL loading for HAVE_WOLFSSL. A revoked certificate could still authenticate a REQUIRE X509 / REQUIRE SUBJECT account. WolfSSL's OpenSSL-compat layer already implements X509_STORE_load_locations/X509_STORE_set_flags with real CRL enforcement, so remove both restrictions. Also fix X509_STORE_set_flags()'s success check: unlike OpenSSL, WolfSSL can return negative error codes on failure, which == 0 missed. Existing CRL tests were fixed to run with WolfSSL as well. Note: on Windows, a revoked cert sometimes surfaces as ECONNRESET instead of a TLS alert; the test's regex handles this too. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Improve comments, formatting. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40382 Support --ssl-crl on server builds with WolfSSL --ssl-crl was silently discarded on WolfSSL: sslopt-case.h nulled opt_ssl_crl after option parsing, and new_VioSSLFd() skipped CRL loading for HAVE_WOLFSSL. A revoked certificate could still authenticate a REQUIRE X509 / REQUIRE SUBJECT account. WolfSSL's OpenSSL-compat layer already implements X509_STORE_load_locations/X509_STORE_set_flags with real CRL enforcement, so remove both restrictions. Also fix X509_STORE_set_flags()'s success check: unlike OpenSSL, WolfSSL can return negative error codes on failure, which == 0 missed. Existing CRL tests were fixed to run with WolfSSL as well. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41012 Galera appliers hang with foreign key of types UUID, INET4, INET6 wsrep_store_key_val_for_row() built the certification key of a row by collating the column value whenever the field reports MYSQL_TYPE_STRING or MYSQL_TYPE_VAR_STRING, taking the collation from Field::charset(). The data types implemented on Field_fbt - UUID, INET6 and INET4 report MYSQL_TYPE_STRING, and their charset() is my_charset_numeric, which is latin1. Their values are however plain binary and accordingly get_innobase_type_from_mysql_type() maps them to DATA_FIXBINARY. Their keys were therefore run through latin1_swedish_ci, which folds them. That corrupts the key in two ways: 1. A key mismatch for one and the same row. The reference key that wsrep_rec_get_foreign_key() appends for the parent of a child INSERT is built from the InnoDB record and is not collated, so it no longer matched the primary key carried by the parent row's own writeset. Certification saw no dependency between a child INSERT and a concurrent parent UPDATE, and two appliers could apply them in parallel causing a hang or crash. 2. A key collision between distinct rows. The folding is many to one, so different values collapse onto one key, Certification compares keys byte for byte, so unrelated rows were treated as the same row. Concurrent transactions on them certified as a conflict and one was aborted with ER_LOCK_DEADLOCK. Fix is for wsrep_store_key_val_for_row() to skip the collation for fields that InnoDB stores as binary, using the same condition as get_innobase_type_from_mysql_type(). This is a no-op for the types that worked before. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hemant Dangi
hemant.dangi@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40501: Assertion `info->type == READ_CACHE || info->type == WRITE_CACHE' failed in reinit_io_cache upon CHANGE MASTER Issue: CHANGE MASTER ... FOR CHANNEL with a channel name within MAX_CONNECTION_NAME can still overflow the OS file name limit once escaped into the relay log file name. Relay_log_info::init() then fails to open the relay log, leaving its index file unopened, but Master_info_index::remove_master_info() unconditionally calls reset_logs() on it during CHANGE MASTER's error cleanup, which hits the assertion in reinit_io_cache(). Solution: Guard the reset_logs() call in remove_master_info() with is_open(), so a relay log that was never opened is never passed to it. Also use MY_SAFE_PATH in open_index_file() so an over-length name fails deterministically instead of silently falling back to a mangled one. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thirunarayanan Balathandayuthapani
thiru@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41022 Wildcard term returns no rows when combined with a phrase in boolean mode search Problem: ======== MATCH(..) AGAINST('"zzzz" aut*' IN BOOLEAN MODE) fails to return the rows which match the wildcard term when the matching word is still in the FTS cache. fts_query_phrase_search() sets query->flags to FTS_PHRASE or FTS_PROXIMITY, but never resets it before returning. The subsequent wildcard term of the same query then finds the stale flag in fts_query_cache() and fts_query_difference(), takes the exact word lookup path instead of fts_cache_find_wildcard() and misses the words present in the cache. Solution: ========= - Reset query->flags in fts_query_phrase_search() before returning, so that the phrase or proximity state doesn't leak into the terms evaluated later in the same query. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension In one of the practical cloud MariaDB setups, a server node accesses its datadir over the network, but also has a fast local SSD storage for temporary data. The content of such temporary storage is lost when the server container is destroyed. The commit uses this ephemeral fast local storage (SSD) as an extension of the portion of InnoDB buffer pool (DRAM) that caches persistent data pages. This cache is separated from the persistent storage of data files and ib_logfile0 and ignored during backup. The following system variables were introduced: innodb_extended_buffer_pool_size - the size of external buffer pool file, if it equals to 0, external buffer pool will not be used; innodb_extended_buffer_pool_path - the directory in which the external buffer pool file is created, the data directory is used if the variable is not set. If innodb_extended_buffer_pool_size is not equal to 0, external buffer pool file will be created on startup of a normal server instance. It is created as a temporary file, so that the operating system removes it when the server closes it or exits, and no stale file is left behind after a crash. For this purpose row_merge_file_create_mode() was generalized into pfs_create_temp_file() and moved to fil0fil.cc, so that both the merge sort files and the external buffer pool file are created by the same function. Only clean pages will be flushed to external buffer pool file. There is no need to flush dirty pages, as such pages will become clean after flushing, and then will be evicted when they reach the tail of LRU list. Freed pages are not written to the external buffer pool file either, they are just evicted. The general idea of this commit is to flush clean pages to external buffer pool file when they are evicted. A page can be evicted either by transaction thread or by background thread of page cleaner. In some cases transaction thread is waiting for page cleaner thread to finish its job. We can't do flushing in external buffer pool file when transaction threads are waiting for eviction, that would hurt performance. That's why the only case for flushing is when page cleaner thread evicts pages in background and there are no waiters. For this purpose buf_pool_t::done_flush_list_waiters_count variable was introduced, we flush evicted clean pages only if the variable is zeroed. Clean pages are evicted in buf_flush_LRU_list_batch() to keep some amount of pages in buffer pool's free list. That's why we flush every second page to external buffer pool file, otherwise there could be not enough amount of pages in free list to let transaction threads to allocate buffer pool pages without page cleaner waiting. This might be not a good solution, but this is enough for prototyping. External buffer pool page is introduced to store information in buffer pool page hash about the certain page can be read from external buffer pool file. The first several members of such page must be the same as the members of internal page. External page frame must be equal to the certain value to distinguish external page from internal one. External buffer pages are preallocated on startup in external pages array. We could get rid of the frame in external page, and check if the page's address belongs to the array to distinguish external and internal pages. There are also external pages free and LRU lists. When some internal page is decided to be flushed in external buffer pool file, a new external page is allocated either from the head of external free list, or from the tail of external LRU list. Both lists are protected with buf_pool.mutex. It makes sense, because a page is removed from internal LRU list during eviction under buf_pool.mutex. Then internal page is locked and the allocated external page is attached to io request for external buffer pool file, and when write request is completed, the internal page is replaced with external one in page hash, external page is pushed to the head of external LRU list and internal page is unlocked. After internal page was removed from external free list, it was not placed in external LRU, and placed there only after write completion, so the page can't be used by the other threads until write is completed. Page hash chain get element function has additional template parameter, which notifies the function if external pages must be ignored or not. We don't ignore external pages in page hash in two cases, when some page is initialized for read and when one is reinitialized for new page creating. When an internal page is initialized for read and external page with the same page id is found in page hash, the internal page is locked, the external page in replaced with newly initialized internal page in the page hash chain, the external page is removed from external LRU list and attached to io request to external buffer pool file. When the io request is completed, external page is returned to external free list, internal page is unlocked. So during read external page is absent in both external LRU and free lists and can't be reused. When an internal page is initialized for new page creating and external pages with the same page id is found in page hash, we just remove external page from the page hash chain and external LRU list and push it to the head of external free list. So the external page can be used for future flushing. The external buffer pool file is not represented by a fil_space_t. The requests to it are issued by fil_system_t::ext_bp_io(), and the external buffer pool page which a request refers to is stored in IORequest in place of the fil_node_t. The pages are written to and read from the external buffer pool file in the same form in which they are written to their tablespaces, i.e. compressed and encrypted pages stay compressed and encrypted in external buffer pool file. If a write to the external buffer pool file fails, a warning is written to the error log and the external buffer pool is disabled for the rest of the server lifetime. Currently the commit passed some local smoke tests, mtr and RQG tests with external buffer pool turned on. TODO: 1. Add some monitoring, i.e. how much pages are currently in external buffer pool, the percent of hits during reading, take a look at the current buffer pool monitoring and implement the general monitoring tools for external buffer pool. 2. Think about partial initialization of external pages array, as it was done for internal pages. 3. Take a look at compressed LRU list, it looks like currently it's not covered with eviction to external buffer pool file (I don't currently understand if we need it at all). 4. Think about more suitable algorithm for eviction to external buffer pool, currently just every second page is flushed. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alexander Barkov
bar@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39518 Allow prepared statements in stored functions in assignment right hand Allowing prepared statements in stored functions when a stored function is used in an assignment right hand. Both DEFAULT clause of a variable initialization and the right side of the SET statement are supported: CREATE PROCEDURE p1() BEGIN -- case 1: DEFAULT clause DECLARE spvar1 INT DEFAULT f1_with_ps(); -- OK -- case 2: SP variable assignment statement DECLARE spvar2 INT; SET spvar2= f1_with_ps(); -- OK END; - Only assignments to SP variables works for now: * SET spvar= func_with_ps(); -- OK * SET @uvar= func_with_ps(); -- Error - Only bare function calls are supported for now. Using a function in an expression does not make it PS-safe yet: SET v= f1()+0; - The parser now does not reject PS statements in stored functions. PS applicability in stored functions is now detected at run time. Note, PS statements in triggers are still prohibited by the parser. - Functions with PS do not acquire MDL locks on tables, and no MDL is taken on the routines themselves either. They work like procedures in terms of table opening and routine locking: a concurrent DROP FUNCTION can complete while such a function is executing. - Functions with PS are not replicated as a single `SELECT f1()` call. They are replicated per-statement, like procedures. Helper changes: - Changing the return result for LEX::sp_variable_declarations_init() from void to bool to catch errors in the caller properly. Misc: - This patch incorporates fixes for the following bugs found during debugging: MDEV-40224,MDEV-40225,MDEV-40226,MDEV-40227,MDEV-40240, MDEV-40285,MDEV-40288,MDEV-40315,MDEV-40318,MDEV-40890,MDEV-40900, MDEV-40901,MDEV-40913,MDEV-40914,MDEV-41013,MDEV-41015,MDEV-41019 Assisted-by: Claude - reviews and minor clean-ups |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MDEV-40168 nested array handling and json validation | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
This is a combination of 5 commits. Undo whitespace changes to reduce diff size Inline the mvi_key variable into its only use mysql_prepare_alter_table() computed it at the top of the key loop and read it ~290 lines below, at the one place that wants it. Nothing in between can change the answer, and the early call also ran for keys that never reach the assignment. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Move the multi-valued key part DDL out of the grammar multi_valued_key_part: built a whole schema object in its action: the hidden DB_MVI_<n> column with its MVI_ENCODE() vcol, the rewrite of the key into an invisible fulltext one, and the key part naming the column. That is DDL, and it belongs with the rest of the multi-valued index code, not in sql_yacc.yy where nobody reading opt_multi_valued_index.cc will find it. It becomes add_mvi_key_part(), which returns the key part or NULL if it raised an error, and the production is three lines. check_mvi_key_type() follows its only caller and turns static, so sql_table.h loses a declaration and the forward `class Key;` that existed only for it. No behaviour change. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> opt_mvi_jsonfuncs.cc: Move the code, const-ify, add comments. Improve comments, formatting. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MDEV-40168 Add testcases for when there's both a ft index and an mvi | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Move the multi-valued key part DDL out of the grammar multi_valued_key_part: built a whole schema object in its action: the hidden DB_MVI_<n> column with its MVI_ENCODE() vcol, the rewrite of the key into an invisible fulltext one, and the key part naming the column. That is DDL, and it belongs with the rest of the multi-valued index code, not in sql_yacc.yy where nobody reading opt_multi_valued_index.cc will find it. It becomes add_mvi_key_part(), which returns the key part or NULL if it raised an error, and the production is three lines. check_mvi_key_type() follows its only caller and turns static, so sql_table.h loses a declaration and the forward `class Key;` that existed only for it. No behaviour change. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alessandro Vetere
iminelink@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40408 btr_page_reorganize_low() uses the buffer pool just to obtain a scratch block Add buf_pool.scratch_buf, a pool of page frames that the page reorganization operations use instead of taking a block from the global buffer pool. buf_pool_t::scratch_buffer: A singly linked list of chunks of buf_tmp_buffer_t slots. A thread holds at most one slot at a time, and it holds a page latch while doing so, which is why reserve() must not wait for the buffer pool or for I/O. When every slot is in use, reserve() appends a chunk that holds twice the slots of the last one, up to a limit on the size of one chunk. Only grow() waits, on the mutex that serializes it and on the allocator. A chunk is never moved or freed before close(), so a thread can keep using the slot that it reserved while another thread appends a chunk. Debug builds start with a single slot, so that a second concurrent page reorganization exercises grow(). buf_pool_t::scratch_buffer::shrink(): Free the page frames of the slots that are not in use. The slots and the chunks are kept, because a slot costs 32 bytes while a page frame costs srv_page_size. A frame survives the first pass, because that pass only clears the used flag. The master thread calls this often while the server is idle and rarely while it is active, and buf_pool_t::garbage_collect() calls it under memory pressure, where it ignores the used flag, because releasing these frames is much cheaper than shrinking the buffer pool. buf_pool_t::io_buf_t::acquire(): Factor out the scan for an unreserved slot, which io_buf_t::reserve() ran twice and the scratch buffer reuses. btr_page_reorganize_low(), page_zip_reorganize(): Obtain the scratch page frame from buf_pool.scratch_buf. This removes the buf_pool.mutex acquisition and the free block wait that buf_block_alloc() could perform while at least a page X-latch was being held. btr_page_reorganize_low() also used to leak the block on its error paths; a single exit now releases the slot. page_copy_rec_list_end_no_locks(), lock_move_reorganize_page(): Take the source page frame instead of a source buf_block_t, because the source is no longer a buffer pool block. In page_copy_rec_list_end_no_locks() the source page is page_align(rec) at every call site, so only the record is passed. buf_tmp_buffer_t::acquire(), buf_tmp_buffer_t::release(): Use acquire and release memory ordering, so that the page frame pointer of a slot is published to the next thread that reserves the slot. acquire() also reads the flag before the exchange, so that a scan across an array of slots does not write to the slots that it finds reserved. release() also marks the page frame undefined for Valgrind and MSan, because the frame stays allocated for the next reserver and no deallocation marks it. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MDEV-40168 Add some DDL tests | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| fixup! 215c4e74e17bf5fb03af376c08159210330b9950 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| opt_mvi_jsonfuncs.cc: Move the code, const-ify, add comments. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40382 Support --ssl-crl on server builds with WolfSSL --ssl-crl was silently discarded on WolfSSL: sslopt-case.h nulled opt_ssl_crl after option parsing, and new_VioSSLFd() skipped CRL loading for HAVE_WOLFSSL. A revoked certificate could still authenticate a REQUIRE X509 / REQUIRE SUBJECT account. WolfSSL's OpenSSL-compat layer already implements X509_STORE_load_locations/X509_STORE_set_flags with real CRL enforcement, so remove both restrictions. Also fix X509_STORE_set_flags()'s success check: unlike OpenSSL, WolfSSL can return negative error codes on failure, which == 0 missed. Existing CRL tests were fixed to run with WolfSSL as well. Note: on Windows, a revoked cert sometimes surfaces as ECONNRESET instead of a TLS alert; the test's regex handles this too. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hemant Dangi
hemant.dangi@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40501: Assertion `info->type == READ_CACHE || info->type == WRITE_CACHE' failed in reinit_io_cache upon CHANGE MASTER Issue: CHANGE MASTER ... FOR CHANNEL with a channel name within MAX_CONNECTION_NAME can still overflow the OS file name limit once escaped into the relay log file name. Relay_log_info::init() then fails to open the relay log, leaving its index file unopened, but Master_info_index::remove_master_info() unconditionally calls reset_logs() on it during CHANGE MASTER's error cleanup, which hits the assertion in reinit_io_cache(). Solution: Guard the reset_logs() call in remove_master_info() with is_open(), so a relay log that was never opened is never passed to it. Also use MY_SAFE_PATH in open_index_file() so an over-length name fails deterministically instead of silently falling back to a mangled one. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MDEV-40168 Add some DDL tests | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40382 Narrow ssl_crl.test's regex, don't mask unrelated failures The previous fix matched any text after "TLS/SSL error:", too broad. Require "revoked" or "10054" (WSAECONNRESET, seen on AppVeyor) so an unrelated 2026 error still fails the test. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Fix the version | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Undo whitespace changes to reduce diff size | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thirunarayanan Balathandayuthapani
thiru@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-19574: innodb_stats_method is not honored when innodb_stats_persistent=ON Problem: ======= When persistent statistics are enabled (innodb_stats_persistent=ON), the innodb_stats_method setting is not properly utilized during statistics calculation. The statistics collection functions always use a hardcoded default behavior for NULL value comparison instead of respecting the configured stats method. This affects the accuracy of n_diff_key_vals (distinct key count), particularly for indexes with nullable columns containing NULL values. Moreover, stat_n_non_null_key_vals[] was never computed for persistent statistics; it stayed at the 0 that dict_stats_empty_index() assigns. With innodb_stats_method=nulls_ignored, innodb_rec_per_key() therefore always found n_diff <= n_null and reported one record per key for every index. This impacts the query optimizer, which makes decisions based on inaccurate cardinality estimates. Solution: ======== Introduced IndexLevelStats to collect statistics at a specific B-tree level during index analysis. Introduced PageStats to collect statistics for leaf page analysis. Refactored the following functions: dict_stats_analyze_index_level() to IndexLevelStats::analyze_level() dict_stats_analyze_index_for_n_prefix() to IndexLevelStats::sample_leaf_pages() dict_stats_analyze_index_below_cur() to PageStats::scan_below() dict_stats_scan_page() to PageStats::scan() The innodb_stats_method value is read once per table in dict_stats_update_persistent() and passed down, so that all indexes of a table are analyzed with the same method. Add the stats method name to stat_description when innodb_stats_method has a non-default value. The suffix is dropped when the description is already full. Added the new stat name n_nonnull_fld01, n_nonnull_fld02, etc. with a stats description, to indicate how many non-null values exist for the nth field of the index. This value is retrieved and stored in the index statistics in dict_stats_fetch_index_stats_step(). The counts are per column, not per n-column prefix. rec_get_n_blob_pages(): Calculate the number of externally stored pages for a record, using ceiling division by the usable BLOB page payload (blob_part_size), which differs between ROW_FORMAT=COMPRESSED (zip_size minus FIL_PAGE_DATA) and the other formats (srv_page_size minus the BLOB header and the page trailer). For ROW_FORMAT=COMPRESSED the length in the field reference is the uncompressed length, so the result is an upper bound. When the leaf level is scanned in full, the number of leaf pages that were scanned is reported as n_leaf_pages for a multi level index. Before, result.n_leaf_pages was overwritten with index->stat_n_leaf_pages, which dict_stats_empty_index() had just set to 1, so every index that took the full scan path reported n_leaf_pages=1. Single page indexes report 1. This changes cardinality estimates and therefore leads to multiple changes in existing test cases. Non-null values are counted only at the leaf level, since only leaf pages hold actual records. A full scan of the leaf level counts them exactly. When the level is sampled, the per column count is derived from the sampled leaves with the same formula as n_diff: n_ordinary_leaf_pages * n_non_null_all_analyzed_pages / n_leaf_pages_to_analyze This is an estimate for NOT NULL columns as well: the sampled leaves may hold fewer or more records than the average, and a dive that stops at a boring page contributes nothing to the sum while still counting in the divisor. innodb_rec_per_key(): stat_n_non_null_key_vals[i] holds the number of records in which the i-th indexed column alone is not NULL, while what has to be excluded here is the number of records whose first i+1 columns are all not NULL, because that is the population which the n-column prefix statistic stat_n_diff_key_vals[i] has to be corrected against when innodb_stats_method=nulls_ignored: with NULLs compared as unequal, every record carrying a NULL anywhere in the prefix adds a distinct value of its own to n_diff. PageStats::scan(): n_non_null is accumulated and assigned only for leaf pages, so that a non-leaf scan cannot leave a node pointer count behind when scan_below() stops at a boring page without reaching a leaf. IndexLevelStats::reset_for_level() also clears n_diff[], and dict_stats_analyze_index() zero initializes the buffer backing it, so that a level scan which finds no records (a failed btr_pcur_open_level(), or a non-leaf page whose first record is not marked as the leftmost one on the level) leaves n_diff[] at 0 instead of stale values. IndexLevelStats::sample_leaf_pages() returns early when the group boundaries for the prefix are empty, which is the same condition. IndexLevelStats::analyze_level(): Instead of copying the last record of the page, retain the latch on the page until the record has been compared with the first record of the next page dict_stats_fetch_index_stats_step() no longer resets stat_n_non_null_key_vals[] while processing an n_diff_pfxNN row: dict_stats_empty_table() has already cleared the array before the fetch, and with n_nonnull_fldNN rows now being read too, that reset would make the result depend on the order in which the rows arrive. dict_stats_save(): now static function in dict0stats.cc that takes the innodb_stats_method value, and is removed from dict0stats.h. dict_stats_update_persistent() saves the statistics itself, so its callers no longer have to. Replaced btr_rec_get_externally_stored_len() with rec_get_n_blob_pages() in dict0stats.cc. btr_rec_get_field_ref_offs() and btr_rec_get_field_ref(), together with the BTR_BLOB_HDR_* macros, were moved from btr0cur.cc to btr0cur.h so that rec_get_n_blob_pages() can reuse them; btr_rec_get_field_ref_offs() is now a noexcept function returning size_t. Changed stat_n_diff_key_vals and stat_n_non_null_key_vals from ib_uint64_t* to uint64_t* len_is_stored(): simplified to a single comparison, which is equivalent for the unsigned lengths that it is used with. Removed the unused UNIV_STATS_DEBUG build macro (univ.i) and turned the DEBUG_PRINTF() helper in dict0stats.cc into an unconditional no-op |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40382 Fix ssl_crl.test regex, AppVeyor got a connection reset On AppVeyor's Windows CI, the client saw a raw WSAECONNRESET (via libmariadb's Schannel backend, reported as error 2026) rather than a parsed TLS alert. Match any text after "TLS/SSL error:" instead of requiring "revoked", using [^\r\n]* instead of .* so the match can't swallow the line's trailing newline. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thirunarayanan Balathandayuthapani
thiru@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-26057 Assertion `!vcol->v_indexes.empty() in trx_undo_log_v_idx Problem: ======== - Rollback of an INPLACE ALTER TABLE is executed while holding only a shared metadata lock on the table, so DML can run concurrently. rollback_inplace_alter_table() resets dict_col_t::ord_part in a critical section of its own, after row_merge_drop_indexes() already removed the aborted indexes from the dictionary cache and emptied dict_v_col_t::v_indexes. During this time, DML statement can see a virtual column with ord_part set and an empty v_indexes, which makes assert failure in trx_undo_report_insert_virtual(). Solution: ======== row_merge_reset_ord_part(): Added a function to reset dict_col_t::ord_part for the columns that are no longer a field of any index remaining in the dictionary cache. For virtual columns the decision is based on dict_v_col_t::v_indexes being empty, and no element is ever removed from that list. row_merge_drop_indexes(): Added a call to row_merge_reset_ord_part() in the branch that removes the indexes from the cache, in the same dict_sys.latch critical section. That branch is taken only when MDL_EXCLUSIVE is held or when this is the only handle to the table, so no concurrent DML can observe the intermediate state. In the lazy drop branch the indexes and their v_indexes entries stay in the cache and nothing is reset; that is done later, when the indexes are dropped while holding MDL_EXCLUSIVE. check_col_exists_in_indexes(): Removed the only_committed parameter, which no longer has any caller. row_quiesce_col_ord_part(): Added a function to get dict_col_t::ord_part and dict_col_t::max_prefix of a column from the committed indexes that are present in the dictionary cache. row_quiesce_write_table(): Write the row_quiesce_col_ord_part() return values to the .cfg file instead of the cached dict_col_t fields, because a rolled back ADD INDEX leaves ord_part set until the aborted index is removed by a later DDL, and max_prefix is never reset when an index is dropped, which makes IMPORT TABLESPACE reject the tablespace with a bogus schema mismatch. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41012 Galera appliers hang with foreign key of types UUID, INET4, INET6 Added second test for testing key collisions from transactions modifying separate rows |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41012 Galera appliers hang with foreign key of types UUID, INET4, INET6 Added a deterministic test for reproducing the issue |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41012 Galera appliers hang with foreign key of types UUID, INET4, INET6 Bumped application protocol version to level 5 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Otto Kekäläinen
otto@kekalainen.net |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Promote getting GitHub stars in client prompt Ask users to give MariaDB a star by having an extra line in the MariaDB client prompt: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is X Server version: Y Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Help others discover MariaDB. Star it on GitHub: https://github.com/MariaDB/server Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> Test file updated with: nano --noconvert --nonewlines mysql-test/main/mysql-interactive.result This change is similar to 346c7afe9b7071ce9c47892a83d69944b608b3da applied on 'main', but without the SERVER_MATURITY_LEVEL check in order to have this text visible in stable releases and actually help get the GitHub star count up, and without the server log entry. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
CONC-852 GnuTLS: actually load CRLs from ssl_crlpath directory GnuTLS has no gnutls_certificate_set_x509_crl_dir(), so ssl_crlpath was silently ignored, letting revoked server certificates through. Enumerate the directory and load each regular file as a CRL instead. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
squash! b0944c9a38c3a3d5d7b9c87d3b137f67236bd744 BACKUP SERVER WITH 'command': Prepend the command with mariadb-backup- and expect it to be in the search PATH. We distribute no such scripts; the user has to write one. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thirunarayanan Balathandayuthapani
thiru@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38942 i_s_dict_fill_sys_tables() aborts when reading INNODB_SYS_TABLES after innodb_force_recovery Problem: ======== A query on INFORMATION_SCHEMA.INNODB_SYS_TABLES crashes when SYS_TABLES contains a record that was inserted by a transaction which has not been committed. This can happen after a crash while a CREATE TABLE was in progress, if the server is restarted with innodb_force_recovery=4 or greater, because trx_rollback_recovered() is then skipped and the recovered transaction remains ACTIVE. dict_sys_tables_rec_read() returns READ_NOT_FOUND for such a record, and dict_load_table_low() returns that as success with no error message and setting *table to nullptr. i_s_sys_tables_fill_table() checks only the error message and passes the nullptr table to i_s_dict_fill_sys_tables(), which dereferences it. Solution: ======== i_s_sys_tables_fill_table(): Skip the SYS_TABLES record when dict_load_table_low() reports success but returns no table, because such a record is not visible. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thirunarayanan Balathandayuthapani
thiru@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39061 mariadb-backup compatible wrapper for BACKUP SERVER This adds a shell script that lets users keep using their existing mariadb-backup commands while the real work is done by the new server-side BACKUP SERVER command. The goal is "drop-in": users should not have to change their backup scripts. extra/mariabackup/scripts/mariadb-backup-server.sh (plain POSIX sh) understands the usual mariadb-backup modes and translates each one. A companion helper, extra/mariabackup/scripts/mbstream-server.sh, lets streamed backups be unpacked by pipelines that expect the mbstream CLI. Both are documented in extra/mariabackup/scripts/README.md. --backup ======== Connects with the mariadb client and runs "BACKUP SERVER TO '<dir>'". Connection options (--user, --host, --port, --socket, --defaults-file, ssl, ...) are passed through to the client; --parallel=N becomes the "<N + 1> CONCURRENT": mariadb-backup runs a dedicated log_copying_thread() besides its N-data copy threads. Minimum value is 2 After the backup it writes backup-prepare.cnf into the backup directory, recording what --prepare needs later: where mariadbd lives, the InnoDB parameters (page size, data file path, undo tablespaces, checksum algorithm, log file size), and if the server is encrypted then how to reload the encryption key plugin (the file_key_management variables), so an encrypted backup can be prepared without extra input. --backup --stream ================= Runs "BACKUP SERVER WITH '<command>'", where <command> writes the tar into a fifo that the wrapper drains to its own stdout, so the backup reaches the consumer as it is produced and never lands on local disk. backup-prepare.cnf is appended as a final tar afterwards. The server's tar carries no end-of-archive marker; only that trailing archive adds one, so the whole stream extracts with a plain "tar -x". --parallel is ignored here, with a warning. CONCURRENT selects the number of output sinks as well as workers, and N self-contained tars cannot be interleaved on one stdout, so a stream needs one writer. For parallel streaming use BACKUP SERVER WITH N CONCURRENT directly, giving each stream its own destination via the appended index. Three properties follow from how BACKUP SERVER streams, all differing from mariadb-backup: - local: the stream command runs inside the server, so the wrapper must share its filesystem; ---target-dir is optional in stream mode (scratch for the per-stream. - tar only: any --stream=<format> (including xbstream) yields tar; - single-threaded: one worker, so no parallel read either. --target-dir is optional in stream mode; nothing is spooled there, it only holds the helper script, the fifo and backup-prepare.cnf (a mktemp dir is used otherwise). mbstream-server.sh maps the mbstream CLI onto a plain "tar -x"/"tar -c", so existing "mbstream -x"/"-c" pipelines keep working on the wrapper's stream. mbstream-only flags (-p/--parallel, ...) are accepted and ignored; any other unknown option is rejected. Environment overrides (mainly for testing): MARIADB (client), MARIADBD (the --prepare bootstrap server) and TAR (the tar implementation, e.g. TAR=bsdtar) can each be overridden. To run the bootstrap under rr, put it in MARIADBD and let rr's own _RR_TRACE_DIR choose the trace location, e.g. _RR_TRACE_DIR=/dev/shm/rr MARIADBD='rr record mariadbd' --prepare ========= Starts "mariadbd --bootstrap" on the backup directory using backup-prepare.cnf as its defaults file, replays the archived redo log between the start and target LSN read from backup.cnf, then builds a fresh ib_logfile0 so a normal server can start on the directory. mariadbd is taken from the path recorded in backup-prepare.cnf if that binary exists, otherwise by searching /libexec, /sbin, /bin and the configured install directories. PATH is not searched; set MARIADBD to point elsewhere. User --defaults-file/-extra-file and encryption options are layered onto the bootstrap. --copy-back / --move-back ========================= Copy or move a prepared backup into the datadir. The datadir is created if missing, a non-empty datadir is refused unless --force-non-empty-directories is given, and a chown reminder is printed. If --aria-log-dir-path is given, the Aria logs (aria_log_control, aria_log.*) are relocated into that directory. Packaging ========= The wrapper is not installed by default and never replaces the real mariadb-backup / mbstream binaries. 1. cmake -DWITH_MARIABACKUP_WRAPPER=ON (default OFF) controls it. 2. When ON, the scripts install as /usr/bin/mariadb-backup-server and /usr/bin/mbstream-server, tagged COMPONENT Backup so they ship in the mariadb-backup package. 3. RPM: nothing extra to do. the component handles it. 4. DEB: not wired. debian/rules uses --fail-missing and does not enable the option, so the -server binaries are not listed. To ship via DEB, make a paired change: add -DWITH_MARIABACKUP_WRAPPER=ON in debian/rules and list both usr/bin/mariadb-backup-server and usr/bin/mbstream-server in debian/mariadb-backup.install together. 5. The real mariadb-backup/mbstream binaries and the mariabackup symlink are left untouched; opt in via an alias or a symlink early in PATH. Limitations (not supported yet) =============================== 1) Incremental backup & prepare (--incremental-basedir, --incremental-dir, --apply-log-only) 2) --rollback-xa 3) Partial backup (--databases, --tables, --tables-file) 4) Output compression and encryption (--compress, --encrypt) 5) --export is accepted but only warns and runs a plain recovery 6) --extra-lsndir is ignored 7) --parallel is ignored with --stream 8) Windows: POSIX sh only, not installed on Windows Behaviour differences from native mariadb-backup ================================================ - The wrapper needs the mariadb client on PATH for --backup, and mariadbd on PATH (or recorded in backup-prepare.cnf) --backup; --prepare needs mariadbd recorded in backup-prepare.cnf, in a standard install directory, or named by MARIADBD - BACKUP SERVER refuses an already-existing target directory - BACKUP SERVER does copy the data file as raw pages without checksum validation, so a corrupted table is not detected at backup time - --prepare only works on a wrapper-made backup. It needs backup-prepare.cnf) - --stream is tar, not xbstream, local-only and single-threaded Tests ===== include/have_mariabackup_wrapper.inc redirects $XTRABACKUP to mariadb-backup-server.sh and $XBSTREAM to mbstream-server.sh, skipping when a wrapper or the mariadb client is unavailable. include/have_mariabackup_combination.inc runs a test under both the [CLIENT] mariadb-backup binary and the [SERVER] wrapper. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vladislav Vaintroub
vvaintroub@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40287 AES_ENCRYPT() and KDF() return NULL with OpenSSL 4.0 MyCTX faked an EVP_CIPHER_CTX in a stack buffer instead of allocating it with EVP_CIPHER_CTX_new(). OpenSSL 4.0 rejects a context that was not created that way for ciphers that use an IV: EVP_CipherInit_ex() fails with "invalid iv length", so AES_ENCRYPT() and KDF() in CBC/CTR/GCM modes return NULL. ECB has no IV and still works, which is why only the non-ECB modes broke. Allocate the context with EVP_CIPHER_CTX_new()/EVP_CIPHER_CTX_free(). Also remove check_openssl_compatibility() (it only validated the old buffer's size, which was never the problem here - the context is 184 bytes) and the EVP_CIPHER_CTX_SIZE / EVP_CIPHER_CTX_init macros. Verified against OpenSSL 4.0.1: the mysys aes-t test fails on the CBC/CTR/GCM cases with the stack buffer and passes with EVP_CIPHER_CTX_new(). No visible performance degradation: the extra allocation costs ~12 ns/call (WolfSSL) and ~30 ns (OpenSSL) on Windows at a 30-byte payload, nothing at 16 KB, and nothing on Linux/glibc; sysbench OLTP over encrypted tables and redo log is unchanged. Assisted-by: Claude:claude-opus-4-8 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dimitri John Ledkov
dimitri.ledkov@surgut.co.uk |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-26015 ssl: remove insecure fixed DH params (mostly unused)- #5639 WolfSSL code path already operates without fixed DH parameters. OpenSSL code path still sets fixed static precomputed DH params, which is now prohibited by IETF. Also OPENSSL_init_ssl is not required since OpenSSL 1.1.0, for over 10 years now. Also cleaned up at the same time. https://www.rfc-editor.org/rfc/rfc10015.html#section-2: > Clients MUST NOT offer and servers MUST NOT select non-ephemeral FFDH cipher suites in (D)TLS 1.2 connections. https://www.rfc-editor.org/rfc/rfc10015.html#section-3: > Clients MUST NOT offer and servers MUST NOT select FFDHE cipher suites in (D)TLS 1.2 connections. And the depreciated tables include all ciphersuites that can use SSL_CTX_set_tmp_dh as part of the connection. Also for a very long time OpenSSL was handling these automatically anyway, back when DHE was still recommended. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-36356 Server crash in Item::save_int_in_field with Window functions A tail (ORDER BY / LIMIT / locking clause) that follows a parenthesized query expression is parsed while the select inside the parentheses is the current one, so everything the tail registers - its window functions, the window specifications they introduce and the units of its subqueries - ends up registered in that inner select. When the parenthesized query expression already has a tail of its own, LEX::add_tail_to_query_expression_body_ext_parens() wraps it into a derived table and attaches the new tail to the wrapping select. The registrations were left behind in the inner select, so the ORDER BY of the wrapping select contained window function items that no select had registered: they never got a Window_funcs_sort, were never computed, and their result field was read unset. The subqueries of the tail kept the inner select as their master and name resolution context. Move these registrations to the wrapping select in the new move_tail_registrations(): - The window functions of the tail are the ones its ORDER BY items contain, looked up with Item::walk() and walk_subquery == FALSE so that a window function belonging to a subquery of the tail is not found and stays registered where it was parsed. Each one takes along the window specification it introduced; "OVER win_name" has none of its own, it is looked up by name at fix_fields() time and never reaches window_specs. - Item::walk() does not descend into a window specification, so the items of a moved specification's PARTITION BY and ORDER BY lists are not re-targeted by Lex_order_limit_lock::set_to() and are pointed at the wrapping select's name resolution context here. - The subquery units of the tail are the ones registered in front of the first unit the inner select had when the tail started to be parsed; the grammar remembers it in a mid-rule action. There is no equivalent of the walk above for them: a unit records neither the clause it came from nor its position in the parsed text. They are excluded from the inner select, re-registered in the wrapping select and their Item_subselect::parent_select is updated. Per-select parse state that is not a registration (n_sum_items, with_sum_func, with_rownum, ftfunc_list, uncacheable, and the select_n_where_fields of the tail's own fields) is deliberately left behind, where it is merely over-counted. Also remove the unreachable !unit check - unit is dereferenced on entry - and fix the indentation of the surrounding block. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||