Console View
|
Categories: connectors experimental galera main |
|
| connectors | experimental | galera | main | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension In one of the practical cloud MariaDB setups, a server node accesses its datadir over the network, but also has a fast local SSD storage for temporary data. The content of such temporary storage is lost when the server container is destroyed. The commit uses this ephemeral fast local storage (SSD) as an extension of the portion of InnoDB buffer pool (DRAM) that caches persistent data pages. This cache is separated from the persistent storage of data files and ib_logfile0 and ignored during backup. The following system variables were introduced: innodb_extended_buffer_pool_size - the size of external buffer pool file, if it equals to 0, external buffer pool will not be used; innodb_extended_buffer_pool_path - the directory in which the external buffer pool file is created, the data directory is used if the variable is not set. If innodb_extended_buffer_pool_size is not equal to 0, external buffer pool file will be created on startup of a normal server instance. It is created as a temporary file, so that the operating system removes it when the server closes it or exits, and no stale file is left behind after a crash. For this purpose row_merge_file_create_mode() was generalized into pfs_create_temp_file() and moved to fil0fil.cc, so that both the merge sort files and the external buffer pool file are created by the same function. Only clean pages will be flushed to external buffer pool file. There is no need to flush dirty pages, as such pages will become clean after flushing, and then will be evicted when they reach the tail of LRU list. Freed pages are not written to the external buffer pool file either, they are just evicted. The general idea of this commit is to flush clean pages to external buffer pool file when they are evicted. A page can be evicted either by transaction thread or by background thread of page cleaner. In some cases transaction thread is waiting for page cleaner thread to finish its job. We can't do flushing in external buffer pool file when transaction threads are waiting for eviction, that would hurt performance. That's why the only case for flushing is when page cleaner thread evicts pages in background and there are no waiters. For this purpose buf_pool_t::done_flush_list_waiters_count variable was introduced, we flush evicted clean pages only if the variable is zeroed. Clean pages are evicted in buf_flush_LRU_list_batch() to keep some amount of pages in buffer pool's free list. That's why we flush every second page to external buffer pool file, otherwise there could be not enough amount of pages in free list to let transaction threads to allocate buffer pool pages without page cleaner waiting. This might be not a good solution, but this is enough for prototyping. External buffer pool page is introduced to store information in buffer pool page hash about the certain page can be read from external buffer pool file. The first several members of such page must be the same as the members of internal page. External page frame must be equal to the certain value to distinguish external page from internal one. External buffer pages are preallocated on startup in external pages array. We could get rid of the frame in external page, and check if the page's address belongs to the array to distinguish external and internal pages. There are also external pages free and LRU lists. When some internal page is decided to be flushed in external buffer pool file, a new external page is allocated either from the head of external free list, or from the tail of external LRU list. Both lists are protected with buf_pool.mutex. It makes sense, because a page is removed from internal LRU list during eviction under buf_pool.mutex. Then internal page is locked and the allocated external page is attached to io request for external buffer pool file, and when write request is completed, the internal page is replaced with external one in page hash, external page is pushed to the head of external LRU list and internal page is unlocked. After internal page was removed from external free list, it was not placed in external LRU, and placed there only after write completion, so the page can't be used by the other threads until write is completed. Page hash chain get element function has additional template parameter, which notifies the function if external pages must be ignored or not. We don't ignore external pages in page hash in two cases, when some page is initialized for read and when one is reinitialized for new page creating. When an internal page is initialized for read and external page with the same page id is found in page hash, the internal page is locked, the external page in replaced with newly initialized internal page in the page hash chain, the external page is removed from external LRU list and attached to io request to external buffer pool file. When the io request is completed, external page is returned to external free list, internal page is unlocked. So during read external page is absent in both external LRU and free lists and can't be reused. When an internal page is initialized for new page creating and external pages with the same page id is found in page hash, we just remove external page from the page hash chain and external LRU list and push it to the head of external free list. So the external page can be used for future flushing. The external buffer pool file is not represented by a fil_space_t. The requests to it are issued by fil_system_t::ext_bp_io(), and the external buffer pool page which a request refers to is stored in IORequest in place of the fil_node_t. The pages are written to and read from the external buffer pool file in the same form in which they are written to their tablespaces, i.e. compressed and encrypted pages stay compressed and encrypted in external buffer pool file. If a write to the external buffer pool file fails, a warning is written to the error log and the external buffer pool is disabled for the rest of the server lifetime. Currently the commit passed some local smoke tests, mtr and RQG tests with external buffer pool turned on. TODO: 1. Add some monitoring, i.e. how much pages are currently in external buffer pool, the percent of hits during reading, take a look at the current buffer pool monitoring and implement the general monitoring tools for external buffer pool. 2. Think about partial initialization of external pages array, as it was done for internal pages. 3. Take a look at compressed LRU list, it looks like currently it's not covered with eviction to external buffer pool file (I don't currently understand if we need it at all). 4. Think about more suitable algorithm for eviction to external buffer pool, currently just every second page is flushed. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MDEV-25848 Allow "js IS NOT NULL AND <JSON_SARGABLE_PREDICATE on js>" to use range access | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Georgi (Joro) Kodinov
joro@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42338: Treat openssl,pcre and zlib library headers as "system" Mark the 3d party library headers as "system" headers so that the warnings in these files are suppressed for supporting compilers. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Andrei Elkin
andrei.elkin@pp.inet.fi |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42337 no semi-sync kill of a closed connection This is MDEV-42004 fixes follow-up. Repl_semi_sync_slave::kill_connection() reconnected to the master at the IO thread exit even though end_server() had already closed the connection (e.g. STOP SLAVE interrupted a reconnect), so the stop waited rpl_semi_sync_slave_kill_conn_timeout once more. Return early when mysql->net.vio is NULL. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Andrei Elkin
andrei.elkin@pp.inet.fi |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42004 STOP SLAVE waits for slave_net_timeout when master is unreachable STOP SLAVE could take up to slave_net_timeout to stop the slave IO thread when the master accepts the connection but does not answer, e.g. when its host is frozen or its packets are dropped. The IO thread was blocked in a reconnect try_to_reconnect() -> safe_reconnect() -> connect_to_master() -> mysql_reconnect() -> mysql_real_connect() waiting for the greeting of the master, and nothing could wake it up in the base of this patch, 11.4 at c5272f132d3. The stall is not specific to semi-sync: an asynchronous slave waits the same, and a semi-sync one waits in addition for rpl_semi_sync_slave_kill_conn_timeout at the IO thread exit, which this commit does not change. The wait was interruptible before MDEV-32567 removed the pthread_kill() of the slave thread in terminate_slave_thread() (commit 013fc02a23e) in order to rely on the THD::awake_no_mutex() -> vio_shutdown(active_vio) mechanism. This bug exposes that it does not cover slave reconnect, as the slave's IO thread does not bind its THD::active_vio to a new socket vio at the time of the reconnect attempt. This commit makes that "tentative" vio reachable for awake_no_mutex(). For that mysql_real_connect() now publishes the vio to the slave thread right after it is created. See the new hook slave_io_thread_attach_vio() in the TCP branch (the Unix one is irrelevant for slave) that pairs an existing slave_io_thread_detach_vio(). So now in the patch when STOP SLAVE shuts the vio down, which wakes up the poll(). The detach hook is called when the bound to THD::active_vio is about to be destroyed. connect_to_master() reports the interrupt to the error log when log_warnings > 2 and the slave has been stopped after the failed connect: Slave I/O thread: reconnect to master '[email protected]:19000' was interrupted by the slave stop: Lost connection to server at 'reading initial communication packet', system error: 0 ... Tested with rpl.rpl_stop_slave_unreachable_master. The test simulates an unresponsive master: it is not shut down but frozen with SIGSTOP, so the kernel still completes the TCP handshake of a new connection, while no greeting is sent, and the IO thread gets into its reconnect. The test waits for that to most probably happen and then runs STOP SLAVE, which normally (not on an overloaded server) must not wait for slave_net_timeout. A shut-down master would not serve for that: its port is closed, so the connect fails at once with "Connection refused", and the IO thread, which then sleeps connect_retry in slave_sleep() (it checks io_slave_killed()), stops promptly without this fix. The test is not run on Windows, as it needs SIGSTOP. With --let $debug_test= 1 the test prints how long STOP SLAVE took. Measured this way, it returned after 3.5 s before the fix and 1.0 s after, with slave_net_timeout=3 and rpl_semi_sync_slave_kill_conn_timeout=1; the same test with a plain asynchronous slave took 2.5 s and 0.0 s. For a reliable run on BuildBot the time is not checked. The fix is proven by the error text in the note above. A reconnect that waits for the greeting until the timeout is out fails with "waiting for initial communication packet ... Connection timed out", while the one whose vio is shut down fails with "reading initial communication packet". The test greps the slave's error log for the latter with search_pattern_in_file.inc, which reports FOUND 1 /Slave I/O thread: reconnect to master .* was interrupted by the slave stop: .*reading initial communication packet/ in mysqld.2.err |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42324 CONNECTION of a MULTIPLE table is used as a format string For ODBC tables with MULTIPLE=1 TDBODBC::GetFile() replaced the DBQ value in CONNECTION with "%s" and TDBODBC::SetFile() passed the result to snprintf() as the format string. Store the parts of CONNECTION before and after the file name as two strings and concatenate them with the file name. Assisted-by: Claude:claude-opus-5-5 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Kristian Nielsen
knielsen@knielsen-hq.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Include binlog in backup when --binlog-storage-engine=innodb Collect the set of binlog files to include in the backup at backup commit time, consistent with the backup LSN. Then copy the binlog files during the FINISH stage, where locks are released; this is safe because binlog files are append-only and recovery clears any extra data written after the backup LSN. This can be used to provision a slave from a backup of a master, taking the starting GTID position from the @@gtid_binlog_pos. An mtr test case binlog_in_engine.server_backup_slave_provision demonstrates this use case. Signed-off-by: Kristian Nielsen <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39091 BACKUP SERVER of ENGINE=RocksDB RocksDB is based on log structured merge trees (LSM trees) that are immutable files. A consistent point-in-time snapshot is obtained at BACKUP_LOCK_COMMIT cheaply via rdb_create_checkpoint(), which is also what mariadb-backup is invoking via SET GLOBAL rocksdb_create_checkpoint. Under BACKUP_LOCK_COMMIT, a backup snapshot is created in #rocksdb under rocksdb_datadir (by default, #rocksdb/#rocksdb in the server datadir). This snapshot will be made available in #rocksdb subdirectory of the backup target. The default rocksdb_datadir="./#rocksdb" makes restore transparent: just point datadir to the backup directory or a restored copy of it. The transfer of the snapshot is deferred to BACKUP_FINISHED, after the backup lock has been released, and finally the checkpoint is removed from rocksdb_datadir. Co-developed by: Thirunarayanan Balathandayuthapani |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-25848 bound range optimizer work for many ARRAY index points With one range slot per ARRAY index, the number of points in a SEL_ARG_ARRAY is no longer bounded by MAX_KEY, and a JSON_OVERLAPS or JSON_CONTAINS constant with N elements cost (see the TODOs): * N records_in_range() index dives, one per leaf, even when the estimate is never used; * O(N^2) work in key_or_array(), which copied both point lists into a new SEL_ARG_ARRAY on every OR and deduplicated with a linear scan. Now: * key_or_array() appends the smaller point list to the larger SEL_ARG_ARRAY in place, or to a copy of it if it is shared (use_count > 0), without sorting or deduplicating. * SEL_ARG_ARRAY::normalize() sorts and deduplicates the points once, when they are needed: estimates, quick select, optimizer trace. * SEL_ARG_ARRAY::estimate() computes rows and cost lazily, on first use (key_and() or check_quick_select_array()). With eq_range_index_dive_limit or more points it dives on that many points spread over the sorted list and extrapolates. * JSON_CONTAINS skips duplicate elements of the constant and uses at most eq_range_index_dive_limit distinct ones for the range, since each AND costs a dive and fewer conjuncts give a superset. Points are now read in sorted order, so the order of rows from an ARRAY index range scan may differ. Co-Authored-By: Claude Opus 5.5 <[email protected]> MDEV-25848 deduplicate JSON_OVERLAPS/JSON_CONTAINS elements when parsing Skip duplicate elements of the constant in json_get_range_elements(), using a hash on the element's JSON text, instead of in json_get_func_mm_tree() for JSON_CONTAINS only. Duplicates now never become Items, SEL_ARG_ARRAY leaves or make_key() calls, for JSON_OVERLAPS as well, and json_get_func_mm_tree() only counts the distinct elements against eq_range_index_dive_limit. The text includes the quotes of strings, so "1" and 1 are different. Equal values with different text, such as 1 and 1.0, are still deduplicated later by SEL_ARG_ARRAY::normalize(). Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Aleksey Midenkov
midenok@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41645 Auto-create of SYSTEM_TIME history partitions checks the partition limit without existing partitions and subpartitions partition_info::vers_set_hist_part() counts the history partitions to auto-create for INTERVAL ... AUTO and stopped when the count reached MAX_PARTITIONS - 2. The limit compared only the number of new partitions with MAX_PARTITIONS - 2. It ignored the partitions the table already has and the number of subpartitions, so the real total ((num_parts + create_count) * num_subparts) could be well over MAX_PARTITIONS while this check passed. The early error was then not given, and the limit was left to whatever checks the actual partition creation does. The fix compares the total of the existing and the new partitions, times the number of subpartitions, with MAX_PARTITIONS. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Add my_dir_open() written by Monty | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Georgi (Joro) Kodinov
joro@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42338: Treat openssl,pcre and zlib library headers as "system" Mark the 3d party library headers as "system" headers so that the warnings in these files are suppressed for supporting compilers. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39091 BACKUP SERVER of ENGINE=RocksDB RocksDB is based on log structured merge trees (LSM trees) that are immutable files. A consistent point-in-time snapshot is obtained at BACKUP_LOCK_COMMIT cheaply via rdb_create_checkpoint(), which is also what mariadb-backup is invoking via SET GLOBAL rocksdb_create_checkpoint. Under BACKUP_LOCK_COMMIT, a backup snapshot is created in #rocksdb under rocksdb_datadir (by default, #rocksdb/#rocksdb in the server datadir). This snapshot will be made available in #rocksdb subdirectory of the backup target. The default rocksdb_datadir="./#rocksdb" makes restore transparent: just point datadir to the backup directory or a restored copy of it. The transfer of the snapshot is deferred to BACKUP_FINISHED, after the backup lock has been released, and finally the checkpoint is removed from rocksdb_datadir. Co-developed by: Thirunarayanan Balathandayuthapani Reviewed by: Alessandro Vetere |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
kjarir
kjarir23@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-10526: Add binary string support to bitwise operators and aggregates Previously all bitwise operators and BIT_AND/BIT_OR/BIT_XOR converted their arguments to BIGINT, silently truncating values wider than 64 bits. This made them useless for INET6 addresses, UUIDs and other binary data. When the arguments are binary strings (BINARY, VARBINARY, BLOB, but not hex hybrids like 0xFF), the operators &, |, ^, ~, << and >> and the aggregate functions BIT_AND, BIT_OR and BIT_XOR now work byte by byte and return a binary string of the length of the operand(s). Scalar operators: - Implemented as Handler_str subclasses of the existing Item_handled_func handlers. Func_handler_bit_bin_to_bin and Func_handler_bit_op2_bin_to_bin hold the common code. - Operands of different lengths give ER_INVALID_BITWISE_OPERANDS_SIZE as a warning and the result is NULL. - The result collation is aggregated from the binary string argument(s), as for other string functions. Aggregate functions: - A binary accumulator initialized to the neutral value (0xFF for BIT_AND, 0x00 for BIT_OR and BIT_XOR). - The result length is the length of the aggregated values, which may be shorter than the declared length (e.g. INET6_ATON() of an IPv4 address). All values of a group must have the same length, otherwise ER_INVALID_BITWISE_OPERANDS_SIZE is raised. - An empty set returns the neutral value of the declared length. - The operand is limited to 512 bytes, ER_INVALID_BITWISE_AGGREGATE_OPERANDS_SIZE otherwise. - The result data type is VARBINARY. val_int(), val_real(), val_decimal() and get_date() convert the binary string the same way as for the scalar operators. - Window functions use per-bit counters, as for the 64-bit mode. - reset_field()/update_field() support the binary mode for GROUP BY with a temporary table. Rows rejected by a FILTER clause (MDEV-24943) leave the field unchanged. - direct_add() for a binary partial result, used by Spider. MDEV-10526: Add Spider pushdown for bitwise aggregates Push BIT_AND, BIT_OR and BIT_XOR down to the data nodes and merge the partial results locally with direct_add(), in both integer and binary string mode. The functions are associative and commutative, so the partial results can be combined like for SUM, MIN and MAX. In integer mode the partial result is parsed with strntoull10rnd(): row->val_int() overflows for values above LONGLONG_MAX. A partial result that can not be parsed is an error, as any substitute value would silently corrupt the merged result. Aggregates with a FILTER clause are not pushed down. MDEV-10526: Address review: test coverage, Spider empty data nodes Add tests for the code paths reported as not covered: - scalar operators: NULL in the first operand, empty strings, shift counts inside a byte, across bytes, out of range and negative - aggregates: the binary result used as an integer, a boolean and a date, the integer result as a string, a double, a decimal and a date, data types that are not allowed, values of different lengths in a temporary table group and in a window, sliding frames removing values (also with FILTER), window functions over WITH ROLLUP - Spider: groups whose partial results come from several data nodes (update_field() with direct_add()), empty data nodes, partial results that are not an integer, too long or NULL - out of memory when allocating the result of an operator and the bit counters of a window function (debug builds) Fix the bugs found by the new tests: - A binary aggregate pushed down to Spider failed with ER_INVALID_BITWISE_OPERANDS_SIZE if the first data node had no rows and the values were shorter than the declared length. The neutral value of the declared length returned by such a node was taken as the first value. A neutral partial result is now always skipped, and ordinary values are not checked for it any more, so values of different lengths are always an error. - A partial result longer than the declared length of the argument (the data node has a wider column) was written past the end of the accumulator. It is now rejected. Remove the code that could not be reached: - BIT_AND, BIT_OR and BIT_XOR never return NULL, so Item_sum_bit does not need to handle a NULL partial result or a NULL temporary table field. Spider returns an error for a NULL. - The binary accumulator and the partial result are now kept in buffers inside Item_sum_bit: they are at most 512 bytes long. There is no allocation that can fail any more. - The operators allocate the result in one common method, which also handles empty strings. - Window functions never aggregate values of different lengths, and are never copied by copy_or_same(), this is now asserted. MDEV-10526: Fix Spider bitwise aggregate tests for --ps-protocol The prepared statement protocol changes the value of the Spider_direct_aggregate status variable for the GROUP BY query, and the stored procedure protocol adds a note to SHOW WARNINGS after an error. Run that query without the prepared statement protocol and show only the warnings of the query. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alexey Botchkov
holyfoot@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Excessive json_nice() calls eliminated. Item_json_func descendants now have val_json() doing the job, returning the valid JSON as it is. The val_str() gets the val_json() return and trims it with json_nice(). Functions that surely return the nice json just return the val_json() in the val_str(). |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thirunarayanan Balathandayuthapani
thiru@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39061 mariadb-backup compatible wrapper for BACKUP SERVER This adds a shell script that lets users keep using their existing mariadb-backup commands while the real work is done by the new server-side BACKUP SERVER command. The goal is "drop-in": users should not have to change their backup scripts. mariadb-backup-server.sh : Understands the usual mariadb-backup modes and translates each one. mbstream-server.sh : lets streamed backups be unpacked by pipelines that expect the mbstream CLI mariadb-backup-pipe.sh is the stream sink that BACKUP SERVER itself runs. All three are documented in README.md. --backup ======== Connects with the mariadb client and runs "BACKUP SERVER TO '<dir>'". Connection options (--user, --host, --port, --socket, --defaults-file, ssl, ...) are passed through to the client. --parallel=N becomes "<N+1> CONCURRENT": mariadb-backup runs a dedicated log_copying_thread() besides its N data-copy threads, while CONCURRENT counts every BACKUP SERVER worker. The result is clamped to the 1..256 the parser accepts, and to a minimum of 2. After the backup it writes backup-prepare.cnf into the backup directory, recording what --prepare needs later: - where mariadbd lives - InnoDB parameters (page size, data file path, undo tablespaces, checksum algorithm, log file size) - If the server is encrypted then how to reload the encryption key plugin (the file_key_management variables), so an encrypted backup can be prepared without extra input. - every active provider_* compression plugin, with plugin-dir. The providers matter because --prepare runs "mariadbd --bootstrap --defaults-file=backup-prepare.cnf", which suppresses the /etc/my.cnf.d snippet that normally enables a provider. Recovery has to decompress a PAGE_COMPRESSED page before it can apply redo to it, so without the provider it aborts with "Unable to apply log to corrupted page". mariadb-backup carries the same plugins through backup-my.cnf (encryption_plugin.cc QUERY_PLUGIN collects the ENCRYPTION plugin and the provider_* DAEMON plugins together), so this is parity with it rather than a new capability. --backup --stream ================= Runs "BACKUP SERVER WITH 'pipe'". The server only accepts a bare command name, which it resolves n the PATH of the mariadbd process after prepending mariadb-backup-, so uses the installed mariadb-backup-pipe instead. mariadb-backup-pipe writes the tar into .mariadb-backup-pipe-<stream>.tar, a named pipe the wrapper created in the server @@datadir and is already draining to its own stdout. The backup therefore reaches the consumer as it is produced and never lands on local disk. The wrapper appends backup-prepare.cnf as a final tar afterwards; the server's tar carries no end-of-archive marker, so that trailing archive supplies the only one and the whole stream extracts with a plain "tar -x". The trailing archive is written with "tar -b 1" so that nothing follows its end-of-archive marker. At the default blocking factor tar pads the archive to 10240 bytes; the reader stops at the marker and exits with the padding still unwritten, which on a small pipe is a short write. tar treats a short write as fatal, so a backup that had in fact completed was reported as failed. --parallel is ignored here, with a warning. Four properties follow from how BACKUP SERVER streams, all differing from mariadb-backup: - local: the stream command runs inside the server, so the wrapper must share its filesystem; - fifo lives in @@datadir, because mariadb-backup-pipe resolves it relative to the working directory of mariadbd, so --stream needs write access there; - tar only: any --stream=<format> yields tar; - single-threaded: one worker, so no parallel read either. --target-dir is optional in stream mode; mbstream-server.sh maps the mbstream CLI onto a plain "tar -x"/"tar -c", so existing "mbstream -x"/"-c" pipelines keep working on the wrapper's stream. mbstream-only flags (-p/--parallel, ...) are accepted and ignored; any other unknown option is rejected. Environment overrides: MARIADB (client), MARIADBD (the --prepare bootstrap server) and TAR (the tar implementation, e.g. TAR=bsdtar) can each be overridden. To run the bootstrap under rr, put it in MARIADBD and let rr's own _RR_TRACE_DIR choose the trace location, e.g. _RR_TRACE_DIR=/dev/shm/rr MARIADBD='rr record mariadbd' --prepare ========= Starts "mariadbd --bootstrap" on the backup directory using backup-prepare.cnf as its defaults file, replays the archived redo log between the start and target LSN read from backup.cnf, then builds a fresh ib_logfile0 so a normal server can start on the directory. mariadbd is taken from the path recorded in backup-prepare.cnf if that binary exists, otherwise by searching /libexec, /sbin, /bin and the configured install directories. PATH is not searched; set MARIADBD to point elsewhere. User --defaults-file/-extra-file and encryption options are layered onto the bootstrap. --copy-back / --move-back ========================= Copy or move a prepared backup into the datadir. The datadir is created if missing, a non-empty datadir is refused unless --force-non-empty-directories is given, and a chown reminder is printed. If --aria-log-dir-path is given, the Aria logs (aria_log_control, aria_log.*) are relocated into that directory. Packaging ========= The wrapper is not installed by default and never replaces the real mariadb-backup / mbstream binaries. 1. cmake -DWITH_MARIABACKUP_WRAPPER=ON (default OFF) controls it. 2. When ON, the scripts install as /usr/bin/mariadb-backup-server, /usr/bin/mbstream-server and /usr/bin/mariadb-backup-pipe, tagged COMPONENT Backup so they ship in the mariadb-backup package. mariadb-backup-pipe must end up in the PATH of the mariadbd process, not merely in the PATH of whoever runs the wrapper. mariadb-backup-pipe is generated unconditionally, because the mariabackup test suite needs it on the server's PATH even when the wrapper is not installed; only its install rule is gated. 3. RPM: nothing extra to do. the component handles it. 4. DEB: not wired. debian/rules uses --fail-missing and does not enable the option, so the -server binaries are not listed. To ship via DEB, make a paired change: add -DWITH_MARIABACKUP_WRAPPER=ON in debian/rules and list all three of usr/bin/mariadb-backup-server, usr/bin/mbstream-server and usr/bin/mariadb-backup-pipe in debian/mariadb-backup.install together. 5. The real mariadb-backup/mbstream binaries and the mariabackup symlink are left untouched; opt in via an alias or a symlink early in PATH. Limitations (not supported yet) =============================== 1) Incremental backup & prepare (--incremental-basedir, --incremental-dir, --apply-log-only) 2) --rollback-xa 3) Partial backup (--databases, --tables, --tables-file) 4) Output compression and encryption (--compress, --encrypt) 5) --export is accepted but only warns and runs a plain recovery 6) --extra-lsndir is ignored 7) --parallel is ignored with --stream 8) Windows: POSIX sh only, not installed on Windows Behaviour differences from native mariadb-backup ================================================ - The wrapper needs the mariadb client on PATH for --backup; --prepare needs mariadbd recorded in backup-prepare.cnf, in a standard install directory, or named by MARIADBD - BACKUP SERVER refuses an already-existing target directory - BACKUP SERVER does copy the data file as raw pages without checksum validation, so a corrupted table is not detected at backup time - --prepare only works on a wrapper-made backup: it needs backup-prepare.cnf - --stream is tar, not xbstream, local-only and single-threaded, and needs write access to @@datadir for the fifo Tests ===== include/have_mariabackup_wrapper.inc redirects $XTRABACKUP to mariadb-backup-server.sh and $XBSTREAM to mbstream-server.sh, skipping when a wrapper or the mariadb client is unavailable. include/have_mariabackup_combination.inc runs a test under both the [CLIENT] mariadb-backup binary and the [SERVER] wrapper. suite/mariabackup/suite.pm prepends the directory holding mariadb-backup-pipe to PATH. BACKUP SERVER resolves the command in the PATH of the mariadbd process, which mtr starts from the suite's environment, so it has to be set there rather than with a --let at --exec time. have_mariabackup_wrapper.inc skips the [SERVER] combination when the helper is not on PATH. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Andrei Elkin
andrei.elkin@pp.inet.fi |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42337 no semi-sync kill of a closed connection This is MDEV-42004 fixes follow-up. Repl_semi_sync_slave::kill_connection() reconnected to the master at the IO thread exit even though end_server() had already closed the connection (e.g. STOP SLAVE interrupted a reconnect), so the stop waited rpl_semi_sync_slave_kill_conn_timeout once more. Return early when mysql->net.vio is NULL. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Kristian Nielsen
knielsen@knielsen-hq.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
After-review fixes Signed-off-by: Kristian Nielsen <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Kristian Nielsen
knielsen@knielsen-hq.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Add testcase for streamed backup with binlog-in-engine Signed-off-by: Kristian Nielsen <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-25848 one range slot per ARRAY index Before, every MEMBER OF / JSON_OVERLAPS / JSON_CONTAINS point got its own cloned KEY_PART and key slot, and points were combined only through index_merge (OR) or by picking one slot (AND). Now all points on an ARRAY key part share the original slot: * SEL_ARG_ARRAY holds a list of make_key() points ("the array contains any of them") and the summed records_in_range() estimate and cost, taken once when the point is built. * key_or() of two ARRAY ranges takes the union of the points; key_and() keeps the operand with the lower estimate, the WHERE condition checks the rest. * KEY_PART cloning (get_claimed_key_part()) is now RTREE-only, so ARRAY-only queries size SEL_TREE::keys by param->keys again. * QUICK_RANGE_SELECT_ARRAY gets one QUICK_RANGE per point and passes all points to hlindex_read_first(); json_index reads them in turn and skips trefs it has already returned. * check_quick_select_array() uses the stored estimate, and rejects non-ARRAY SEL_ARGs (e.g. IS NULL) on the ARRAY key part. ORed MEMBER OF and JSON_OVERLAPS now give a single range scan instead of index_merge sort_union, with no MAX_KEY limit on the number of points. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-14992/MDEV-38362/MDEV-39092 BACKUP SERVER The following SQL statements will be introduced: BACKUP SERVER TO '/path/to/directory' [ 1 CONCURRENT ]; BACKUP SERVER WITH [ 1 CONCURRENT ] 'command'; In place of the 1, any positive number of threads may be specified. For the first variant, '/path/to' must exist and '/path/to/directory' must not exist; that is where the backup will be written to. For BACKUP SERVER WITH, the string mariadb-backup- will be prepended to the specified command and quoted. Path separators or quotes are not allowed in command. We expect the mariadb-backup-command to be in the search PATH of the mariadbd process. For now, we distribute no such script or program; the user has to write one, something like this: zstdmt|exec ssh [email protected] "exec cat > $1.tar.zstd" The standard input of that command will be in a format compatible with GNU tar --format=oldgnu (and also BSD tar variants that are also part of Microsoft Windows and Apple macOS). The command is expected to optionally compress and encrypt the stream and redirect it to a file on a local or a remote server. The BACKUP SERVER WITH will append an additional argument, a positive base-ten number in ASCII, starting with 1, to identify the current thread. In this way, there can be multiple concurrent streams. The InnoDB write-ahead log will be streamed near the end, because the stream format requires the file sizes to be declared upfront. In BACKUP SERVER TO ... 2 CONCURRENT or more, one thread will be responsible for copying the InnoDB write-ahead log from the beginning of the backup, running InnoDB_backup::log_track(). Other files will be copied by other threads. Note: In single-threaded BACKUP SERVER TO, the parameter innodb_log_recovery_start that is written out to backup.cnf is STRICTLY NECESSARY TO AVOID CORRUPTION during recovery! By default, InnoDB crash recovery starts from the latest available log checkpoint. However, for restoring a backup, recovery must start from the checkpoint that was the latest when the backup was started. Starting recovery from a possible later checkpoint will result in a corrupted database! The backup or the first stream will contain a file backup.cnf, which includes parameters needed for restoring the backup. Currently, these are innodb_log_recovery_start and innodb_log_recovery_target. If innodb_log_recovery_target>0, InnoDB will be in read-only mode, not allowing any writes to persistent files other than via the log application. To restore a streaming backup made with BACKUP SERVER WITH, an empty directory needs to be created and all streams be extracted there using the standard tar utility of the operating system, optionally after undoing any encryption or compression that had been added by the backup command. Then, the backup is prepared or MariaDB server started up on the extracted directory, similar to as if the BACKUP SERVER TO statement had been used. The following will be implemented separately: MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER MDEV-40163 Partial backup and restore MDEV-39091 Back up ENGINE=RocksDB MDEV-40333 Concurrent DDL for Aria tables in BACKUP SERVER MDEV-39987 Improve BACKUP SERVER performance for ENGINE=Aria The implementation introduces a basic multi-threaded driver Sql_cmd_backup, storage engine interfaces, and basic copying of the storage engines InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV. backup_target: A target directory and a set of operations, such as backup_target::create_file(). On Microsoft Windows, we must use directory paths because there is no variant of CopyFileEx() that would work on file handles. backup_sink: Per-thread storage engine context and output stream, and a set of operations such as backup_sink::create_file() that are valid if an output stream exists. The member "id" identifies the thread, between 0 and CONCURRENT-1. backup_context: Process-wide BACKUP SERVER context. Handles the directory traversal and copying of files for built-in storage engines that do not implement this backup interface. backup_target_stage: Wrap backup_context. backup_target_stage::step(), backup_context::step(): Process a file from a directory scan, or by invoking handlerton::backup_step(). handlerton::backup_file(): Check if a file should be included in the backup. Implemented for ENGINE=Aria in maria_backup_file(). handlerton::backup_start(), handlerton::backup_end(): Invoked at the start or end of a backup phase, in the thread that executes a BACKUP SERVER statement. handlerton::backup_step(): A backup step that can be invoked from multiple threads concurrently, between the execution of the corresponding handlerton::backup_start() and handlerton::backup_end() of the same phase. my_clone_file_range(): A wrapper for Linux block-cloning ioctl(2) FICLONERANGE, which may fail depending on the parameters and thus require a fallback to my_copy_file_range(). my_copy_file_range(): A partial or sparse file-copying service. On FreeBSD, this simply wraps copy_file_range(2). On Linux, for small files we attempt copy_file_range(2) before falling back to specialised a pread/pwrite loop, which turned out to be the fastest way to copy large files. Even though sendfile(2) would work for copying between files, it as well as copy_file_range(2) can be much slower than the alternatives. For large files on Linux, we always invoke a pread/pwrite loop with special cache control. It is also needed as a fallback, because the Linux copy_file_range(2) can fail depending on the file system. On Windows, my_copy_file_range() is not usable on files that have been opened for asynchronous I/O, such as InnoDB data files. Such files have to be copied via backup::copy(), which uses the native_file_handle wrapper of HANDLE. my_copy_file(): Copy files between two POSIX file descriptors. On Microsoft Windows, my_copy() or CopyFileEx() should be used instead. On Linux, we attempt the ioctl(2) FICLONE before falling back to my_copy_file_range(). my_copy(): Make use of CopyFileEx() on Microsoft Windows. backup::copy(): Wraps my_copy_file_range() or the Windows native_file_handle based copied outside mysys. backup::copy_or_stream(): Copy or stream a file. backup_sink::append(): The streaming streaming equivalent of backup::copy(). backup_sink::append_async(): A variant of backup_sink::append() where the source file region is guaranteed to be immutable after the call returns. Zero-copy Linux sendfile(2) is inherently risky for copying data files that may be modified in place, because it could introduce a race condition between a page write that runs concurrently with a child process that is reading the data from the pipe. This could be improved with io_uring later. backup_sink::zeropad(): Zero-pad the last tar block if needed. backup_sink::write(): Write to a stream or a file, with a retry mechanism in case a partial write occurred. pread_write(): Allocate a buffer of up to 4 MiB. This is the "slow path" of copying or streaming files. For InnoDB data files, BACKUP SERVER TO invokes this on output that has been opened in unbuffered I/O (O_DIRECT) mode, to avoid polluting the Linux file system cache (block cache). For the InnoDB log, regular buffered I/O performed best in our tests. maria_backup_start(): The entry point for ENGINE=Aria. Currently, the directory traversal is single-threaded and a memory hog. This is to be improved in MDEV-39987. The Aria log and data file copying were implemented by Michael 'Monty' Widenius and Alessandro Vetere. maria_backup_log_tail(): Copy the necessary part of the last ENGINE=Aria log, by my_clone_file_range() or my_copy_file_range(). The copy will be trimmed to the logical end of the backup. The logic was developed by Alessandro Vetere. InnoDB_backup::init(): Wait for a possible previous BACKUP SERVER operation to reach the very end of InnoDB_backup::context::cleanup() so that the context can be safely reused. InnoDB_backup::queue: Collection of tablespace IDs and payload sizes at the start of the backup, and the log_sys.first_lsn of log files that have to be included in the backup. If any data file is created or extended while the backup is executing, we must have the corresponding write-ahead-log entries that we are copying since the latest checkpoint that was completed when the backup started. If any tablespaces are deleted during the backup, we may or may not copy them, and the application of a FILE_DELETE record will remove them. Similarly, applying FILE_RENAME or FILE_CREATE records will rename or create files during recovery as needed. InnoDB_backup::context::tracked: Log file queue for the dedicated log-copying task InnoDB_backup::log_track(). innodb_backup_log_tracking(): Determine if log-tracking backup is active, which blocks concurrent changes of innodb_log_file_buffering and innodb_log_file_write_through. log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this is to make BACKUP SERVER prevent the concurrent execution of SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size when innodb_log_archive=OFF. log_sys.archived_checkpoint: Keep track of the earliest available checkpoint, corresponding to log_sys.archived_lsn. This reflects SET GLOBAL innodb_log_recovery_start (which is settable now), for incremental backup. fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed to be known. To speed up startup, InnoDB does not normally open all tablespace files. fil_space_t::create_lsn: Change to Atomic_relaxed and use this to indicate tablespace creation LSN, in addition to indicate undo tablespace rebuild LSN. fil_space_t::backup_end: The first page number that is not being backed up (by default 0, to indicate that no backup is in progress). fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be covered by fil_space_t::backup_end. This is the unit of "page range locking" during InnoDB backup. fil_node_t::backup_name: A snapshot of the file name, sampled in InnoDB_backup::init(). For backing up any files that were created before the backup start checkpoint LSN, these names will be written to the backup. On restore, the files will be renamed or deleted as needed, by applying FILE_RENAME or FILE_DELETE records. fil_node_t::rename(), fil_node_t::set_backup_name(), fil_node_t::get_backup_name(), fil_node_t::~fil_node_t(), fil_node_t::clear_backup_name(): Deal with fil_node_t::backup_name. buf_page_t::write_fix_try(), buf_page_t::write_unfix_try(): Try to set or unset a fake "write fix" on a page, to prevent concurrent flush() during a backup batch. The atomic operations may run concurrently with set_reinit() and set_freed(). The fake "write fix" does not prevent any concurrent read or write of the page data in the buffer pool; it only blocks writes to the underlying data file. buf_page_t::flush(): Atomically test and set write fix, and skip the operation if the fake "write fix" was set. buf_page_t::set_freed(), buf_page_t::set_reinit(): Employ a compare-and-exchange loop to accommodate for the "write fix". innodb_backup_batch_wait(): Look up any pages that we are about to back up. For any dirty pages, invoke buf_page_t::write_fix_try() to try to set a fake "write fix" lock-free. If the page is currently write-fixed between buf_page_t::flush() and buf_page_t::write_complete(), acquire and release a page U-latch to wait for the conflicting write to complete. InnoDB_backup::backup_batch_start(), InnoDB_backup::backup_batch_stop(): Adjust fil_space_t::backup_end and fake "write fix" of dirty pages to protect the copying of a range of pages from the underlying file. InnoDB_backup::commit(): Enqueue the remaining log to be copied. InnoDB_backup::checkpoint_complete(): If backup is running and commit() has not been called, add each completed innodb_archive_log=ON file to the queue. Else, skip or delete, as appropriate. InnoDB_backup::archive_create(), innodb_backup_archive_create(): Hook for log_t::archive_create(). log_t::backup_start(): If we were running with innodb_log_archive=ON, ensure that the latest file is a valid recovery starting point. That is, wait for the latest log checkpoint to be within the file. buf_flush_list_space(): Check for concurrent backup before writing each page. This is inefficient, but this function may be invoked from multiple threads concurrently, and it cannot be changed easily, especially for fil_crypt_thread(). fil_ibd_create(): Set fil_space_t::create_lsn after the file has been created. dict_load_tablespaces(): Determine the size of each file if upgrade==true. Backup depends on that. buf_dblwr_t::begin(), buf_dblwr_t::end(), buf_dblwr_t::size(): Accessors to allow BACKUP SERVER to skip the contents of the doublewrite buffer in the system tablespace. It is only useful for crash recovery in case a data page had been incompletely written by the time the server was killed. If the server is killed during a backup, the backup will be incomplete and unusable anyway. Furthermore, the page range locking makes page writes and backup mutually exclusive. Tested by: Matthias Leich Tested by: Saahil Alam Tested by: Rahul Raj Co-authored by: Michael 'Monty' Widenius (mysys and ENGINE=Aria changes) Co-authored by: Alessandro Vetere (some ENGINE=Aria changes) Reviewed by: Thirunarayanan Balathandayuthapani (the ENGINE=InnoDB part) |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MDEV-25848 Revert SEL_TREE::keys allocation size for B-tree indexes | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Kristian Nielsen
knielsen@knielsen-hq.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Include binlog in backup when --binlog-storage-engine=innodb Collect the set of binlog files to include in the backup at backup commit time, consistent with the backup LSN. Then copy the binlog files during the FINISH stage, where locks are released; this is safe because binlog files are append-only and recovery clears any extra data written after the backup LSN. This can be used to provision a slave from a backup of a master, taking the starting GTID position from the @@gtid_binlog_pos. An mtr test case binlog_in_engine.server_backup_slave_provision demonstrates this use case. Signed-off-by: Kristian Nielsen <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| fixup! 77311fbaf3e42d429c6c2f7ac3de1d9842b71d1b | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Aleksey Midenkov
midenok@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41644 date_add_interval(): signed overflow on huge YEAR/MONTH interval values INTERVAL::year and INTERVAL::month are unsigned, but date_add_interval() cast them to long and did signed arithmetic before the range check. A value above LONG_MAX became negative, and the multiplication or addition could overflow, which is undefined behaviour. The result could also wrap into the valid range: the year sum is stored in the 32-bit MYSQL_TIME::year, so INTERVAL 4294967301 YEAR added 5 years instead of failing. The fix checks the year and month magnitude before the arithmetic, as is already done for days with MAX_DAY_NUMBER: no interval of 10000 years or more, or of 120000 months or more, gives a valid date. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Kristian Nielsen
knielsen@knielsen-hq.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Include binlog in backup when --binlog-storage-engine=innodb Collect the set of binlog files to include in the backup at backup commit time, consistent with the backup LSN. Then copy the binlog files during the FINISH stage, where locks are released; this is safe because binlog files are append-only and recovery clears any extra data written after the backup LSN. This can be used to provision a slave from a backup of a master, taking the starting GTID position from the @@gtid_binlog_pos. An mtr test case binlog_in_engine.server_backup_slave_provision demonstrates this use case. Signed-off-by: Kristian Nielsen <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Rex Johnston
rex.johnston@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42318 InnoDB range estimate doubled for ranges wider than 9 leaf pages Prior to the MDEV-21136, the old records_in_range estimator undercounted tall trees because it sampled only around the borders. A compensation doubled the estimate. if (root_height > divergence_height && !is_n_rows_exact) /* In trees whose height is > 1 our algorithm tends to underestimate: multiply the estimate by 2: */ n_rows*= 2; After MDEV-21136, this compensation is no longer needed. This commit removes the above lines. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Jan Lindström
jan.lindstrom@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42284 fix galera_sr.mysql-wsrep#215 ER_LOCK_WAIT_TIMEOUT Node_1's own causality wait (wsrep_sync_wait, on by default) blocked its plain SELECT on t1 while the test held node_1's applier paused at the sync.wsrep_apply_cb debug sync point, so the SELECT could not reach the GTID the paused applier had not yet reached and timed out with ER_LOCK_WAIT_TIMEOUT instead of returning. Set wsrep_sync_wait = 0 on node_1 for that SELECT and the INSERT that follows it, saved and restored around each of the test's three blocks. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Kristian Nielsen
knielsen@knielsen-hq.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Add testcase to provision a new slave from a backup of an old slave Signed-off-by: Kristian Nielsen <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Aleksey Midenkov
midenok@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41643 Temporal to_string() returns a String with unset length when allocation fails Temporal_hybrid::to_string() returned the destination String with a stale length when String::alloc() failed, instead of signalling the failure. The sibling to_string() methods of Interval_DDhhmmssff, Time, Datetime and Date shared the identical flaw. The fix returns NULL on allocation failure in all five, so they report OOM consistently. The val_str() implementations that return the to_string() result directly did not set null_value, so a NULL result would be taken for a value. They now go through the new Item::check_str(), which sets null_value from the result, so a NULL does not stick to the item for the next execution. The time_to_string_oom debug keyword makes Time::to_string() fail as on out of memory. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Georgi (Joro) Kodinov
joro@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42349: my_json_writer-t.cc fails to compile on MacOSX/clang under maintainer_mode Converted all the stack JsonObject instances into pointers and dynamically allocated/deallocated these. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Fix crash caused by incorrectly assigning MULTIPLE to an ARRAY key when adding an index to a table with an existing ARRAY INDEX Missing ARRAY key type causes hliton to not be assigned in mysql_prepare_create_table_finalize. This further causes the ARRAY key not to be placed at the end of the key list, which it should be. Later in TABLE_SHARE::init_from_binary_frm_image it still assumes the last key is an hlindex key. When it is not, this non-hlindex last key has NULL hliton, causing options() to access field of the NULL t argument passed to it by the caller thus crash. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| my_dir_open() wrapper for POSIX and Windows, added by Monty | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-25848 do not let get_mm_parts() match cloned key parts get_claimed_key_part() appended clones at param->key_parts_end, so every later get_mm_parts() call iterated over them too and cloned each clone again. N independent predicates on the same RTREE/ARRAY field thus produced 2^(N-1) key parts, exhausting MAX_KEY at the 8th predicate (e.g. 8 ORed "x MEMBER OF (js)" lost range access). Keep clones in a separate region after key_parts_end, tracked by RANGE_OPT_PARAM::key_part_clones_end. get_mm_parts() only walks key_parts..key_parts_end, so each predicate creates at most one clone and N predicates use N key parts. Assisted-By: Claude Opus 5.5 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| fixup! 7cfefd08e195bda8e03b0687632953656128ce7d | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
squash! 7cfefd08e195bda8e03b0687632953656128ce7d Move most file-copying functions to mysys and some file APIs to member functions of backup_target and backup_sink. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-25848 Range optimizer support for JSON_OVERLAPS and JSON_CONTAINS Implemented get_mm_tree for JSON_OVERLAPS and JSON_CONTAINS. For each JSON_OVERLAPS/JSON_CONTAINS(field, value), parse the array or scalar element into a list. Map with get_mm_parts over the list and tree_or/tree_and the results. At leaf level uses a helper extracted from MEMBER OF. Also moved JSON_OVERLAPS to inherit from Item_bool_func2_with_rev due to its symmetry. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42070 handlersocket crashes after open_index with huge number of return fields alloca() sizes in do_exec_on_index() and cmd_find_internal() depend on client-controlled counts (return fields, filters) bounded only by the request line length. * make DENA_ALLOCA_ALLOCATE always use malloc * free key_buf/filter_buf on the early "filterblob" return Also, a huge statement id in open_index or IN list length in exec resized a vector to 4G elements, aborting on std::bad_alloc. Limit both to 1024. Assisted-by: Claude:claude-opus-5-5 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-42350 innodb_log_archive=ON recovery fails at the end of the log file recv_sys_t::find_checkpoint(): Relax an assertion. recv_sys_t::find_checkpoint_archived(): If the log ends at a file boundary, set n_checkpoint=0 so that the next checkpoint will correctly written at the start of the next log file. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Fix crash caused by incorrectly assigning MULTIPLE to an ARRAY key when adding an index to a table with an existing ARRAY INDEX Missing ARRAY key type causes hliton to not be assigned in mysql_prepare_create_table_finalize. This further causes the ARRAY key not to be placed at the end of the key list, which it should be. Later in TABLE_SHARE::init_from_binary_frm_image it still assumes the last key is an hlindex key. When it is not, this non-hlindex last key has NULL hliton, causing options() to access field of the NULL t argument passed to it by the caller thus crash. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||