Console View
|
Categories: connectors experimental galera main |
|
| connectors | experimental | galera | main | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
CONCPP-164 Savepoint names in setSavepoint were not quoted and escaped They are not quoted and escaped also in release and rollabck methods. It could cause incorrect work and created theoretical opportunity for SQL injection. The testcase has been added and savepoint suite added to ctest tests. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Factor out common code into get_mvi_index() | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
CONCPP-159 Adding infer to the Github Actions workflow Also, some minimal formatting/whitespaces changes made along the way |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
CONCCP-165 Making PS more resilient to forged metadata PS here means server side prepared statements. With forged metadata driver could read past the allocated buffer. The fix make the read area is not longer than buffer_length. Atm I can't see any possibility for a testcase. We need to be able to run "evil server" for that. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Monty
monty@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-25292 Atomic CREATE OR REPLACE TABLE Atomic CREATE OR REPLACE allows to keep an old table intact if the command fails or during the crash. That is done by renaming the original table to temporary name, as a backup and restoring it if the CREATE fails. When the command is complete and logged the backup table is deleted. Atomic replace algorithm Two DDL chains are used for CREATE OR REPLACE: ddl_log_state_create (C) and ddl_log_state_rm (D). 1. (C) Log rename of ORIG to TMP table (Rename TMP to original). 2. Rename orignal to TMP. 3. (C) Log CREATE_TABLE_ACTION of ORIG (drops ORIG); 4. Do everything with ORIG (like insert data) 5. (D) Log drop of TMP 6. Write query to binlog (this marks (C) to be closed in case of failure) 7. Execute drop of TMP through (D) 8. Close (C) and (D) If there is a failure before 6) we revert the changes in (C) Chain (D) is only executed if 6) succeded (C is closed on crash recovery). Foreign key errors will be found at the 1) stage. Additional notes - CREATE TABLE without REPLACE and temporary tables is not affected by this commit. set @@drop_before_create_or_replace=1 can be used to get old behaviour where existing tables are dropped in CREATE OR REPLACE. - CREATE TABLE is reverted if binlogging the query fails. - Engines having HTON_EXPENSIVE_RENAME flag set are not affected by this commit. Conflicting tables marked with this flag will be deleted with CREATE OR REPLACE. - Replication execution is not affected by this commit. - Replication will first drop the conflicting table and then creating the new one. - CREATE TABLE .. SELECT XID usage is fixed and now there is no need to log DROP TABLE via DDL_CREATE_TABLE_PHASE_LOG (see comments in do_postlock()). XID is now correctly updated so it disables DDL_LOG_DROP_TABLE_ACTION. Note that binary log is flushed at the final stage when the table is ready. So if we have XID in the binary log we don't need to drop the table. - Three variations of CREATE OR REPLACE handled: 1. CREATE OR REPLACE TABLE t1 (..); 2. CREATE OR REPLACE TABLE t1 LIKE t2; 3. CREATE OR REPLACE TABLE t1 SELECT ..; - Test case uses 6 combinations for engines (aria, aria_notrans, myisam, ib, lock_tables, expensive_rename) and 2 combinations for binlog types (row, stmt). Combinations help to check differences between the results. Error failures are tested for the above three variations. - expensive_rename tests CREATE OR REPLACE without atomic replace. The effect should be the same as with the old behaviour before this commit. - Triggers mechanism is unaffected by this change. This is tested in create_replace.test. - LOCK TABLES is affected. Lock restoration must be done after new table is created or TMP is renamed back to ORIG - Moved ddl_log_complete() from send_eof() to finalize_ddl(). This checkpoint was not executed before for normal CREATE TABLE but is executed now. - CREATE TABLE will now rollback also if writing to the binary logging failed. See rpl_gtid_strict.test backup ddl log changes - In case of a successfull CREATE OR REPLACE we only log the CREATE event, not the DROP TABLE event of the old table. ddl_log.cc changes ddl_log_execute_action() now properly return error conditions. ddl_log_disable_entry() added to allow one to disable one entry. The entry on disk is still reserved until ddl_log_complete() is executed. On XID usage Like with all other atomic DDL operations XID is used to avoid inconsistency between master and slave in the case of a crash after binary log is written and before ddl_log_state_create is closed. On recovery XIDs are taken from binary log and corresponding DDL log events get disabled. That is done by ddl_log_close_binlogged_events(). On linking two chains together Chains are executed in the ascending order of entry_pos of execute entries. But entry_pos assignment order is undefined: it may assign bigger number for the first chain and then smaller number for the second chain. So the execution order in that case will be reverse: second chain will be executed first. To avoid that we link one chain to another. While the base chain (ddl_log_state_create) is active the secondary chain (ddl_log_state_rm) is not executed. That is: only one chain can be executed in two linked chains. The interface ddl_log_link_chains() was defined in "MDEV-22166 ddl_log_write_execute_entry() extension". Atomic info parameters in HA_CREATE_INFO Many functions in CREATE TABLE pass the same parameters. These parameters are part of table creation info and should be in HA_CREATE_INFO (or whatever). Passing parameters via single structure is much easier for adding new data and refactoring. Aria changes: - Fixed issue in Aria engine with CREATE + locked tables that data was not properly commited in some cases in case of crashes. InnoDB related changes (by Marko Mäkelä): - table_name_t::is_create_or_replace(): A new predicate to check for CREATE OR REPLACE TABLE will rename an old table to and eventually drop after creating the replacement. - dict_table_t::parse_name(): Do acquire MDL on #sql-create- names for partitioned tables. - dict_table_rename_in_cache(): On CREATE OR REPLACE TABLE ... SELECT, forget the original dict_table_t::mdl_name so that purge will acquire MDL on the #sql-create- name instead. In this way, the MDL_EXCLUSIVE that the CREATE OR REPLACE TABLE holds on the user-visible name will not unnecessarily block any purge of old history until the very end when the #sql-create- table will be dropped. - ha_innobase::delete_table(): Do not check FOREIGN KEY consistency when dropping an #sql-create- table. - row_rename_table_for_mysql(): Update SYS_FOREIGN.ID also when renaming to #sql-create- in order to avoid any duplicate key error when CREATE OR REPLACE TABLE is creating some FOREIGN KEY constraints by names that existed in the old table. Other changes: - Removed some auto variables in log.cc for better code readability. - Fixed old bug that CREATE ... SELECT would not be able to auto repair a table that is part of the SELECT. - Marked MyISAM that it does not support ROLLBACK (not required but done for better consistency with other engines). - maria_create_trn_for_mysql() does not register a new transaction handler for commits. This was needed to ensure create or replace will not end with an active transaction. - We do not get anymore warnings about "Engine not supporting atomic create" when doing a legal CREATE OR REPLACE on a table with foreign key constraints. - Updated VIDEX engine flags to disable CREATE SEQUENCE. - Removed mysql_mutex_unlock(&LOCK_gdl) / mysql_mutex_lock(&LOCK_gdl) around calls to binlog as these are unsafe. The binlog code uses global variables that needs protection from other caller. - EITS data is preserved if create or replace fails if drop_before_create_or_replace=OFF. If ON, then create or replace will drop EITS before the drop of the original table (as before). - Using CREATE OR REPLACE on a encrypted table that the user cannot decrypt will fail instead of replacing the encrypted table. The encrypted table will unchanged. Known issues: - One cannot use create or replace on an InnoDB tables that has foreign key references point to it - CREATE OR REPLACE TEMPORARY table is not full atomic. Any conflicting table will always be dropped before creating a new one. (Old behaviour). Bug fixes related to this MDEV: MDEV-36435 Assertion failure in finalize_locked_tables() MDEV-36439 Assertion `thd_arg->lex->sql_command != SQLCOM_CREATE_SEQUENCE... MDEV-36498 Failed CoR in non-atomic mode no longer generates DROP in RBR... MDEV-36508 Temporary files #sql-create-....frm occasionally stay after crash recovery MDEV-38479 Crash in CREATE OR REPLACE SEQUENCE when new sequence cannot be created MDEV-36497 Assertion failure after atomic CoR with Aria under lock in transactional context MDEV-36501 EITS data is lost after failed attempt to CREATE OR REPLACE table MDEV-36493 Atomic CREATE OR REPLACE ... SELECT blocks InnoDB purge MDEV-39367 MSAN/valgrind errors in temp_file_size_cb_func, main.tmp_space_usage fails MDEV-39446 Atomic CREATE OR REPLACE fails if a table cannot be decrypted MDEV-40776 Atomic CREATE OR REPLACE silently breaks the foreign key MDEV-40765 Assertion `"unexpected references" == 0' failed upon failing CREATE OR REPLACE Reverted commits: - MDEV-36685 "CREATE-SELECT may lose in binlog side-effects of stored-routine" as it did not take into account that it safe to clear binlogs if the created table is non transactional and there are no other non transactional tables used. This was done because it caused extra logging when it is not needed (not using any non transactional tables) and it also did not solve side effects when using statement based loggging. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Increased min required smake version to 3.10 To eliminate the cmake warning and to trigger the build |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
ODBC-506 Paramarray operation error can cause buffer overrun in the next such operation. In 3.1 the error has to happen on the last paramset, 3.2 does not have this limitation. That happened because the variable used for array offset was not reset in such case. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Merge branch 'cpp-1.0' into cpp-1.1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
CONCPP-167 Changing of auto_increment_increment with crashed the driver This could happen only if rewriteBatchedStatements on - only in this case auto_increment_increment was tracked. If that actually happened, the bug in the session tracking processing caused unhandled exception that aborted the program. The test has been added. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
ODBC-507 Param arrays with DAE parameters did not work We did not have a test for this case, so not clear if it worked and was broken at some point or did not work at all. The test is being added along with the fix. Also the fix almost completely re-writes ODBC-506 fix as the code causing it(and the fix) is completely removed and/or rewritten. In particular, ArrayOffset has been removed from the Stmt handle as it's meaning/use was almost duplicating DaeRowNumber. Buffers for positioned update(WHERE CURRENT OF) parameter values now stored in the handle as their lifetime in case of DAE should be longer than statement execution function. We had the testcase for it, though. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| C/C has been moved to the commit that is supposed to be 3.3.21 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Fix of the last merge - the test in Actions has to be with latest C/C 3.4 and not 3.3. Also, fixed one test in preparedstatement for the MySQL |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Daniel Bartholomew
db@dbart.us |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| bump the VERSION | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
InnoDB review changes table_name_t::is_create_or_replace(): A new predicate to check for CREATE OR REPLACE TABLE will rename an old table to and eventually drop after creating the replacement. dict_table_t::parse_name(): Do acquire MDL on #sql-create- names for partitioned tables. dict_table_rename_in_cache(): On CREATE OR REPLACE TABLE ... SELECT, forget the original dict_table_t::mdl_name so that purge will acquire MDL on the #sql-create- name instead. In this way, the MDL_EXCLUSIVE that the CREATE OR REPLACE TABLE holds on the user-visible name will not unnecessarily block any purge of old history until the very end when the #sql-create- table will be dropped. ha_innobase::delete_table(): Do not check FOREIGN KEY consistency when dropping an #sql-create- table. row_rename_table_for_mysql(): Update SYS_FOREIGN.ID also when renaming to #sql-create- in order to avoid any duplicate key error when CREATE OR REPLACE TABLE is creating some FOREIGN KEY constraints by names that existed in the old table. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40608 MariaDB-devel is incomplete for plugins This works on Linux and on Windows, with rpm/deb/tar.gz/zip installations. For rpm/deb it just works, for tar.gz/zip there is no standard location, so one needs to configure plugin with -DCMAKE_PREFIX_PATH=/pah/to/mariadb/basedir after that, `cmake --install .` works too, installing in the same basedir. `cmake --build . --target package` works, creating rpm/deb/targz/zip depending on whether it's Linux or Windows and whether -DRPM or -DDEB was specified. * create and install mariadb-plugin-config.cmake * deb: move all headers that plugins need to libmariadb-dev, together with libmysqlservices.a. At least until we'll create mariadb-plugin-dev. Nobody should need huge libmariadbd-dev to develop a plugin * rpm: all in MariaDB-devel already, no changes here * adjust plugin.cmake to work for external plugins * move server-internal part to top-level CMakeLists.txt * remove WITH_WSREP from my_config.h (it upsets external plugins) * disable DBUG in plugins, can be enabled with -UDBUG_OFF, if needed * remove double-defined macros from unireg.h (the guard doesn't help if unireg.h is included first) ColumnStore, until fixed, needs a backward-compatibility workaround |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-14992/MDEV-38362/MDEV-39092 BACKUP SERVER The following SQL statements will be introduced: BACKUP SERVER TO '/path/to/directory' [ 1 CONCURRENT ]; BACKUP SERVER WITH [ 1 CONCURRENT ] 'command'; In place of the 1, any positive number of threads may be specified. For the first variant, '/path/to' must exist and '/path/to/directory' must be compatible with secure_file_priv and not exist; that is where the backup will be written to. For the second variant, 'command' must be the name of a script or command that will be executed in a child process. The standard input of that command will be in a format that is compatible with GNU tar --format=oldgnu (and also BSD tar variants that are also part of Microsoft Windows and Apple macOS). The command is expected to optionally compress and encrypt the stream and redirect it to a file on a local or a remote server. The BACKUP SERVER WITH will append an additional argument, a positive base-ten number in ASCII, starting with 1, to identify the current thread. In this way, each concurrent stream can write a separate file. The InnoDB write-ahead log will be streamed near the end, because the stream format requires the file sizes to be declared upfront. TODO: implement the following: In BACKUP SERVER TO ... 2 CONCURRENT or more, one thread will be responsible for copying the InnoDB write-ahead log from the beginning of the backup. Other files will be copied by other threads. Note: In single-threaded BACKUP SERVER TO, the parameter innodb_log_recovery_start that is written out to backup.cnf is STRICTLY NECESSARY TO AVOID CORRUPTION during recovery! By default, InnoDB crash recovery starts from the latest available log checkpoint. However, for restoring a backup, recovery must start from the checkpoint that was the latest when the backup was started. Starting recovery from a possible later checkpoint will result in a corrupted database! The backup or the first stream will contain a file backup.cnf, which includes parameters needed for restoring the backup. Currently, these are innodb_log_recovery_start and innodb_log_recovery_target. If innodb_log_recovery_target>0, InnoDB will be in read-only mode, not allowing any writes to persistent files other than via the log application. To restore a streaming backup made with BACKUP SERVER WITH, an empty directory needs to be created and all streams be extracted there using the standard tar utility of the operating system, optionally after undoing any encryption or compression that had been added by the backup command. Then, the backup is prepared or MariaDB server started up on the extracted directory, similar to as if the BACKUP SERVER TO statement had been used. The following will be implemented separately: MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER MDEV-40163 Partial backup and restore MDEV-39091 Back up ENGINE=RocksDB MDEV-40333 Concurrent DDL for Aria tables in BACKUP SERVER The implementation introduces a basic multi-threaded driver Sql_cmd_backup, storage engine interfaces, and basic copying of the storage engines InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV. backup_target: A structured data type to represent a target directory. On Microsoft Windows, we must use directory paths because there is no variant of CopyFileEx() that would work on file handles. backup_sink: Wraps a per-thread output stream as well as storage engine specific context. handlerton::backup_start(), handlerton::backup_end(): Invoked at the start or end of a backup phase, in the thread that executes a BACKUP SERVER statement. handlerton::backup_step(): A backup step that can be invoked from multiple threads concurrently, between the execution of the corresponding handlerton::backup_start() and handlerton::backup_end() of the same phase. copy_entire_file(): A file copying service for POSIX systems. copy_mmap(): A zero-copy alternative to backup::copy(), to copy from a memory-mapped buffer. copy_file_range_try(): A wrapper for Linux copy_file_range(2), which may fail with EOPNOTSUPP or EXDEV and thus require a fallback to copy_mmap() or backup::copy(). backup::copy(): A partial or sparse file-copying service. On other platforms than FreeBSD or Microsoft Windows, there are shortcut alternatives to this. Note: On Linux we never invoke sendfile(2) for copying between files, because can be much slower than the alternatives. backup_stream_append_plain(): A wrapper of backup::append(), which is the streaming equivalent of backup::copy(). backup_stream_zeropad(): Zero-pad the last tar block if needed. backup_stream_append_async(): A variant of backup_stream_append_plain() where the source file region is guaranteed to be immutable after the call returns. Zero-copy mmap(2) or Linux sendfile(2) are inherently risky for copying data files that may be modified in place, because it could introduce a race condition between a page write that runs concurrently with a child process that is reading the data from the pipe. backup::append(): On systems where we can determine the size of the pipe buffer, invoke backup_stream_append_async() for the initial write, and pread_write() for the last part, to guarantee that the data written by the zero-copy shortcut will have been consumed before the call returns and the caller is able to resume writes to the source region. pread_write(): On 64-bit systems, allocate a buffer of up to 1 MiB. This is the "slow path" of copying or streaming files. struct Aria_backup: Context for multi-threaded backup, comprising directory handles, a mutex and enum Aria_backup_status. aria_backup_start(): Prepare the context for aria_backup_step(). Most files are copied in BACKUP_PHASE_NO_DDL after flush_tables(thd, FLUSH_NON_TRANS_TABLES) has been invoked. All ENGINE=Aria files (including TRANSACTIONAL=0) are copied in BACKUP_PHASE_NO_COMMIT. Thanks to Andrzej Jarząbek for writing test cases and suggesting this logic. aria_backup_step(): Copy one non-ACID file. Acquires Aria_backup::mutex, traverses directories to construct one file name, releases the mutex, and copies the file if one was found. aria_backup_data(): Copy one data file. On Microsoft Windows, this assumes that the current directory is the datadir. This assumption would not hold in the Embedded Server library, which is not supported on Microsoft Windows. aria_backup_log(): Copy one ENGINE=Aria log file. aria_backup_end(): Finish a copying phase and clean up the context. InnoDB_backup::init(): Wait for a possible previous BACKUP SERVER operation to reach the very end of InnoDB_backup::context::cleanup() so that the context can be safely reused. InnoDB_backup::queue: Collection of tablespace IDs and payload sizes at the start of the backup, and the log_sys.first_lsn of log files that have to be included in the backup. If any data file is created or extended while the backup is executing, we must have the corresponding write-ahead-log entries that we are copying since the latest checkpoint that was completed when the backup started. If any tablespaces are deleted during the backup, we may or may not copy them, and the application of a FILE_DELETE record will remove them. Similarly, applying FILE_RENAME or FILE_CREATE records will rename or create files during recovery as needed. log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this is to make BACKUP SERVER prevent the concurrent execution of SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size when innodb_log_archive=OFF. log_sys.archived_checkpoint: Keep track of the earliest available checkpoint, corresponding to log_sys.archived_lsn. This reflects SET GLOBAL innodb_log_recovery_start (which is settable now), for incremental backup. fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed to be known. To speed up startup, InnoDB does not normally open all tablespace files. fil_space_t::create_lsn: Change to Atomic_relaxed and use this to indicate tablespace creation LSN, in addition to indicate undo tablespace rebuild LSN. fil_space_t::backup_end: The first page number that is not being backed up (by default 0, to indicate that no backup is in progress). fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be covered by fil_space_t::backup_end. This is the unit of "page range locking" during InnoDB backup. buf_page_t::write_fix_try(), buf_page_t::write_unfix_try(): Try to set or unset a fake "write fix" on a page, to prevent concurrent flush() during a backup batch. The atomic operations may run concurrently with set_reinit() and set_freed(). The fake "write fix" does not prevent any concurrent read or write of the page data in the buffer pool; it only blocks writes to the underlying data file. buf_page_t::flush(): Atomically test and set write fix, and skip the operation if the fake "write fix" was set. buf_page_t::set_freed(), buf_page_t::set_reinit(): Employ a compare-and-exchange loop to accommodate for the "write fix". innodb_backup_batch_wait(): Look up any pages that we are about to back up. For any dirty pages, invoke buf_page_t::write_fix_try() to try to set a fake "write fix" lock-free. If the page is currently write-fixed between buf_page_t::flush() and buf_page_t::write_complete(), acquire and release a page U-latch to wait for the conflicting write to complete. InnoDB_backup::backup_batch_start(), InnoDB_backup::backup_batch_stop(): Adjust fil_space_t::backup_end and fake "write fix" of dirty pages to protect the copying of a range of pages from the underlying file. InnoDB_backup::commit(): Enqueue the remaining log to be copied. InnoDB_backup::checkpoint_complete(): If backup is running and commit() has not been called, add each completed innodb_archive_log=ON file to the queue. Else, skip or delete, as appropriate. log_t::backup_start(): If we were running with innodb_log_archive=ON, ensure that the latest file is a valid recovery starting point. That is, wait for the latest log checkpoint to be within the file. buf_flush_list_space(): Check for concurrent backup before writing each page. This is inefficient, but this function may be invoked from multiple threads concurrently, and it cannot be changed easily, especially for fil_crypt_thread(). fil_ibd_create(): Set fil_space_t::create_lsn after the file has been created. dict_load_tablespaces(): Determine the size of each file if upgrade==true. Backup depends on that. buf_dblwr_t::begin(), buf_dblwr_t::end(), buf_dblwr_t::size(): Accessors to allow BACKUP SERVER to skip the contents of the doublewrite buffer in the system tablespace. It is only useful for crash recovery in case a data page had been incompletely written by the time the server was killed. If the server is killed during a backup, the backup will be incomplete and unusable anyway. Furthermore, the page range locking makes page writes and backup mutually exclusive. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
CONCPP-161, CONCPP-162 Fix and test of the wrong DB metadata methods parameters escaping There were 2 separate issues - some hepler functions had hardcoded parameter for "BACKSLASH_NO_ESCAPE", and 2nd - other helper functions that actually escaped quots, due to the bug could process own output and in this way multiply quots/backslashes. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Factor out common code into Item_func_json_contains::get_mvi_access() Item_func_json_contains::mvi_analyze() and ::create_ft_for_mvi() were near-identical: both checked the arguments, looked up the matching MVI, parsed the constant second argument and ran the same scan loop calling encode_mvi_key(). They differed only in what they did with each encoded key. Move all of that into get_mvi_access(), which returns an Mvi_access, and give Mvi_access two methods: - add_key(), to collect one encoded element key, - create_ft_item(), to build the MATCH vcol AGAINST ('+encoded_foo +encoded_bar ...' IN BOOLEAN MODE) item. It honors Mvi_access::conjunctive, so JSON_OVERLAPS will get the OR form for free. mvi_analyze() and create_ft_for_mvi() are now thin wrappers around get_mvi_access(). This also fixes a memory leak: the encoded keys were copied with String::copy(), giving each String in Mvi_access::encoded a heap buffer that is never freed (the Strings live on the MEM_ROOT, so their destructors never run). Copy the keys onto the MEM_ROOT instead. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Test FLUSH TABLES and BACKUP SERVER | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Monty
monty@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-25292 Atomic CREATE OR REPLACE TABLE Atomic CREATE OR REPLACE allows to keep an old table intact if the command fails or during the crash. That is done by renaming the original table to temporary name, as a backup and restoring it if the CREATE fails. When the command is complete and logged the backup table is deleted. Atomic replace algorithm Two DDL chains are used for CREATE OR REPLACE: ddl_log_state_create (C) and ddl_log_state_rm (D). 1. (C) Log rename of ORIG to TMP table (Rename TMP to original). 2. Rename orignal to TMP. 3. (C) Log CREATE_TABLE_ACTION of ORIG (drops ORIG); 4. Do everything with ORIG (like insert data) 5. (D) Log drop of TMP 6. Write query to binlog (this marks (C) to be closed in case of failure) 7. Execute drop of TMP through (D) 8. Close (C) and (D) If there is a failure before 6) we revert the changes in (C) Chain (D) is only executed if 6) succeded (C is closed on crash recovery). Foreign key errors will be found at the 1) stage. Additional notes - CREATE TABLE without REPLACE and temporary tables is not affected by this commit. set @@drop_before_create_or_replace=1 can be used to get old behaviour where existing tables are dropped in CREATE OR REPLACE. - CREATE TABLE is reverted if binlogging the query fails. - Engines having HTON_EXPENSIVE_RENAME flag set are not affected by this commit. Conflicting tables marked with this flag will be deleted with CREATE OR REPLACE. - Replication execution is not affected by this commit. - Replication will first drop the conflicting table and then creating the new one. - CREATE TABLE .. SELECT XID usage is fixed and now there is no need to log DROP TABLE via DDL_CREATE_TABLE_PHASE_LOG (see comments in do_postlock()). XID is now correctly updated so it disables DDL_LOG_DROP_TABLE_ACTION. Note that binary log is flushed at the final stage when the table is ready. So if we have XID in the binary log we don't need to drop the table. - Three variations of CREATE OR REPLACE handled: 1. CREATE OR REPLACE TABLE t1 (..); 2. CREATE OR REPLACE TABLE t1 LIKE t2; 3. CREATE OR REPLACE TABLE t1 SELECT ..; - Test case uses 6 combinations for engines (aria, aria_notrans, myisam, ib, lock_tables, expensive_rename) and 2 combinations for binlog types (row, stmt). Combinations help to check differences between the results. Error failures are tested for the above three variations. - expensive_rename tests CREATE OR REPLACE without atomic replace. The effect should be the same as with the old behaviour before this commit. - Triggers mechanism is unaffected by this change. This is tested in create_replace.test. - LOCK TABLES is affected. Lock restoration must be done after new table is created or TMP is renamed back to ORIG - Moved ddl_log_complete() from send_eof() to finalize_ddl(). This checkpoint was not executed before for normal CREATE TABLE but is executed now. - CREATE TABLE will now rollback also if writing to the binary logging failed. See rpl_gtid_strict.test backup ddl log changes - In case of a successfull CREATE OR REPLACE we only log the CREATE event, not the DROP TABLE event of the old table. ddl_log.cc changes ddl_log_execute_action() now properly return error conditions. ddl_log_disable_entry() added to allow one to disable one entry. The entry on disk is still reserved until ddl_log_complete() is executed. On XID usage Like with all other atomic DDL operations XID is used to avoid inconsistency between master and slave in the case of a crash after binary log is written and before ddl_log_state_create is closed. On recovery XIDs are taken from binary log and corresponding DDL log events get disabled. That is done by ddl_log_close_binlogged_events(). On linking two chains together Chains are executed in the ascending order of entry_pos of execute entries. But entry_pos assignment order is undefined: it may assign bigger number for the first chain and then smaller number for the second chain. So the execution order in that case will be reverse: second chain will be executed first. To avoid that we link one chain to another. While the base chain (ddl_log_state_create) is active the secondary chain (ddl_log_state_rm) is not executed. That is: only one chain can be executed in two linked chains. The interface ddl_log_link_chains() was defined in "MDEV-22166 ddl_log_write_execute_entry() extension". Atomic info parameters in HA_CREATE_INFO Many functions in CREATE TABLE pass the same parameters. These parameters are part of table creation info and should be in HA_CREATE_INFO (or whatever). Passing parameters via single structure is much easier for adding new data and refactoring. Aria changes: - Fixed issue in Aria engine with CREATE + locked tables that data was not properly commited in some cases in case of crashes. InnoDB related changes (by Marko Mäkelä): - table_name_t::is_create_or_replace(): A new predicate to check for CREATE OR REPLACE TABLE will rename an old table to and eventually drop after creating the replacement. - dict_table_t::parse_name(): Do acquire MDL on #sql-create- names for partitioned tables. - dict_table_rename_in_cache(): On CREATE OR REPLACE TABLE ... SELECT, forget the original dict_table_t::mdl_name so that purge will acquire MDL on the #sql-create- name instead. In this way, the MDL_EXCLUSIVE that the CREATE OR REPLACE TABLE holds on the user-visible name will not unnecessarily block any purge of old history until the very end when the #sql-create- table will be dropped. - ha_innobase::delete_table(): Do not check FOREIGN KEY consistency when dropping an #sql-create- table. - row_rename_table_for_mysql(): Update SYS_FOREIGN.ID also when renaming to #sql-create- in order to avoid any duplicate key error when CREATE OR REPLACE TABLE is creating some FOREIGN KEY constraints by names that existed in the old table. Other changes: - Removed some auto variables in log.cc for better code readability. - Fixed old bug that CREATE ... SELECT would not be able to auto repair a table that is part of the SELECT. - Marked MyISAM that it does not support ROLLBACK (not required but done for better consistency with other engines). - maria_create_trn_for_mysql() does not register a new transaction handler for commits. This was needed to ensure create or replace will not end with an active transaction. - We do not get anymore warnings about "Engine not supporting atomic create" when doing a legal CREATE OR REPLACE on a table with foreign key constraints. - Updated VIDEX engine flags to disable CREATE SEQUENCE. - Removed mysql_mutex_unlock(&LOCK_gdl) / mysql_mutex_lock(&LOCK_gdl) around calls to binlog as these are unsafe. The binlog code uses global variables that needs protection from other caller. - EITS data is preserved if create or replace fails if drop_before_create_or_replace=OFF. If ON, then create or replace will drop EITS before the drop of the original table (as before). - Using CREATE OR REPLACE on a encrypted table that the user cannot decrypt will fail instead of replacing the encrypted table. The encrypted table will unchanged. Known issues: - One cannot use create or replace on an InnoDB tables that has foreign key references point to it - CREATE OR REPLACE TEMPORARY table is not full atomic. Any conflicting table will always be dropped before creating a new one. (Old behaviour). Bug fixes related to this MDEV: MDEV-36435 Assertion failure in finalize_locked_tables() MDEV-36439 Assertion `thd_arg->lex->sql_command != SQLCOM_CREATE_SEQUENCE... MDEV-36498 Failed CoR in non-atomic mode no longer generates DROP in RBR... MDEV-36508 Temporary files #sql-create-....frm occasionally stay after crash recovery MDEV-38479 Crash in CREATE OR REPLACE SEQUENCE when new sequence cannot be created MDEV-36497 Assertion failure after atomic CoR with Aria under lock in transactional context MDEV-36501 EITS data is lost after failed attempt to CREATE OR REPLACE table MDEV-36493 Atomic CREATE OR REPLACE ... SELECT blocks InnoDB purge MDEV-39367 MSAN/valgrind errors in temp_file_size_cb_func, main.tmp_space_usage fails MDEV-39446 Atomic CREATE OR REPLACE fails if a table cannot be decrypted MDEV-40776 Atomic CREATE OR REPLACE silently breaks the foreign key MDEV-40765 Assertion `"unexpected references" == 0' failed upon failing CREATE OR REPLACE Reverted commits: - MDEV-36685 "CREATE-SELECT may lose in binlog side-effects of stored-routine" as it did not take into account that it safe to clear binlogs if the created table is non transactional and there are no other non transactional tables used. This was done because it caused extra logging when it is not needed (not using any non transactional tables) and it also did not solve side effects when using statement based loggging. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Petrunia
sergey@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Add comments | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Khaled Riyad
khaled57.dev@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38004 Double free on re-execution of prepared aggregate function Between two executions of a prepared statement, Item_sp::cleanup() freed the stored function's memory root but left the arena's free list pointing into it. That list is filled when the function call returns and the active arena is restored. On the next execution Item_sum_sp::clear() walked the stale list and destroyed already freed items. Free the items before freeing the memory they live in, the order Item_sum_sp::clear() already uses. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
PranavKTiwari
pranav.tiwari@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-25515 User Account Host Names using CIDR notation CREATE USER u@'192.168.0.0/24'; -- same as '192.168.0.0/255.255.255.0' CIDR is an input spelling only, normalised to the netmask form while parsing. A malformed mask is now rejected instead of stored, and one already in the privilege tables is skipped at load with a warning instead of being honoured. Nothing rejected these before, and they are not merely useless: a non-contiguous mask such as 10.0.0.0/255.0.255.0 matches the scattered set 10.*.0.*. Skipping rather than erroring keeps such a row droppable and renamable. New error ER_INVALID_HOST_NETMASK. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
CONCPP-163 Introducing allowMultiQueries option to control if multistatement queries are allowed Use of multistatements was harcoded - it wasn't meant to be this way. Now the option `allowMultiQueries` controls it and the default is off. Also, useAffectedOptions is added. Again, the code was there but it was not used/documented. The default for it is changed to be true, unlike JDBC wants it, to preserve the current default. Setting to false will make the driver to return for upserts number of found rows rather than affected In save way the option interactiveClient has been brought to life - the code was there, but the option was not used. Default is false. True will make the driver to use interactive client wait timeout. The test has been added. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41012 Galera appliers hang with foreign key of types UUID, INET4, INET6 wsrep_store_key_val_for_row() built the certification key of a row by collating the column value whenever the field reports MYSQL_TYPE_STRING or MYSQL_TYPE_VAR_STRING, taking the collation from Field::charset(). The data types implemented on Field_fbt - UUID, INET6 and INET4 report MYSQL_TYPE_STRING, and their charset() is my_charset_numeric, which is latin1. Their values are however plain binary and accordingly get_innobase_type_from_mysql_type() maps them to DATA_FIXBINARY. Their keys were therefore run through latin1_swedish_ci, which folds them. That corrupts the key in two ways: 1. A key mismatch for one and the same row. The reference key that wsrep_rec_get_foreign_key() appends for the parent of a child INSERT is built from the InnoDB record and is not collated, so it no longer matched the primary key carried by the parent row's own writeset. Certification saw no dependency between a child INSERT and a concurrent parent UPDATE, and two appliers could apply them in parallel causing a hang or crash. 2. A key collision between distinct rows. The folding is many to one, so different values collapse onto one key, Certification compares keys byte for byte, so unrelated rows were treated as the same row. Concurrent transactions on them certified as a conflict and one was aborted with ER_LOCK_DEADLOCK. Fix is for wsrep_store_key_val_for_row() to skip the collation for fields that InnoDB stores as binary, using the same condition as get_innobase_type_from_mysql_type(). This is a no-op for the types that worked before. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bsrikanth-mariadb
srikanth.bondalapati@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39226: Push whole multi-table update/delete down into engines Give storage engines a way to take over an entire multi-table UPDATE/DELETE, the way they can already take over a SELECT. Without it the join, the row matching and every modification run in the SQL layer even when an engine could do the whole statement itself in one step; a single-table UPDATE/DELETE already avoids this via direct_update_rows()/direct_delete_rows(), but a multi-table statement has no primary handler object to drive that path. This adds a generic, engine-agnostic pushdown interface: the SQL layer offers the statement to the engine, and if the engine accepts it, it performs the whole thing and reports only the row counts. - Split select_handler into a pushdown_handler base with select_handler (result set) and a new multi_upddel_handler (runs a whole UPDATE/DELETE, reports row counts, reported as PUSHED UPDATE/PUSHED DELETE); add handlerton::create_multi_upddel, looked up in Sql_cmd_dml::execute_inner(). - multi_update/multi_delete gain direct_update_delete_done(), which records the engine's counts so send_eof() binlogs and replies without the SQL-layer loop; it forces statement-format binlogging so the change still replicates under binlog_format=ROW, and errors out instead of silently dropping counts for an unsupported result object. - FederatedX implements the interface as the reference engine used to test correctness: it prints the statement back and runs it remotely, passes the engine's error code/SQLSTATE through, reads the matched count from the remote info string, executes IGNORE locally, and only pushes down when all tables share one remote server (same as SELECT/derived/unit pushdown). Test: federated.federatedx_pushdown_upd_del. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
copilot-swe-agent[bot]
198982749+Copilot@users.noreply.github.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Fix test_resultsetmetadata for MariaDB 12.3 display size change Co-authored-by: lawrinn <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Monty
monty@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Added --debug-dbug option to mysqltest.cc This was to get rid of warnings when using mtr --debug |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Monty
monty@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
ha_table_exists() cleanup and improvement This is part of MDEV-25292 Atomic CREATE OR REPLACE TABLE. Removed default values for arguments, added flags argument to specify filename flags (FN_TO_IS_TMP, FN_FROM_IS_TMP) and forward the flag to build_table_name(). Original patch from: Aleksey Midenkov <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Fixed the test for the case of SSPS | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
fixup: do not remove WITH_WSREP this creates ABI incompatiility. install wsrep headers instead |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| fixup! f06baeaf4fdbe51ef95b7aeb96f46622a0c2247b | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
CONCPP-160 Binary RS metadata was re-read before mysql_stmt_store_result That prevented the driver to have info about max field length. That made calculation of that length by C/C useless + made it allocate larger buffers for row and RS cache. The test does not cover that. I can not think of one. But leaving it in plcace - won't hurt. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Monty
monty@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-23298 Assertion `table_list->prelocking_placeholder == TABLE_LIST::PRELOCK_NONE' failed in check_lock_and_start_stmt on CREATE OR REPLACE TABLE Fixed by removing wrong assert Review: Sanja Byelkin |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hemant Dangi
hemant.dangi@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-40944: Galera test failure on galera_sst_mariabackup_ssl_role_certs Issue: mariadb-backup SST unconditionally passes socat's "commonname=" option; some socat builds don't register it, so parseopts() rejects it as unknown regardless of value, breaking all SSL-encrypted SST. Solution: probe the socat binary once for commonname support and drop the option when unsupported. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
CONCPP-165 the driver made more resilient to unexpected data in date/time fields The functions parsing Date and Timestamp fields(in text protocol) expected data in normal format only. Evil server could make them write to unallocated vector members and with specially crafted strings could control offset and whad driver writes there |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Moved C/C to what is supposed to be 3.3.21 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Lawrin Novitsky
lawrin.novitsky@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Removed last remains of the _snprintf | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Monty
monty@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-25292 Atomic CREATE OR REPLACE TABLE Atomic CREATE OR REPLACE allows to keep an old table intact if the command fails or during the crash. That is done by renaming the original table to temporary name, as a backup and restoring it if the CREATE fails. When the command is complete and logged the backup table is deleted. Atomic replace algorithm Two DDL chains are used for CREATE OR REPLACE: ddl_log_state_create (C) and ddl_log_state_rm (D). 1. (C) Log rename of ORIG to TMP table (Rename TMP to original). 2. Rename orignal to TMP. 3. (C) Log CREATE_TABLE_ACTION of ORIG (drops ORIG); 4. Do everything with ORIG (like insert data) 5. (D) Log drop of TMP 6. Write query to binlog (this marks (C) to be closed in case of failure) 7. Execute drop of TMP through (D) 8. Close (C) and (D) If there is a failure before 6) we revert the changes in (C) Chain (D) is only executed if 6) succeded (C is closed on crash recovery). Foreign key errors will be found at the 1) stage. Additional notes - CREATE TABLE without REPLACE and temporary tables is not affected by this commit. set @@drop_before_create_or_replace=1 can be used to get old behaviour where existing tables are dropped in CREATE OR REPLACE. - CREATE TABLE is reverted if binlogging the query fails. - Engines having HTON_EXPENSIVE_RENAME flag set are not affected by this commit. Conflicting tables marked with this flag will be deleted with CREATE OR REPLACE. - Replication execution is not affected by this commit. - Replication will first drop the conflicting table and then creating the new one. - CREATE TABLE .. SELECT XID usage is fixed and now there is no need to log DROP TABLE via DDL_CREATE_TABLE_PHASE_LOG (see comments in do_postlock()). XID is now correctly updated so it disables DDL_LOG_DROP_TABLE_ACTION. Note that binary log is flushed at the final stage when the table is ready. So if we have XID in the binary log we don't need to drop the table. - Three variations of CREATE OR REPLACE handled: 1. CREATE OR REPLACE TABLE t1 (..); 2. CREATE OR REPLACE TABLE t1 LIKE t2; 3. CREATE OR REPLACE TABLE t1 SELECT ..; - Test case uses 6 combinations for engines (aria, aria_notrans, myisam, ib, lock_tables, expensive_rename) and 2 combinations for binlog types (row, stmt). Combinations help to check differences between the results. Error failures are tested for the above three variations. - expensive_rename tests CREATE OR REPLACE without atomic replace. The effect should be the same as with the old behaviour before this commit. - Triggers mechanism is unaffected by this change. This is tested in create_replace.test. - LOCK TABLES is affected. Lock restoration must be done after new table is created or TMP is renamed back to ORIG - Moved ddl_log_complete() from send_eof() to finalize_ddl(). This checkpoint was not executed before for normal CREATE TABLE but is executed now. - CREATE TABLE will now rollback also if writing to the binary logging failed. See rpl_gtid_strict.test backup ddl log changes - In case of a successfull CREATE OR REPLACE we only log the CREATE event, not the DROP TABLE event of the old table. ddl_log.cc changes ddl_log_execute_action() now properly return error conditions. ddl_log_disable_entry() added to allow one to disable one entry. The entry on disk is still reserved until ddl_log_complete() is executed. On XID usage Like with all other atomic DDL operations XID is used to avoid inconsistency between master and slave in the case of a crash after binary log is written and before ddl_log_state_create is closed. On recovery XIDs are taken from binary log and corresponding DDL log events get disabled. That is done by ddl_log_close_binlogged_events(). On linking two chains together Chains are executed in the ascending order of entry_pos of execute entries. But entry_pos assignment order is undefined: it may assign bigger number for the first chain and then smaller number for the second chain. So the execution order in that case will be reverse: second chain will be executed first. To avoid that we link one chain to another. While the base chain (ddl_log_state_create) is active the secondary chain (ddl_log_state_rm) is not executed. That is: only one chain can be executed in two linked chains. The interface ddl_log_link_chains() was defined in "MDEV-22166 ddl_log_write_execute_entry() extension". Atomic info parameters in HA_CREATE_INFO Many functions in CREATE TABLE pass the same parameters. These parameters are part of table creation info and should be in HA_CREATE_INFO (or whatever). Passing parameters via single structure is much easier for adding new data and refactoring. Aria changes: - Fixed issue in Aria engine with CREATE + locked tables that data was not properly commited in some cases in case of crashes. InnoDB related changes (by Marko Mäkelä): - table_name_t::is_create_or_replace(): A new predicate to check for CREATE OR REPLACE TABLE will rename an old table to and eventually drop after creating the replacement. - dict_table_t::parse_name(): Do acquire MDL on #sql-create- names for partitioned tables. - dict_table_rename_in_cache(): On CREATE OR REPLACE TABLE ... SELECT, forget the original dict_table_t::mdl_name so that purge will acquire MDL on the #sql-create- name instead. In this way, the MDL_EXCLUSIVE that the CREATE OR REPLACE TABLE holds on the user-visible name will not unnecessarily block any purge of old history until the very end when the #sql-create- table will be dropped. - ha_innobase::delete_table(): Do not check FOREIGN KEY consistency when dropping an #sql-create- table. - row_rename_table_for_mysql(): Update SYS_FOREIGN.ID also when renaming to #sql-create- in order to avoid any duplicate key error when CREATE OR REPLACE TABLE is creating some FOREIGN KEY constraints by names that existed in the old table. Other changes: - Removed some auto variables in log.cc for better code readability. - Fixed old bug that CREATE ... SELECT would not be able to auto repair a table that is part of the SELECT. - Marked MyISAM that it does not support ROLLBACK (not required but done for better consistency with other engines). - maria_create_trn_for_mysql() does not register a new transaction handler for commits. This was needed to ensure create or replace will not end with an active transaction. - We do not get anymore warnings about "Engine not supporting atomic create" when doing a legal CREATE OR REPLACE on a table with foreign key constraints. - Updated VIDEX engine flags to disable CREATE SEQUENCE. - Removed mysql_mutex_unlock(&LOCK_gdl) / mysql_mutex_lock(&LOCK_gdl) around calls to binlog as these are unsafe. The binlog code uses global variables that needs protection from other caller. - EITS data is preserved if create or replace fails if drop_before_create_or_replace=OFF. If ON, then create or replace will drop EITS before the drop of the original table (as before). - Using CREATE OR REPLACE on a encrypted table that the user cannot decrypt will fail instead of replacing the encrypted table. The encrypted table will unchanged. Known issues: - One cannot use create or replace on an InnoDB tables that has foreign key references point to it - CREATE OR REPLACE TEMPORARY table is not full atomic. Any conflicting table will always be dropped before creating a new one. (Old behaviour). Bug fixes related to this MDEV: MDEV-36435 Assertion failure in finalize_locked_tables() MDEV-36439 Assertion `thd_arg->lex->sql_command != SQLCOM_CREATE_SEQUENCE... MDEV-36498 Failed CoR in non-atomic mode no longer generates DROP in RBR... MDEV-36508 Temporary files #sql-create-....frm occasionally stay after crash recovery MDEV-38479 Crash in CREATE OR REPLACE SEQUENCE when new sequence cannot be created MDEV-36497 Assertion failure after atomic CoR with Aria under lock in transactional context MDEV-36501 EITS data is lost after failed attempt to CREATE OR REPLACE table MDEV-36493 Atomic CREATE OR REPLACE ... SELECT blocks InnoDB purge MDEV-39367 MSAN/valgrind errors in temp_file_size_cb_func, main.tmp_space_usage fails MDEV-39446 Atomic CREATE OR REPLACE fails if a table cannot be decrypted MDEV-40776 Atomic CREATE OR REPLACE silently breaks the foreign key MDEV-40765 Assertion `"unexpected references" == 0' failed upon failing CREATE OR REPLACE Reverted commits: - MDEV-36685 "CREATE-SELECT may lose in binlog side-effects of stored-routine" as it did not take into account that it safe to clear binlogs if the created table is non transactional and there are no other non transactional tables used. This was done because it caused extra logging when it is not needed (not using any non transactional tables) and it also did not solve side effects when using statement based loggging. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
sjaakola
seppo.jaakola@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-41012 Galera appliers hang with foreign key of types UUID, INET4, INET6 Added second test for testing key collisions from transactions modifying separate rows |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||