Console View
|
Categories: connectors experimental galera main |
|
| connectors | experimental | galera | main | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-37949: Implement innodb_log_archive The InnoDB write-ahead log file in the old innodb_log_archive=OFF format is named ib_logfile0, pre-allocated to innodb_log_file_size and written as a ring buffer. This is good for write performance and space management, but unsuitable for arbitrary point-in-time recovery or for facilitating efficient incremental backup. innodb_log_archive=ON: A new format where InnoDB will create and preallocate files ib_%016x.log, instead of writing a circular file ib_logfile0. Each file will be pre-allocated to innodb_log_file_size. Once a log fills up, we will create and pre-allocate another log file, to which log records will be written. Upon the completion of the first log checkpoint in a recently created log file, the old log file will be marked read-only, signaling that there will be no further writes to that file, and that the file may safely be moved to long-term storage. The file name includes the log sequence number (LSN) at file offset 12288 (log_t::START_OFFSET). Each checkpoint is identified by storing a 64-bit big-endian offset into an optional sequence of FILE_MODIFY records followed by a FILE_CHECKPOINT record, between 12288 and the end of the file. The innodb_encrypt_log format is identified by storing the encryption information at the start of the log file. The first 64-bit value will be 1, which is an invalid checkpoint offset. Each innodb_log_archive=ON log must use the same encryption parameters. Changing innodb_encrypt_log or related parameters is only possible by setting innodb_log_archive=OFF and restarting the server, which will permanently lose the history of the archived log. The maximum number of log checkpoints that the innodb_log_archive=ON file header can represent is limited to 12288/8=1536 when using innodb_encrypt_log=OFF. If we run out of slots in a log file, each subsequently completed checkpoint in that log file will overwrite the last slot in the checkpoint header, until we switch to the next log. innodb_log_recovery_start: The checkpoint LSN to start recovery from. This will be useful when recovering from an archived log. This is useful for restoring an incremental backup (applying InnoDB log files that were copied since the previous restore). innodb_log_recovery_target: The requested LSN to end recovery at. When this is set, all persistent InnoDB tables will be read-only, and no writes to the log are allowed. The intended purpose of this setting is to prepare an incremental backup, as well as to allow data retrieval as of a particular logical point of time. Setting innodb_log_recovery_target>0 is much like setting innodb_read_only=ON, with the exception that the data files may be written to by crash recovery, and locking reads will conflict with any incomplete transactions as necessary, and all transaction isolation levels will work normally (not hard-wired to READ UNCOMMITTED). srv_read_only_mode: When this is set (innodb_read_only=ON), also recv_sys.rpo (innodb_log_recovery_target) will be set to the current LSN. This ensures that it will suffice to check only one of these variables when blocking writes to persistent tables. The status variable innodb_lsn_archived will reflect the LSN since when a complete InnoDB log archive is available. Its initial value will be that of the new parameter innodb_log_archive_start. If that variable is 0 (the default), the innodb_lsn_archived will be recovered from the available log files. If innodb_log_archive=OFF, innodb_lsn_archived will be adjusted to the latest checkpoint every time a log checkpoint is executed. If innodb_log_archive=ON, the value should not change. SET GLOBAL innodb_log_archive=!@@GLOBAL.innodb_log_archive will take effect as soon as possible, possibly after a log checkpoint has been completed. The log file will be renamed between ib_logfile0 and ib_%016x.log as appropriate. When innodb_log_archive=ON, the setting SET GLOBAL innodb_log_file_size will affect subsequently created log files when the file that is being currently written is running out. If we are switching log files exactly at the same time, then a somewhat misleading error message "innodb_log_file_size change is already in progress" will be issued. no_checkpoint_prepare.inc: A new file, to prepare for subsequent inclusion of no_checkpoint_end.inc. We will invoke the server to parse the log and to determine the latest checkpoint. All --suite=encryption tests that use innodb_encrypt_log will be skipped for innodb_log_archive=ON, because enabling or disabling encryption on the log is not possible without temporarily setting innodb_log_archive=OFF and restarting the server. The idea is to add the following arguments to an invocation of mysql-test/mtr: --mysqld=--loose-innodb-log-archive \ --mysqld=--loose-innodb-log-recovery-start=12288 \ --mysqld=--loose-innodb-log-file-mmap=OFF \ --skip-test=mariabackup Alternatively, specify --mysqld=--loose-innodb-log-file-mmap=ON to cover both code paths. The mariabackup test suite must be skipped when using the innodb_log_archive=ON format, because mariadb-backup will only support the old ib_logfile0 format (innodb_log_archive=OFF). A number of tests would fail when the parameter innodb_log_recovery_start=12288 is present, which is forcing recovery to start from the beginning of the history (the database creation). The affected tests have been adjusted with explicit --innodb-log-recovery-start=0 to override that: (0) Some injected corruption may be "healed" by replaying the log from the beginning. Some tests expect an empty buffer pool after a restart, with no page I/O due to crash recovery. (1) Any test that sets innodb_read_only=ON would fail with an error message that the setting prevents crash recovery, unless innodb_log_recovery_start=0. (2) Any test that changes innodb_undo_tablespaces would fail in crash recovery, because crash recovery assumes that the undo tablespace ID that is available from the undo* files corresponds with the start of the log. This is an unforunate design bug which we cannot fix easily. log_sys.first_lsn: The start of the current log file, to be consulted in log_t::write_checkpoint() when renaming files. log_sys.archived_lsn: New field: The value of innodb_lsn_archived. log_sys.end_lsn: New field: The log_sys.get_lsn() when the latest checkpoint was initiated. That is, the start LSN of a possibly empty sequence of FILE_MODIFY records followed by FILE_CHECKPOINT. log_sys.resize_target: The value of innodb_log_file_size that will be used for creating the next archive log file once the current file (of log_sys.file_size) fills up. log_sys.archive: New field: The value of innodb_log_archive. log_sys.next_checkpoint_no: Widen to uint16_t. There may be up to 12288/8=1536 checkpoints in the header. log_sys.log: If innodb_log_archive=ON, this file handle will be kept open also in the PMEM code path. log_sys.resize_log: If innodb_log_archive=ON, we may have two log files open both during normal operation and when parsing the log. This will store the other handle (old or new file). log_sys.resize_buf: In the memory-mapped code path, this will point to the file resize_log when innodb_log_archive=ON. recv_sys.log_archive: All innodb_log_archive=ON files that will be considered in recovery. recv_sys.was_archive: A flag indicating that an innodb_log_archive=ON file is in innodb_log_archive=OFF format. log_sys.is_pmem, log_t::is_mmap_writeable(): A new predicate. If is_mmap_writeable(), we assert and guarantee buf_size == capacity(). log_t::checkpoint_age_max(): Return the maximum checkpoint age. When innodb_log_archive=ON the limit is innodb_log_file_size larger. log_t::archive_new_write(): Create and allocate a new log file, and write the outstanding data to both the current and the new file, or only to the new file, until write_checkpoint() completes the first checkpoint in the new file. If the log file would overflow, increase innodb_log_file_size to the next multiple of 4096 bytes. log_t::archive_create(bool): Create and allocate an archive log file. On POSIX, ensure that we get the exact correct file size by invoking ftruncate() on POSIX, because os_file_set_size() would only truncate files on Microsoft Windows. Remember the file handle of the current log in resize_log, so that write_checkpoint() will be able to make it read-only. log_t::archived_mmap_switch_prepare(): Invoke archive_create() unless another thread already did it. log_t::archived_mmap_switch_complete(): Switch to the buffer that was created in log_t::archived_mmap_switch_prepare(). Remember the old log_sys.buf in log_sys.checkpoint_buf, to be unmapped in log_t::write_checkpoint(). buf_flush_archive_create(): Create a spare archive log file if needed. buf_flush_page_cleaner(): Invoke buf_flush_archive_create() about one second after a checkpoint where we switched archived log files. This allows any writer threads to resume activity after we completed the previous checkpoint. log_t::write_checkpoint(): Allow an old checkpoint to be completed in the old log file even after a new one has been created. If we are writing the first checkpoint in a new log file, we will mark the old log file read-only. We will also update log_sys.first_lsn unless it was already updated in ARCHIVED_MMAP code path. In that code path, there is the special case where log_sys.resize_buf == nullptr and log_sys.checkpoint_buf points to log_sys.resize_log (the old log file that is about to be made read-only). In this case, log_sys.first_lsn will already point to the start of the current log_sys.log, even though the switch has not been fully completed yet. Try to preallocate the next archive file if needed, with the goal that when we need the file it will already be ready for use. log_t::write_checkpoint(), log_checkpoint_low(), log_checkpoint(), buf_flush_sync_for_checkpoint(): Return the LSN on which to invoke archive_create(), or 0 if no spare archive log file needs to be created. log_t::header_rewrite(my_bool): Rewrite the log file header before or after renaming the log file, and write a message about the change, so that there will be a chance to recover in case the server is being killed during this operation. The recovery of the last ib_%016%.log does tolerate also the ib_logfile0 format. log_t::set_archive(my_bool,THD): Implement SET GLOBAL innodb_log_archive. An error will be returned if non-archived SET GLOBAL innodb_log_file_size (log file resizing) is in progress. Wait for checkpoint if necessary. The current log file will be renamed to either ib_logfile0 or ib_%016x.log, as appropriate. In SET GLOBAL innodb_log_archive=OFF, we trigger a write-ahead of the log if necessary, to prevent overrun. log_t::archive_rename(): Rename an archived log to ib_logfile0 on recovery in case there had been a crash during set_archive(). log_t::archive_set_size(): A new function, to ensure that log_sys.resize_target is set on startup. log_checkpoint_low(): Do not prevent a checkpoint at the start of a file. We want the first innodb_log_archive=ON file to start with a checkpoint. log_t::create(lsn_t): Initialize last_checkpoint_lsn. Initialize the log header as specified by log_sys.archive (innodb_log_archive). log_write_buf(): Add the parameter max_length, the file wrap limit. log_write_up_to(), mtr_t::commit_log_release<bool mmap=true>(): If we are switching log files, invoke buf_flush_ahead(lsn, false) to ensure that a log checkpoint will be completed in the new file. mtr_t::finish_writer(): Specialize for innodb_log_archive=ON. mtr_t::commit_file(): Ensure that log archive rotation will complete. log_t::append_prepare<log_t::ARCHIVED_MMAP>(): Special case. log_t::append_prepare(): Remove the call to set_check_for_checkpoint() that was duplicating logic in log_close(). log_t::get_path(): Get the name of the current log file. log_t::get_circular_path(size_t): Get the path name of a circular file. Replaces get_log_file_path(). log_t::get_archive_path(lsn_t): Return a name of an archived log file. log_t::append_archive_name(): Append the archive log file name to a path string. log_t::checkpoint_margin(): Replaces log_checkpoint_margin(). If a new archived log file has been created, wait for the first checkpoint in that file. Revise the formula when log_sys.need_checkpoint needs to hold. When innodb_log_archive=ON, we can allow the checkpoint to be older by innodb_log_file_size. log_close(): Relax the log_sys.need_checkpoint condition for innodb_log_archive=ON. srv_log_rebuild_if_needed(): Never rebuild if innodb_log_archive=ON. The setting innodb_log_file_size will affect the creation of subsequent log files. The parameter innodb_encrypt_log cannot be changed while the log is in the innodb_log_archive=ON format. log_t::attach(), log_mmap(): Add the parameter log_access, to distinguish memory-mapped or read-only access. log_t::attach(): When disabling innodb_log_file_mmap, read checkpoint_buf from the last innodb_log_archive=ON file. log_t::clear_mmap(): Clear the tail of the checkpoint buffer if is_mmap_writeable(). log_t::set_recovered(): Invoke clear_mmap(), and restore the log buffer to the correct position. recv_sys_t::apply(): Let log_t::clear_mmap() enable log writes. log_file_is_zero(): Check if a log file starts with NUL bytes (is a preallocated file). recv_sys_t::find_checkpoint(): Find and remember the checkpoint position in the last file when innodb_log_recovery_start points to an older file. When innodb_log_file_mmap=OFF, restore log_sys.checkpoint_buf from the latest log file. If the last archive log file is actually in innodb_log_archive=OFF format despite being named ib_%016.log, try to recover it in that format. If the circular ib_logfile0 is missing, determine the oldest archived log file with contiguous LSN. If innodb_log_archive=ON, refuse to start if ib_logfile0 exists. Open non-last non-preallocated archived log file in read/write mode. recv_sys_t::find_checkpoint_archived(): Validate each checkpoint in the current file header, and by default aim to recover from the last valid one. Terminate the search if the last validated checkpoint spanned two files. If innodb_log_recovery_start has been specified, attempt to validate it even if there is no such information stored in the checkpoint header. log_parse_file(): Do not invoke fil_name_process() during recv_sys_t::find_checkpoint_archived(), when we tolerate FILE_MODIFY records while looking for a FILE_CHECKPOINT record. recv_scan_log(): Invoke log_t::archived_switch_recovery() upon reaching the end of the current archived log file. log_t::archived_switch_recovery_prepare(): Make use of recv_sys.log_archive and open all but the last file read-only. log_t::archived_switch_recovery(): Switch files in the pread() code path. log_t::archived_mmap_switch_recovery_complete(): Switch files in the memory-mapped code path. recv_warp: A pointer wrapper for memory-mapped parsing that spans two archive log files. recv_sys_t::parse_mmap(): Use recv_warp for innodb_log_archive=ON. recv_sys_t::parse(): Tweak some logic for innodb_log_archive=ON. log_t::set_recovered_checkpoint(): Set the checkpoint on recovery. Updates also the end_lsn. log_t::set_recovered_lsn(): Also update flush_lock and write_lock, to ensure that log_write_up_to() will be a no-op. Invoke unstash_archive_file() in case there was a garbage (pre-allocated) file at the end which was not parsed at all. log_t::persist(): Even if the flushed_to_disk_lsn does not change, we may want to reset the write_lsn_offset. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Daniel Black
daniel@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-36453 UBSAN: main.mysqltest - runtime error: null pointer ..passed as argument 2, which is declared to never be null (dynstr_append_mem). dynstr_append_mem, uses memcpy to append to the string which in the glibc library is declared to not take null string as the src. Defensively we declare dynstr_append_mem to not take a null argument too. mariadb-test, to prevent it calling dynstr_append_mem with a null mtr variable value, one that hasn't been initialized, we return an error if the value is null. The result of this is the $6 in the test case is never assigned a value and would error. With this resolved, strchr, if the string isn't found, a null pointer is returned. UBSAN will complain both about incrementing a pointer beyond the boundary, and also incrementing if it is null. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yuchen Pei
ycp@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38752 Check that virtual column is a supertype to its expression before substitution New optimizations were introduced that substitute virtual column expressions (abbr. vcol expr) with index virtual columns (abbr. vcol field) in ORDER BY and GROUP BY. In this patch we introduce checks that the type of a vcol field is the supertype to its vcol expr, and if not, do not proceed the optimization. This ensures that the substitution is safe and does not for example lose information due to truncation. For simplicity, the version of the check implemented is strict with 100% precision, meaning that there are instances where vcol field is a supertype to vcol expr, but the check returns false. Type not covered in tests: Type_handler_null Thanks to Jarir Khan <[email protected]> for initial patches addressing this issue. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Merge 12.3 into main | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-37949: Implement innodb_log_archive The InnoDB write-ahead log file in the old innodb_log_archive=OFF format is named ib_logfile0, pre-allocated to innodb_log_file_size and written as a ring buffer. This is good for write performance and space management, but unsuitable for arbitrary point-in-time recovery or for facilitating efficient incremental backup. innodb_log_archive=ON: A new format where InnoDB will create and preallocate files ib_%016x.log, instead of writing a circular file ib_logfile0. Each file will be pre-allocated to innodb_log_file_size. Once a log fills up, we will create and pre-allocate another log file, to which log records will be written. Upon the completion of the first log checkpoint in a recently created log file, the old log file will be marked read-only, signaling that there will be no further writes to that file, and that the file may safely be moved to long-term storage. The file name includes the log sequence number (LSN) at file offset 12288 (log_t::START_OFFSET). Each checkpoint is identified by storing a 64-bit big-endian offset into an optional sequence of FILE_MODIFY records followed by a FILE_CHECKPOINT record, between 12288 and the end of the file. The innodb_encrypt_log format is identified by storing the encryption information at the start of the log file. The first 64-bit value will be 1, which is an invalid checkpoint offset. Each innodb_log_archive=ON log must use the same encryption parameters. Changing innodb_encrypt_log or related parameters is only possible by setting innodb_log_archive=OFF and restarting the server, which will permanently lose the history of the archived log. The maximum number of log checkpoints that the innodb_log_archive=ON file header can represent is limited to 12288/8=1536 when using innodb_encrypt_log=OFF. If we run out of slots in a log file, each subsequently completed checkpoint in that log file will overwrite the last slot in the checkpoint header, until we switch to the next log. innodb_log_recovery_start: The checkpoint LSN to start recovery from. This will be useful when recovering from an archived log. This is useful for restoring an incremental backup (applying InnoDB log files that were copied since the previous restore). innodb_log_recovery_target: The requested LSN to end recovery at. When this is set, all persistent InnoDB tables will be read-only, and no writes to the log are allowed. The intended purpose of this setting is to prepare an incremental backup, as well as to allow data retrieval as of a particular logical point of time. Setting innodb_log_recovery_target>0 is much like setting innodb_read_only=ON, with the exception that the data files may be written to by crash recovery, and locking reads will conflict with any incomplete transactions as necessary, and all transaction isolation levels will work normally (not hard-wired to READ UNCOMMITTED). srv_read_only_mode: When this is set (innodb_read_only=ON), also recv_sys.rpo (innodb_log_recovery_target) will be set to the current LSN. This ensures that it will suffice to check only one of these variables when blocking writes to persistent tables. The status variable innodb_lsn_archived will reflect the LSN since when a complete InnoDB log archive is available. Its initial value will be that of the new parameter innodb_log_archive_start. If that variable is 0 (the default), the innodb_lsn_archived will be recovered from the available log files. If innodb_log_archive=OFF, innodb_lsn_archived will be adjusted to the latest checkpoint every time a log checkpoint is executed. If innodb_log_archive=ON, the value should not change. SET GLOBAL innodb_log_archive=!@@GLOBAL.innodb_log_archive will take effect as soon as possible, possibly after a log checkpoint has been completed. The log file will be renamed between ib_logfile0 and ib_%016x.log as appropriate. When innodb_log_archive=ON, the setting SET GLOBAL innodb_log_file_size will affect subsequently created log files when the file that is being currently written is running out. If we are switching log files exactly at the same time, then a somewhat misleading error message "innodb_log_file_size change is already in progress" will be issued. no_checkpoint_prepare.inc: A new file, to prepare for subsequent inclusion of no_checkpoint_end.inc. We will invoke the server to parse the log and to determine the latest checkpoint. All --suite=encryption tests that use innodb_encrypt_log will be skipped for innodb_log_archive=ON, because enabling or disabling encryption on the log is not possible without temporarily setting innodb_log_archive=OFF and restarting the server. The idea is to add the following arguments to an invocation of mysql-test/mtr: --mysqld=--loose-innodb-log-archive \ --mysqld=--loose-innodb-log-recovery-start=12288 \ --mysqld=--loose-innodb-log-file-mmap=OFF \ --skip-test=mariabackup Alternatively, specify --mysqld=--loose-innodb-log-file-mmap=ON to cover both code paths. The mariabackup test suite must be skipped when using the innodb_log_archive=ON format, because mariadb-backup will only support the old ib_logfile0 format (innodb_log_archive=OFF). A number of tests would fail when the parameter innodb_log_recovery_start=12288 is present, which is forcing recovery to start from the beginning of the history (the database creation). The affected tests have been adjusted with explicit --innodb-log-recovery-start=0 to override that: (0) Some injected corruption may be "healed" by replaying the log from the beginning. Some tests expect an empty buffer pool after a restart, with no page I/O due to crash recovery. (1) Any test that sets innodb_read_only=ON would fail with an error message that the setting prevents crash recovery, unless innodb_log_recovery_start=0. (2) Any test that changes innodb_undo_tablespaces would fail in crash recovery, because crash recovery assumes that the undo tablespace ID that is available from the undo* files corresponds with the start of the log. This is an unforunate design bug which we cannot fix easily. log_sys.first_lsn: The start of the current log file, to be consulted in log_t::write_checkpoint() when renaming files. log_sys.archived_lsn: New field: The value of innodb_lsn_archived. log_sys.end_lsn: New field: The log_sys.get_lsn() when the latest checkpoint was initiated. That is, the start LSN of a possibly empty sequence of FILE_MODIFY records followed by FILE_CHECKPOINT. log_sys.resize_target: The value of innodb_log_file_size that will be used for creating the next archive log file once the current file (of log_sys.file_size) fills up. log_sys.archive: New field: The value of innodb_log_archive. log_sys.next_checkpoint_no: Widen to uint16_t. There may be up to 12288/8=1536 checkpoints in the header. log_sys.log: If innodb_log_archive=ON, this file handle will be kept open also in the PMEM code path. log_sys.resize_log: If innodb_log_archive=ON, we may have two log files open both during normal operation and when parsing the log. This will store the other handle (old or new file). log_sys.resize_buf: In the memory-mapped code path, this will point to the file resize_log when innodb_log_archive=ON. recv_sys.log_archive: All innodb_log_archive=ON files that will be considered in recovery. recv_sys.was_archive: A flag indicating that an innodb_log_archive=ON file is in innodb_log_archive=OFF format. log_sys.is_pmem, log_t::is_mmap_writeable(): A new predicate. If is_mmap_writeable(), we assert and guarantee buf_size == capacity(). log_t::checkpoint_age_max(): Return the maximum checkpoint age. When innodb_log_archive=ON the limit is innodb_log_file_size larger. log_t::archive_new_write(): Create and allocate a new log file, and write the outstanding data to both the current and the new file, or only to the new file, until write_checkpoint() completes the first checkpoint in the new file. If the log file would overflow, increase innodb_log_file_size to the next multiple of 4096 bytes. log_t::archive_create(bool): Create and allocate an archive log file. On POSIX, ensure that we get the exact correct file size by invoking ftruncate() on POSIX, because os_file_set_size() would only truncate files on Microsoft Windows. Remember the file handle of the current log in resize_log, so that write_checkpoint() will be able to make it read-only. log_t::archived_mmap_switch_prepare(): Invoke archive_create() unless another thread already did it. log_t::archived_mmap_switch_complete(): Switch to the buffer that was created in log_t::archived_mmap_switch_prepare(). Remember the old log_sys.buf in log_sys.checkpoint_buf, to be unmapped in log_t::write_checkpoint(). buf_flush_archive_create(): Create a spare archive log file if needed. buf_flush_page_cleaner(): Invoke buf_flush_archive_create() about one second after a checkpoint where we switched archived log files. This allows any writer threads to resume activity after we completed the previous checkpoint. log_t::write_checkpoint(): Allow an old checkpoint to be completed in the old log file even after a new one has been created. If we are writing the first checkpoint in a new log file, we will mark the old log file read-only. We will also update log_sys.first_lsn unless it was already updated in ARCHIVED_MMAP code path. In that code path, there is the special case where log_sys.resize_buf == nullptr and log_sys.checkpoint_buf points to log_sys.resize_log (the old log file that is about to be made read-only). In this case, log_sys.first_lsn will already point to the start of the current log_sys.log, even though the switch has not been fully completed yet. Try to preallocate the next archive file if needed, with the goal that when we need the file it will already be ready for use. log_t::write_checkpoint(), log_checkpoint_low(), log_checkpoint(), buf_flush_sync_for_checkpoint(): Return the LSN on which to invoke archive_create(), or 0 if no spare archive log file needs to be created. log_t::header_rewrite(my_bool): Rewrite the log file header before or after renaming the log file, and write a message about the change, so that there will be a chance to recover in case the server is being killed during this operation. The recovery of the last ib_%016%.log does tolerate also the ib_logfile0 format. log_t::set_archive(my_bool,THD): Implement SET GLOBAL innodb_log_archive. An error will be returned if non-archived SET GLOBAL innodb_log_file_size (log file resizing) is in progress. Wait for checkpoint if necessary. The current log file will be renamed to either ib_logfile0 or ib_%016x.log, as appropriate. In SET GLOBAL innodb_log_archive=OFF, we trigger a write-ahead of the log if necessary, to prevent overrun. log_t::archive_rename(): Rename an archived log to ib_logfile0 on recovery in case there had been a crash during set_archive(). log_t::archive_set_size(): A new function, to ensure that log_sys.resize_target is set on startup. log_checkpoint_low(): Do not prevent a checkpoint at the start of a file. We want the first innodb_log_archive=ON file to start with a checkpoint. log_t::create(lsn_t): Initialize last_checkpoint_lsn. Initialize the log header as specified by log_sys.archive (innodb_log_archive). log_write_buf(): Add the parameter max_length, the file wrap limit. log_write_up_to(), mtr_t::commit_log_release<bool mmap=true>(): If we are switching log files, invoke buf_flush_ahead(lsn, false) to ensure that a log checkpoint will be completed in the new file. mtr_t::finish_writer(): Specialize for innodb_log_archive=ON. mtr_t::commit_file(): Ensure that log archive rotation will complete. log_t::append_prepare<log_t::ARCHIVED_MMAP>(): Special case. log_t::append_prepare(): Remove the call to set_check_for_checkpoint() that was duplicating logic in log_close(). log_t::get_path(): Get the name of the current log file. log_t::get_circular_path(size_t): Get the path name of a circular file. Replaces get_log_file_path(). log_t::get_archive_path(lsn_t): Return a name of an archived log file. log_t::append_archive_name(): Append the archive log file name to a path string. log_t::checkpoint_margin(): Replaces log_checkpoint_margin(). If a new archived log file has been created, wait for the first checkpoint in that file. Revise the formula when log_sys.need_checkpoint needs to hold. When innodb_log_archive=ON, we can allow the checkpoint to be older by innodb_log_file_size. log_close(): Relax the log_sys.need_checkpoint condition for innodb_log_archive=ON. srv_log_rebuild_if_needed(): Never rebuild if innodb_log_archive=ON. The setting innodb_log_file_size will affect the creation of subsequent log files. The parameter innodb_encrypt_log cannot be changed while the log is in the innodb_log_archive=ON format. log_t::attach(), log_mmap(): Add the parameter log_access, to distinguish memory-mapped or read-only access. log_t::attach(): When disabling innodb_log_file_mmap, read checkpoint_buf from the last innodb_log_archive=ON file. log_t::clear_mmap(): Clear the tail of the checkpoint buffer if is_mmap_writeable(). log_t::set_recovered(): Invoke clear_mmap(), and restore the log buffer to the correct position. recv_sys_t::apply(): Let log_t::clear_mmap() enable log writes. log_file_is_zero(): Check if a log file starts with NUL bytes (is a preallocated file). recv_sys_t::find_checkpoint(): Find and remember the checkpoint position in the last file when innodb_log_recovery_start points to an older file. When innodb_log_file_mmap=OFF, restore log_sys.checkpoint_buf from the latest log file. If the last archive log file is actually in innodb_log_archive=OFF format despite being named ib_%016.log, try to recover it in that format. If the circular ib_logfile0 is missing, determine the oldest archived log file with contiguous LSN. If innodb_log_archive=ON, refuse to start if ib_logfile0 exists. Open non-last non-preallocated archived log file in read/write mode. recv_sys_t::find_checkpoint_archived(): Validate each checkpoint in the current file header, and by default aim to recover from the last valid one. Terminate the search if the last validated checkpoint spanned two files. If innodb_log_recovery_start has been specified, attempt to validate it even if there is no such information stored in the checkpoint header. log_parse_file(): Do not invoke fil_name_process() during recv_sys_t::find_checkpoint_archived(), when we tolerate FILE_MODIFY records while looking for a FILE_CHECKPOINT record. recv_scan_log(): Invoke log_t::archived_switch_recovery() upon reaching the end of the current archived log file. log_t::archived_switch_recovery_prepare(): Make use of recv_sys.log_archive and open all but the last file read-only. log_t::archived_switch_recovery(): Switch files in the pread() code path. log_t::archived_mmap_switch_recovery_complete(): Switch files in the memory-mapped code path. recv_warp: A pointer wrapper for memory-mapped parsing that spans two archive log files. recv_sys_t::parse_mmap(): Use recv_warp for innodb_log_archive=ON. recv_sys_t::parse(): Tweak some logic for innodb_log_archive=ON. log_t::set_recovered_checkpoint(): Set the checkpoint on recovery. Updates also the end_lsn. log_t::set_recovered_lsn(): Also update flush_lock and write_lock, to ensure that log_write_up_to() will be a no-op. Invoke unstash_archive_file() in case there was a garbage (pre-allocated) file at the end which was not parsed at all. log_t::persist(): Even if the flushed_to_disk_lsn does not change, we may want to reset the write_lsn_offset. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
cleanup: sphinx tests run sphinx tests even if no sphinx is installed require installed sphinx per-test, not for the whole suite, in case there are tests that can run without it |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Daniel Black
daniel@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-35545 UBSAN Gis_geometry_collection::init_from_opresult From the UBSAN error: sql/spatial.cc:3364:10: runtime error: applying non-zero offset 1 to null pointer In Gis_geometry_collection::init_from_opresult, a pointer argument was being treated as a counter for the Special case of GEOMETRYCOLLECTION EMPTY. The memory location was never accessed. Rather than use points to count and return a difference at the end, the code is replace to use g_len_total as a counter. This gets a 1 value for the GEOMETRYCOLLECTION EMPTY case and no ointer undefined behaviours occur. As other init_from_opresult functions return uint both g_len and result use that type. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dave Gosselin
dave.gosselin@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-23278: Incorrect Calculation with AVG() Function Type_handler_int_result::Item_decimal_precision assumes that max_length includes space for a sign, but Item_int literals don't include such a space. When aggregate_attributes_int propagates these into functions via a Item_type_holder, the formula computes precision-1 (or 0 for 1-digit cases), producing the incorrect calculation with AVG. This fix changes aggregate_attributes_int to follow the convention the Type_handler expects, that max_length = max digit-count + (signed ? 1 : 0). This matches the TODO comment that begins with "TODO: rewrite aggregate_attributes_int()... " which suggested this idea. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Daniel Black
daniel@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-26814: UBSAN: offset to nullptr in JSON_ARRAY_INSERT SELECT JSON_ARRAY_INSERT (0,NULL,1); triggered a UBSAN error. Specification of JSON_ARRAY_INSERT should return NULL if any arguments are null. SQL NULL, aka Item_null::val_str will return a nullptr so check this and then return a NULL value. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Merge branch '10.11' into bb-10.11-release | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Daniel Black
daniel@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-36451: blackhole float-cast-overflow As UBSAN error, the attempt of evaluating a best_acess_path in the optimizer was using -nan as its worst_seeks value. This didn't cast to an integer for a rows estimate value resulting in the UBSAN error. The blackhole engine had a worst_seeks derived from read_time (same value). This was derived in the default handler::scan_time as stats.data_file_length / stats.block_size expression where both where 0. Corrected this by giving the default handler::scan_time an implementation that just returns 0 for the case where stats.block_size was 0, to avoid returning a NaN values for all storage engines that leave their stats block_size as 0, including the backhole. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
ayush-jha123
kumarayushjha123@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38010: Master & relay log info files ignore trailing garbage in numeric lines This patch fixes an issue where Int_IO_CACHE::from_chars stops parsing at the first invalid character but fails to consume the remainder of the line. This caused trailing garbage on a numeric field (like Master_Port) to be interpreted as the value for the subsequent field. The fix introduces a strict validation helper is_string_blank_or_empty which ensures that only whitespace or control characters follow the parsed numeric value. The init_*_from_file functions now zero-initialize variables, perform error checking immediately after string conversion, and safely reject files with trailing garbage. The test master_info_numeric_validation has been updated to use --move_file for robust backup and restoration of the master.info file. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Daniel Black
daniel@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38913: pam_mariadb_mtr.so installation path change This reverts the effect of MDEV-21654 and includes pam_mariadb_mtr in the mariadb-test-data Deb package. This allows our CI systems to run tests on the pam implementation on Debian systems. As the pam configuration can include a full path, we've change the installation of the pam module pam_mariadb_mtr.so to under the /usr/lib*/plugins/test_pam_modules directory and configured the pam configuration file, suite/plugins/pam/mariadb_mtr, to include the full path to pam_mariadb_mtr.so. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39512 SIGSEGV in ha_sphinx::create on TRUNCATE regression dd_recreate_table() had NULL create_info->option_struct. Let's get it from the table share. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Merge branch 'bb-10.11-release' into bb-11.4-release | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
a18792721831
735809899@qq.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-33814 Wrong error message "Can't create table" on "ALTER TABLE" When ALTER TABLE uses the copy algorithm, ha_create_table() is invoked to build the shadow table. A failure there was previously reported as "Can't create table", which is misleading because the user never issued a CREATE. Fix (per review by @midenok): keep errno 1005 / SQLSTATE HY000 unchanged and only swap the message text when HA_CREATE_TMP_ALTER is set, via a new translatable template ER_CANT_CREATE_TMP_ALTER_TABLE in errmsg-utf8.txt (no SQLSTATE clause -- it is a message template rather than a wire error code). The user-visible error becomes: ERROR 1005 (HY000): Can't create temporary table for ALTER TABLE \ `db`.`tbl` (errno: N "...") No existing error handler, stored procedure, tool, or application that matches on errno 1005 or SQLSTATE HY000 is affected. Notes on the 12.3 rebase: * In 12.3 the ER_CANT_CREATE_TABLE reporting logic has been moved from ha_create_table() into its new helper ha_create_table_from_share(). The HA_CREATE_TMP_ALTER branch is added in the helper so that both the main shadow table and any high-level (vector) index sub-tables go through the same path. Vector index sub-tables use a zero- initialised index_cinfo, so HA_CREATE_TMP_ALTER is never set for them and they still get the plain "Can't create table" message. Review follow-ups (Round 6, @midenok): * Moved the test content from suite/innodb/t/alter_table_error_message.test into main/alter_table.test (no duplicate --include directives); dropped the extra .test/.result files accordingly. * Added the standard "End of 12.3 tests" section marker at the end. * Removed the unnecessary DROP TABLE IF EXISTS cluster_t; in the cleanup step (the test is deterministic, the table no longer exists at that point). Review follow-ups (Round 7, test re-record + errmsg formatting): * Fixed ER_CANT_CREATE_TMP_ALTER_TABLE template to use the exact ER_CANT_CREATE_TABLE format specifiers, so error frames produced by my_error() are byte-identical to the existing error (only the fixed prefix changes). * Re-recorded innodb.innodb-index.result and innodb.foreign_key.result which now observe the new "Can't create temporary table for ALTER TABLE" message in ALTER TABLE error paths that used to print "Can't create table". * No code changes outside of the intentional message swap; all other .result files remain untouched. Review follow-ups (Round 8, additional test re-record from CI): * Re-recorded innodb.table_flags.result for the MDEV-38079 scenario (ALTER TABLE t1 PAGE_COMPRESSED=1, ALGORITHM=COPY under INNODB_DEFAULT_ROW_FORMAT=REDUNDANT) -- this ALTER path now reports the new "Can't create temporary table for ALTER TABLE" message. Verified on all 10 page size / checksum combinations (4k/8k/16k/ 32k/64k x strict_crc32/strict_full_crc32). * Re-recorded parts.engine_defined_part_attributes.result for the ALTER TABLE `t4` PARTITION BY RANGE(...) ENCRYPTED path. The four remaining "Can't create table" occurrences in the same file refer to pure CREATE TABLE failures and correctly retain the original message. * No code or test-case logic changes; only .result re-recordings. Signed-off-by: a18792721831 <[email protected]> CLA: a18792721831 <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Andrey Sokolov
sokolov.andrey.yurevich@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-39487: mroonga: remove flags local variable from grn_io_open This variable is unused. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thirunarayanan Balathandayuthapani
thiru@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| - Addressing the review comments | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Merge branch 'bb-10.11-release' into bb-11.4-release | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
find pcre2.h in a non-default location for macos builder |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-37949: Implement innodb_log_archive The InnoDB write-ahead log file in the old innodb_log_archive=OFF format is named ib_logfile0, pre-allocated to innodb_log_file_size and written as a ring buffer. This is good for write performance and space management, but unsuitable for arbitrary point-in-time recovery or for facilitating efficient incremental backup. innodb_log_archive=ON: A new format where InnoDB will create and preallocate files ib_%016x.log, instead of writing a circular file ib_logfile0. Each file will be pre-allocated to innodb_log_file_size. Once a log fills up, we will create and pre-allocate another log file, to which log records will be written. Upon the completion of the first log checkpoint in a recently created log file, the old log file will be marked read-only, signaling that there will be no further writes to that file, and that the file may safely be moved to long-term storage. The file name includes the log sequence number (LSN) at file offset 12288 (log_t::START_OFFSET). Each checkpoint is identified by storing a 64-bit big-endian offset into an optional sequence of FILE_MODIFY records followed by a FILE_CHECKPOINT record, between 12288 and the end of the file. The innodb_encrypt_log format is identified by storing the encryption information at the start of the log file. The first 64-bit value will be 1, which is an invalid checkpoint offset. Each innodb_log_archive=ON log must use the same encryption parameters. Changing innodb_encrypt_log or related parameters is only possible by setting innodb_log_archive=OFF and restarting the server, which will permanently lose the history of the archived log. The maximum number of log checkpoints that the innodb_log_archive=ON file header can represent is limited to 12288/8=1536 when using innodb_encrypt_log=OFF. If we run out of slots in a log file, each subsequently completed checkpoint in that log file will overwrite the last slot in the checkpoint header, until we switch to the next log. innodb_log_recovery_start: The checkpoint LSN to start recovery from. This will be useful when recovering from an archived log. This is useful for restoring an incremental backup (applying InnoDB log files that were copied since the previous restore). innodb_log_recovery_target: The requested LSN to end recovery at. When this is set, all persistent InnoDB tables will be read-only, and no writes to the log are allowed. The intended purpose of this setting is to prepare an incremental backup, as well as to allow data retrieval as of a particular logical point of time. Setting innodb_log_recovery_target>0 is much like setting innodb_read_only=ON, with the exception that the data files may be written to by crash recovery, and locking reads will conflict with any incomplete transactions as necessary, and all transaction isolation levels will work normally (not hard-wired to READ UNCOMMITTED). srv_read_only_mode: When this is set (innodb_read_only=ON), also recv_sys.rpo (innodb_log_recovery_target) will be set to the current LSN. This ensures that it will suffice to check only one of these variables when blocking writes to persistent tables. The status variable innodb_lsn_archived will reflect the LSN since when a complete InnoDB log archive is available. Its initial value will be that of the new parameter innodb_log_archive_start. If that variable is 0 (the default), the innodb_lsn_archived will be recovered from the available log files. If innodb_log_archive=OFF, innodb_lsn_archived will be adjusted to the latest checkpoint every time a log checkpoint is executed. If innodb_log_archive=ON, the value should not change. SET GLOBAL innodb_log_archive=!@@GLOBAL.innodb_log_archive will take effect as soon as possible, possibly after a log checkpoint has been completed. The log file will be renamed between ib_logfile0 and ib_%016x.log as appropriate. When innodb_log_archive=ON, the setting SET GLOBAL innodb_log_file_size will affect subsequently created log files when the file that is being currently written is running out. If we are switching log files exactly at the same time, then a somewhat misleading error message "innodb_log_file_size change is already in progress" will be issued. no_checkpoint_prepare.inc: A new file, to prepare for subsequent inclusion of no_checkpoint_end.inc. We will invoke the server to parse the log and to determine the latest checkpoint. All --suite=encryption tests that use innodb_encrypt_log will be skipped for innodb_log_archive=ON, because enabling or disabling encryption on the log is not possible without temporarily setting innodb_log_archive=OFF and restarting the server. The idea is to add the following arguments to an invocation of mysql-test/mtr: --mysqld=--loose-innodb-log-archive \ --mysqld=--loose-innodb-log-recovery-start=12288 \ --mysqld=--loose-innodb-log-file-mmap=OFF \ --skip-test=mariabackup Alternatively, specify --mysqld=--loose-innodb-log-file-mmap=ON to cover both code paths. The mariabackup test suite must be skipped when using the innodb_log_archive=ON format, because mariadb-backup will only support the old ib_logfile0 format (innodb_log_archive=OFF). A number of tests would fail when the parameter innodb_log_recovery_start=12288 is present, which is forcing recovery to start from the beginning of the history (the database creation). The affected tests have been adjusted with explicit --innodb-log-recovery-start=0 to override that: (0) Some injected corruption may be "healed" by replaying the log from the beginning. Some tests expect an empty buffer pool after a restart, with no page I/O due to crash recovery. (1) Any test that sets innodb_read_only=ON would fail with an error message that the setting prevents crash recovery, unless innodb_log_recovery_start=0. (2) Any test that changes innodb_undo_tablespaces would fail in crash recovery, because crash recovery assumes that the undo tablespace ID that is available from the undo* files corresponds with the start of the log. This is an unforunate design bug which we cannot fix easily. log_sys.first_lsn: The start of the current log file, to be consulted in log_t::write_checkpoint() when renaming files. log_sys.archived_lsn: New field: The value of innodb_lsn_archived. log_sys.end_lsn: New field: The log_sys.get_lsn() when the latest checkpoint was initiated. That is, the start LSN of a possibly empty sequence of FILE_MODIFY records followed by FILE_CHECKPOINT. log_sys.resize_target: The value of innodb_log_file_size that will be used for creating the next archive log file once the current file (of log_sys.file_size) fills up. log_sys.archive: New field: The value of innodb_log_archive. log_sys.next_checkpoint_no: Widen to uint16_t. There may be up to 12288/8=1536 checkpoints in the header. log_sys.log: If innodb_log_archive=ON, this file handle will be kept open also in the PMEM code path. log_sys.resize_log: If innodb_log_archive=ON, we may have two log files open both during normal operation and when parsing the log. This will store the other handle (old or new file). log_sys.resize_buf: In the memory-mapped code path, this will point to the file resize_log when innodb_log_archive=ON. recv_sys.log_archive: All innodb_log_archive=ON files that will be considered in recovery. recv_sys.was_archive: A flag indicating that an innodb_log_archive=ON file is in innodb_log_archive=OFF format. log_sys.is_pmem, log_t::is_mmap_writeable(): A new predicate. If is_mmap_writeable(), we assert and guarantee buf_size == capacity(). log_t::checkpoint_age_max(): Return the maximum checkpoint age. When innodb_log_archive=ON the limit is innodb_log_file_size larger. log_t::archive_new_write(): Create and allocate a new log file, and write the outstanding data to both the current and the new file, or only to the new file, until write_checkpoint() completes the first checkpoint in the new file. If the log file would overflow, increase innodb_log_file_size to the next multiple of 4096 bytes. log_t::archive_create(bool): Create and allocate an archive log file. On POSIX, ensure that we get the exact correct file size by invoking ftruncate() on POSIX, because os_file_set_size() would only truncate files on Microsoft Windows. Remember the file handle of the current log in resize_log, so that write_checkpoint() will be able to make it read-only. log_t::archived_mmap_switch_prepare(): Invoke archive_create() unless another thread already did it. log_t::archived_mmap_switch_complete(): Switch to the buffer that was created in log_t::archived_mmap_switch_prepare(). Remember the old log_sys.buf in log_sys.checkpoint_buf, to be unmapped in log_t::write_checkpoint(). buf_flush_archive_create(): Create a spare archive log file if needed. buf_flush_page_cleaner(): Invoke buf_flush_archive_create() about one second after a checkpoint where we switched archived log files. This allows any writer threads to resume activity after we completed the previous checkpoint. log_t::write_checkpoint(): Allow an old checkpoint to be completed in the old log file even after a new one has been created. If we are writing the first checkpoint in a new log file, we will mark the old log file read-only. We will also update log_sys.first_lsn unless it was already updated in ARCHIVED_MMAP code path. In that code path, there is the special case where log_sys.resize_buf == nullptr and log_sys.checkpoint_buf points to log_sys.resize_log (the old log file that is about to be made read-only). In this case, log_sys.first_lsn will already point to the start of the current log_sys.log, even though the switch has not been fully completed yet. Try to preallocate the next archive file if needed, with the goal that when we need the file it will already be ready for use. log_t::write_checkpoint(), log_checkpoint_low(), log_checkpoint(), buf_flush_sync_for_checkpoint(): Return the LSN on which to invoke archive_create(), or 0 if no spare archive log file needs to be created. log_t::header_rewrite(my_bool): Rewrite the log file header before or after renaming the log file, and write a message about the change, so that there will be a chance to recover in case the server is being killed during this operation. The recovery of the last ib_%016%.log does tolerate also the ib_logfile0 format. log_t::set_archive(my_bool,THD): Implement SET GLOBAL innodb_log_archive. An error will be returned if non-archived SET GLOBAL innodb_log_file_size (log file resizing) is in progress. Wait for checkpoint if necessary. The current log file will be renamed to either ib_logfile0 or ib_%016x.log, as appropriate. In SET GLOBAL innodb_log_archive=OFF, we trigger a write-ahead of the log if necessary, to prevent overrun. log_t::archive_rename(): Rename an archived log to ib_logfile0 on recovery in case there had been a crash during set_archive(). log_t::archive_set_size(): A new function, to ensure that log_sys.resize_target is set on startup. log_checkpoint_low(): Do not prevent a checkpoint at the start of a file. We want the first innodb_log_archive=ON file to start with a checkpoint. log_t::create(lsn_t): Initialize last_checkpoint_lsn. Initialize the log header as specified by log_sys.archive (innodb_log_archive). log_write_buf(): Add the parameter max_length, the file wrap limit. log_write_up_to(), mtr_t::commit_log_release<bool mmap=true>(): If we are switching log files, invoke buf_flush_ahead(lsn, false) to ensure that a log checkpoint will be completed in the new file. mtr_t::finish_writer(): Specialize for innodb_log_archive=ON. mtr_t::commit_file(): Ensure that log archive rotation will complete. log_t::append_prepare<log_t::ARCHIVED_MMAP>(): Special case. log_t::append_prepare(): Remove the call to set_check_for_checkpoint() that was duplicating logic in log_close(). log_t::get_path(): Get the name of the current log file. log_t::get_circular_path(size_t): Get the path name of a circular file. Replaces get_log_file_path(). log_t::get_archive_path(lsn_t): Return a name of an archived log file. log_t::append_archive_name(): Append the archive log file name to a path string. log_t::checkpoint_margin(): Replaces log_checkpoint_margin(). If a new archived log file has been created, wait for the first checkpoint in that file. Revise the formula when log_sys.need_checkpoint needs to hold. When innodb_log_archive=ON, we can allow the checkpoint to be older by innodb_log_file_size. log_close(): Relax the log_sys.need_checkpoint condition for innodb_log_archive=ON. srv_log_rebuild_if_needed(): Never rebuild if innodb_log_archive=ON. The setting innodb_log_file_size will affect the creation of subsequent log files. The parameter innodb_encrypt_log cannot be changed while the log is in the innodb_log_archive=ON format. log_t::attach(), log_mmap(): Add the parameter log_access, to distinguish memory-mapped or read-only access. log_t::attach(): When disabling innodb_log_file_mmap, read checkpoint_buf from the last innodb_log_archive=ON file. log_t::clear_mmap(): Clear the tail of the checkpoint buffer if is_mmap_writeable(). log_t::set_recovered(): Invoke clear_mmap(), and restore the log buffer to the correct position. recv_sys_t::apply(): Let log_t::clear_mmap() enable log writes. log_file_is_zero(): Check if a log file starts with NUL bytes (is a preallocated file). recv_sys_t::find_checkpoint(): Find and remember the checkpoint position in the last file when innodb_log_recovery_start points to an older file. When innodb_log_file_mmap=OFF, restore log_sys.checkpoint_buf from the latest log file. If the last archive log file is actually in innodb_log_archive=OFF format despite being named ib_%016.log, try to recover it in that format. If the circular ib_logfile0 is missing, determine the oldest archived log file with contiguous LSN. If innodb_log_archive=ON, refuse to start if ib_logfile0 exists. Open non-last non-preallocated archived log file in read/write mode. recv_sys_t::find_checkpoint_archived(): Validate each checkpoint in the current file header, and by default aim to recover from the last valid one. Terminate the search if the last validated checkpoint spanned two files. If innodb_log_recovery_start has been specified, attempt to validate it even if there is no such information stored in the checkpoint header. log_parse_file(): Do not invoke fil_name_process() during recv_sys_t::find_checkpoint_archived(), when we tolerate FILE_MODIFY records while looking for a FILE_CHECKPOINT record. recv_scan_log(): Invoke log_t::archived_switch_recovery() upon reaching the end of the current archived log file. log_t::archived_switch_recovery_prepare(): Make use of recv_sys.log_archive and open all but the last file read-only. log_t::archived_switch_recovery(): Switch files in the pread() code path. log_t::archived_mmap_switch_recovery_complete(): Switch files in the memory-mapped code path. recv_warp: A pointer wrapper for memory-mapped parsing that spans two archive log files. recv_sys_t::parse_mmap(): Use recv_warp for innodb_log_archive=ON. recv_sys_t::parse(): Tweak some logic for innodb_log_archive=ON. log_t::set_recovered_checkpoint(): Set the checkpoint on recovery. Updates also the end_lsn. log_t::set_recovered_lsn(): Also update flush_lock and write_lock, to ensure that log_write_up_to() will be a no-op. Invoke unstash_archive_file() in case there was a garbage (pre-allocated) file at the end which was not parsed at all. log_t::persist(): Even if the flushed_to_disk_lsn does not change, we may want to reset the write_lsn_offset. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38802 MariaDB server start emits error but continues anyway: Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist it's not an error, as the server continues anyway |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Oleksandr Byelkin
sanja@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Merge branch 'bb-10.6-release' into 10.11 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
PranavKTiwari
pranav.tiwari@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| used updatble flag. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MDEV-39266 Stack Overflow via alloca() in Privilege Table JSON Parser | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-32745 followup for 7828fb475b0 * add a test for new --cat_file feature * fix off-by-one error in --cat_file lines limit * fix broken INCLUDE_DIRECTORIES in extra/ * fix typos * remove mtr-specific workaround from the tool, solve it in the test * remove dead code * fixed memory leak in mariadb-migration-config-file [client-server] section * fixed memory leak in mariadbd --character_set_server=xxx (execution time for main.mariadb-migrate-config-file went down from 150s to 2s) * moved tool header to the tool dir * avoid tripple-initialization of plugins * disable tool builds in ASAN builds (because of RocksDB) * compilation failure on x86 (signedness) |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||