Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Sergei Golubchik
MDEV-39288 SHOW CREATE ROUTINE does not apply to roles

Reported by Aisle Research
Sergei Golubchik
MDEV-38802 MariaDB server start emits error but continues anyway: Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist

it's not an error, as the server continues anyway
Marko Mäkelä
MDEV-37949: Implement innodb_log_archive

The InnoDB write-ahead log file in the old innodb_log_archive=OFF
format is named ib_logfile0, pre-allocated to innodb_log_file_size and
written as a ring buffer. This is good for write performance and space
management, but unsuitable for arbitrary point-in-time recovery or for
facilitating efficient incremental backup.

innodb_log_archive=ON: A new format where InnoDB will create and
preallocate files ib_%016x.log, instead of writing a circular file
ib_logfile0. Each file will be pre-allocated to innodb_log_file_size.
Once a log fills up, we will create and pre-allocate another log file,
to which log records will be written.  Upon the completion of the first
log checkpoint in a recently created log file, the old log file will
be marked read-only, signaling that there will be no further writes to
that file, and that the file may safely be moved to long-term storage.

The file name includes the log sequence number (LSN) at file offset
12288 (log_t::START_OFFSET). Each checkpoint is identified by storing
a 64-bit big-endian offset into an optional sequence of FILE_MODIFY
records followed by a FILE_CHECKPOINT record, between 12288 and the
end of the file.

The innodb_encrypt_log format is identified by storing the encryption
information at the start of the log file. The first 64-bit value will
be 1, which is an invalid checkpoint offset. Each
innodb_log_archive=ON log must use the same encryption parameters.
Changing innodb_encrypt_log or related parameters is only possible by
setting innodb_log_archive=OFF and restarting the server, which will
permanently lose the history of the archived log.

The maximum number of log checkpoints that the innodb_log_archive=ON
file header can represent is limited to 12288/8=1536 when using
innodb_encrypt_log=OFF. If we run out of slots in a log file, each
subsequently completed checkpoint in that log file will overwrite the
last slot in the checkpoint header, until we switch to the next log.

innodb_log_recovery_start: The checkpoint LSN to start recovery from.
This will be useful when recovering from an archived log. This is useful
for restoring an incremental backup (applying InnoDB log files that were
copied since the previous restore).

innodb_log_recovery_target: The requested LSN to end recovery at.
When this is set, all persistent InnoDB tables will be read-only, and
no writes to the log are allowed. The intended purpose of this setting
is to prepare an incremental backup, as well as to allow data
retrieval as of a particular logical point of time.

Setting innodb_log_recovery_target>0 is much like setting
innodb_read_only=ON, with the exception that the data files may be
written to by crash recovery, and locking reads will conflict with any
incomplete transactions as necessary, and all transaction isolation
levels will work normally (not hard-wired to READ UNCOMMITTED).

srv_read_only_mode: When this is set (innodb_read_only=ON), also
recv_sys.rpo (innodb_log_recovery_target) will be set to the current LSN.
This ensures that it will suffice to check only one of these variables
when blocking writes to persistent tables.

The status variable innodb_lsn_archived will reflect the LSN
since when a complete InnoDB log archive is available. Its initial
value will be that of the new parameter innodb_log_archive_start.
If that variable is 0 (the default), the innodb_lsn_archived will
be recovered from the available log files. If innodb_log_archive=OFF,
innodb_lsn_archived will be adjusted to the latest checkpoint every
time a log checkpoint is executed. If innodb_log_archive=ON, the value
should not change.

SET GLOBAL innodb_log_archive=!@@GLOBAL.innodb_log_archive will take
effect as soon as possible, possibly after a log checkpoint has been
completed. The log file will be renamed between ib_logfile0 and
ib_%016x.log as appropriate.

When innodb_log_archive=ON, the setting SET GLOBAL innodb_log_file_size
will affect subsequently created log files when the file that is being
currently written is running out. If we are switching log files exactly
at the same time, then a somewhat misleading error message
"innodb_log_file_size change is already in progress" will be issued.

no_checkpoint_prepare.inc: A new file, to prepare for subsequent
inclusion of no_checkpoint_end.inc. We will invoke the server to
parse the log and to determine the latest checkpoint.

All --suite=encryption tests that use innodb_encrypt_log
will be skipped for innodb_log_archive=ON, because enabling
or disabling encryption on the log is not possible without
temporarily setting innodb_log_archive=OFF and restarting
the server. The idea is to add the following arguments to an
invocation of mysql-test/mtr:

--mysqld=--loose-innodb-log-archive \
--mysqld=--loose-innodb-log-recovery-start=12288 \
--mysqld=--loose-innodb-log-file-mmap=OFF \
--skip-test=mariabackup

Alternatively, specify --mysqld=--loose-innodb-log-file-mmap=ON
to cover both code paths.

The mariabackup test suite must be skipped when using the
innodb_log_archive=ON format, because mariadb-backup will only
support the old ib_logfile0 format (innodb_log_archive=OFF).

A number of tests would fail when the parameter
innodb_log_recovery_start=12288 is present, which is forcing
recovery to start from the beginning of the history
(the database creation). The affected tests have been adjusted with
explicit --innodb-log-recovery-start=0 to override that:

(0) Some injected corruption may be "healed" by replaying the log
from the beginning. Some tests expect an empty buffer pool after
a restart, with no page I/O due to crash recovery.
(1) Any test that sets innodb_read_only=ON would fail with an error
message that the setting prevents crash recovery, unless
innodb_log_recovery_start=0.
(2) Any test that changes innodb_undo_tablespaces would fail in crash
recovery, because crash recovery assumes that the undo tablespace ID
that is available from the undo* files corresponds with the start of
the log. This is an unforunate design bug which we cannot fix easily.

log_sys.first_lsn: The start of the current log file, to be consulted
in log_t::write_checkpoint() when renaming files.

log_sys.archived_lsn: New field: The value of innodb_lsn_archived.

log_sys.end_lsn: New field: The log_sys.get_lsn() when the latest
checkpoint was initiated. That is, the start LSN of a possibly empty
sequence of FILE_MODIFY records followed by FILE_CHECKPOINT.

log_sys.resize_target: The value of innodb_log_file_size that will be
used for creating the next archive log file once the current file (of
log_sys.file_size) fills up.

log_sys.archive: New field: The value of innodb_log_archive.

log_sys.next_checkpoint_no: Widen to uint16_t. There may be up to
12288/8=1536 checkpoints in the header.

log_sys.log: If innodb_log_archive=ON, this file handle will be kept
open also in the PMEM code path.

log_sys.resize_log: If innodb_log_archive=ON, we may have two log
files open both during normal operation and when parsing the log. This
will store the other handle (old or new file).

log_sys.resize_buf: In the memory-mapped code path, this will point
to the file resize_log when innodb_log_archive=ON.

recv_sys.log_archive: All innodb_log_archive=ON files that will be
considered in recovery.

recv_sys.was_archive: A flag indicating that an innodb_log_archive=ON
file is in innodb_log_archive=OFF format.

log_sys.is_pmem, log_t::is_mmap_writeable(): A new predicate.
If is_mmap_writeable(), we assert and guarantee buf_size == capacity().

log_t::archive_new_write(): Create and allocate a new log file, and
write the outstanding data to both the current and the new file, or
only to the new file, until write_checkpoint() completes the first
checkpoint in the new file.

log_t::archived_mmap_switch_prepare(): Create and memory-map a new log
file, and update file_size to resize_target. Remember the file handle
of the current log in resize_log, so that write_checkpoint() will be
able to make it read-only.

log_t::archived_mmap_switch_complete(): Switch to the buffer that was
created in archived_mmap_switch_prepare().

log_t::archive_create(bool): Create and allocate an archive log file.
On POSIX, ensure that we get the exact correct file size by invoking
ftruncate() on POSIX, because os_file_set_size() would only
truncate files on Microsoft Windows.

buf_flush_archive_create(): Create a spare archive log file if needed.

buf_flush_page_cleaner(): Invoke buf_flush_archive_create() about
one second after a checkpoint where we switched archived log files.
This allows any writer threads to resume activity after we completed
the previous checkpoint.

log_t::write_checkpoint(): Allow an old checkpoint to be completed in
the old log file even after a new one has been created. If we are
writing the first checkpoint in a new log file, we will mark the old
log file read-only. We will also update log_sys.first_lsn unless it
was already updated in ARCHIVED_MMAP code path. In that code path,
there is the special case where log_sys.resize_buf == nullptr and
log_sys.checkpoint_buf points to log_sys.resize_log (the old log file
that is about to be made read-only). In this case, log_sys.first_lsn
will already point to the start of the current log_sys.log, even
though the switch has not been fully completed yet.
Try to preallocate the next archive file if needed, with the goal that
when we need the file it will already be ready for use.

log_t::write_checkpoint(), log_checkpoint_low(), log_checkpoint(),
buf_flush_sync_for_checkpoint(): Return the LSN on which to invoke
archive_create(), or 0 if no spare archive log file needs to be created.

log_t::header_rewrite(my_bool): Rewrite the log file header before or
after renaming the log file, and write a message about the change,
so that there will be a chance to recover in case the server is being
killed during this operation.  The recovery of the last ib_%016%.log
does tolerate also the ib_logfile0 format.

log_t::set_archive(my_bool,THD): Implement SET GLOBAL innodb_log_archive.
An error will be returned if non-archived SET GLOBAL innodb_log_file_size
(log file resizing) is in progress. Wait for checkpoint if necessary.
The current log file will be renamed to either ib_logfile0 or
ib_%016x.log, as appropriate. In SET GLOBAL innodb_log_archive=OFF,
we trigger a write-ahead of the log if necessary, to prevent overrun.

log_t::archive_rename(): Rename an archived log to ib_logfile0 on recovery
in case there had been a crash during set_archive().

log_t::archive_set_size(): A new function, to ensure that
log_sys.resize_target is set on startup.

log_checkpoint_low(): Do not prevent a checkpoint at the start of a file.
We want the first innodb_log_archive=ON file to start with a checkpoint.

log_t::create(lsn_t): Initialize last_checkpoint_lsn. Initialize the
log header as specified by log_sys.archive (innodb_log_archive).

log_write_buf(): Add the parameter max_length, the file wrap limit.

log_write_up_to(), mtr_t::commit_log_release<bool mmap=true>():
If we are switching log files, invoke buf_flush_ahead(lsn, false)
to ensure that a log checkpoint will be completed in the new file.

mtr_t::finish_writer(): Specialize for innodb_log_archive=ON.

mtr_t::commit_file(): Ensure that log archive rotation will complete.

log_t::append_prepare<log_t::ARCHIVED_MMAP>(): Special case.

log_t::append_prepare(): Remove the call to set_check_for_checkpoint()
that was duplicating logic in log_close().

log_t::get_path(): Get the name of the current log file.

log_t::get_circular_path(size_t): Get the path name of a circular file.
Replaces get_log_file_path().

log_t::get_archive_path(lsn_t): Return a name of an archived log file.

log_t::append_archive_name(): Append the archive log file name
to a path string.

log_t::checkpoint_margin(): Replaces log_checkpoint_margin().
If a new archived log file has been created, wait for the
first checkpoint in that file. Revise the formula when
log_sys.need_checkpoint needs to hold. When innodb_log_archive=ON,
we can allow the checkpoint to be older by innodb_log_file_size.

log_close(): Relax the log_sys.need_checkpoint condition
for innodb_log_archive=ON.

srv_log_rebuild_if_needed(): Never rebuild if innodb_log_archive=ON.
The setting innodb_log_file_size will affect the creation of
subsequent log files. The parameter innodb_encrypt_log cannot be
changed while the log is in the innodb_log_archive=ON format.

log_t::attach(), log_mmap(): Add the parameter log_access,
to distinguish memory-mapped or read-only access.

log_t::attach(): When disabling innodb_log_file_mmap, read
checkpoint_buf from the last innodb_log_archive=ON file.

log_t::clear_mmap(): Clear the tail of the checkpoint buffer
if is_mmap_writeable().

log_t::set_recovered(): Invoke clear_mmap(), and restore the
log buffer to the correct position.

recv_sys_t::apply(): Let log_t::clear_mmap() enable log writes.

log_file_is_zero(): Check if a log file starts with NUL bytes
(is a preallocated file).

recv_sys_t::find_checkpoint(): Find and remember the checkpoint position
in the last file when innodb_log_recovery_start points to an older file.
When innodb_log_file_mmap=OFF, restore log_sys.checkpoint_buf from
the latest log file. If the last archive log file is actually
in innodb_log_archive=OFF format despite being named ib_%016.log,
try to recover it in that format. If the circular ib_logfile0 is missing,
determine the oldest archived log file with contiguous LSN.
If innodb_log_archive=ON, refuse to start if ib_logfile0 exists.
Open non-last non-preallocated archived log file in read/write mode.

recv_sys_t::find_checkpoint_archived(): Validate each checkpoint in
the current file header, and by default aim to recover from the last
valid one. Terminate the search if the last validated checkpoint
spanned two files. If innodb_log_recovery_start has been specified,
attempt to validate it even if there is no such information stored
in the checkpoint header.

log_parse_file(): Do not invoke fil_name_process() during
recv_sys_t::find_checkpoint_archived(), when we tolerate FILE_MODIFY
records while looking for a FILE_CHECKPOINT record.

recv_scan_log(): Invoke log_t::archived_switch_recovery() upon
reaching the end of the current archived log file.

log_t::archived_switch_recovery_prepare(): Make use of
recv_sys.log_archive and open all but the last file read-only.

log_t::archived_switch_recovery(): Switch files in the pread() code path.

log_t::archived_mmap_switch_recovery_complete(): Switch files in the
memory-mapped code path.

recv_warp: A pointer wrapper for memory-mapped parsing that spans two
archive log files.

recv_sys_t::parse_mmap(): Use recv_warp for innodb_log_archive=ON.

recv_sys_t::parse(): Tweak some logic for innodb_log_archive=ON.

log_t::set_recovered_checkpoint(): Set the checkpoint on recovery.
Updates also the end_lsn.

log_t::set_recovered_lsn(): Also update flush_lock and write_lock,
to ensure that log_write_up_to() will be a no-op. Invoke
unstash_archive_file() in case there was a garbage (pre-allocated)
file at the end which was not parsed at all.

log_t::persist(): Even if the flushed_to_disk_lsn does not change,
we may want to reset the write_lsn_offset.

Avoid setting log_sys.need_checkpoint unnecessarily (MDEV-39162).
Marko Mäkelä
squash! 7599bed66d0abcefc07ee2175bfac1e6fdac9abb

log_close(), log_t::checkpoint_margin(): Revise the formula when
log_sys.need_checkpoint needs to hold. When innodb_log_archive=ON,
we can allow the checkpoint to be older by innodb_log_file_size.

log_t::append_prepare(): Remove the call to set_check_for_checkpoint()
that was duplicating logic in log_close().
Lawrin Novitsky
Merge branch 'cpp-1.0' into cpp-1.1
Oleg Smirnov
MDEV-39304 QB_Name hint with path is silently ignored inside view definition

This commit adds an explicit warning message indicating
that QB_NAME hints with path are not supported inside view definitions
Oleg Smirnov
MDEV-38045: Implement implicit query block names for optimizer hints

This patch implements support for implicit query block (QB) names in
optimizer hints, allowing hints to reference query blocks and tables
within derived tables, views and CTEs without requiring explicit
QB_NAME hints.

Examples.
-- Addressing a table inside a derived table using implicit QB name
select /*+ no_index(t1@dt) */ *
  from (select * from t1 where a > 10) as DT;
-- this is an equivalent to:
select /*+ no_index(t1@dt) */ * from
  (select /*+ qb_name(dt)*/ * from t1 where a > 10) as DT;
-- Addressing a query block corresponding to the derived table
select /*+ no_bnl(@dt) */ *
  from (select * from t1, t2 where t.1.a > t2.a) as DT;

-- View
create view v1 as select * from t1 where a > 10 and b > 100;
-- referencing a table inside a view by implicit QB name:
select /*+ index_merge(t1@v1 idx_a, idx_b) */ *
  from v1, t2 where v1.a = t2.a;
-- equivalent to:
create view v1 as select /*+ qb_name(qb_v1) */ *
  from t1 where a > 10 and b > 100;
select /*+ index_merge(t1@qb_v1 idx_a, idx_b) */ *
  from v1, t2 where v1.a = t2.a;

-- CTE
with aless100 as (select a from t1 where b <100)
  select /*+ index(t1@aless100) */ * from aless100;
-- equivalent to:
with aless100 as (select /*+ qb_name(aless100) */ a from t1 where b <100)
  select /*+ index(t1@aless100) */ * from aless100;

Limitations:
  - Only SELECT statements support implicit QB names. DML operations
    (UPDATE, DELETE, INSERT) only support explicit QB names
Rucha Deodhar
MDEV-39179: Incorrect NULL handling in UPDATE ... RETURNING result

Analysis:
OLD_VALUE() swapped only field->ptr, leaving null_ptr pointing to the
current record. This caused incorrect NULL results.

Fix:
Store null_ptr_old for record[1] and swap it together with ptr to
preserve correct NULL semantics.
Lawrin Novitsky
Introduced UBSAN.supp to supress UBSAN issues beyond our control

(on C/C side). They are quite the same as in C/ODBC
  • ccpp-linux-amd64-msan: build linux-connector_cpp failed -  stdio
Marko Mäkelä
MDEV-37949: Implement innodb_log_archive

The InnoDB write-ahead log file in the old innodb_log_archive=OFF
format is named ib_logfile0, pre-allocated to innodb_log_file_size and
written as a ring buffer. This is good for write performance and space
management, but unsuitable for arbitrary point-in-time recovery or for
facilitating efficient incremental backup.

innodb_log_archive=ON: A new format where InnoDB will create and
preallocate files ib_%016x.log, instead of writing a circular file
ib_logfile0. Each file will be pre-allocated to innodb_log_file_size.
Once a log fills up, we will create and pre-allocate another log file,
to which log records will be written.  Upon the completion of the first
log checkpoint in a recently created log file, the old log file will
be marked read-only, signaling that there will be no further writes to
that file, and that the file may safely be moved to long-term storage.

The file name includes the log sequence number (LSN) at file offset
12288 (log_t::START_OFFSET). Each checkpoint is identified by storing
a 64-bit big-endian offset into an optional sequence of FILE_MODIFY
records followed by a FILE_CHECKPOINT record, between 12288 and the
end of the file.

The innodb_encrypt_log format is identified by storing the encryption
information at the start of the log file. The first 64-bit value will
be 1, which is an invalid checkpoint offset. Each
innodb_log_archive=ON log must use the same encryption parameters.
Changing innodb_encrypt_log or related parameters is only possible by
setting innodb_log_archive=OFF and restarting the server, which will
permanently lose the history of the archived log.

The maximum number of log checkpoints that the innodb_log_archive=ON
file header can represent is limited to 12288/8=1536 when using
innodb_encrypt_log=OFF. If we run out of slots in a log file, each
subsequently completed checkpoint in that log file will overwrite the
last slot in the checkpoint header, until we switch to the next log.

innodb_log_recovery_start: The checkpoint LSN to start recovery from.
This will be useful when recovering from an archived log. This is useful
for restoring an incremental backup (applying InnoDB log files that were
copied since the previous restore).

innodb_log_recovery_target: The requested LSN to end recovery at.
When this is set, all persistent InnoDB tables will be read-only, and
no writes to the log are allowed. The intended purpose of this setting
is to prepare an incremental backup, as well as to allow data
retrieval as of a particular logical point of time.

Setting innodb_log_recovery_target>0 is much like setting
innodb_read_only=ON, with the exception that the data files may be
written to by crash recovery, and locking reads will conflict with any
incomplete transactions as necessary, and all transaction isolation
levels will work normally (not hard-wired to READ UNCOMMITTED).

srv_read_only_mode: When this is set (innodb_read_only=ON), also
recv_sys.rpo (innodb_log_recovery_target) will be set to the current LSN.
This ensures that it will suffice to check only one of these variables
when blocking writes to persistent tables.

The status variable innodb_lsn_archived will reflect the LSN
since when a complete InnoDB log archive is available. Its initial
value will be that of the new parameter innodb_log_archive_start.
If that variable is 0 (the default), the innodb_lsn_archived will
be recovered from the available log files. If innodb_log_archive=OFF,
innodb_lsn_archived will be adjusted to the latest checkpoint every
time a log checkpoint is executed. If innodb_log_archive=ON, the value
should not change.

SET GLOBAL innodb_log_archive=!@@GLOBAL.innodb_log_archive will take
effect as soon as possible, possibly after a log checkpoint has been
completed. The log file will be renamed between ib_logfile0 and
ib_%016x.log as appropriate.

When innodb_log_archive=ON, the setting SET GLOBAL innodb_log_file_size
will affect subsequently created log files when the file that is being
currently written is running out. If we are switching log files exactly
at the same time, then a somewhat misleading error message
"innodb_log_file_size change is already in progress" will be issued.

no_checkpoint_prepare.inc: A new file, to prepare for subsequent
inclusion of no_checkpoint_end.inc. We will invoke the server to
parse the log and to determine the latest checkpoint.

All --suite=encryption tests that use innodb_encrypt_log
will be skipped for innodb_log_archive=ON, because enabling
or disabling encryption on the log is not possible without
temporarily setting innodb_log_archive=OFF and restarting
the server. The idea is to add the following arguments to an
invocation of mysql-test/mtr:

--mysqld=--loose-innodb-log-archive \
--mysqld=--loose-innodb-log-recovery-start=12288 \
--mysqld=--loose-innodb-log-file-mmap=OFF \
--skip-test=mariabackup

Alternatively, specify --mysqld=--loose-innodb-log-file-mmap=ON
to cover both code paths.

The mariabackup test suite must be skipped when using the
innodb_log_archive=ON format, because mariadb-backup will only
support the old ib_logfile0 format (innodb_log_archive=OFF).

A number of tests would fail when the parameter
innodb_log_recovery_start=12288 is present, which is forcing
recovery to start from the beginning of the history
(the database creation). The affected tests have been adjusted with
explicit --innodb-log-recovery-start=0 to override that:

(0) Some injected corruption may be "healed" by replaying the log
from the beginning. Some tests expect an empty buffer pool after
a restart, with no page I/O due to crash recovery.
(1) Any test that sets innodb_read_only=ON would fail with an error
message that the setting prevents crash recovery, unless
innodb_log_recovery_start=0.
(2) Any test that changes innodb_undo_tablespaces would fail in crash
recovery, because crash recovery assumes that the undo tablespace ID
that is available from the undo* files corresponds with the start of
the log. This is an unforunate design bug which we cannot fix easily.

log_sys.first_lsn: The start of the current log file, to be consulted
in log_t::write_checkpoint() when renaming files.

log_sys.archived_lsn: New field: The value of innodb_lsn_archived.

log_sys.end_lsn: New field: The log_sys.get_lsn() when the latest
checkpoint was initiated. That is, the start LSN of a possibly empty
sequence of FILE_MODIFY records followed by FILE_CHECKPOINT.

log_sys.resize_target: The value of innodb_log_file_size that will be
used for creating the next archive log file once the current file (of
log_sys.file_size) fills up.

log_sys.archive: New field: The value of innodb_log_archive.

log_sys.next_checkpoint_no: Widen to uint16_t. There may be up to
12288/8=1536 checkpoints in the header.

log_sys.log: If innodb_log_archive=ON, this file handle will be kept
open also in the PMEM code path.

log_sys.resize_log: If innodb_log_archive=ON, we may have two log
files open both during normal operation and when parsing the log. This
will store the other handle (old or new file).

log_sys.resize_buf: In the memory-mapped code path, this will point
to the file resize_log when innodb_log_archive=ON.

recv_sys.log_archive: All innodb_log_archive=ON files that will be
considered in recovery.

recv_sys.was_archive: A flag indicating that an innodb_log_archive=ON
file is in innodb_log_archive=OFF format.

log_sys.is_pmem, log_t::is_mmap_writeable(): A new predicate.
If is_mmap_writeable(), we assert and guarantee buf_size == capacity().

log_t::archive_new_write(): Create and allocate a new log file, and
write the outstanding data to both the current and the new file, or
only to the new file, until write_checkpoint() completes the first
checkpoint in the new file.

log_t::archived_mmap_switch_prepare(): Create and memory-map a new log
file, and update file_size to resize_target. Remember the file handle
of the current log in resize_log, so that write_checkpoint() will be
able to make it read-only.

log_t::archived_mmap_switch_complete(): Switch to the buffer that was
created in archived_mmap_switch_prepare().

log_t::archive_create(bool): Create and allocate an archive log file.
On POSIX, ensure that we get the exact correct file size by invoking
ftruncate() on POSIX, because os_file_set_size() would only
truncate files on Microsoft Windows.

buf_flush_archive_create(): Create a spare archive log file if needed.

buf_flush_page_cleaner(): Invoke buf_flush_archive_create() about
one second after a checkpoint where we switched archived log files.
This allows any writer threads to resume activity after we completed
the previous checkpoint.

log_t::write_checkpoint(): Allow an old checkpoint to be completed in
the old log file even after a new one has been created. If we are
writing the first checkpoint in a new log file, we will mark the old
log file read-only. We will also update log_sys.first_lsn unless it
was already updated in ARCHIVED_MMAP code path. In that code path,
there is the special case where log_sys.resize_buf == nullptr and
log_sys.checkpoint_buf points to log_sys.resize_log (the old log file
that is about to be made read-only). In this case, log_sys.first_lsn
will already point to the start of the current log_sys.log, even
though the switch has not been fully completed yet.
Try to preallocate the next archive file if needed, with the goal that
when we need the file it will already be ready for use.

log_t::write_checkpoint(), log_checkpoint_low(), log_checkpoint(),
buf_flush_sync_for_checkpoint(): Return the LSN on which to invoke
archive_create(), or 0 if no spare archive log file needs to be created.

log_t::header_rewrite(my_bool): Rewrite the log file header before or
after renaming the log file, and write a message about the change,
so that there will be a chance to recover in case the server is being
killed during this operation.  The recovery of the last ib_%016%.log
does tolerate also the ib_logfile0 format.

log_t::set_archive(my_bool,THD): Implement SET GLOBAL innodb_log_archive.
An error will be returned if non-archived SET GLOBAL innodb_log_file_size
(log file resizing) is in progress. Wait for checkpoint if necessary.
The current log file will be renamed to either ib_logfile0 or
ib_%016x.log, as appropriate. In SET GLOBAL innodb_log_archive=OFF,
we trigger a write-ahead of the log if necessary, to prevent overrun.

log_t::archive_rename(): Rename an archived log to ib_logfile0 on recovery
in case there had been a crash during set_archive().

log_t::archive_set_size(): A new function, to ensure that
log_sys.resize_target is set on startup.

log_checkpoint_low(): Do not prevent a checkpoint at the start of a file.
We want the first innodb_log_archive=ON file to start with a checkpoint.

log_t::create(lsn_t): Initialize last_checkpoint_lsn. Initialize the
log header as specified by log_sys.archive (innodb_log_archive).

log_write_buf(): Add the parameter max_length, the file wrap limit.

log_write_up_to(), mtr_t::commit_log_release<bool mmap=true>():
If we are switching log files, invoke buf_flush_ahead(lsn, false)
to ensure that a log checkpoint will be completed in the new file.

mtr_t::finish_writer(): Specialize for innodb_log_archive=ON.

mtr_t::commit_file(): Ensure that log archive rotation will complete.

log_t::append_prepare<log_t::ARCHIVED_MMAP>(): Special case.

log_t::append_prepare(): Remove the call to set_check_for_checkpoint()
that was duplicating logic in log_close().

log_t::get_path(): Get the name of the current log file.

log_t::get_circular_path(size_t): Get the path name of a circular file.
Replaces get_log_file_path().

log_t::get_archive_path(lsn_t): Return a name of an archived log file.

log_t::append_archive_name(): Append the archive log file name
to a path string.

log_t::checkpoint_margin(): Replaces log_checkpoint_margin().
If a new archived log file has been created, wait for the
first checkpoint in that file. Revise the formula when
log_sys.need_checkpoint needs to hold. When innodb_log_archive=ON,
we can allow the checkpoint to be older by innodb_log_file_size.

log_close(): Relax the log_sys.need_checkpoint condition
for innodb_log_archive=ON.

srv_log_rebuild_if_needed(): Never rebuild if innodb_log_archive=ON.
The setting innodb_log_file_size will affect the creation of
subsequent log files. The parameter innodb_encrypt_log cannot be
changed while the log is in the innodb_log_archive=ON format.

log_t::attach(), log_mmap(): Add the parameter log_access,
to distinguish memory-mapped or read-only access.

log_t::attach(): When disabling innodb_log_file_mmap, read
checkpoint_buf from the last innodb_log_archive=ON file.

log_t::clear_mmap(): Clear the tail of the checkpoint buffer
if is_mmap_writeable().

log_t::set_recovered(): Invoke clear_mmap(), and restore the
log buffer to the correct position.

recv_sys_t::apply(): Let log_t::clear_mmap() enable log writes.

log_file_is_zero(): Check if a log file starts with NUL bytes
(is a preallocated file).

recv_sys_t::find_checkpoint(): Find and remember the checkpoint position
in the last file when innodb_log_recovery_start points to an older file.
When innodb_log_file_mmap=OFF, restore log_sys.checkpoint_buf from
the latest log file. If the last archive log file is actually
in innodb_log_archive=OFF format despite being named ib_%016.log,
try to recover it in that format. If the circular ib_logfile0 is missing,
determine the oldest archived log file with contiguous LSN.
If innodb_log_archive=ON, refuse to start if ib_logfile0 exists.
Open non-last non-preallocated archived log file in read/write mode.

recv_sys_t::find_checkpoint_archived(): Validate each checkpoint in
the current file header, and by default aim to recover from the last
valid one. Terminate the search if the last validated checkpoint
spanned two files. If innodb_log_recovery_start has been specified,
attempt to validate it even if there is no such information stored
in the checkpoint header.

log_parse_file(): Do not invoke fil_name_process() during
recv_sys_t::find_checkpoint_archived(), when we tolerate FILE_MODIFY
records while looking for a FILE_CHECKPOINT record.

recv_scan_log(): Invoke log_t::archived_switch_recovery() upon
reaching the end of the current archived log file.

log_t::archived_switch_recovery_prepare(): Make use of
recv_sys.log_archive and open all but the last file read-only.

log_t::archived_switch_recovery(): Switch files in the pread() code path.

log_t::archived_mmap_switch_recovery_complete(): Switch files in the
memory-mapped code path.

recv_warp: A pointer wrapper for memory-mapped parsing that spans two
archive log files.

recv_sys_t::parse_mmap(): Use recv_warp for innodb_log_archive=ON.

recv_sys_t::parse(): Tweak some logic for innodb_log_archive=ON.

log_t::set_recovered_checkpoint(): Set the checkpoint on recovery.
Updates also the end_lsn.

log_t::set_recovered_lsn(): Also update flush_lock and write_lock,
to ensure that log_write_up_to() will be a no-op. Invoke
unstash_archive_file() in case there was a garbage (pre-allocated)
file at the end which was not parsed at all.

log_t::persist(): Even if the flushed_to_disk_lsn does not change,
we may want to reset the write_lsn_offset.
Rucha Deodhar
MDEV-39119: Improve error handling when using OLD_VALUE as alias name

Analysis:
Since OLD_VALUE_SYM was part of reserved keywords, it did not allow
old_value in the alias.
Fix:
Change OLD_VALUE_SYM from reserved keyword to keyword_sp_var_and_label.
Oleg Smirnov
MDEV-38045: Preparation commit

1. `TABLE_LIST::init_derived` is called twice during derived tables
processing: first time from `mysql_derived_init` and the second
time from `mysql_derived_prepare`. Both times there is a check
for whether an optimizer hint or switch setting allows a derived
table to be merged.
However, it is not necessary to restrict merging during the
initialization call, it is enough to apply hints/switch setting
during the preparation phase.

2. `open_normal_and_derived_tables()` runs processing of all phases
of derived tables handling with a single call. But for future
commits it is required to separate DT initialization from other
phases. This commit implements the separation.
Oleksandr Byelkin
Merge branch '10.11' into bb-10.11-release
Lawrin Novitsky
One more UBSAN silencing.

Same nature "function called using wrong pointer type" for C/C parameters
callback call. In this case besides inclusion in the namespace the
function had accidentally wrong return type.
  • ccpp-linux-amd64-msan: build linux-connector_cpp failed -  stdio
Rucha Deodhar
MDEV-5092: Implement UPDATE with result set (UPDATE ... RETURNING)

The patch introduces the OLD_VALUE() expression to reference the value
of a column before it was updated. The parser is extended to support
RETURNING and OLD_VALUE(), and RETURNING expressions are stored in a
separate returning_list in SELECT_LEX with independent wildcard tracking.
RETURNING is rejected for multi-table UPDATE.

During setup of RETURNING fields, THD::is_setting_returning is used
when resolving fields, particularly for updates through views.
When resolving view fields, Item_direct_view_ref may point to the
view's item_list, losing the information about whether the value
should be old or new. The original item in returning_list still
contains the correct is_old_value_reference flag, which is copied
back to the resolved item.

OLD_VALUE() is implemented by extending Item_field with a new
Item_old_field class. Item_field::set_field() initializes
Field::ptr_old to the corresponding location in record[1],
which stores the old row during UPDATE execution.

When sending result rows, Item_old_field::send() temporarily switches
the field pointer from record[0] (current row) to ptr_old so
OLD_VALUE() returns the value before the update.

The UPDATE execution path is modified to send a result set when
RETURNING is present instead of an OK packet.
Sergei Golubchik
MDEV-39498 mroonga_command() incorrect result length metadata
Andrey Sokolov
MDEV-39487: mroonga: remove flags local variable from grn_io_open

This variable is unused.
Sergei Golubchik
MDEV-39498 more fixes

* use max_length=640
* also fix mroonga_highlight_html, mroonga_normalize, mroonga_snippet_html
* remove disable_cursor_protocol from all mroonga tests
Oleg Smirnov
MDEV-38045 Fix type conversion errors
Oleg Smirnov
MDEV-38045 Add code coverage markers to untestable code paths

MariaDB test suite does not provide tools to test out-of-memory (OOM)
errors, so it is not feasible to cover some paths with test cases.
This commit adds markers /* purecov: inspected */ and
/* purecov: deadcode */ to such paths.
Oleksandr Byelkin
Merge branch '10.6' into bb-10.11-release
Razvan Liviu Varzaru
Support fedora for static_test linking
  • ccpp-linux-amd64-msan: build linux-connector_cpp failed -  stdio
rusher
[misc] use DOCKER_LOGIN as user if secret available
Rucha Deodhar
MDEV-39127: UBSAN : downcast of address X which does not point to an object
of type 'multi_update' in sql/sql_update.cc
| Sql_cmd_update::update_single_table

Analysis:
the 'result' object was being incorrectly used which maybe of the type
multi_update.This caused UBSAN error due to an invalid downcast in
Sql_cmd_update::update_single_table().

Fix:
Introduce a dedicated returning_result object for handling RETURNING output
instead of reusing result. This ensures the correct result handler is used
and avoids unsafe casts.
Sergei Golubchik
MDEV-38802 MariaDB server start emits error but continues anyway: Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist

it's not an error, as the server continues anyway
Lawrin Novitsky
Increased cmake min required version to enable build with newest cmake's

Also, fixed UBSAN issue caused by including mysql header inside the
namespace and using pointer to destroyer functions in the smart
pointers. Tha was in ConnectProtocol and TextRowProtocol classes with
MYSQL and MYSQL_RES handles, respectively. Removed unique_ptr use
completely as it's pretty safe and easy to destroy them in the the
respective class destructor.
Added check and exception throw if MYSQL handle could not be allocated -
taht was missing.
Oleg Smirnov
MDEV-38045: Implement QB_NAME hint with path syntax for nested query blocks

Extended QB_NAME hint to support path-based addressing of query blocks
nested within views, derived tables, and CTEs, following TiDB's syntax.

New syntax:
  QB_NAME(name, query_block_path), where
    query_block_path ::= query_block_path_element
                          [ {. query_block_path_element }... ]
    query_block_path_element ::= @ qb_path_element_select_num |
                                    qb_path_element_view_sel
    qb_path_element_view_sel ::= qb_path_element_view_name
                                  [ @ qb_path_element_select_num ]

For example,
  `SELECT /*+ qb_name(qb_v1, v1) */* FROM v1`
The name `qb_v1` is assigned to the inner query block of the view `v1`.

  `SELECT /*+ qb_name(qb_v1, v1@sel_1) */* FROM v1`
Means the same but specifies that `v1` is present in SELECT#1 of the current
query block.

  `SELECT /*+ qb_name(qb_v1, v1@sel_1 .@sel_2) */* FROM v1`
This means SELECT#2 of view `v1`, which is present in SELECT#1 of
the current query block, gets the name `qb_v1`.

It is possible to specify not only view names but also derived tables
and CTE's in the path.

Views and derived tables may be nested on multiple levels, for example:
  `SELECT /*+ qb_name(dt2_dt1_v1_1, dt1 .dt2 .v2 .@SEL_2)
              no_index(t1@dt2_dt1_v1_1)*/ v1.*
  FROM v1 JOIN (SELECT v1.* FROM v1 JOIN (SELECT * FROM v2) dt2) dt1`

Limitations:
  - Only SELECT statements support QB names with path. DML operations
    (UPDATE, DELETE, INSERT) only support explicitly defined QB names
Sergei Golubchik
MDEV-39266 Stack Overflow via alloca() in Privilege Table JSON Parser
Lawrin Novitsky
More fixes to please UBSAN and MSAN.

For UBSAN it's the similar problem as in prvious commit, for MSAN - it
suddenly had issues afte move of min required cmake version to 3.5.
Seemingly it is connected to policy 0056 and it's new behavior.
  • ccpp-linux-amd64-msan: build linux-connector_cpp failed -  stdio