Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
bsrikanth-mariadb
MDEV-36523: Load basic stats from trace into optimizer

This task loads the stats of the tables that are used in a query,
from the trace into the optimizer. This feature is controlled
by optimizer_replay_context, and points to an user defined variable name,
that has the context information.
The stats such as num_of_records present in the table, indexes if present,
their names, along with the average number of records_per_key
with in each index, cost of reading an index are loaded from the trace.
Additionally, stats from range analysis i.e. ranges, and the
corresponding number of records, along with their costs are also
loaded from the trace.

The trace context which is in JSON format is firstly set into the
user defined session variable, and this variable name is set to
optimizer_replay_context system variable.
Later, when a user issues a query, the contents of the json context is
parsed and an in memory representation is built using the class
Optimizer_context_replay. This class is then used by the optimizer
to update and save original values of the tables, indexes, and range stats
in the methods "set_statistics_for_table()" of sql_statistics.cc, and
"check_quick_select()" of opt_range.cc.
After the query gets finished, the statistics that were updated
in the optimizer are restored back to the saved original values.

The entry point for parsing the json structure is in
"mysql_execute_command()" of sql_parse.cc, and similarly exit point
i.e. to restore the saved stats of the optimizer is at the end
of the same method.

Two new warning/error messages are introduced:
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_PARSE_FAILED, and
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_MATCH_FAILED
Andrei Elkin
Partial all but 3 hunks revert has_/lock/unlock- temporary_tables hunks of temporary_tables.cc.
ParadoxV5
Merge branch 'main' into MDEV-37146
Andrei Elkin
Revert "MDEV-37686 rpl.create_or_replace_mix2 fails in MDEV-35915 branch"

This reverts commit 3241798214b066d62ba3274ba5dc29549349ca65.
Due to MDEV-38212.
bsrikanth-mariadb
MDEV-36511: Dump basic stats of a table to trace

This feature stores the basic stats of the base tables that are used in
a query, to the optimizer trace. This feature is also controlled by
optimizer_record_context, and is not enabled by default. The stats
such as num_of_records present in the table, indexes if present
then their names, along with the average number of records_per_key
with in each index are dumped to the trace. Additionally, stats from
range analysis of the queries are also dumped into the trace

The approach taken here is to extend the existing function
store_tables_context_in_trace() and add new dump_table_stats_to_trace()
to opt_trace_ddl_info.cc. For storing the range analysis stats into the
trace, they are first recorded in the hash defined in
Optimizer_Stats_Context_Recorder, and is used later while dumping the
stats into the trace from dump_table_stats_to_trace().
Several new tests are also added to opt_trace_store_stats.test
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

fts_optimize_table() : Assigns thd to transaction even it is
called via user_thread or fulltext optimize thread.
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Add FTSQueryExecutor class as a thin abstraction over QueryExecutor.
This class takes care of open, lock, read, insert, delete
for all auxiliary tables INDEX_[1..6], common FTS tables
(DELETED, DELETED_CACHE, BEING_DELETED, CONFIG..)

FTSQueryExecutor Class which has the following function:
Auxiliary table functions : insert_aux_record(), delete_aux_record(),
    read_aux(), read_aux_all()

FTS common table functions : insert_common_record(), delete_common_record(),
    delete_all_common_records(), read_all_common()

FTS CONFIG table functions : insert_config_record(), update_config_record(),
                            delete_config_record(), read_config_with_lock()
Nikita Malyavin
MDEV-38151 GTT: missing FUNCTION support

open_temporary_table[s] defaulted to searching only local temporary
tables (Tmp_table_kind::TMP). When in function, a temporary table was
"carefully re-opened": it was closed and then opened again, with
open_temporary_table. This resulted in "table not found".

The correct key in most cases is Tmp_table_kind::ANY.
In some cases it should be Tmp_table_kind::TMP:
* When operation is not supported for GTT (ANALYZE/REPAIR)
* When a global handle should be opened (CREATE VIEW)
* If it's a re-open

Apart from this bug, it caused a global temporary table to be always
opened through open_tables, meaning that tdc was queried first. This
means that the earlier code never relied on pre-opening
temporary tables.

This means that all the commands that follow the default
open_temporary_tables path in mysql_execute_command (the standard
temporary tables pre-opening) should be able to handle child GTT share.

This wasn't so for TRUNCATE, hence the changes in sql_truncate.cc.

Also, this broke the lookup order: first a local temporary table should
be looked up. If it doesn't exist, only then a global temporary table
should be looked up.
To fix the latter -- push back global temporary tables' local shares and
push front local temporary tables.

To support push_front, All_tmp_tables_list declaration is changed. It
should use I_P_List_fast_push_back adapter, which adds T **m_last to the
list body.
Nikita Malyavin
cleanup: register global_temporary_tp handlerton only ON COMMIT DELETE
ParadoxV5
MDEV-28302 configurable defaults for CHANGE MASTER

Many CHANGE MASTER fields typically have the same
configurations between multi-source connections – namely:
master_connect_retry
master_ssl
master_ssl_ca
master_ssl_capath
master_ssl_cert
master_ssl_cipher
master_ssl_key
master_ssl_verify_server_cert
master_ssl_crl
master_ssl_crlpath
master_use_gtid
master_retry_count
master_heartbeat_period

When MDEV-25674 added `master_retry_count` to CHANGE MASTER, it kept
the server option `--master-retry-count` to be its default value.
This commit back-adds corresponding server options
for the defaults of the rest of those fields.
With them, the command line or config files can set up common
configurations across replication sources (and even replicas).

CHANGE MASTER can override unset
(defaulted) configurations per connection.
This commit also adds `DEFAULT` keyword support for all of those fields,
so overridden configurations can reset
to the default without RESET REPLICA.

Supporting passing the `DEFAULT` keyword also enables setting
`master_connect_retry` and `master_retry_count` to 0,
which was previously disregarded.
While here, the commit also increases `master_retry_count`
to 64-bit on LLP64 (e.g., Windows) to match LP64 (e.g., Linux).
Rucha Deodhar
MDEV-37950: INSERT ... RETURNING exposes columns for which the user lacks
SELECT privilege

Analysis:
When setup_fields() is called, the want_privilege is set to NO_ACL, so
correct priveleges are not checked.
Fix:
Since RETURNING requires SELECT_ACL privelige, when we are setting up
the returning fields for the given query, set want_privilege to SELECT_ACL.
Reset to original value of want_privilege once done.
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Introduce QueryExecutor to perform direct InnoDB record scans
with a callback interface and consistent-read handling.
Also handles basic DML operation on clustered index of the table

Newly Added file row0query.h & row0query.cc

QueryExecutor class the following apis
read(): iterate clustered index with RecordCallback
read_by_index(): scan secondary index and fetch clustered row
lookup_clustered_record(): resolve PK from secondary rec
process_record_with_mvcc(): build version via read view
and skip deletes
insert_record(): Insert tuple into table's clustered index
select_for_update(): Lock the record which matches with search_tuple
update_record():  Update the currently selected and X-locked
clustered record.
delete_record():  Delete the clustered record identified by tuple
delete_all(): Delete all clustered records in the table
replace_record(): Tries update via select_for_update() +
  update_record(); if not found, runs insert_record.
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Introduce AuxCompareMode and AuxRecordReader to scan FTS auxiliary
indexes with compare+process callbacks.

Replace legacy SQL-graph APIs with typed executor-based ones:
-Add fts_index_fetch_nodes(trx, index, word, user_arg,
                          FTSRecordProcessor,compare_mode).
-Redefine fts_write_node() to use FTSQueryExecutor and fts_aux_data_t.

Implement write path via delete_aux_record (or) insert_aux_record.
Keep lock-wait retry handling and memory limit checks.

Change fts_select_index{,_by_range,_by_hash} return type
from ulint to uint8_t and simplify return flow.
Include fts0exec.h in fts0priv.h and update declarations accordingly.
Marko Mäkelä
WIP MDEV-37949: Implement innodb_log_archive, innodb_lsn_archived

TODO: Make innodb_log_archive=ON writes fully work!

TODO: Rename log files and set innodb_lsn_archived on checkpoint.

TODO: Implement recovery from archived log.

The new setting SET GLOBAL innodb_log_archive=ON will enable log
archiving as soon as the current ib_logfile0 is about to wrap around.

The read-only variable innodb_lsn_archived will reflect the LSN
since when a complete InnoDB log archive is available. If it is
0 at startup, it will be recovered from the log files. If
innodb_log_archive=OFF, this variable will be adjusted to the
latest checkpoint every time a log checkpoint is executed.
If innodb_log_archive=ON, the value should not change.

When innodb_log_archive=ON, the setting SET GLOBAL innodb_log_file_size
will affect subsequently created log files when the file that is being
currently written is running out.

log_t::archive_new_write(): Create and allocate a new log file,
and write the outstanding data to both the current and the new file.

log_t::archive_new_mmap(): Create and memory-map a new log file,
and update file_size to resize_target.

log_t::archive_set_size(): Ensure that resize_target is set for
new archived log files.

log_t::first_lsn: This should be the start of the current log file,
to be consulted in log_t::write_checkpoint() when renaming files.

log_write_buf(): Add the parameter max_length, the file wrap limit.

mtr_t::finish_writer(): Specialize for innodb_log_archive=ON

innodb_log_archive_update(): Assign log_sys.archived_lsn to the latest
checkpoint; it should be usable. If the log buffer is about to wrap
around, back off and retry.

log_t::archived_mmap_switch_prepare(): Create and memory-map a new log
file. TODO: Do we need a separate function from append_prepare_wait()?

log_t::append_prepare<log_t::ARCHIVED_MMAP>(): Special case.

log_t::archived_mmap_switch_complete(): Switch to the buffer that was
created in archived_mmap_switch_prepare().

mtr_t::finish_writer(): Invoke log_close() only if innodb_log_archive=OFF.
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Introduce ConfigReader callback to extract key, value from
fulltext config common table (CONFIG). This table has
<key, value> schema.

Simplifield all function which uses CONFIG tale. These
functions uses executor.insert_config_record(), update_config_record()
instead of SQL or query graph.
Yuchen Pei
MDEV-9826 skip ci [demo] not compiling

/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x18): undefined reference to `my_hasher_base31'
/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x20): undefined reference to `my_hasher_crc32c'
/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x28): undefined reference to `my_hasher_xxh32'
/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x30): undefined reference to `my_hasher_xxh3'
/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x18): undefined reference to `my_hasher_base31'
/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x20): undefined reference to `my_hasher_crc32c'
/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x28): undefined reference to `my_hasher_xxh32'
/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x30): undefined reference to `my_hasher_xxh3'
collect2: error: ld returned 1 exit status
make[2]: *** [unittest/sql/CMakeFiles/explain_filename-t.dir/build.make:137: unittest/sql/explain_filename-t] Error 1
make[2]: Leaving directory '/home/ycp/source/mariadb-server/main/build'
make[1]: *** [CMakeFiles/Makefile2:3830: unittest/sql/CMakeFiles/explain_filename-t.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
collect2: error: ld returned 1 exit status
make[2]: *** [unittest/sql/CMakeFiles/my_json_writer-t.dir/build.make:137: unittest/sql/my_json_writer-t] Error 1
make[2]: Leaving directory '/home/ycp/source/mariadb-server/main/build'
make[1]: *** [CMakeFiles/Makefile2:3902: unittest/sql/CMakeFiles/my_json_writer-t.dir/all] Error 2
Creating mysqld link
cd /home/ycp/source/mariadb-server/main/build/sql && /usr/bin/cmake -E create_symlink mariadbd mysqld
make[2]: Leaving directory '/home/ycp/source/mariadb-server/main/build'
[ 98%] Built target mariadbd
Creating mariabackup link
cd /home/ycp/source/mariadb-server/main/build/extra/mariabackup && /usr/bin/cmake -E create_symlink mariadb-backup mariabackup
make[2]: Leaving directory '/home/ycp/source/mariadb-server/main/build'
[100%] Built target mariadb-backup
make[1]: Leaving directory '/home/ycp/source/mariadb-server/main/build'
make: *** [Makefile:166: all] Error 2
bsrikanth-mariadb
MDEV-36523: Load basic stats from trace into optimizer

This task loads the stats of the tables that are used in a query,
from the trace into the optimizer. This feature is also controlled
by optimizer_record_context, and is not enabled by default.
The stats such as num_of_records present in the table, indexes if present,
their names, along with the average number of records_per_key
with in each index are loaded from the trace.
Additionally, stats from range analysis i.e. ranges, and the
corresponding number of records are also loaded from the trace.

The trace context which is in JSON format is firstly set into the
session variable optimizer_trace_stored_context.
Later, when a user issues a query, the contents of this variable is
parsed and an in memory representation is built using the class
Optimizer_context_replay. This class is then used by the optimizer
to update and save original values of the tables, indexes, and range stats
in the methods "set_statistics_for_table()" of sql_statistics.cc, and
"check_quick_select()" of opt_range.cc.
After the query gets finished, the statistics that were updated
in the optimizer are restored back to the saved original values.

The entry point for parsing the json structure is in
"mysql_execute_command()" of sql_parse.cc, and similarly exit point
i.e. to restore the saved stats of the optimizer is at the end
of the same method.

Two new warning/error messages are introduced:
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_PARSE_FAILED, and
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_MATCH_FAILED
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Refactor fetch, optimize to QueryExecutor and
standardize processor API. Replaced legacy SQL-graph
paths with QueryExecutor-based reads/writes:
fts_query code now uses QueryExecutor::read(), read_by_index()
with RecordCallback (updating fts_query_match_document(),
fts_query_is_in_proximity_range(), and fts_expand_query() to call
fts_query_fetch_document() instead of fts_doc_fetch_by_doc_id(),
which was removed along with FTS_FETCH_DOC_BY_DOC_ID_* macros);

Rewrote fts_optimize_write_word() to delete (or) insert via
FTSQueryExecutor::delete_aux_record()/insert_aux_record()
using fts_aux_data_t;
Kristian Nielsen
Fix hang on master when disabling semi-sync

There is a global variable global_ack_signal_fd used to signal the receiver
thread to wake up when disabling semi-sync. This variable was cleared to -1
in the Ack_listener destructor, which ran at the end of the
Ack_receiver::run() function without any locking. If the thread was delayed
at that point, it could end up overwriting the new value set by a new
receiver thread. This would leave the server in a state with an invalid
global_ack_signal_fd and could cause a subsequent disable of semisync to
fail due to the wakeup not arriving at the receiver thread.

This was seen as a sporadic failure of the test case
rpl.rpl_semi_sync_cond_var_per_thd.

Fix by not modifying the global in constructor/destructor; instead set and
clear the global explicitly, allowing to clear the fd with proper locking
while the mutex is still being held.

Also fix a missing pthread_join(), which would leak thread descriptors and
allow to start a new receiver thread before the old one shut down fully.

(Either of these two changes fix the hang bug).

Signed-off-by: Kristian Nielsen <[email protected]>
ParadoxV5
aw shoot
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

- Removed fts0sql.cc file.
- Removed commented fts funtions
- Removed fts_table_t from fts_query_t and fts_optimize_t
Alexander Barkov
FETCH anchored_cursor_with_return INTO var1,var2,var3
Yuchen Pei
MDEV-9826 skip ci [demo] not compiling

/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x18): undefined reference to `my_hasher_base31'
/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x20): undefined reference to `my_hasher_crc32c'
/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x28): undefined reference to `my_hasher_xxh32'
/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x30): undefined reference to `my_hasher_xxh3'
/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x18): undefined reference to `my_hasher_base31'
/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x20): undefined reference to `my_hasher_crc32c'
/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x28): undefined reference to `my_hasher_xxh32'
/usr/bin/ld: ../../sql/libpartition.a(ha_partition.cc.o):(.data.rel+0x30): undefined reference to `my_hasher_xxh3'
collect2: error: ld returned 1 exit status
make[2]: *** [unittest/sql/CMakeFiles/explain_filename-t.dir/build.make:137: unittest/sql/explain_filename-t] Error 1
make[2]: Leaving directory '/home/ycp/source/mariadb-server/main/build'
make[1]: *** [CMakeFiles/Makefile2:3830: unittest/sql/CMakeFiles/explain_filename-t.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
collect2: error: ld returned 1 exit status
make[2]: *** [unittest/sql/CMakeFiles/my_json_writer-t.dir/build.make:137: unittest/sql/my_json_writer-t] Error 1
make[2]: Leaving directory '/home/ycp/source/mariadb-server/main/build'
make[1]: *** [CMakeFiles/Makefile2:3902: unittest/sql/CMakeFiles/my_json_writer-t.dir/all] Error 2
Creating mysqld link
cd /home/ycp/source/mariadb-server/main/build/sql && /usr/bin/cmake -E create_symlink mariadbd mysqld
make[2]: Leaving directory '/home/ycp/source/mariadb-server/main/build'
[ 98%] Built target mariadbd
Creating mariabackup link
cd /home/ycp/source/mariadb-server/main/build/extra/mariabackup && /usr/bin/cmake -E create_symlink mariadb-backup mariabackup
make[2]: Leaving directory '/home/ycp/source/mariadb-server/main/build'
[100%] Built target mariadb-backup
make[1]: Leaving directory '/home/ycp/source/mariadb-server/main/build'
make: *** [Makefile:166: all] Error 2
Kristian Nielsen
Semi-sync: Clean up incorrect file/pos comparisons

The semi-sync code has a number of places where it compares pairs of
(filename,offset) for which is larger than the other.

The filename comparisons are done using strcmp(), which is wrong. Filenames
will compare wrong when they wrap from eg. bin-999999 to bin-1000000, and
user can also rename log files which can likewise break comparisons.

Further, the comparisons are completely unnecessary, as all the transactions
to be waited for are already stored in a linear list in order _and_ in a
hash table. So the code can simply use the existing hash table look and list
traversal to determine status and sequence of the waited-for transactions.

So this patch removes all the comparisons for larger/smaller, leaving only
comparisons for equality and effectively making the (filename,offset) pairs
just opaque transaction identifiers. And also removes a few other related
pieces of dead/unnecessary code.

Signed-off-by: Kristian Nielsen <[email protected]>
bsrikanth-mariadb
MDEV-36523: Load basic stats from trace into optimizer

This task loads the stats of the tables that are used in a query,
from the trace into the optimizer. This feature is also controlled
by optimizer_record_context, and is not enabled by default.
The stats such as num_of_records present in the table, indexes if present,
their names, along with the average number of records_per_key
with in each index are loaded from the trace.
Additionally, stats from range analysis i.e. ranges, and the
corresponding number of records are also loaded from the trace.

The trace context which is in JSON format is firstly set into the
session variable optimizer_trace_stored_context.
Later, when a user issues a query, the contents of this variable is
parsed and an in memory representation is built using the class
Optimizer_context_replay. This class is then used by the optimizer
to update and save original values of the tables, indexes, and range stats
in the methods "set_statistics_for_table()" of sql_statistics.cc, and
"check_quick_select()" of opt_range.cc.
After the query gets finished, the statistics that were updated
in the optimizer are restored back to the saved original values.

The entry point for parsing the json structure is in
"mysql_execute_command()" of sql_parse.cc, and similarly exit point
i.e. to restore the saved stats of the optimizer is at the end
of the same method.

Two new warning/error messages are introduced:
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_PARSE_FAILED, and
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_MATCH_FAILED
ParadoxV5
MDEV-37530 Refactor Master & Relay Log info to iterable tuples

The persistence code of CHANGE MASTER values needs to match file
lines with the formatting for the corresponding field’s type.
This is unstructured, repetitive (not DRY), and makes feature expansions
(such as MDEV-28302 `CHANGE MASTER …=DEFAULT`)
error-prone if not difficult.

This commit moves these functions and global constants from
`slave.cc`/`.h`, as well as the Master and Relay Log Info File
entries from `Rpl_mi` and `Rpl_rli`, to dedicated `rpl_*info_file.h`
files and under corresponding structs to facilitate organization.
Namely, this commit wraps those file entries with transparent
structs that inherit from a `Persistent` interface,
or shared helper structs that themselves inherit from `Persistent`.
By moving the file read/write helper functions to or behind
implementations of `Persistent`’s virtual methods,
reading or writing the file only takes a type-agnostic loop
over the (wrapped) CHANGE MASTER values.

* This commit also includes preemptive support
  for preserving MDEV-28302’s `=DEFAULT`.
  As such, unset fields (namely `master_connect_retry`)
  now remember their `DEFAULT` states
  rather than whatever the default is at CHANGE MASTER time.
* For consistency’s sake, `master_heartbeat_period` is
  now reset at RESET REPLICA instead of CHANGE MASTER.

As this refactor will disconnect it from fixes for
some open bugs in prior versions, this commit also:
* Reimplements the value reader functions to be strict with their input
  * Fixes MDEV-38010
    number parsing ignores trailing garbage and overflows
  * Supercedes MDEV-38020 integer overflow
* Changes master_heartbeat_period from a `float` of seconds
  to a `uint32_t` of milliseconds (adding `/1000.0`s as needed)
  * Fixes MDEV-35879 `Slave_heartbeat_period` is imprecise
  * The maximum of `master_heartbeat_period` has been
    increased to 4294967.295, i.e., (2³²-1)÷1000.
  * `master_heartbeat_period` now rounds
    instead of truncates (rounds down).
* Prepares to make `master_retry_count` 64-bit on
  LLP64 (e.g., Windows) to match LP64 (e.g., Linux)
Alexander Barkov
FETCH anchored_cursor_with_return INTO var1,var2,var3
Yuchen Pei
MDEV-9826 More hash algorithms for PARTITION BY [LINEAR] KEY

PARTITION BY [LINEAR] KEY ALGORITHM={MYSQL51|MYSQL55|BASE31|CRC32C|XXH32|XXH3}

- The BASE31 algorithm uses a base-31 representation of the bytes, see
  Modular hashing in https://algs4.cs.princeton.edu/34hash/. It serves
  as a simple baseline
- CRC32C uses my_crc32c.
- XXH32 and XXH3 are xxhash algorithms - xxhash.h copied from latest
  release (0.8.3) of https://github.com/Cyan4973/xxHash

For performance (esp. xxh) we use one-shot hash functions in binary
hash_sort, and streaming hash function otherwise for byte-by-byte
hashing. XXH is the only stateful hash function. The other hash
algorithms are stateless and homomorphic, so streaming and one-shot
functions are identical, which is reflected in the logic of fallback
from NULL m_hash_byte to m_hash_str applied to a byte.

Tested with

mtr --suite main --do-test=.*partition
mtr --suite parts

Also ran above tests with the following patch that changes the default
algorithm from MYSQL55 to CRC32C, and XXH32 (changing the patch
accordingly)

modified  sql/ha_partition.cc
@@ -10336,6 +10336,8 @@ uint32 ha_partition::calculate_key_hash_value(Field **field_array)
  switch ((*field_array)->table->part_info->key_algorithm)
  {
  case partition_info::KEY_ALGORITHM_NONE:
+    hasher.set_algorithm(HASH_ALGORITHM_CRC32C);
+    break;
  case partition_info::KEY_ALGORITHM_55:
    /* Hasher default to mysql55 */
    break;
modified  sql/partition_info.cc
@@ -2328,7 +2328,7 @@ bool partition_info::fix_parser_data(THD *thd)
      if ((thd_sql_command(thd) == SQLCOM_CREATE_TABLE ||
            thd_sql_command(thd) == SQLCOM_ALTER_TABLE) &&
          key_algorithm == KEY_ALGORITHM_NONE)
-        key_algorithm= KEY_ALGORITHM_55;
+        key_algorithm= PARTITION_INFO_DEFAULT_ALGORITHM;
    }
    DBUG_RETURN(FALSE);
  }
@@ -2344,7 +2344,7 @@ bool partition_info::fix_parser_data(THD *thd)
    if ((thd_sql_command(thd) == SQLCOM_CREATE_TABLE ||
          thd_sql_command(thd) == SQLCOM_ALTER_TABLE) &&
        key_algorithm == KEY_ALGORITHM_NONE)
-      key_algorithm= KEY_ALGORITHM_55;
+      key_algorithm= PARTITION_INFO_DEFAULT_ALGORITHM;
  }
  defined_max_value= FALSE; // in case it already set (CREATE TABLE LIKE)
  do
modified  sql/partition_info.h
@@ -446,6 +446,8 @@ class partition_info : public DDL_LOG_STATE, public Sql_alloc
  int gen_part_type(THD *thd, String *str) const;
};

+#define PARTITION_INFO_DEFAULT_ALGORITHM partition_info::KEY_ALGORITHM_CRC32C
+
void part_type_error(THD *thd, partition_info *work_part_info,
                      const char *part_type, partition_info *tab_part_info);

modified  sql/sql_partition.cc
@@ -2471,7 +2471,7 @@ static int add_key_with_algorithm(String *str, const partition_info *part_info)
  err+= str->append(STRING_WITH_LEN("KEY "));

  if (part_info->key_algorithm != partition_info::KEY_ALGORITHM_NONE &&
-      part_info->key_algorithm != partition_info::KEY_ALGORITHM_55)
+      part_info->key_algorithm != PARTITION_INFO_DEFAULT_ALGORITHM)
  {
    err+= str->append(STRING_WITH_LEN("ALGORITHM = "));
    switch (part_info->key_algorithm)
@@ -2479,6 +2479,9 @@ static int add_key_with_algorithm(String *str, const partition_info *part_info)
      case partition_info::KEY_ALGORITHM_51:
        err+= str->append(STRING_WITH_LEN("MYSQL51"));
        break;
+      case partition_info::KEY_ALGORITHM_55:
+        err+= str->append(STRING_WITH_LEN("MYSQL55"));
+        break;
      case partition_info::KEY_ALGORITHM_BASE31:
        err+= str->append(STRING_WITH_LEN("BASE31"));
        break;
Nikita Malyavin
MDEV-38151 GTT: missing FUNCTION support

open_temporary_tables defaulted to searching only local temporary tables
(Tmp_table_kind::TMP), which is wrong -- apart from this bug, it caused
  a global temporary table to be always opened through open_tables,
  meaning that tdc was queried first.

The correct key in most cases is Tmp_table_kind::ANY.
In some cases it should be Tmp_table_kind::TMP:
* When operation is not supported for GTT (ANALYZE/REPAIR)
* When a global handle should be opened (CREATE VIEW)
* If it's a re-open

This means that all the commands that follow the default
open_temporary_tables path in mysql_execute_command (the standard
temporary tables pre-opening) should be able to handle child GTT share.

This wasn't so for TRUNCATE, hence the changes in sql_truncate.cc.

Also, this broke the lookup order: first a local temporary table should
be looked up. If it doesn't exist, only then a global temporary table
should be looked up.
To fix the latter -- push back global temporary tables' local shares and
push front local temporary tables.

To support push_front, All_tmp_tables_list declaration is changed. It
should use I_P_List_fast_push_back adapter, which adds T **m_last to the
list body.
bsrikanth-mariadb
MDEV-36523: Load basic stats from trace into optimizer

This task loads the stats of the tables that are used in a query,
from the trace into the optimizer. This feature is also controlled
by optimizer_record_context, and is not enabled by default.
The stats such as num_of_records present in the table, indexes if present,
their names, along with the average number of records_per_key
with in each index are loaded from the trace.
Additionally, stats from range analysis i.e. ranges, and the
corresponding number of records are also loaded from the trace.

The trace context which is in JSON format is firstly set into the
session variable optimizer_trace_stored_context.
Later, when a user issues a query, the contents of this variable is
parsed and an in memory representation is built using the class
Optimizer_context_replay. This class is then used by the optimizer
to update and save original values of the tables, indexes, and range stats
in the methods "set_statistics_for_table()" of sql_statistics.cc, and
"check_quick_select()" of opt_range.cc.
After the query gets finished, the statistics that were updated
in the optimizer are restored back to the saved original values.

The entry point for parsing the json structure is in
"mysql_execute_command()" of sql_parse.cc, and similarly exit point
i.e. to restore the saved stats of the optimizer is at the end
of the same method.

Two new warning/error messages are introduced:
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_PARSE_FAILED, and
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_MATCH_FAILED
bsrikanth-mariadb
MDEV-36523: Load basic stats from trace into optimizer

This task loads the stats of the tables that are used in a query,
from the trace into the optimizer. This feature is controlled
by optimizer_replay_context, and points to an user defined variable name,
that has the context information.
The stats such as num_of_records present in the table, indexes if present,
their names, along with the average number of records_per_key
with in each index, cost of reading an index are loaded from the trace.
Additionally, stats from range analysis i.e. ranges, and the
corresponding number of records, along with their costs are also
loaded from the trace.

The trace context which is in JSON format is firstly set into the
user defined session variable, and this variable name is set to
optimizer_replay_context system variable.
Later, when a user issues a query, the contents of the json context is
parsed and an in memory representation is built using the class
Optimizer_context_replay. This class is then used by the optimizer
to update and save original values of the tables, indexes, and range stats
in the methods "set_statistics_for_table()" of sql_statistics.cc, and
"check_quick_select()" of opt_range.cc.
After the query gets finished, the statistics that were updated
in the optimizer are restored back to the saved original values.

The entry point for parsing the json structure is in
"mysql_execute_command()" of sql_parse.cc, and similarly exit point
i.e. to restore the saved stats of the optimizer is at the end
of the same method.

Two new warning/error messages are introduced:
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_PARSE_FAILED, and
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_MATCH_FAILED
ParadoxV5
Improve `mariadb-dump --dump-slave`’s desc. output

Improve the wording and spacing of the description comment generated
by `mariadb-dump --dump-slave` for its CHANGE MASTER statement.
Nikita Malyavin
MDEV-38151 GTT: missing FUNCTION support
Andrei Elkin
MDEV-38212 MDEV-37686 Breaks Parallel Replication

Re-fixing MDEV-37686.

NULLifying `THD::rgi_slave` pointer in `start_new_trans` ctor harmed
the parallel slave conflict detection.
Now instead of `NULL`ifying of this member a finer approach is taken to
optionally screen `THD::rgi_slave` when it is attempted to be accessed
within `start_new_trans` context, that is when such out-of-band
transaction is run by a slave thread.

The start_new_trans is allowed of course to access server local
non-replicated temporary tables, should it ever need that.

The original MDEV-37686 aspect is tested with 12.3 branch's
rpl.create_or_replace_mix2.
Any possible side effects to temporary table replication is controlled
by existing rpl suite tests.
There is no need for a specific mtr test to prove the parallel slave
side is back to normal, as `THD::rgi_slave` is again available to concurrent
transactions.
ParadoxV5
`SELECT … FROM information_schema.SLAVE_STATUS` is an alternative
to `SHOW SLAVE STATUS` that supports `WHERE` and column selection.
Utilizing this server-side feature allows `mariadb-dump --dump-slave`
to iterate only the required data on the client side.

This commit keeps support for pre-`information_schema.SLAVE_STATUS`
(MDEV-33526 @ 11.6) via `SHOW SLAVE STATUS` by switching column indices.
On the contrary, it additionally drops `--dump-slave`
and `--apply-slave-statements` support for pre-GTID &
pre-multi-source (before 10.0 – long EOL).

Reviewed-by: Brandon Nesterenko <[email protected]>
ParadoxV5
Bring SHOW SLAVE STATUS mode back

BTW, sounds like we are open on dropping `mariadb-dump`
support for pre-10.0. (They are long EOL.)
I’ll limit this drop to only `--dump-slave` for now.
Marko Mäkelä
MDEV-37949: Introduce innodb_log_recovery_start, innodb_log_recovery_target

innodb_log_recovery_start: The checkpoint LSN to start recovery from.
This will be useful when recovering from an archived log.

innodb_log_recovery_target: The requested LSN to end recovery at.
This will be useful when recovering data files that were copied
as of a time that is before end of the available log.
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Introduce CommonTableReader callback to collect doc_id_t from
fulltext common tables (DELETED, BEING_DELETED, DELETED_CACHE,
BEING_DELETED_CACHE). These table share the same schema strucutre.

Simplified all function which uses DELETED, BEING_DELETED,
DELETED_CACHE, BEING_DELETED_CACHE table. These functions
uses executor.insert_common_record(), delete_common_record(),
delete_all_common_records() instead of SQL or query graph.

fts_table_fetch_doc_ids(): Changed the signature of the function
to pass the table name instead of fts_table_t.
Yuchen Pei
MDEV-9826 More hash algorithms for PARTITION BY [LINEAR] KEY

PARTITION BY [LINEAR] KEY ALGORITHM={MYSQL51|MYSQL55|BASE31|CRC32C|XXH32|XXH3}

- The BASE31 algorithm uses a base-31 representation of the bytes, see
  Modular hashing in https://algs4.cs.princeton.edu/34hash/. It serves
  as a simple baseline
- CRC32C uses my_crc32c.
- XXH32 and XXH3 are xxhash algorithms - xxhash.h copied from latest
  release (0.8.3) of https://github.com/Cyan4973/xxHash

For performance (esp. xxh) we use one-shot hash functions in binary
hash_sort, and streaming hash function otherwise for byte-by-byte
hashing. XXH is the only stateful hash function. The other hash
algorithms are stateless and homomorphic, so streaming and one-shot
functions are identical, which is reflected in the logic of fallback
from NULL m_hash_byte to m_hash_str applied to a byte.

Tested with

mtr --suite main --do-test=.*partition
mtr --suite parts

Also ran above tests with the following patch that changes the default
algorithm from MYSQL55 to CRC32C, and XXH32 (changing the patch
accordingly)

modified  sql/ha_partition.cc
@@ -10336,6 +10336,8 @@ uint32 ha_partition::calculate_key_hash_value(Field **field_array)
  switch ((*field_array)->table->part_info->key_algorithm)
  {
  case partition_info::KEY_ALGORITHM_NONE:
+    hasher.set_algorithm(HASH_ALGORITHM_CRC32C);
+    break;
  case partition_info::KEY_ALGORITHM_55:
    /* Hasher default to mysql55 */
    break;
modified  sql/partition_info.cc
@@ -2328,7 +2328,7 @@ bool partition_info::fix_parser_data(THD *thd)
      if ((thd_sql_command(thd) == SQLCOM_CREATE_TABLE ||
            thd_sql_command(thd) == SQLCOM_ALTER_TABLE) &&
          key_algorithm == KEY_ALGORITHM_NONE)
-        key_algorithm= KEY_ALGORITHM_55;
+        key_algorithm= PARTITION_INFO_DEFAULT_ALGORITHM;
    }
    DBUG_RETURN(FALSE);
  }
@@ -2344,7 +2344,7 @@ bool partition_info::fix_parser_data(THD *thd)
    if ((thd_sql_command(thd) == SQLCOM_CREATE_TABLE ||
          thd_sql_command(thd) == SQLCOM_ALTER_TABLE) &&
        key_algorithm == KEY_ALGORITHM_NONE)
-      key_algorithm= KEY_ALGORITHM_55;
+      key_algorithm= PARTITION_INFO_DEFAULT_ALGORITHM;
  }
  defined_max_value= FALSE; // in case it already set (CREATE TABLE LIKE)
  do
modified  sql/partition_info.h
@@ -446,6 +446,8 @@ class partition_info : public DDL_LOG_STATE, public Sql_alloc
  int gen_part_type(THD *thd, String *str) const;
};

+#define PARTITION_INFO_DEFAULT_ALGORITHM partition_info::KEY_ALGORITHM_CRC32C
+
void part_type_error(THD *thd, partition_info *work_part_info,
                      const char *part_type, partition_info *tab_part_info);

modified  sql/sql_partition.cc
@@ -2471,7 +2471,7 @@ static int add_key_with_algorithm(String *str, const partition_info *part_info)
  err+= str->append(STRING_WITH_LEN("KEY "));

  if (part_info->key_algorithm != partition_info::KEY_ALGORITHM_NONE &&
-      part_info->key_algorithm != partition_info::KEY_ALGORITHM_55)
+      part_info->key_algorithm != PARTITION_INFO_DEFAULT_ALGORITHM)
  {
    err+= str->append(STRING_WITH_LEN("ALGORITHM = "));
    switch (part_info->key_algorithm)
@@ -2479,6 +2479,9 @@ static int add_key_with_algorithm(String *str, const partition_info *part_info)
      case partition_info::KEY_ALGORITHM_51:
        err+= str->append(STRING_WITH_LEN("MYSQL51"));
        break;
+      case partition_info::KEY_ALGORITHM_55:
+        err+= str->append(STRING_WITH_LEN("MYSQL55"));
+        break;
      case partition_info::KEY_ALGORITHM_BASE31:
        err+= str->append(STRING_WITH_LEN("BASE31"));
        break;