Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Andrei Elkin
Partial all but 3 hunks revert has_/lock/unlock- temporary_tables hunks of temporary_tables.cc.
Andrei Elkin
Revert "MDEV-37686 rpl.create_or_replace_mix2 fails in MDEV-35915 branch"

This reverts commit 3241798214b066d62ba3274ba5dc29549349ca65.
Due to MDEV-38212.
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

fts_optimize_table() : Assigns thd to transaction even it is
called via user_thread or fulltext optimize thread.
Alexander Barkov
MDEV-38249 Refactoring: Change Item_save_in_value() not to return SQL NULL

Fixing the return type of Type_handler::Item_save_in_value() from
bool to void.

Adding a new method st_value::is_null().

This makes the code less confusing.
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Add FTSQueryExecutor class as a thin abstraction over QueryExecutor.
This class takes care of open, lock, read, insert, delete
for all auxiliary tables INDEX_[1..6], common FTS tables
(DELETED, DELETED_CACHE, BEING_DELETED, CONFIG..)

FTSQueryExecutor Class which has the following function:
Auxiliary table functions : insert_aux_record(), delete_aux_record(),
    read_aux(), read_aux_all()

FTS common table functions : insert_common_record(), delete_common_record(),
    delete_all_common_records(), read_all_common()

FTS CONFIG table functions : insert_config_record(), update_config_record(),
                            delete_config_record(), read_config_with_lock()
Nikita Malyavin
MDEV-38151 GTT: missing FUNCTION support

open_temporary_table[s] defaulted to searching only local temporary
tables (Tmp_table_kind::TMP). When in function, a temporary table was
"carefully re-opened": it was closed and then opened again, with
open_temporary_table. This resulted in "table not found".

The correct key in most cases is Tmp_table_kind::ANY.
In some cases it should be Tmp_table_kind::TMP:
* When operation is not supported for GTT (ANALYZE/REPAIR)
* When a global handle should be opened (CREATE VIEW)
* If it's a re-open

Apart from this bug, it caused a global temporary table to be always
opened through open_tables, meaning that tdc was queried first. This
means that the earlier code never relied on pre-opening
temporary tables.

This means that all the commands that follow the default
open_temporary_tables path in mysql_execute_command (the standard
temporary tables pre-opening) should be able to handle child GTT share.

This wasn't so for TRUNCATE, hence the changes in sql_truncate.cc.

Also, this broke the lookup order: first a local temporary table should
be looked up. If it doesn't exist, only then a global temporary table
should be looked up.
To fix the latter -- push back global temporary tables' local shares and
push front local temporary tables.

To support push_front, All_tmp_tables_list declaration is changed. It
should use I_P_List_fast_push_back adapter, which adds T **m_last to the
list body.
Marko Mäkelä
Cleanup: Merge recv_recovery_read_checkpoint() to srv_start()
Nikita Malyavin
cleanup: register global_temporary_tp handlerton only ON COMMIT DELETE
Rucha Deodhar
MDEV-37950: INSERT ... RETURNING exposes columns for which the user lacks
SELECT privilege

Analysis:
When setup_fields() is called, the want_privilege is set to NO_ACL, so
correct priveleges are not checked.
Fix:
Since RETURNING requires SELECT_ACL privelige, when we are setting up
the returning fields for the given query, set want_privilege to SELECT_ACL.
Reset to original value of want_privilege once done.
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Refactor fetch, optimize to QueryExecutor and
standardize processor API. Replaced legacy SQL-graph
paths with QueryExecutor-based reads/writes:
fts_query code now uses QueryExecutor::read(), read_by_index()
with RecordCallback (updating fts_query_match_document(),
fts_query_is_in_proximity_range(), and fts_expand_query() to call
fts_query_fetch_document() instead of fts_doc_fetch_by_doc_id(),
which was removed along with FTS_FETCH_DOC_BY_DOC_ID_* macros);

Rewrote fts_optimize_write_word() to delete (or) insert via
FTSQueryExecutor::delete_aux_record()/insert_aux_record()
using fts_aux_data_t;
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Introduce QueryExecutor to perform direct InnoDB record scans
with a callback interface and consistent-read handling.
Also handles basic DML operation on clustered index of the table

Newly Added file row0query.h & row0query.cc

QueryExecutor class the following apis
read(): iterate clustered index with RecordCallback
read_by_index(): scan secondary index and fetch clustered row
lookup_clustered_record(): resolve PK from secondary rec
process_record_with_mvcc(): build version via read view
and skip deletes
insert_record(): Insert tuple into table's clustered index
select_for_update(): Lock the record which matches with search_tuple
update_record():  Update the currently selected and X-locked
clustered record.
delete_record():  Delete the clustered record identified by tuple
delete_all(): Delete all clustered records in the table
replace_record(): Tries update via select_for_update() +
  update_record(); if not found, runs insert_record.
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Introduce AuxCompareMode and AuxRecordReader to scan FTS auxiliary
indexes with compare+process callbacks.

Replace legacy SQL-graph APIs with typed executor-based ones:
-Add fts_index_fetch_nodes(trx, index, word, user_arg,
                          FTSRecordProcessor,compare_mode).
-Redefine fts_write_node() to use FTSQueryExecutor and fts_aux_data_t.

Implement write path via delete_aux_record (or) insert_aux_record.
Keep lock-wait retry handling and memory limit checks.

Change fts_select_index{,_by_range,_by_hash} return type
from ulint to uint8_t and simplify return flow.
Include fts0exec.h in fts0priv.h and update declarations accordingly.
Marko Mäkelä
WIP MDEV-37949: Implement innodb_log_archive, innodb_lsn_archived

TODO: Make innodb_log_archive=ON writes fully work!

TODO: Rename log files and set innodb_lsn_archived on checkpoint.

TODO: Implement recovery from archived log.

The new setting SET GLOBAL innodb_log_archive=ON will enable log
archiving as soon as the current ib_logfile0 is about to wrap around.

The read-only variable innodb_lsn_archived will reflect the LSN
since when a complete InnoDB log archive is available. If it is
0 at startup, it will be recovered from the log files. If
innodb_log_archive=OFF, this variable will be adjusted to the
latest checkpoint every time a log checkpoint is executed.
If innodb_log_archive=ON, the value should not change.

When innodb_log_archive=ON, the setting SET GLOBAL innodb_log_file_size
will affect subsequently created log files when the file that is being
currently written is running out.

log_t::archive_new_write(): Create and allocate a new log file,
and write the outstanding data to both the current and the new file.

log_t::archive_new_mmap(): Create and memory-map a new log file,
and update file_size to resize_target.

log_t::archive_set_size(): Ensure that resize_target is set for
new archived log files.

log_t::first_lsn: This should be the start of the current log file,
to be consulted in log_t::write_checkpoint() when renaming files.

log_write_buf(): Add the parameter max_length, the file wrap limit.

mtr_t::finish_writer(): Specialize for innodb_log_archive=ON

innodb_log_archive_update(): Assign log_sys.archived_lsn to the latest
checkpoint; it should be usable. If the log buffer is about to wrap
around, back off and retry.

log_t::archived_mmap_switch_prepare(): Create and memory-map a new log
file. TODO: Do we need a separate function from append_prepare_wait()?

log_t::append_prepare<log_t::ARCHIVED_MMAP>(): Special case.

log_t::archived_mmap_switch_complete(): Switch to the buffer that was
created in archived_mmap_switch_prepare().

mtr_t::finish_writer(): Invoke log_close() only if innodb_log_archive=OFF.
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Introduce ConfigReader callback to extract key, value from
fulltext config common table (CONFIG). This table has
<key, value> schema.

Simplifield all function which uses CONFIG tale. These
functions uses executor.insert_config_record(), update_config_record()
instead of SQL or query graph.
bsrikanth-mariadb
MDEV-36523: Load basic stats from trace into optimizer

This task loads the stats of the tables that are used in a query,
from the trace into the optimizer. This feature is also controlled
by optimizer_record_context, and is not enabled by default.
The stats such as num_of_records present in the table, indexes if present,
their names, along with the average number of records_per_key
with in each index are loaded from the trace.
Additionally, stats from range analysis i.e. ranges, and the
corresponding number of records are also loaded from the trace.

The trace context which is in JSON format is firstly set into the
session variable optimizer_trace_stored_context.
Later, when a user issues a query, the contents of this variable is
parsed and an in memory representation is built using the class
Optimizer_context_replay. This class is then used by the optimizer
to update and save original values of the tables, indexes, and range stats
in the methods "set_statistics_for_table()" of sql_statistics.cc, and
"check_quick_select()" of opt_range.cc.
After the query gets finished, the statistics that were updated
in the optimizer are restored back to the saved original values.

The entry point for parsing the json structure is in
"mysql_execute_command()" of sql_parse.cc, and similarly exit point
i.e. to restore the saved stats of the optimizer is at the end
of the same method.

Two new warning/error messages are introduced:
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_PARSE_FAILED, and
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_MATCH_FAILED
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Refactor fetch, optimize to QueryExecutor and
standardize processor API. Replaced legacy SQL-graph
paths with QueryExecutor-based reads/writes:
fts_query code now uses QueryExecutor::read(), read_by_index()
with RecordCallback (updating fts_query_match_document(),
fts_query_is_in_proximity_range(), and fts_expand_query() to call
fts_query_fetch_document() instead of fts_doc_fetch_by_doc_id(),
which was removed along with FTS_FETCH_DOC_BY_DOC_ID_* macros);

Rewrote fts_optimize_write_word() to delete (or) insert via
FTSQueryExecutor::delete_aux_record()/insert_aux_record()
using fts_aux_data_t;
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

- Removed fts0sql.cc file.
- Removed commented fts funtions
- Removed fts_table_t from fts_query_t and fts_optimize_t
Kristian Nielsen
Fix hang on master when disabling semi-sync

There is a global variable global_ack_signal_fd used to signal the receiver
thread to wake up when disabling semi-sync. This variable was cleared to -1
in the Ack_listener destructor, which ran at the end of the
Ack_receiver::run() function without any locking. If the thread was delayed
at that point, it could end up overwriting the new value set by a new
receiver thread. This would leave the server in a state with an invalid
global_ack_signal_fd and could cause a subsequent disable of semisync to
fail due to the wakeup not arriving at the receiver thread.

This was seen as a sporadic failure of the test case
rpl.rpl_semi_sync_cond_var_per_thd.

Fix by not modifying the global in constructor/destructor; instead set and
clear the global explicitly, allowing to clear the fd with proper locking
while the mutex is still being held.

Also fix a missing pthread_join(), which would leak thread descriptors and
allow to start a new receiver thread before the old one shut down fully.

(Either of these two changes fix the hang bug).

Signed-off-by: Kristian Nielsen <[email protected]>
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Introduce ConfigReader callback to extract key, value from
fulltext config common table (CONFIG). This table has
<key, value> schema.

Simplifield all function which uses CONFIG tale. These
functions uses executor.insert_config_record(), update_config_record()
instead of SQL or query graph.
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Introduce QueryExecutor to perform direct InnoDB record scans
with a callback interface and consistent-read handling.
Also handles basic DML operation on clustered index of the table

Newly Added file row0query.h & row0query.cc

QueryExecutor class the following apis
read(): iterate clustered index with RecordCallback
read_by_index(): scan secondary index and fetch clustered row
lookup_clustered_record(): resolve PK from secondary rec
process_record_with_mvcc(): build version via read view
and skip deletes
insert_record(): Insert tuple into table's clustered index
select_for_update(): Lock the record which matches with search_tuple
update_record():  Update the currently selected and X-locked
clustered record.
delete_record():  Delete the clustered record identified by tuple
delete_all(): Delete all clustered records in the table
replace_record(): Tries update via select_for_update() +
  update_record(); if not found, runs insert_record.
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Add FTSQueryExecutor class as a thin abstraction over QueryExecutor.
This class takes care of open, lock, read, insert, delete
for all auxiliary tables INDEX_[1..6], common FTS tables
(DELETED, DELETED_CACHE, BEING_DELETED, CONFIG..)

FTSQueryExecutor Class which has the following function:
Auxiliary table functions : insert_aux_record(), delete_aux_record(),
    read_aux(), read_aux_all()

FTS common table functions : insert_common_record(), delete_common_record(),
    delete_all_common_records(), read_all_common()

FTS CONFIG table functions : insert_config_record(), update_config_record(),
                            delete_config_record(), read_config_with_lock()
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

- Removed fts0sql.cc file.
- Removed commented fts funtions
- Removed fts_table_t from fts_query_t and fts_optimize_t
Alexander Barkov
FETCH anchored_cursor_with_return INTO var1,var2,var3
Kristian Nielsen
Semi-sync: Clean up incorrect file/pos comparisons

The semi-sync code has a number of places where it compares pairs of
(filename,offset) for which is larger than the other.

The filename comparisons are done using strcmp(), which is wrong. Filenames
will compare wrong when they wrap from eg. bin-999999 to bin-1000000, and
user can also rename log files which can likewise break comparisons.

Further, the comparisons are completely unnecessary, as all the transactions
to be waited for are already stored in a linear list in order _and_ in a
hash table. So the code can simply use the existing hash table look and list
traversal to determine status and sequence of the waited-for transactions.

So this patch removes all the comparisons for larger/smaller, leaving only
comparisons for equality and effectively making the (filename,offset) pairs
just opaque transaction identifiers. And also removes a few other related
pieces of dead/unnecessary code.

Signed-off-by: Kristian Nielsen <[email protected]>
bsrikanth-mariadb
MDEV-36523: Load basic stats from trace into optimizer

This task loads the stats of the tables that are used in a query,
from the trace into the optimizer. This feature is also controlled
by optimizer_record_context, and is not enabled by default.
The stats such as num_of_records present in the table, indexes if present,
their names, along with the average number of records_per_key
with in each index are loaded from the trace.
Additionally, stats from range analysis i.e. ranges, and the
corresponding number of records are also loaded from the trace.

The trace context which is in JSON format is firstly set into the
session variable optimizer_trace_stored_context.
Later, when a user issues a query, the contents of this variable is
parsed and an in memory representation is built using the class
Optimizer_context_replay. This class is then used by the optimizer
to update and save original values of the tables, indexes, and range stats
in the methods "set_statistics_for_table()" of sql_statistics.cc, and
"check_quick_select()" of opt_range.cc.
After the query gets finished, the statistics that were updated
in the optimizer are restored back to the saved original values.

The entry point for parsing the json structure is in
"mysql_execute_command()" of sql_parse.cc, and similarly exit point
i.e. to restore the saved stats of the optimizer is at the end
of the same method.

Two new warning/error messages are introduced:
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_PARSE_FAILED, and
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_MATCH_FAILED
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Introduce AuxCompareMode and AuxRecordReader to scan FTS auxiliary
indexes with compare+process callbacks.

Replace legacy SQL-graph APIs with typed executor-based ones:
-Add fts_index_fetch_nodes(trx, index, word, user_arg,
                          FTSRecordProcessor,compare_mode).
-Redefine fts_write_node() to use FTSQueryExecutor and fts_aux_data_t.

Implement write path via delete_aux_record (or) insert_aux_record.
Keep lock-wait retry handling and memory limit checks.

Change fts_select_index{,_by_range,_by_hash} return type
from ulint to uint8_t and simplify return flow.
Include fts0exec.h in fts0priv.h and update declarations accordingly.
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

fts_optimize_table() : Assigns thd to transaction even it is
called via user_thread or fulltext optimize thread.
Alexander Barkov
FETCH anchored_cursor_with_return INTO var1,var2,var3
Nikita Malyavin
MDEV-38151 GTT: missing FUNCTION support

open_temporary_tables defaulted to searching only local temporary tables
(Tmp_table_kind::TMP), which is wrong -- apart from this bug, it caused
  a global temporary table to be always opened through open_tables,
  meaning that tdc was queried first.

The correct key in most cases is Tmp_table_kind::ANY.
In some cases it should be Tmp_table_kind::TMP:
* When operation is not supported for GTT (ANALYZE/REPAIR)
* When a global handle should be opened (CREATE VIEW)
* If it's a re-open

This means that all the commands that follow the default
open_temporary_tables path in mysql_execute_command (the standard
temporary tables pre-opening) should be able to handle child GTT share.

This wasn't so for TRUNCATE, hence the changes in sql_truncate.cc.

Also, this broke the lookup order: first a local temporary table should
be looked up. If it doesn't exist, only then a global temporary table
should be looked up.
To fix the latter -- push back global temporary tables' local shares and
push front local temporary tables.

To support push_front, All_tmp_tables_list declaration is changed. It
should use I_P_List_fast_push_back adapter, which adds T **m_last to the
list body.
bsrikanth-mariadb
MDEV-36523: Load basic stats from trace into optimizer

This task loads the stats of the tables that are used in a query,
from the trace into the optimizer. This feature is also controlled
by optimizer_record_context, and is not enabled by default.
The stats such as num_of_records present in the table, indexes if present,
their names, along with the average number of records_per_key
with in each index are loaded from the trace.
Additionally, stats from range analysis i.e. ranges, and the
corresponding number of records are also loaded from the trace.

The trace context which is in JSON format is firstly set into the
session variable optimizer_trace_stored_context.
Later, when a user issues a query, the contents of this variable is
parsed and an in memory representation is built using the class
Optimizer_context_replay. This class is then used by the optimizer
to update and save original values of the tables, indexes, and range stats
in the methods "set_statistics_for_table()" of sql_statistics.cc, and
"check_quick_select()" of opt_range.cc.
After the query gets finished, the statistics that were updated
in the optimizer are restored back to the saved original values.

The entry point for parsing the json structure is in
"mysql_execute_command()" of sql_parse.cc, and similarly exit point
i.e. to restore the saved stats of the optimizer is at the end
of the same method.

Two new warning/error messages are introduced:
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_PARSE_FAILED, and
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_MATCH_FAILED
Yuchen Pei
MDEV-9826 Use proper XXH streaming functions
bsrikanth-mariadb
MDEV-36523: Load basic stats from trace into optimizer

This task loads the stats of the tables that are used in a query,
from the trace into the optimizer. This feature is controlled
by optimizer_replay_context, and points to an user defined variable name,
that has the context information.
The stats such as num_of_records present in the table, indexes if present,
their names, along with the average number of records_per_key
with in each index, cost of reading an index are loaded from the trace.
Additionally, stats from range analysis i.e. ranges, and the
corresponding number of records, along with their costs are also
loaded from the trace.

The trace context which is in JSON format is firstly set into the
user defined session variable, and this variable name is set to
optimizer_replay_context system variable.
Later, when a user issues a query, the contents of the json context is
parsed and an in memory representation is built using the class
Optimizer_context_replay. This class is then used by the optimizer
to update and save original values of the tables, indexes, and range stats
in the methods "set_statistics_for_table()" of sql_statistics.cc, and
"check_quick_select()" of opt_range.cc.
After the query gets finished, the statistics that were updated
in the optimizer are restored back to the saved original values.

The entry point for parsing the json structure is in
"mysql_execute_command()" of sql_parse.cc, and similarly exit point
i.e. to restore the saved stats of the optimizer is at the end
of the same method.

Two new warning/error messages are introduced:
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_PARSE_FAILED, and
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_MATCH_FAILED
Nikita Malyavin
MDEV-38151 GTT: missing FUNCTION support
Andrei Elkin
MDEV-38212 MDEV-37686 Breaks Parallel Replication

Re-fixing MDEV-37686.

NULLifying `THD::rgi_slave` pointer in `start_new_trans` ctor harmed
the parallel slave conflict detection.
Now instead of `NULL`ifying of this member a finer approach is taken to
optionally screen `THD::rgi_slave` when it is attempted to be accessed
within `start_new_trans` context, that is when such out-of-band
transaction is run by a slave thread.

The start_new_trans is allowed of course to access server local
non-replicated temporary tables, should it ever need that.

The original MDEV-37686 aspect is tested with 12.3 branch's
rpl.create_or_replace_mix2.
Any possible side effects to temporary table replication is controlled
by existing rpl suite tests.
There is no need for a specific mtr test to prove the parallel slave
side is back to normal, as `THD::rgi_slave` is again available to concurrent
transactions.
Alessandro Vetere
fixup MDEV-19123 Change default charset from latin1 to utf8mb4

Adjust the 128 indexes build test result.

Reviewed by: Alexander Barkov
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Introduce CommonTableReader callback to collect doc_id_t from
fulltext common tables (DELETED, BEING_DELETED, DELETED_CACHE,
BEING_DELETED_CACHE). These table share the same schema strucutre.

Simplified all function which uses DELETED, BEING_DELETED,
DELETED_CACHE, BEING_DELETED_CACHE table. These functions
uses executor.insert_common_record(), delete_common_record(),
delete_all_common_records() instead of SQL or query graph.

fts_table_fetch_doc_ids(): Changed the signature of the function
to pass the table name instead of fts_table_t.
Marko Mäkelä
MDEV-37949: Introduce innodb_log_recovery_start, innodb_log_recovery_target

innodb_log_recovery_start: The checkpoint LSN to start recovery from.
This will be useful when recovering from an archived log.

innodb_log_recovery_target: The requested LSN to end recovery at.
This will be useful when recovering data files that were copied
as of a time that is before end of the available log.
Thirunarayanan Balathandayuthapani
MDEV-28730 Remove internal parser usage from InnoDB fts

Introduce CommonTableReader callback to collect doc_id_t from
fulltext common tables (DELETED, BEING_DELETED, DELETED_CACHE,
BEING_DELETED_CACHE). These table share the same schema strucutre.

Simplified all function which uses DELETED, BEING_DELETED,
DELETED_CACHE, BEING_DELETED_CACHE table. These functions
uses executor.insert_common_record(), delete_common_record(),
delete_all_common_records() instead of SQL or query graph.

fts_table_fetch_doc_ids(): Changed the signature of the function
to pass the table name instead of fts_table_t.