Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Rucha Deodhar
MDEV-40127: Server hangs when setting NEW=OLD on a multi-row table

Analysis:
m_fields was keeping old fields around across executions instead of
being cleared, causing stale state and hangs/crashes on re-execution.

Fix:
Clear m_fields in cleanup().
Rucha Deodhar
MDEV-40124: Assertion `m_sp == m_thd->spcont->m_sp' failed virtual
Item *Item_splocal::this_item(), UBSAN : member call on null pointer of
type 'Sp_rcontext_handler' in sql/item.cc

Analysis:
Comparing trigger row objects (IF NEW = OLD) calls cmp_row_type(),
which uses element_index(i). Item_trigger_row was missing its own
element_index() override, falling back to Item_splocal's version
and crashing/asserting on local context lookup.

Fix:
Override element_index(i) in Item_trigger_row to return
m_fields.elem(i) directly.
Kristian Nielsen
Fix hang on master when disabling semi-sync

There is a global variable global_ack_signal_fd used to signal the receiver
thread to wake up when disabling semi-sync. This variable was cleared to -1
in the Ack_listener destructor, which ran at the end of the
Ack_receiver::run() function without any locking. If the thread was delayed
at that point, it could end up overwriting the new value set by a new
receiver thread. This would leave the server in a state with an invalid
global_ack_signal_fd and could cause a subsequent disable of semisync to
fail due to the wakeup not arriving at the receiver thread.

This was seen as a sporadic failure of the test case
rpl.rpl_semi_sync_cond_var_per_thd.

Fix by not modifying the global in constructor/destructor; instead set and
clear the global explicitly, allowing to clear the fd with proper locking
while the mutex is still being held.

Also fix a missing pthread_join(), which would leak thread descriptors and
allow to start a new receiver thread before the old one shut down fully.

(Either of these two changes fix the hang bug).

Signed-off-by: Kristian Nielsen <[email protected]>
Monty
fixup! 18bf6b1b16f8c55b0d7d75ec860c0cf0fb88b7be
Sergei Golubchik
fix errmsg-utf8.txt dependencies for Ninja generator

GenError's custom command must specify headers as OUTPUT,
otherwise ninja cannot deduce that mysqld.cc depends on errmsg-utf8.txt

As a bonus, BYPRODUCTS lists generated files for `ninja clean`
Marko Mäkelä
Suggested changes from Monty
Kristian Nielsen
Semi-sync: Implement support for using GTID in semi-sync ACKs

Implement the necessary logic in class Repl_semi_sync_master_gtid and
related code, so that the semi-sync master can request the slave to put the
GTID in the reply ACK packet, instead of the filename/offset.

In GTID mode, when a slave connects (which it should do using a GTID start
position), the latest GTID in the starting position (if any), as determined
by the list of transactions pending acks, is used as the point at which to
implicit ACK anything pending.

The semi-sync logic is otherwise unchanged, in GTID mode it just uses the
GTID as the transaction identifier instead of the file/pos of the end of the
event group.

This patch only enables the GTID-based semisync for binlog-in-engine, but
the mode works with old binlog implementation as well, and passes all tests
(if enabled by code change).

Signed-off-by: Kristian Nielsen <[email protected]>
Khaled Riyad
MDEV-40377 Change Server source code to point to new docs (11.4 part)

Replace the remaining Knowledge Base links with their MariaDB
Documentation equivalents, including the 838 URLs in the help tables.
Only URLs change in fill_help_tables.sql.

Merging upward: fill_help_tables.sql conflicts at 10.11->11.4 and
11.8->12.3. At those two merges keep the target branch's version,
since 11.4, 11.8 and 12.3 each carry their own URL fix. From 12.3 to
13.1 and 13.1 to main it merges cleanly; take the incoming change.
.github/pull_request_template.md is deleted in 12.3; keep the deletion.
Daniel Bartholomew
bump the VERSION
Oleksandr Byelkin
Fix memory leack in mysqltest
Kristian Nielsen
mysqltest: Implement --enable_sync_gtid option

The --enable_sync_gtid option switches to use GTID-based
--sync_slave_with_master (eg. using MASTER_GTID_WAIT() instead of
MASTER_POS_WAIT()).

This is useful to run adapt existing test cases for use with
--binlog-storage-engine. But it is also useful in general for replication
using GTID (which is the default).

The option is off by default to not randomly break existing tests.

Signed-off-by: Kristian Nielsen <[email protected]>
Kristian Nielsen
Implement support for semi-sync with --binlog-storage-engine

Call report_binlog_update() when semi-synchronous replication is enabled and
using --binlog-storage-engine.

Only AFTER_COMMIT is available. The AFTER_SYNC in legacy binlog has the
property that changes in a committing transaction does not get visible to
other transactions until the slave has acknowledged, avoiding phantom reads
if the master fails permanently just after. This requires the two-phase
commit between binlog and storage engine so that the binlog is written
before the transaction is engine-committed. However, the whole point of
--binlog-storage-engine is to avoid the expensive two-phase commit.

(The ability to avoid phantom reads could be later implemented as an
AFTER_PREPARE option, which would send binlog to slave and await ack before
it is written/committed into the engine).

Some existing semi-synchronous replication tests are adapted to also run in
the binlog_in_engine suite.

Signed-off-by: Kristian Nielsen <[email protected]>
bsrikanth-mariadb
MDEV-39226: Push whole multi-table update/delete down into engines

Give storage engines a way to take over an entire multi-table
UPDATE/DELETE, the way they can already take over a SELECT. Without it the
join, the row matching and every modification run in the SQL layer even
when an engine could do the whole statement itself in one step; a
single-table UPDATE/DELETE already avoids this via
direct_update_rows()/direct_delete_rows(), but a multi-table statement has
no primary handler object to drive that path.

This adds a generic, engine-agnostic pushdown interface: the SQL layer
offers the statement to the engine, and if the engine accepts it, it
performs the whole thing and reports only the row counts.

- Split select_handler into a pushdown_handler base with select_handler
  (result set) and a new multi_upddel_handler (runs a whole UPDATE/DELETE,
  reports row counts, reported as PUSHED UPDATE/PUSHED DELETE); add
  handlerton::create_multi_upddel, looked up in Sql_cmd_dml::execute_inner().
- multi_update/multi_delete gain direct_update_delete_done(), which records
  the engine's counts so send_eof() binlogs and replies without the
  SQL-layer loop; it forces statement-format binlogging so the change still
  replicates under binlog_format=ROW, and errors out instead of silently
  dropping counts for an unsupported result object.
- FederatedX implements the interface as the reference engine used to test
  correctness: it prints the statement back and runs it remotely, passes
  the engine's error code/SQLSTATE through, reads the matched count from the
  remote info string, executes IGNORE locally, and only pushes down when all
  tables share one remote server (same as SELECT/derived/unit pushdown).

Test: federated.federatedx_pushdown_upd_del.
Marko Mäkelä
MDEV-41152: Fix FILE_CREATE recovery

fil_name_process(): Treat FILE_CREATE in the same way as FILE_MODIFY
that led to a FIL_LOAD_DEFER return.
Khaled Riyad
MDEV-40377 Change Server source code to point to new docs (10.11 part)

Replace the remaining Knowledge Base links with their MariaDB
Documentation equivalents, and fix the 14 help table URLs pointing at
/README pages that do not exist. Only URLs change in
fill_help_tables.sql.

Merging upward: fill_help_tables.sql conflicts at 10.11->11.4 and
11.8->12.3. At those two merges keep the target branch's version,
since 11.4, 11.8 and 12.3 each carry their own URL fix. From 12.3 to
13.1 and 13.1 to main it merges cleanly; take the incoming change.
.github/pull_request_template.md is deleted in 12.3; keep the deletion.
Oleksandr Byelkin
fix typo
Monty
Added new mysys functions my_open_dir and improved my_copy

- Improved my_copy() using copy_file_range and memmap

- New mysys functions:
int my_copy_file(File from, File to, myf MyFlags);
int my_copy_file_range(File from, File to, my_off_t start,
                        my_off_t end, myf MyFlags);

- New functions for looping over files in a directory:
MY_NO_CACHE_DIR *my_dir_open();
int my_dir_read_next()
int my_dir_rewind();
int my_dir_close();

Other things
- Fixed #ifdef's in sql_backup.cc to use the new define
  HAVE_COPY_FILE_RANGE
Rucha Deodhar
MDEV-34723: NEW and OLD in a trigger as row variables

Implementation:
NEW and OLD represent the entire table row. So it can be thought of as
list of Item_trigger_field. When we are in a trigger and NEW or OLD is
encountered, create Item_trigger_row object with same constructor as
Item_trigger_field, it will also be used later while creating
Item_trigger_field objects. Populate the m_fields list while
fixing fields. Create a corresponding instruction sp_instr_set_trigger_row
which will be used to set the values
Marko Mäkelä
MDEV-41152: Fix FILE_CREATE recovery

fil_name_process(): Treat FILE_CREATE in the same way as FILE_MODIFY
that led to a FIL_LOAD_DEFER return.

(cherry picked from commit a898d2b1ae1719fdbd11c4221b9a9ae8ec5334a6)
Rucha Deodhar
MDEV-40478: main.func_json_unicode_escape test unpredicatable under
view protocol

Analysis:
The function with arguments is too long for view column name. So view
protocol just renames it.
Fix:
Use alias to avoid renaming.
Monty
fixup! 18bf6b1b16f8c55b0d7d75ec860c0cf0fb88b7be
Kristian Nielsen
Semi-sync: Some few after-review fixes

Signed-off-by: Kristian Nielsen <[email protected]>
Monty
fixup! 5bd9778829a151239c8b96136bc05e5af6652b22
Kristian Nielsen
Semi-sync: Refactor in preparation for using GTID in semi-sync acks

This is a refactor patch that contains no/little logic changes but a lot of
mostly mechanic code changes to prepare for allowing to use either old-style
filename/offset or new-style GTID to identify an event group in the
semi-sync ack.

The idea is to replace all explicit filename/offset function arguments with
a generic Repl_semi_sync_trx_info *inf to identify an event group (aka
"transaction"). This object can then be used to look up in the semi-sync
hash table by either file/pos or by GTID.

The classes Active_tranx and Repl_semi_sync_master are sub-classed into
Active_tranx_file_pos/Active_tranx_gtid and
Repl_semi_sync_master_file_pos/Repl_semi_sync_master_gtid. Virtual functions
are implemented in each for comparing identifiers and for calculating hash
keys, using either the file/pos or the GTID as appropriate. This way, the
existing logic can now be used with either (only file/pos is actually used
in this patch; adding GTID is for a subsequent patch).

For the binlog writing side, report_binlog_update(), wait_after_sync(), and
THD::semisync_info are extended to also take the GTID of the event group,
which is already available in the calling code.

For the dump thread / slave connection side, update_sync_header() is
extended to take also the GTID. The dump thread code is extended to keep
track of the GTID of the current event group (slightly extending the logic
already there to keep track of event groups). Also, the dump thread now only
passes the last event of an event group into the semisync layer (the other
events are redundant, as they are never semi-sync ack'ed, saving needless
semisync locking and hash lookups).

Signed-off-by: Kristian Nielsen <[email protected]>
Rucha Deodhar
MDEV-34723: NEW and OLD in a trigger as row variables

Implementation:
NEW and OLD represent the entire table row. So it can be thought of as
list of Item_trigger_field. When we are in a trigger and NEW or OLD is
encountered, create Item_trigger_row object with same constructor as
Item_trigger_field, it will also be used later while creating
Item_trigger_field objects. Populate the m_fields list while
fixing fields. Create a corresponding instruction sp_instr_set_trigger_row
which will be used to set the values
Alexander Barkov
MDEV-39563 Implement UPDATE ... RETURNING ... INTO

Adding support for UPDATE .. RETURNING .. INTO queries.

For example:

  UPDATE t1 SET a=10,b=20 RETURNING a,b INTO va,vb;
  UPDATE t1 SET a=10,b=20 RETURNING a,b INTO @a,@b;

Limitations:
1. These types of queries:
  - REPLACE .. RETURNING .. INTO
  - DELETE .. RETURNING .. INTO
  - INSERT .. RETURNING .. INTO
  do not work - they return an error.
  They will be implemented separately, when needed.

2. UPDATE..RETURNING..INTO with --binlog_format=statement is not allowed
  and an error is raised.

3. Using OLD_VALUE(col) inside UPDATE..RETURNING..INTO is not allowed
  and an error is raised.

4. Multi-table updates, as well as single table updates with a subquery
  to the same table in WHERE (which get converted to multi-table) do
  not work and an error is raised.

Notes:

1. ANALYZE and EXPLAIN
  Both
    ANALYZE UPDATE .. RETURNING .. INTO ..
    EXPLAIN UPDATE .. RETURNING .. INTO ..
  return this error:
    'RETURNING..INTO' is not allowed in this context

2. Behavior on no data

  a. In case of degenerated plans (WHERE 1=0, LIMIT 0),
    no errors are raised.

  b. If the updated table contains no rows, the behavior depends on the engine,
    for example:
    - MyISAM returns no errors
    - InnoDB raises
        No data - zero rows fetched, selected, or processed
    This behavior is engine dependent because some engines (e.g. MyISAM)
    quickly know that the table has no records and execute the statement
    using a degenerated plan.

  c. If there are some rows, but non of them match the WHERE condition,
    then this error is raised:
      No data - zero rows fetched, selected, or processed

  d. If some rows where found but none of them actually
    got changed by the SET, still this error is raised:
      No data - zero rows fetched, selected, or processed
    The error message might be misleading. However, if we read
    it as "zero rows [that required updates] fetched", it looks OK.
    Let's not introduce a new error message for now.

Helper changes:

1. The grammar in analyze_stmt_command was changed to have
  LEX::analyze_stmt set to true earlier, so
  LEX::set_returning_into_result() already knows if this
  is an ANALYZE statement.

2. The Sql_cmd_update constructor is now called earlier in the grammar,
  to be able to call Sql_cmd_update::set_with_old_value_items()
  in the SET and RETURNING clauses.

3. Sql_cmd_dml::lex is now set during the constructor time.
  It makes things easier:
  - Sql_cmd_update::returns_result_set() needs the lex.
  - Sql_cmd_delete::orig_multitable and Sql_cmd_update::orig_multitable
    are not needed any more.
    They were used only in Sql_cmd_delete::sql_command_code() and
    Sql_cmd_update::sql_command_code().
    Sql_cmd_dml::sql_command_code() now returns lex->sql_command.
    The overrides Sql_cmd_delete::sql_command_code() and
    Sql_cmd_update::sql_command_code() were removed.
Rucha Deodhar
MDEV-40124: Assertion `m_sp == m_thd->spcont->m_sp' failed virtual
Item *Item_splocal::this_item(), UBSAN : member call on null pointer of
type 'Sp_rcontext_handler' in sql/item.cc

Analysis:
Comparing trigger row objects (IF NEW = OLD) calls cmp_row_type(),
which uses element_index(i). Item_trigger_row was missing its own
element_index() override, falling back to Item_splocal's version
and crashing/asserting on local context lookup.

Fix:
Override element_index(i) in Item_trigger_row to return
m_fields.elem(i) directly.
Rex Johnston
MDEV-38801  implement Item_cache_year shallow_copy()

Implement shallow_copy() for Item_cache_year, same as for Item_cache_bool
to avoid typeid mismatch from inheriting this method from Item_cache_int.
Triggered when creating a clone of a condition for pushdown into a
derived table.

(Testcase amended by Sergei Petrunia)
Monty
Added new mysys functions my_open_dir and improved my_copy

- Improved my_copy() using copy_file_range and memmap

- New mysys functions:
int my_copy_file(File from, File to, myf MyFlags);
int my_copy_file_range(File from, File to, my_off_t start,
                        my_off_t end, myf MyFlags);

- New functions for looping over files in a directory:
MY_NO_CACHE_DIR *my_dir_open();
int my_dir_read_next()
int my_dir_rewind();
int my_dir_close();

Other things
- Fixed #ifdef's in sql_backup.cc to use the new define
  HAVE_COPY_FILE_RANGE
Rucha Deodhar
MDEV-40124: Assertion `m_sp == m_thd->spcont->m_sp' failed virtual
Item *Item_splocal::this_item(), UBSAN : member call on null pointer of
type 'Sp_rcontext_handler' in sql/item.cc

Analysis:
Comparing trigger row objects (IF NEW = OLD) calls cmp_row_type(),
which uses element_index(i). Item_trigger_row was missing its own
element_index() override, falling back to Item_splocal's version
and crashing/asserting on local context lookup.

Fix:
Override element_index(i) in Item_trigger_row to return
m_fields.elem(i) directly.
Khaled Riyad
MDEV-40377 Change Server source code to point to new docs (11.8 part)

Replace the remaining Knowledge Base links with their MariaDB
Documentation equivalents, including the 838 URLs in the help tables.
Only URLs change in fill_help_tables.sql.

Merging upward: fill_help_tables.sql conflicts at 10.11->11.4 and
11.8->12.3. At those two merges keep the target branch's version,
since 11.4, 11.8 and 12.3 each carry their own URL fix. From 12.3 to
13.1 and 13.1 to main it merges cleanly; take the incoming change.
.github/pull_request_template.md is deleted in 12.3; keep the deletion.
Oleksandr Byelkin
Fix compiler pronlems
Monty
fixup! 5bd9778829a151239c8b96136bc05e5af6652b22
Kristian Nielsen
Semi-sync: Clean up incorrect file/pos comparisons

The semi-sync code has a number of places where it compares pairs of
(filename,offset) for which is larger than the other.

The filename comparisons are done using strcmp(), which is wrong. Filenames
will compare wrong when they wrap from eg. bin-999999 to bin-1000000, and
user can also rename log files which can likewise break comparisons.

Further, the comparisons are completely unnecessary, as all the transactions
to be waited for are already stored in a linear list in order _and_ in a
hash table. So the code can simply use the existing hash table look and list
traversal to determine status and sequence of the waited-for transactions.

So this patch removes all the comparisons for larger/smaller, leaving only
comparisons for equality and effectively making the (filename,offset) pairs
just opaque transaction identifiers. And also removes a few other related
pieces of dead/unnecessary code.

Signed-off-by: Kristian Nielsen <[email protected]>
Marko Mäkelä
WIP: log tracking BACKUP SERVER TO ... CONCURRENT (for HAVE_INNODB_PMEM)

backup_sink::id: The thread identifier (0 to CONCURRENT-1)

innodb_backup_checkpoint_pmem(): Copy the old log file.

InnoDB_backup::log_track(), InnoDB_backup::log_track_pmem():
Keep copying the log until we run out of InnoDB data files to copy.

InnoDB_backup::checkpoint_complete_pmem(): Copy the remaining
part of an old log file right before it is being released.

InnoDB_backup::commit(): In log tracking backup, copy the rest of
the HAVE_INNODB_PMEM log.

FIXME: Implement the non-PMEM code path with minimal blocking.
Monty
Integrate the old and new backup vofr

Fixes a lot of issues in current backup:
- Galera
- Enables ddl logging (so we can use it in the future)
- Flushes binary logs (we still must add code to copy them)
- mdl locks are consistent between maria-backup and backup command
- startup backup code for innodb moved to innodb_prepare_for_backup() called by prepare_for_backup hton handler.
- Give errors if backup command is done under a transaction, global read lock or lock tables.
- Retry for MDL_BACKUP_WAIT_DDL (needed for backup.backup_ddl_concurrent_verify)
- Removed wrong log locks in Aria

Things to do (in addition to the things in my review) :

At backup_stage stage start, force rotate of aria log files. This allows us to copy all old logs without any locks
Copy all transactional tables and old aria logs under BACKUP_START (as maria-backup does)
Copy the active aria log file under block commit.
Improve speed of copying aria tables by copy files in up to 1M blocks and run checksum on the blocks and only re-read blocks with wrong checksum.
Copy non transactional files under BACKUP_PHASE_NO_BEGIN_NON_TRANS.
 Note that Aria does not support the documented BACKUP_PHASE_NO_DML_NON_TRANS . The BACKUP_PHASE_NO_BEGIN_NON_TRANS state is already blocking changes to non transactional tables
Monty
Update backup code to use new my_dir and my_copy interfaces
Rucha Deodhar
MDEV-40127: Server hangs when setting NEW=OLD on a multi-row table

Analysis:
m_fields was keeping old fields around across executions instead of
being cleared, causing stale state and hangs/crashes on re-execution.

Fix:
Clear m_fields in cleanup().
Rucha Deodhar
MDEV-40127: Server hangs when setting NEW=OLD on a multi-row table

Analysis:
m_fields was keeping old fields around across executions instead of
being cleared, causing stale state and hangs/crashes on re-execution.

Fix:
Clear m_fields in cleanup().
Rucha Deodhar
MDEV-34723: NEW and OLD in a trigger as row variables

Implementation:
NEW and OLD represent the entire table row. So it can be thought of as
list of Item_trigger_field. When we are in a trigger and NEW or OLD is
encountered, create Item_trigger_row object with same constructor as
Item_trigger_field, it will also be used later while creating
Item_trigger_field objects. Populate the m_fields list while
fixing fields. Create a corresponding instruction sp_instr_set_trigger_row
which will be used to set the values