Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Marko Mäkelä
MDEV-38126: Use InnoDB srw_lock_low for plugin/server_audit

Replace the contention prone mysql_prlock_t with a simpler
rw-lock wrapper from InnoDB, and fix several potential
race conditions by making consistent use of lock_operations.

This will also omit any PERFORMANCE_SCHEMA instrumentation.
Monty
MDEV-24 Segmented key cache for Aria

Added option 'aria-pagecache-segments', default 1.

For values > 1, this split the aria-pagecache-buffer into the given
number of segments, each independent from each other.  Having multiple
pagecaches improve performance when multiple connections runs queries
concurrently using different tables.

Each pagecache will use aria-pageache-buffer/segments amount of
memory, however at least 128K.

Each opened table has its index and data file use the segments in a
a round-robin fashion.

Internal changes:
- All programs allocating the maria pagecache themselves should now
  call multi_init_pagecache() instead of init_pagecache().
- pagecache statistics is now stored in 'pagecache_stats' instead of
  maria_pagecache. One must call multi_update_pagecache_stats() to
  update the statistics.
- Added into PAGECACHE_FILE a pointer to files pagecache. This was
  done to ensure that index and data file are using the same
  pagecache and simplified the checkpoint code.
  I kept pagecache in TABLE_SHARE to minimize the changes.
- really_execute_checkpoint() was update to handle a dynamic number of
  pagecaches.
- pagecache_collect_changed_blocks_with_lsn() was slight changed to
  allow it to be called for each pagecache.
- undefined not used functions maria_assign_pagecache() and
  maria_change_pagecache()
- ma_pagecaches.c is totally rewritten. It now contains all multi_pagecache
  functions.

Errors found be QA that are fixed:
MDEV-36872 UBSAN errors in ma_checkpoint.c
MDEV-36874 Behavior upon too small aria_pagecache_buffer_size in case of
          multiple segments is not very user-friendly
Sergey Vojtovich
MDL_ticket cleanup

Removed useless MDL_ticket::create() and MDL_ticket::destroy()
indirection. It didn't serve any purpose.

Moved mysql_mdl_create() PFS call out of critical section.

Moved ticket->m_time assignment out of MDL_lock::m_rwlock protection.
It increases critical section size for no good reason. Assigning it
before critical section must be good enough for statistics purposes,
since we must not do long waits here anyway.

This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Dave Gosselin
MDEV-37933: Rewrite [NATURAL] FULL OUTER to LEFT, RIGHT, or INNER JOIN

Rewrite FULL OUTER JOIN queries as either LEFT, RIGHT, or INNER JOIN
by checking if and how the WHERE clause rejects nulls.

For example, the following two queries are equivalent because the
WHERE condition rejects nulls from the left table and allows matches
in the right table (or NULL from the right table) for the remaining
rows:

  SELECT * FROM t1 FULL JOIN t2 ON t1.v = t2.v WHERE t1.v IS NOT NULL;
  SELECT * FROM t1 LEFT JOIN t2 ON t1.v = t2.v;
Vladislav Vaintroub
MDEV-38029 my_tzinfo-t fails for certain TZ values on musl

The test fails for TZ values such as `PST8PDT` (present but outdated in
tzdb) and custom forms like `GST-1GDT`. On musl, these values do not
trigger the expected DST transitions, leading to incorrect DST offsets
or abbreviations.

This appears to be a musl libc bug; the same TZ values behave correctly
elsewhere, including Windows. We work around it by skipping the
affected tests when musl is detected.
Sergey Vojtovich
Move m_wait.reset_status() out of critical section

MDL_wait::m_wait_status has to be reset to EMPTY state before going
into waiting routines. There're three options to get it done:

1. Before MDL_lock::rw_lock critical section of
  MDL_context::acquire_lock(). In this case MDL_context is not exposed
  neither via MDL_lock::m_waiting nor has it MDL_context::m_waiting_for
  set.
  Cons: m_wait.reset_status() brings unwanted overhead when lock can
  be served immediately.

2. Current solution. Within MDL_lock::rw_lock critical section of
  MDL_context::acquire_lock(). MDL_context::m_waiting_for is not yet set
  however MDL_context was already exposed via MDL_lock::m_waiting list.
  The latter is not a problem since we're still holding exclusive lock
  on MDL_lock::rw_lock.
  Cons: increases critical section size for no good reason.

3. Whenever MDL_wait is created and after wait in
  MDL_context::acquire_lock() is completed. At this point MDL_context
  is not exposed via MDL_lock::m_waiting anymore and
  MDL_context::m_waiting_for is reset.
  Cons: none, it is just plain beauty.

Now MDL_wait::m_wait_status is manipulated as following:

EMPTY - set whenever MDL_wait object is created and after each wait
GRANTED - can be set by a thread that releases incompatible lock
VICTIM - can be set either by owner thread or by concurrent higher
        priority thread during deadlock detection
TIMEOUT - always set by owner thread
KILLED - always set by owner thread

This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Vladislav Vaintroub
MDEV-38059 - skip sp-bugs2 test, if compiled without perfschema
Monty
MDEV-37808 "Local temporary space limit reached" on not so rare occasions

This bug happens when using galera, max_tmp_session_space_usage and a
transaction that uses temporary file of a size between 'binlog-cache-size'
(default 32K) and 64K followed by a change user command.

In the case of a transaction of size between 32k and 64k, the server don't
truncate the binary log file on disk to speed up things, which is fine.

The bug was that change_user() reset the tmp_space_used variable and the
next time the temporary_file was truncated, we got a negative value for
tmp_space_used, which caused the error "Local temporary space limit
reached".

Fixed by not resetting tmp_space_used in init(), which is called by
change_user().

Other things
- Truncate binary log cache files when change_user() is called. This
  makes the new users session not depending on log file sizes by previous
  user.
- Some new ASSERT and DBUG_ENTER
- Fixed typo in DBUG output in ma_pagecache.c
Daniel Black
Merge branch '11.4' into MDEV-38029
Sergey Vojtovich
MDL_lock encapsulation: notify_conflicting_locks()

Avoid accessing MDL_lock::m_rwlock from MDL_context::acquire_lock(),
use MDL_lock::notify_conflicting_locks_if_needed() instead.

Also MDL_lock::needs_notification() doesn't require MDL_lock::m_rwlock
protection, so it is moved out of critical section.

This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Vladislav Vaintroub
MDEV-38029 my_tzinfo-t fails for certain TZ values on musl

The test fails for TZ values such as `PST8PDT` (present but outdated in
tzdb) and custom forms like `GST-1GDT`. On musl, these values do not
trigger the expected DST transitions, leading to incorrect DST offsets
or abbreviations.

This appears to be a musl libc bug; the same TZ values behave correctly
elsewhere, including Windows. We work around it by skipping the
affected tests when musl is detected.
Sergey Vojtovich
MDL_lock encapsulation: add_cloned_ticket()

Avoid accessing MDL_lock::m_granted and MDL_lock::m_rwlock from
MDL_context::clone_ticket(), use MDL_lock::add_cloned_ticket() instead.

This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Rex Johnston
MDEV-37220  Allow UPDATE/DELETE to read from a CTE

We extend from the SQL standard to match the functionality of other
databases that allow the inclusion of a CTE definition prior to update
and delete statements.

These CTEs are currently read only, like other derived tables, so
cannot have their columns updated in updates set clause, nor have rows
removed in the delete statement.
Sergey Vojtovich
MDL_lock encapsulation: MDL_lock::iterate()

Avoid accessing MDL_lock::m_waiting, MDL_lock::m_granted and
MDL_lock::m_rwlock from mdl_iterate_lock(), use MDL_lock::iterate()
instead.

This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Sergey Vojtovich
MDL_lock encapsulation: try_acquire_lock()

Avoid accessing MDL_lock::m_rwlock from MDL_context::try_acquire_lock()
and MDL_context::acquire_lock(), code moved to
MDL_context::try_acquire_lock_impl() instead. It is an intermediate
change that reduce uses of MDL_lock::m_rwlock out of MDL_lock class.

This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Sergey Vojtovich
MDL_lock encapsulation: try_acquire_lock_impl()

Avoid accessing MDL_lock members from MDL_context::acquire_lock()
MDL_context::try_acquire_lock_impl() and MDL_map::find_or_insert().
This is done by implementing MDL_map::try_acquire_lock() and
MDL_lock::try_acquire_lock(). With this patch MDL_lock members are
not accessed outside of the class.

This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Monty
Add mariadb-plugin-columnstore.install.generated to .gitignore
Monty
MDEV-32266 All queries in stored procedures increment empty_queries counter

Fixed by setting server_status SERVER_STATUS_RETURNED_ROW if send_data
is called for stored procedures.

This make the definition of Empty_queries well defined:
"Empty_queries" is the number of SELECT queries that returns 0 rows.
Vladislav Vaintroub
MDEV-38029 my_tzinfo-t fails for certain TZ values on musl

The test fails for TZ values such as `PST8PDT` (present but outdated in
tzdb) and custom forms like `GST-1GDT`. On musl, these values do not
trigger the expected DST transitions, leading to incorrect DST offsets
or abbreviations.

This appears to be a musl libc bug; the same TZ values behave correctly
elsewhere, including Windows. We work around it by skipping the
affected tests when musl is detected.
Marko Mäkelä
MDEV-38126: Fix -DWITH_UBSAN=ON

logger_init_mutexes(void): Define the function as taking no
arguments, to avoid function pointer type mismatch.

my_b_flush_io_cache(): runtime error: addition of unsigned offset ...
overflowed. Cast to a signed offset.
Marko Mäkelä
fixup! 937c433d3b3b727f712c62d6369aadce69194d15
Monty
MDEV-38130 Add Option File Syntax '?' to ignore unreadable directories

This is useful if one has MariaDB config files in /etc that should only be
visible for the super user and the MariaDB server but one still want to
provide options for normal users, like skip-ssl.
Sergey Vojtovich
MDL_lock encapsulation: MDL_lock::get_lock_owner()

Avoid accessing MDL_lock::m_rwlock from MDL_map::get_lock_owner(),
code moved to MDL_lock::get_lock_owner() instead.

get_lock_owner() doesn't have to check BACKUP namespace since it is
used by user level locks only.

This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Sergey Vojtovich
MDEV-19749 - MDL scalability regression after backup locks

Statements that intend to modify data have to acquire protection
against ongoing backup. Prior to backup locks, protection against
FTWRL was acquired in form of 2 shared metadata locks of GLOBAL
(global read lock) and COMMIT namespaces. These two namespaces
were separate entities, they didn't share data structures and
locking primitives. And thus they were separate contention
points.

With backup locks, introduced by 7a9dfdd, these namespaces were
combined into a single BACKUP namespace. It became a single
contention point, which doubled load on BACKUP namespace data
structures and locking primitives compared to GLOBAL and COMMIT
namespaces. In other words system throughput has halved.

MDL fast lanes solve this problem by allowing multiple contention
points for single MDL_lock. Fast lane is scalable multi-instance
registry for leightweight locks. Internally it is just a list of
granted tickets, close counter and a mutex.

Number of fast lanes (or contention points) is defined by the
metadata_locks_instances system variable. Value of 1 disables fast
lanes and lock requests are served by conventional MDL_lock data
structures.

Since fast lanes allow arbitrary number of contention points, they
outperform pre-backup locks GLOBAL and COMMIT.

Fast lanes are enabled only for BACKUP namespace. Support for other
namespaces is to be implemented separately.

Lock types are divided in 2 categories: lightweight and heavyweight.

Lightweight lock types represent DML: MDL_BACKUP_DML,
MDL_BACKUP_TRANS_DML, MDL_BACKUP_SYS_DML, MDL_BACKUP_DDL,
MDL_BACKUP_ALTER_COPY, MDL_BACKUP_COMMIT. They are fully compatible
with each other. Normally served by corresponding fast lane, which is
determined by thread_id % metadata_locks_instances.

Heavyweight lock types represent ongoing backup: MDL_BACKUP_START,
MDL_BACKUP_FLUSH, MDL_BACKUP_WAIT_FLUSH, MDL_BACKUP_WAIT_DDL,
MDL_BACKUP_WAIT_COMMIT, MDL_BACKUP_FTWRL1, MDL_BACKUP_FTWRL2,
MDL_BACKUP_BLOCK_DDL. These locks are always served by conventional
MDL_lock data structures. Whenever such lock is requested, fast
lanes are closed and all tickets registered in fast lanes are
moved to conventional MDL_lock data structures. Until such locks
are released or aborted, lightweight lock requests are served by
conventional MDL_lock data structures.

Strictly speaking moving tickets from fast lanes to conventional
MDL_lock data structures is not required. But it allows to reduce
complexity and keep intact methods like: MDL_lock::visit_subgraph(),
MDL_lock::notify_conflicting_locks(), MDL_lock::reschedule_waiters(),
MDL_lock::can_grant_lock().

It is not even required to register tickets in fast lanes. They
can be implemented basing on an atomic variable that holds two
counters: granted lightweight locks and granted/waiting heavyweight
locks. Similarly to MySQL solution, which roughly speaking has
"single atomic fast lane". However it appears to be it won't bring
any better performance, while code complexity is going to be much
higher.
Sergey Vojtovich
MDL_lock encapsulation: MDL_lock::get_key()

Avoid accessing MDL_lock::key from outside of MDL_lock class directly,
use MDL_lock::get_key() instead.

This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Sergey Vojtovich
MDL_lock encapsulation: MDL_lock::remove_ticket()

Avoid accessing MDL_lock::m_waiting from MDL_context::acquire_lock(),
use MDL_lock::abort_wait() instead.

Avoid accessing MDL_lock::m_granted from MDL_context::release_lock(),
use MDL_lock::release() instead.

Avoid accessing MDL_lock::m_strategy and MDL_lock::m_rwlock from
MDL_map::remove(), code moved to MDL_lock::remove_ticket() instead.

This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Monty
Fixed uninitalized memory used error with rpl_gtid

THD::m_last_commit_gtid was not fully intialized, which caused valgrind
warnings when creating a hash of domains.

Fixed by initialzing all members of m_last_commit_gtid
Yuchen Pei
MDEV-37330 [to-squash] support DESC
Sergey Vojtovich
MDL_lock encapsulation: removed redundant methods

Given that MDL_lock::m_strategy is only accessed by MDL_lock methods,
there is no point in having MDL_lock::needs_notification() and
MDL_lock::hog_lock_types_bitmap() getters anymore.

MDL_lock::has_pending_conflicting_lock() moved to MDL_lock class.

This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Brandon Nesterenko
MDEV-37662: Binlog Corruption When tmpdir is Full

The binary log could be corrupted when committing a large transaction
(i.e. one whose data exceeds the binlog_cache_size limit and spills
into a tmp file) in binlog_format=row if the server's --tmp-dir is
full. The corruption that happens is only the GTID of the errored
transaction would be written into the binary log, without any
body/finalizing events.  This would happen because the content of the
transaction wasn't flushed at the proper time, and the transaction's
binlog cache data was not durable while trying to copy the content
from the binlog cache file into the binary log itself. While switching
the tmp file from a WRITE_CACHE to a READ_CACHE, the server would see
there is still data to flush in the cache, and first try to flush it.
This is not a valid time to flush that data to the temporary file
though, as:

  1. The GTID event has already been written directly to the binary
    log. So if this flushing fails, it leaves the binary log in a
    corrupted state.

  2. This is done during group commit, and will slow down other
    concurrent transactions, which are otherwise ready to commit.

This patch fixes these issues by ensuring all transaction data is
fully flushed to its temporary file (if used) before starting any
critical paths, i.e. in binlog_flush_cache(). Note that if the binlog
cache is solely in-memory, this flush-to-temporary-file is skipped.

Reviewed-by: Andrei Elkin <[email protected]>
Signed-off-by: Brandon Nesterenko <[email protected]>
Sergey Vojtovich
MDL_lock encapsulation: MDL_lock::upgrade()

Avoid accessing MDL_lock::m_granted and MDL_lock::m_rwlock from
MDL_ticket::upgrade_shared_lock(), use MDL_lock::upgrade() instead.

This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Rucha Deodhar
Making NEW and OLD as row variables.
Rex Johnston
MDEV-37220  Allow UPDATE/DELETE to read from a CTE

We extend from the SQL standard to match the functionality of other
databases that allow the inclusion of a CTE definition prior to update
and delete statements.

These CTEs are currently read only, like other derived tables, so
cannot have their columns updated in updates set clause, nor have rows
removed in the delete statement.
Sergey Vojtovich
MDL_lock encapsulation: MDL_lock::downgrade()

Avoid accessing MDL_lock::m_granted and MDL_lock::m_rwlock from
MDL_ticket::downgrade_lock(), use MDL_lock::downgrade() instead.

This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Sergey Vojtovich
MDL_lock encapsulation: final

All MDL_lock objects and types are private now. Along with a set
of methods which are not used outside.

This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.