Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
Thirunarayanan Balathandayuthapani
MDEV-39061 mariadb-backup compatible wrapper for BACKUP SERVER

This adds a shell script that lets users keep using their existing
mariadb-backup commands while the real work is done by the new
server-side BACKUP SERVER command. The goal is "drop-in": users should
not have to change their backup scripts.

extra/mariabackup/scripts/mariadb-backup-server.sh (plain POSIX sh)
understands the usual mariadb-backup modes and translates each one.
A companion helper, extra/mariabackup/scripts/mbstream-server.sh,
lets streamed backups be unpacked by pipelines that expect the
mbstream CLI. Both are documented in
extra/mariabackup/scripts/README.md.

--backup
========
Connects with the mariadb client and runs "BACKUP SERVER TO '<dir>'".
Connection options (--user, --host, --port, --socket, --defaults-file,
ssl, ...) are passed through to the client; --parallel=N becomes the
"<N> CONCURRENT" clause.
After the backup it writes backup-prepare.cnf into the backup
directory, recording what --prepare needs later: where
mariadbd lives, the InnoDB parameters (page size, data file path,
undo tablespaces, checksum algorithm, log file size), and if
the server is encrypted then how to reload the encryption key
plugin (the file_key_management variables),
so an encrypted backup can be prepared without extra input.

--backup --stream
=================
Runs "BACKUP SERVER WITH [N CONCURRENT] '<command>'": the server feeds
each stream's tar to <command>, the wrapper collects the parts, writes
them to stdout, then appends backup-prepare.cnf as a final tar. The
per-stream tars carry no end-of-archive marker; only the trailing
backup-prepare.cnf adds the single end marker, so the whole stream
extracts with a plain "tar -x".
Two properties follow from how BACKUP SERVER streams,
both differing from mariadb-backup:
- local: the stream command runs inside the server, so the wrapper
  must share its filesystem;
- tar only: any --stream=<format> (including xbstream) yields tar.
--target-dir is optional in stream mode (scratch for the per-stream
parts; a mktemp dir is used otherwise).

mbstream-server.sh maps the mbstream CLI onto a plain "tar -x"/"tar -c", so
existing "mbstream -x"/"-c" pipelines keep working on the wrapper's
stream. mbstream-only flags (-p/--parallel, ...) are accepted and
ignored; any other unknown option is rejected.

Environment overrides (mainly for testing): MARIADB (client),
MARIADBD (the --prepare bootstrap server) and TAR (the tar
implementation, e.g. TAR=bsdtar) can each be overridden. To run the
bootstrap under rr, put it in MARIADBD and let rr's own _RR_TRACE_DIR
choose the trace location, e.g.
  _RR_TRACE_DIR=/dev/shm/rr MARIADBD='rr record mariadbd'

--prepare
=========
Starts "mariadbd --bootstrap" on the backup directory using
backup-prepare.cnf as its defaults file, replays the archived redo
log between the start and target LSN read from backup.cnf,
then builds a fresh ib_logfile0 so a normal server can start
on the directory. mariadbd is taken from the path recorded in
backup-prepare.cnf if that binary exists, otherwise from PATH.
User --defaults-file/-extra-file and encryption options are
layered onto the bootstrap.

--copy-back / --move-back
=========================
Copy or move a prepared backup into the datadir. The datadir
is created if missing, a non-empty datadir is refused unless
--force-non-empty-directories is given, and a chown
reminder is printed.

If --aria-log-dir-path is given, the Aria logs (aria_log_control,
aria_log.*) are relocated into that directory.

Packaging
=========
The wrapper is not installed by default and never replaces the
real mariadb-backup / mbstream binaries.
1. cmake -DWITH_MARIABACKUP_WRAPPER=ON (default OFF) controls it.
2. When ON, the scripts install as /usr/bin/mariadb-backup-server
and /usr/bin/mbstream-server, tagged COMPONENT Backup so they
ship in the mariadb-backup package.
3. RPM: nothing extra to do. the component handles it.
4. DEB: not wired. debian/rules uses --fail-missing and does not
enable the option, so the -server binaries are not listed.
To ship via DEB, make a paired change: add
-DWITH_MARIABACKUP_WRAPPER=ON in debian/rules and list both
usr/bin/mariadb-backup-server and
usr/bin/mbstream-server in debian/mariadb-backup.install together.
5. The real mariadb-backup/mbstream binaries and the
mariabackup symlink are left untouched; opt in via an alias or a
symlink early in PATH.

Limitations (not supported yet)
===============================
1) Incremental backup & prepare (--incremental-basedir,
  --incremental-dir, --apply-log-only)
2) --rollback-xa
3) Partial backup (--databases, --tables, --tables-file)
4) Output compression and encryption (--compress, --encrypt)
5) --export is accepted but only warns and runs a plain recovery
6) --extra-lsndir is ignored
7) Windows: POSIX sh only, not installed on Windows

Behaviour differences from native mariadb-backup
================================================
- The wrapper needs the mariadb client on PATH for
--backup, and mariadbd on PATH (or recorded in backup-prepare.cnf)
for --prepare
- BACKUP SERVER refuses an already-existing target directory
- BACKUP SERVER does copy the data file as raw pages without
checksum validation, so a corrupted table is not detected
at backup time
- --prepare only works on a wrapper-made backup. It
needs backup-prepare.cnf)
- --stream is tar, not xbstream, and local-only

Tests
=====
include/have_mariabackup_wrapper.inc redirects $XTRABACKUP to
mariadb-backup-server.sh and $XBSTREAM to mbstream-server.sh, skipping
when a wrapper or the mariadb client is unavailable.
include/have_mariabackup_combination.inc runs a test under both the
[CLIENT] mariadb-backup binary and the [SERVER] wrapper.
Marko Mäkelä
fixup! 6c59b142f4acb4a5a7477358a25c2a56ec915386
Daniel Black
MDEV-40176: UBSAN: runtime error: applying non-zero offset in `my_charpos_mb` (2)

my_numchars_mb and my_charpos_mb both used their cs argument
so remove the notused attribute on them.
Alexey (Holyfoot) Botchkov
MDEV-40013 Server crashes in XMLSchema_user_type::validate_prepare/XMLSchema_attribute::validate_prepare.

complexType doesn't have to have any compositor at all. So let's set
'empty_compositor' in this case.
Raghunandan Bhat
MDEV-40176: UBSAN: runtime error: applying non-zero offset in `my_charpos_mb`

Problem:
  When `my_charpos_mb()` is called with pos = end = NULL and the string
  has fewer than `length` characters, the `end + 2 - start` return
  expression evaluates `end+2`, forming the pointer NULL+2. Offsetting
  a null pointer is undefined behavior.

Fix:
  Compute the integer difference before adding the offset. The result is
  identical but no invalid pointer is ever formed.
Georg Richter
Fixed infer NULL dereference reports in unittests
  • cc-x-codbc-windows: 'dojob pwd if '3.4' == '3.4' ls win32/test SET TEST_DSN=master SET TEST_DRIVER=master SET TEST_PORT=3306 SET TEST_SCHEMA=odbcmaster if '3.4' == '3.4' cd win32/test if '3.4' == '3.4' ctest --output-on-failure' failed -  stdio
Marko Mäkelä
squash! 81b3ae71537ca4c67ea4d0f740778f1596fd29a8

InnoDB_backup::queue: Collection of tablespace IDs and payload sizes
at the start of the backup
Yuchen Pei
MDEV-40103 Initialise thd->net in spider_create_sys_thd

net is used by THD::print_aborted_warning when threads are killed so
this needed to be initalised.

This fixes an msan "uninitialized value" bug
Yuchen Pei
MDEV-40103 Initialise thd->net in spider_create_sys_thd

net is used by THD::print_aborted_warning when threads are killed so
this needed to be initalised.

This fixes an msan "uninitialized value" bug
Thirunarayanan Balathandayuthapani
MDEV-39061 mariadb-backup compatible wrapper for BACKUP SERVER

This adds a shell script that lets users keep using their existing
mariadb-backup commands while the real work is done by the new
server-side BACKUP SERVER command. The goal is "drop-in": users should
not have to change their backup scripts.

extra/mariabackup/scripts/mariadb-backup-server.sh (plain POSIX sh)
understands the usual mariadb-backup modes and translates each one.
A companion helper, extra/mariabackup/scripts/mbstream-server.sh,
lets streamed backups be unpacked by pipelines that expect the
mbstream CLI. Both are documented in
extra/mariabackup/scripts/README.md.

--backup
========
Connects with the mariadb client and runs "BACKUP SERVER TO '<dir>'".
Connection options (--user, --host, --port, --socket, --defaults-file,
ssl, ...) are passed through to the client; --parallel=N becomes the
"<N> CONCURRENT" clause.
After the backup it writes backup-prepare.cnf into the backup
directory, recording what --prepare needs later: where
mariadbd lives, the InnoDB parameters (page size, data file path,
undo tablespaces, checksum algorithm, log file size), and if
the server is encrypted then how to reload the encryption key
plugin (the file_key_management variables),
so an encrypted backup can be prepared without extra input.

--backup --stream
=================
Runs "BACKUP SERVER WITH [N CONCURRENT] '<command>'": the server feeds
each stream's tar to <command>, the wrapper collects the parts, writes
them to stdout, then appends backup-prepare.cnf as a final tar. The
per-stream tars carry no end-of-archive marker; only the trailing
backup-prepare.cnf adds the single end marker, so the whole stream
extracts with a plain "tar -x".
Two properties follow from how BACKUP SERVER streams,
both differing from mariadb-backup:
- local: the stream command runs inside the server, so the wrapper
  must share its filesystem;
- tar only: any --stream=<format> (including xbstream) yields tar.
--target-dir is optional in stream mode (scratch for the per-stream
parts; a mktemp dir is used otherwise).

mbstream-server.sh maps the mbstream CLI onto a plain "tar -x"/"tar -c", so
existing "mbstream -x"/"-c" pipelines keep working on the wrapper's
stream. mbstream-only flags (-p/--parallel, ...) are accepted and
ignored; any other unknown option is rejected.

Environment overrides (mainly for testing): MARIADB (client),
MARIADBD (the --prepare bootstrap server) and TAR (the tar
implementation, e.g. TAR=bsdtar) can each be overridden. To run the
bootstrap under rr, put it in MARIADBD and let rr's own _RR_TRACE_DIR
choose the trace location, e.g.
  _RR_TRACE_DIR=/dev/shm/rr MARIADBD='rr record mariadbd'

--prepare
=========
Starts "mariadbd --bootstrap" on the backup directory using
backup-prepare.cnf as its defaults file, replays the archived redo
log between the start and target LSN read from backup.cnf,
then builds a fresh ib_logfile0 so a normal server can start
on the directory. mariadbd is taken from the path recorded in
backup-prepare.cnf if that binary exists, otherwise from PATH.
User --defaults-file/-extra-file and encryption options are
layered onto the bootstrap.

--copy-back / --move-back
=========================
Copy or move a prepared backup into the datadir. The datadir
is created if missing, a non-empty datadir is refused unless
--force-non-empty-directories is given, and a chown
reminder is printed.

If --aria-log-dir-path is given, the Aria logs (aria_log_control,
aria_log.*) are relocated into that directory.

Packaging
=========
The wrapper is not installed by default and never replaces the
real mariadb-backup / mbstream binaries.
1. cmake -DWITH_MARIABACKUP_WRAPPER=ON (default OFF) controls it.
2. When ON, the scripts install as /usr/bin/mariadb-backup-server
and /usr/bin/mbstream-server, tagged COMPONENT Backup so they
ship in the mariadb-backup package.
3. RPM: nothing extra to do. the component handles it.
4. DEB: not wired. debian/rules uses --fail-missing and does not
enable the option, so the -server binaries are not listed.
To ship via DEB, make a paired change: add
-DWITH_MARIABACKUP_WRAPPER=ON in debian/rules and list both
usr/bin/mariadb-backup-server and
usr/bin/mbstream-server in debian/mariadb-backup.install together.
5. The real mariadb-backup/mbstream binaries and the
mariabackup symlink are left untouched; opt in via an alias or a
symlink early in PATH.

Limitations (not supported yet)
===============================
1) Incremental backup & prepare (--incremental-basedir,
  --incremental-dir, --apply-log-only)
2) --rollback-xa
3) Partial backup (--databases, --tables, --tables-file)
4) Output compression and encryption (--compress, --encrypt)
5) --export is accepted but only warns and runs a plain recovery
6) --extra-lsndir is ignored
7) Windows: POSIX sh only, not installed on Windows

Behaviour differences from native mariadb-backup
================================================
- The wrapper needs the mariadb client on PATH for
--backup, and mariadbd on PATH (or recorded in backup-prepare.cnf)
for --prepare
- BACKUP SERVER refuses an already-existing target directory
- BACKUP SERVER does copy the data file as raw pages without
checksum validation, so a corrupted table is not detected
at backup time
- --prepare only works on a wrapper-made backup. It
needs backup-prepare.cnf)
- --stream is tar, not xbstream, and local-only

Tests
=====
include/have_mariabackup_wrapper.inc redirects $XTRABACKUP to
mariadb-backup-server.sh and $XBSTREAM to mbstream-server.sh, skipping
when a wrapper or the mariadb client is unavailable.
include/have_mariabackup_combination.inc runs a test under both the
[CLIENT] mariadb-backup binary and the [SERVER] wrapper.
Georg Richter
Fixed another infer NULL dereference report
  • cc-x-codbc-windows: 'dojob pwd if '3.4' == '3.4' ls win32/test SET TEST_DSN=master SET TEST_DRIVER=master SET TEST_PORT=3306 SET TEST_SCHEMA=odbcmaster if '3.4' == '3.4' cd win32/test if '3.4' == '3.4' ctest --output-on-failure' failed -  stdio
Yuchen Pei
MDEV-39558 Check resulting VECTOR length in type aggregation inference

VECTOR, as a subtype of VARCHAR, has max length 65532, i.e. maximum
dimension of 16383. BLOB/MEDIUMBLOB/LONGBLOB (and corresponding TEXT
types) each has length exceeding 65532. Therefore, when aggregating
VECTOR with one of these BLOB/TEXT types, the aggregated type has
length exceeding the max length of VECTOR, which should result in an
error.

To that end, in this patch we add checks on the resulting vector
length during type aggregation inference
Yuchen Pei
MDEV-40103 Initialise thd->net in spider_create_sys_thd

This fixes an msan uninitialized bug
ParadoxV5
MDEV-39788: Fix `master.info` not upgrading `master_use_gtid`

The line-count line in `master.info` and `relay-log.info`
has been inconsistent (off by one) since its introduction.
MDEV-37530 corrected this by changing `master.info`
to use `relay-log.info`’s line-count definition by
chance, but at the cost of an upgrade incompatibility.
It now reads one more line beyond the pre-upgraded file’s line-based
section, which for MariaDB 10.0–12.2 means an upgrade doesn’t carry
over the first option stored in `key=value`, `master_use_gtid`.

Since older versions can read the new format in a downgrade,
rather than encouraging the old inconsistency,
this commit focuses on the upgrade problem by adding a shim entry to
`master.info`’s list to emulate prior versions’ reading behaviour.
Although this implementation can only restore compatibility with
versions 10.0+, versions before MariaDB 10 have long been EOL.

While here, this commit also fixes code and
comments that contradict the actual effect.
drrtuy
fix(perf): optimize regexp_rewrite perf for CB.
Alessandro Vetere
MDEV-40209 Escalate lock-release via a saturating stall counter

lock_release() and lock_release_on_prepare() release a committing or preparing
transaction's explicit locks under the shared lock_sys.rd_lock(), taking each
per-cell hash latch and per-table lock mutex with a trylock because trx->mutex
is held in the reverse of the normal latch order. A single failed trylock
marked the whole pass unsuccessful, and after a fixed cap of 5 such passes the
code escalated to the exclusive lock_sys.wr_lock() for the whole transaction.
Under concurrency the trylocks fail transiently, so the cap escalated
transactions that were still steadily releasing locks, not just stuck ones; the
exclusive latch then blocks every concurrent lock_sys.rd_lock() acquirer in
lock_rec_lock() and lock_table(), producing a convoy. The chance of hitting the
cap rises with both the contention level and the number of latches a
transaction must trylock per pass.

Replace the fixed cap with a saturating stall counter (LOCK_RELEASE_MAX_STALLS,
incremented on a no-progress pass, decremented on progress, floored at zero)
that escalates a genuinely stuck transaction after 5 net stalls, as the fixed
cap did, while leaving a transaction that keeps making progress to finish under
the shared latch. A hard LOCK_RELEASE_MAX_PASSES ceiling bounds the loop
independently, for the case where concurrent activity keeps adding locks (e.g.
implicit-to-explicit conversion during XA PREPARE) so that progress never
converges. The _try functions report progress through an out-parameter computed
under trx->mutex, so trx->lock.trx_locks is never read unlatched.
Yuchen Pei
MDEV-39558 Check resulting VECTOR length in type aggregation inference

VECTOR, as a subtype of VARCHAR, has max length 65532, i.e. maximum
dimension of 16383. BLOB/MEDIUMBLOB/LONGBLOB (and corresponding TEXT
types) each has length exceeding 65532. Therefore, when aggregating
VECTOR with one of these BLOB/TEXT types, the aggregated type has
length exceeding the max length of VECTOR, which should result in an
error.

To that end, in this patch we add checks on the resulting vector
length during type aggregation inference
Alessandro Vetere
MDEV-40210 Redundant CAS in async_flush_lsn::try_clear_if_at_most()

MDEV-39600 added try_clear_if_at_most() to clear buf_flush_async_lsn with
an atomic CAS that preserves a concurrent bump(). If the snapshot is
already 0, compare_exchange_strong(0, 0) is a no-op, so return early on a
zero snapshot and avoid the atomic read-modify-write. The page cleaner
calls this on every pass, so in the common steady state (no async flush
queued) it drops needless exclusive access to the m_lsn cache line. A zero
value is already the cleared state and a concurrent bump() is preserved
either way, so the result is identical.
drrtuy
chore(docs): architecture and shared resources docs.
Daniel Black
MDEV-40176: UBSAN: runtime error: applying non-zero offset in `my_charpos_mb` (2)

my_numchars_mb and my_charpos_mb both used their cs argument
so remove the notused attribute on them.
drrtuy
fix: use jemalloc as a default allocator for DuckDB.
Sergei Golubchik
MDEV-39776 SHOW CREATE TABLE corrupts `-quoted table options

Two problems:

1. option value specified as `foo bar` was not quoted,
  printed as foo bar, breaking CREATE TABLE.
2. Option value that included */ was printed as is, in unknown
  options when the whole option was commented out, this broke the
  comment: /* opt='val*/ue' */

Fix: simplify value printing, ignore the original quoting,
always print the value quoted as a string. Break comment markers
with an escape (as *\/) if inside a comment.

Note, cannot use `-quoting, because \-escapes don't work inside,
backticks, so */ cannot be broken.
Yuchen Pei
MDEV-39558 Check resulting VECTOR length in type aggregation inference

VECTOR, as a subtype of VARCHAR, has max length 65532, i.e. maximum
dimension of 16383. BLOB/MEDIUMBLOB/LONGBLOB (and corresponding TEXT
types) each has length exceeding 65532. Therefore, when aggregating
VECTOR with one of these BLOB/TEXT types, the aggregated type has
length exceeding the max length of VECTOR, which should result in an
error.

To that end, in this patch we add checks on the resulting vector
length during type aggregation inference
Vladislav Vaintroub
MDEV-40026 acl_insert_db crashes on overlong dbname in "denies" array.

Fix to return error when json_unescape() fails.
Also return error, if length from json_unescape happens to be larger
that size of buffer - 1, to make sure that null-termination of string
does not override memory.

Some cleanup - only complain once about invalid "denies" entry during
FLUSH privileges. Avoid full parsing during when processing
mysql.global_priv in grant_reload() in the first pass.
Vladislav Vaintroub
MDEV-40116 Column-level DENY SELECT hides accessible columns

Allow SHOW if grants on table level indicate "maybe allowed", i.e
there are some grants that are not all denied on this level.

Column level denies are applied in SHOW COLUMS.
They are not applied in SHOW CREATE TABLE, which will succeed if
there are appropriate table level privs, without checking columns.
Yuchen Pei
MDEV-40103 Initialise thd->net in spider_create_sys_thd

This fixes an msan uninitialized bug
drrtuy
chore(test): TPC-H gen outputs in parquet format and queries run by MariaDB native SQL.
Yuchen Pei
MDEV-40088 [to-squash] Disallow subqueries in INTERVAL clause in range interval auto partitioning

This is consistent with system time (versioning) partitioning. Also
consistent is that both allow expressions otherwise.
Georg Richter
Merge branch '3.3' into 3.4
  • cc-x-codbc-windows: 'dojob pwd if '3.4' == '3.4' ls win32/test SET TEST_DSN=master SET TEST_DRIVER=master SET TEST_PORT=3306 SET TEST_SCHEMA=odbcmaster if '3.4' == '3.4' cd win32/test if '3.4' == '3.4' ctest --output-on-failure' failed -  stdio
Vladislav Vaintroub
MDEV-40115 DENY EXECUTE hides unrelated routines in SHOW commands or I_S

Fix check_routine_level_acl() to pass parent access, for accurate
calculation.

Previously, if (db_level_access & privs) returned 0, it was treated as
no-access on DB level in check_some_routine_access().

This is not the complete truth, with DENY, it may mean there is access,
but some routines have DENY. So pass the db level grants/denies to the
function for more accurate checking.
Sergei Golubchik
MDEV-39818 backtick in FK names breaks SHOW CREATE TABLE

InnoDB used to have two versions of innobase_quote_identifier().
One that handled embedded backticks and one that didn't.

Let's use just one implementation that handles backticks.
Keep the other one as a helper that writes to FILE but does not
implement the quoting logic.
Yuchen Pei
MDEV-40088 [to-squash] Disallow subqueries in INTERVAL clause in range interval auto partitioning

This is consistent with system time (versioning) partitioning. Also
consistent is that both allow expressions otherwise.
ParadoxV5
Merge branch 'MDEV-39788.test' into MDEV-39788
Yuchen Pei
MDEV-40103 Initialise thd->net in spider_create_sys_thd

This fixes an msan uninitialized bug
Alexey (Holyfoot) Botchkov
MDEV-40019 Xmlisvalid shows wrong result if calling it multiple times from query with UNION ALL.

Call the validation_prepare() for the tail of the list.
Vladislav Vaintroub
MDEV-40028 Assertion `rights.allow_bits() == merged->cols' failed

Fixed mismatched memroot in new operator, in apply_deny_column.

MDEV-40014 Crash in SHOW GRANTS FOR user after REVOKE

Fixed premature my_hash_reset in update_role_columns, should not be
done if only DENYs are present

Also fixed unintended GRANT on column privileges when DENY should be
applied.
Raghunandan Bhat
MDEV-40176: UBSAN: runtime error: applying non-zero offset in `my_charpos_mb`

Problem:
  When `my_charpos_mb()` is called with pos = end = NULL and the string
  has fewer than `length` characters, the `end + 2 - start` return
  expression evaluates `end+2`, forming the pointer NULL+2. Offsetting
  a null pointer is undefined behavior.

Fix:
  Compute the integer difference before adding the offset. The result is
  identical but no invalid pointer is ever formed.
Georg Richter
Fixed another issue found by rpl_fuzzer
  • cc-x-codbc-windows: 'dojob pwd if '3.4' == '3.4' ls win32/test SET TEST_DSN=master SET TEST_DRIVER=master SET TEST_PORT=3306 SET TEST_SCHEMA=odbcmaster if '3.4' == '3.4' cd win32/test if '3.4' == '3.4' ctest --output-on-failure' failed -  stdio
Vladislav Vaintroub
MDEV-40131 DENY granted on roles exclusively is missing after FLUSH PRIVILEGES.

Fixed: user_table_has_denies was not set for role entries, but for normal
users only.