Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
drrtuy
fix: MDEV-40651 ALTER TABLE and DDL in general is now executed in an atomic manner b/c previously MariaDB DDL triggered multiple autocommit DDL statements in DuckDB some of those can fail leaving table in unsable state.
Jan Lindström
MDEV-40645 : Slave Crash on Malformed User_var_log_event

A crafted DECIMAL user variable event makes bin2decimal() allocate and
copy a size taken entirely from two attacker-supplied bytes.

bin2decimal() has no parameter telling it how many bytes are available
(strings/decimal.c:1424):

  int bin_size= decimal_bin_size(precision, scale);
  sanity(to);
  d_copy= (uchar*) my_alloca(bin_size);
  memcpy(d_copy, from, bin_size);

decimal_bin_size() guards its inputs with DBUG_ASSERT(precision > 0)
and DBUG_ASSERT(scale <= precision) only, so release builds proceed with
anything. my_alloca is real alloca in this build.

Precision and scale arrive as val[0] and val[1]. Three call sites
pass them through:

* the applier, User_var_log_event::do_apply_event
(sql/log_event_server.cc:4563), which checks only val_len < 3
* pack_info (sql/log_event_server.cc:4356), which checks nothing, not
even val_len >= 3
* mysqlbinlog (sql/log_event_client.cc:2489), which checks nothing
The constructor does bound val_len against the event
(sql/log_event.cc:3064), so val_len is honest, but nothing ties
decimal_bin_size(val[0], val[1]) to it. val_len of 3 with a precision
of 65 already asks for 30 bytes where 1 is available.

Fixed by adding guards against malformed precision, scale and
val_len.
Lawrin Novitsky
ODBC-497 Adding infer to the Github Actions workflow
Sergei Golubchik
mariadb-test: don't use strlen/strend, support lines with embedded \0

add a test
Sergei Golubchik
MDEV-39689 update the test
Sergei Golubchik
MDEV-40629 environment injection via wsrep bootstrap in the service file

* don't create mariadb-wsrep-new-cluster in the mariadbd-writable path,
  the server should not be able to poison the environment with OUTFILE.
  Create it in /run
* As in /run it must be deleted by root, let galera_new_cluster delete
  it, not the service
* wsrep-start-position cannot be created by root, so avoid a file
  for it at all

Assisted-By: Claude:claude-5-opus
Ahmad
MDEV-39858: Reloading COSINE metric index from disk degrades search recall due to abs2 quantization noise

When a vector is created in-memory using FVector::create() during normal inserts, its squared magnitude (abs2) under the COSINE metric is hardcoded to 0.5f.

However, when the index is reloaded from disk (after a server restart, FLUSH TABLES, or ALTER TABLE), the index uses FVectorNode::load_from_record(). This method reads the stored scale and quantized int16 coordinates from the database record, and runs postprocess(). Inside postprocess(), abs2 is dynamically recomputed using floating-point math:
abs2 = subabs2 + scale * scale * dot_product(d, d, vec_len) / 2;

Because the coordinates stored on disk are quantized int16 values, this recalculation introduces rounding noise.

This affects high dimensions datasets, and it is increasing as M increases.

Added hardcoded abs2=0.5 to FVectorNode::load_from_record and removed postprocess()
drrtuy
fix: hardening against unknown SQL injection vectors running DuckDB.
forkfun
on rpm: don't set mysql user's $HOME to datadir

useradd set --home to %{mysqldatadir}, matching datadir.
Set --home to /nonexistent (as in deb)
Brandon Nesterenko
MDEV-40648 (Regression): Replication Undefined Behavior on Malformed Rotate Log Event

test
Rex Johnston
MDEV-39499 squashme
Arcadiy Ivanov
MDEV-40634 Const MEMORY table's BLOB outlives the lock protecting it

A single-row table is read once during optimization and its row kept in
`record[0]` for the rest of the statement.  `JOIN::optimize_stage2()`
then releases the lock on every const table, on the premise stated in
its own comment: *"It's safe to ignore result code as all tables where
opened for read only."*

That premise assumes a read leaves a **copy** of the row behind.  MEMORY
with a blob does not.  `hp_read_blobs()` answers the read by pointing
`record[0]` at the blob data inside `HP_SHARE` rather than copying it,
so from the moment the lock is dropped another connection is free to
overwrite, free or recycle those bytes -- and the statement goes on
reading them.  The result is a const table whose value changes in the
middle of the statement using it, and a read of freed memory.

Let a caller that keeps reading a row after the unlock ask for such
tables to be left alone.  `GET_LOCK_SKIP_ZERO_COPY_ROWS` drops them
from the lock set `get_lock_data()` builds, exactly as
`GET_LOCK_SKIP_SEQUENCES` already does, and the const-table unlock in
`JOIN::optimize_stage2()` passes it.

Which tables those are is for the engine to say rather than for the
lock layer to infer.  The 64-bit `table_flags()` space is full, so a
second word `table_flags2()` carries the first such property,
`HA2_CANNOT_ACCESS_ROWDATA_AFTER_UNLOCK`, and `ha_heap::open()`
raises it for any table that has a blob.

The skip has to be opt-in rather than a rule.  `mysql_lock_remove()`
also reaches `mysql_unlock_some_tables()`, and there the unlock is
permanent and must not be skipped.

A MEMORY blob const table now stays read-locked for the whole statement
and blocks writers, which is the price every non-const MEMORY table
already pays.

The regression test parks the reader with `GET_LOCK()` rather than with
a stored function.  A stored function puts the statement into prelocked
mode, and the const-table unlock is skipped entirely in that mode, so
the code path under test would never run.

The gate that parks it is taken with `--disable_ps2_protocol` in force.
`--ps-protocol` executes every complete `SELECT` twice and compares the
two result sets, and `GET_LOCK()` is recursive, so a doubled
acquisition would outlive the single `RELEASE_LOCK()` that opens the
gate again.
Brandon Nesterenko
MDEV-40648 (Regression): Mismatched Event Length can Corrupt Slave Relay Log
Dave Gosselin
Clarify NULL handling comment in next_min()
Marko Mäkelä
MDEV-40667 Infinite loop on SET GLOBAL innodb_log_archive=ON

log_t::set_archive(): Always invoke buf_flush_wait(wait_lsn, false)
on the latest sampled get_lsn(), so that if a checkpoint is
possible, one will be executed.
Sergei Golubchik
MDEV-40637 CONNECT crashes on double(255,50) in DOS table

cap the length correctly
Brandon Nesterenko
MDEV-40648: Replication Undefined Behavior on Malformed Rotate Log Event

A slave stops with a relay log read failure when its master sends an
event whose header declares a length other than the number of bytes the
event arrived in. The file and position the slave reports for that
failure are not where the problem is. A master that logs no checksum
does not stop the slave at all. Such a master can make the slave run one
statement twice, leaving the slave's data holding a row the master's
binary log never carried. A "malicious" master can already send whatever
events it likes, so what this defeats is comparing a slave's applied
stream against the master's binary log.

The slave IO thread reads each event from the master as one network
packet, and writes that packet into the relay log unchanged, using the
packet's own length. Every later reader of that relay log frames the
events by a different length: the one each event's header declares at
EVENT_LEN_OFFSET. A master writing an event sets the two to the same
value. Log_event::read_log_event(), which parses the events that
queue_event() does not construct itself, checks only that the packet
reaches EVENT_LEN_OFFSET, and never compares the declared length with
the length of the packet.

queue_event() never compared the two lengths either. An event declaring
fewer bytes than the packet held reached the relay log with the extra
bytes behind the event. The SQL thread framed its next read from inside
the previous event. Where the master had placed a complete event in
those extra bytes, and no checksum covered the packet, the SQL thread
applied that second event.

This patch adds validation to ensure the lengths are equal. An event
whose lengths disagree will stop the IO thread with
ER_SLAVE_FATAL_ERROR, and the relay log will never receive the event.

Reviewed-by: TODO
Signed-off-by: Brandon Nesterenko <[email protected]>
Oleksandr Byelkin
Merge branch '10.6' into bb-10.11-release
drrtuy
fix: resolve SQL injection imposed by lexer semantics difference b/w
MariaDB and DuckDB.
Sergei Golubchik
cleanup: encryption.filekeys_encfile_badfile

combine all tests for wrong FILE: values into one test
add a test for a wrong key.
Arcadiy Ivanov
MDEV-40591 Unexpected ER_NOT_KEYFILE or MSAN error in heap_check_heap

`ha_heap::external_lock()` verifies the table with `heap_check_heap()`
at `F_UNLCK`.  That is safe on the ordinary unlock path, where
`mysql_unlock_tables()` calls `unlock_external()` before
`thr_multi_unlock()` and the lock is still held.  It is not safe on
either path that unlocks after a *failed* lock attempt, where the
caller holds nothing at all while another connection is writing:

1. `mysql_lock_tables()` calls `unlock_external()` to balance the
  external locks it already took, because `thr_multi_lock()` timed
  out.
2. `lock_external()` unwinds the tables it has already locked, because
  a later table refused -- all before `thr_multi_lock()` runs at all.
  `ha_partition::external_lock()` unwinds its partitions the same way.

MEMORY has no row-level concurrency control, so a scan taken outside
the lock sees a writer's intermediate state by construction:
`hp_alloc_from_tail()` publishes `total_records` at allocation time,
before the slot is written, while the checker scans `[0, total_records
+ deleted)` and reads every slot's flags byte.  Under MSAN that is a
use of uninitialised `my_malloc()` memory; otherwise it is a spurious
`total_records` mismatch.

`heap_check_heap()` ends with `heap_mark_crashed()`, which sets
`HEAP_STATE_CRASHED` in the **shared** `HP_SHARE`, so one bogus
mid-write observation poisons a healthy table for every connection
using it -- the reported `ER_NOT_KEYFILE`.

MDEV-21373 disabled this check in 2021 for exactly this reason, by
gating it on `EXTRA_DEBUG`.  MDEV-38975 changed the gate to
`EXTRA_HEAP_DEBUG` and defined that for every debug build, reviving
the race.

Rather than switch the check off wholesale again, only verify a table
that this handle both holds a lock on and has changed under it:

- `HP_INFO::lock_type` remembers the `ha_heap::external_lock()`
  argument, the way `MARIA_HA` and `MI_INFO` already do;
- `HP_INFO::changed` is set by `heap_write()`, `heap_update()` and
  `heap_delete()`, and cleared by `ha_heap::external_lock()` on every
  grant, so it means "changed since this lock was taken";
- `table_is_locked_and_changed()` requires both.

The change term is what separates the three unlock paths, because the
lock type cannot: `ha_heap::external_lock()` records it before
`thr_multi_lock()` runs, so it is armed on the two failing paths as
well.  Neither of them ever ran a row operation, so neither has
changed anything.  It has to be per handle rather than
`HP_SHARE::changed`, which is true on exactly those paths, another
connection being the one writing.

Requiring a change also makes a debug build cheaper: the verification
scans every record and every index, and now runs only after a
statement that wrote to the table.

Deriving this in the engine rather than repairing `lock_external()`
also covers `ha_partition`, which reimplements the same unwind.

A temporary table gets `F_EXTRA_LCK` and so counts as always locked:
no other connection can reach its share.  This covers the user's
`CREATE TEMPORARY TABLE` and not only the optimizer's internal one --
an internal table frees its blob chains outright, whereas a user
temporary table parks them, and `get_lock_data()` leaves it out of the
lock set entirely, so it never reaches `external_lock()` at all.  The
`ALTER` copy target is temporary too, and additionally takes a direct
`handler::ha_external_lock()` instead of going through the lock set.

Redeeming a parked blob chain puts records back on the shared free
list, so it needs the same protection, and both redemption points
assert it.

`hp_test_unlock_check-t` builds the lock states directly, in the order
`ha_heap::external_lock()` builds them, so nothing here is raced.
Four MTR tests cover the shapes it cannot reach: blob updates and
deletes on a user `TEMPORARY` MEMORY table (`heap.blob_tmp_table`),
`INSERT DELAYED` (`heap.blob_delayed_insert`), the `ALTER` copy target
(`heap.blob_online_alter`), and one share locked twice in a lock set
(`heap.blob_lock_twice`).  No existing test exercised any of them.
Marko Mäkelä
MDEV-40667 Infinite loop on SET GLOBAL innodb_log_archive=ON
Sergei Golubchik
MDEV-40658 file_key_management crash on empty FILE: file
Dave Gosselin
MDEV-25964:  Unexpected bypass of lock

When an uncommitted transaction inserts rows into a table and
another statement locks rows in the same table (SELECT ... FOR UPDATE)
while computing a MIN or MAX, then:
  1. In a Debug build, the server aborts on an assertion
  2. In a Release build, the server returns wrong results
These errors occur because, while reading a group of rows for computing
a MAX, the transaction timeout error was swallowed.

Under the scenario described above and captured in the new test at this
commit, QUICK_GROUP_MIN_MAX_SELECT::next_max() emits a lock timeout error
during QUICK_GROUP_MIN_MAX_SELECT::get_next() but the error was suppressed
if we computed a MIN.

The InnoDB storage engine has an unwritten convention that after it has
returned a fatal error (which is any error except HA_ERR_END_OF_FILE
or HA_ERR_KEY_NOT_FOUND), then the SQL layer should not try to make
any further reads.  This is because InnoDB might have rolled back
the current transaction already.  So in the case of an error, return
immediately from QUICK_GROUP_MIN_MAX_SELECT::get_next().
Sergei Golubchik
mariadb-test: don't use strlen/strend, support lines with embedded \0

add a test
Sergei Golubchik
MDEV-40362 fix test

followup for f277eefa424
Brandon Nesterenko
MDEV-40648: Mismatched Event Length can Corrupt Slave Relay Log

A slave stops with a relay log read failure when its master sends an
event whose header declares a length other than the number of bytes the
event arrived in. The file and position the slave reports for that
failure are not where the problem is. A master that logs no checksum
does not stop the slave at all. Such a master can make the slave run one
statement twice, leaving the slave's data holding a row the master's
binary log never carried. A "malicious" master can already send whatever
events it likes, so what this defeats is comparing a slave's applied
stream against the master's binary log.

The slave IO thread reads each event from the master as one network
packet, and writes that packet into the relay log unchanged, using the
packet's own length. Every later reader of that relay log frames the
events by a different length: the one each event's header declares at
EVENT_LEN_OFFSET. A master writing an event sets the two to the same
value. Log_event::read_log_event(), which parses the events that
queue_event() does not construct itself, checks only that the packet
reaches EVENT_LEN_OFFSET, and never compares the declared length with
the length of the packet.

queue_event() never compared the two lengths either. An event declaring
fewer bytes than the packet held reached the relay log with the extra
bytes behind the event. The SQL thread framed its next read from inside
the previous event. Where the master had placed a complete event in
those extra bytes, and no checksum covered the packet, the SQL thread
applied that second event.

This patch adds validation to ensure the lengths are equal. An event
whose lengths disagree will stop the IO thread with
ER_SLAVE_FATAL_ERROR, and the relay log will never receive the event.

Reviewed-by: TODO
Signed-off-by: Brandon Nesterenko <[email protected]>
Sergei Golubchik
MDEV-40636 CSV crashes on DELETE

chain_size is the number of tina_set elements, not number of bytes

Assisted-By: Claude:claude-5-opus
Daniel Black
MDEV-40630: stunnel as Deb/RPM recommends for rsync sst

stunnel has been a component used by the rsync SST mechanism of galera
but has never been listed as a package dependency.

Since MDEV-28233 corrected a unencrypted fallback in the the case of
stunnel being absent, we add stunnel as a recommendation for Debian and
RPM packages.
forkfun
MDEV-38472 Assertion in Diagnostics_area::set_error_status on UPDATE...LIMIT DEFAULT

Binding DEFAULT to a LIMIT ? placeholder via EXECUTE ... USING is never
rejected at bind time. unit->set_limit() later evaluates it and sets
ER_INVALID_DEFAULT_PARAM in the Diagnostics_area, but SQLCOM_UPDATE (and
SQLCOM_UPDATE_MULTI, SQLCOM_INSERT_SELECT/REPLACE_SELECT) proceeded into
open_tables() regardless. A concurrent metadata change (e.g. ALTER TABLE
FORCE) then triggers Reprepare_observer::report_error(), which tries to
set ER_NEED_REPREPARE on an already-set DA and hits the assert.

Add the same thd->is_error() check after unit->set_limit() (as in
SQLCOM_DELETE), aborting before open_tables() is reached.
Daniel Black
MDEV-31808 crash upon altering table with NEXTVAL for default under exclusive lock

A segfault in mysql_lock_merge waa the result of a null argument
when called from open_and_lock_internal_tables. As the argument
was null there was no lock to merge and thd->lock was already in place,
so no futher action is required in open_and_lock_internal_tables.

After passing the previous correctly on open_table of the
sequence used in the table, we get to the checking the thd->open_table
list where tdc is flushed. This puts the open_table into an error
condition to reopen all tables. When we are just opening a sequence,
this condition isn't one we want to fail on.
Oleksandr Byelkin
Merge branch '10.11' into bb-10.11-release
ARaveala
MDEV-39762 (Regression): Slave Overflow on Malformed Query_compressed_log_event
Sergei Golubchik
MDEV-40670 qc_info OOB read if query contains \0

qc_info cannot parse query cache key, this cannot be done
without changing query cache key format.

let's limit the (possibly incorrect) db_length to make sure
it does not extend beyond the key end, this way there will be no
OOB reads and no information leak.
aleksei.bukhalov
docs: add public dataset tutorials for the DuckDB engine

Add two step-by-step tutorials:

- NYC Taxi Trips: load TLC Parquet via run_in_duckdb() + read_parquet(),
  keep the zone lookup in InnoDB, then join the two engines in a plain
  SELECT. Follows the structure of the ClickHouse taxi tutorial.

- OWID CO2 Emissions: load a commit-pinned CSV revision, run
  aggregations and a LAG() window function.

Both tutorials point at security-model.md before telling the reader to
enable duckdb_allow_run_in_duckdb, and both note that ordinary
INSERT/UPDATE/DELETE also work on ENGINE=DuckDB tables so that
run_in_duckdb() does not look like the only write path.

Also documents TIMESTAMPDIFF in mariadb-duckdb-incompatibilities.md and
adds a tutorials section to the README.

Every SQL block and every expected output was extracted from the
markdown, executed, and compared cell by cell on MariaDB 11.4.13 built
from this branch; both data sets are pinned so results stay
reproducible.
ARaveala
MDEV-39762: Slave Overflow on Malformed Query_compressed_log_event

A replica can crash when a compressed event from its master carries a
corrupted uncompressed length. A compressed Query event whose length
is near 4GB overwrites the IO thread's stack with the event's content.
A compressed rows event of the same shape makes the replica ask for an
allocation of 4GB. A debug build that gets that memory then fails an
assertion in binlog_buf_uncompress(). On a build where a ulong is 32
bits wide, both events overwrite the IO thread's stack.

The IO thread uncompresses a compressed event before writing the event
to the relay log, and offers the uncompress function 4096 bytes of its
own stack to hold the result. The function computes how large the event
will be once uncompressed, and that size decides where the uncompressed
event goes. An event that fits in the stack buffer is uncompressed
there. A larger event is uncompressed into an allocation the function
makes at that size. With either buffer, the function tells zlib that
the room available is the uncompressed length the event declared.

query_event_uncompress() never bounded the length read out of the
event. The function added the header length to that value, rounded the
sum up for alignment, and cast the result to uint32. The cast dropped
the high bits of a sum above 4GB, so a length near 4GB produced a size
of a few dozen bytes. That size fit the stack buffer, so content of any
size went onto the IO thread's stack. row_log_event_uncompress()
computes that sum in a ulong, which wraps where a ulong is 32 bits
wide. The Query_compressed_log_event constructor and
Rows_log_event::uncompress_buf() size their allocations from the same
unbounded length.

Reject an uncompressed length above MAX_MAX_ALLOWED_PACKET at every
point the length is read, before any size is computed from it. No
master writes a larger length, because max_allowed_packet is capped at
1GB. query_event_uncompress() also keeps that size in a size_t, so the
allocation is made from the whole computed value. Both functions return
the error code for the IO thread to report, so a length over the limit
stops the IO thread with ER_TOO_BIG_FOR_UNCOMPRESS rather than the
generic uncompress error. A compressed event with a corrupted
uncompressed length now stops the replica's IO thread with an error
instead of crashing the replica.

Alexandra Raveala wrote the original patch and its first regression
test, contributed through PR 5413. Brandon Nesterenko saw the work
through for the 10.6 release, replacing the test with one that makes
the master write the corrupt length and reworking how the IO thread
reports the error.

Co-authored-by: Brandon Nesterenko <[email protected]>
Reviewed-by: TODO
Signed-off-by: Brandon Nesterenko <[email protected]>
Sergei Golubchik
MDEV-40670 qc_info OOB read if query contains \0

qc_info cannot parse query cache key, this cannot be done
without changing query cache key format.

let's limit the (possibly incorrect) db_length to make sure
it does not extend beyond the db name end, this way there will be no
OOB reads and no information leak.
Alexander Barkov
MDEV-39587 Package-wide TYPE for variable declarations

SET sql_mode=ORACLE;
DELIMITER $$
CREATE OR REPLACE PACKAGE pkg AS
  -- Declare a package public data type
  TYPE varchar_array IS TABLE OF VARCHAR(2000) INDEX BY INTEGER;
END;
$$
DELIMITER ;
DELIMITER $$

CREATE OR REPLACE PROCEDURE p1 AS
  v pkg.varchar_array; -- Use the package public data type
BEGIN
  v(0):='test';
  SELECT v(0);
END;
$$
DELIMITER ;

Note, the change is done only for sql_mode=ORACLE, because the TYPE
declaration is not available for the default mode.

Where package-wide types are available
--------------------------------------
- Variabe list type:
    DECLARE var pkg1.type1;

- RETURN type for a package routine:
    CREATE FUNCTION .. RETURN pkg1.type1 ...

- Parameter type for a package routine:
    PROCEDURE p1(param1 pkg1.type1);

- Assoc array element type:
    TYPE assoc1_t IS TABLE OF pkg1.type1 ...

- REF CURSOR RETURN type:
    TYPE cur1_t IS REF CURSOR RETURN pkg1.type1;

Change details
--------------

- Adding a member Lex_length_and_dec_st::m_foreign_module_type
  It's set to true when the data type was initialized from a TYPE
  in foreign routine (e.g. in PACKAGE spec).
  It's needed to prevent use of qualified identifiers in public contexts,
  i.e. in schema public routine parameter types and schema publuc function
  RETURN types.
  Adding a helper method sp_head::check_applicability() which prevents
  use of qualified types in public context.

- Adding a helper method sp_head::raise_unknown_data_type().

- Adding methods LEX::set_field_type_typedef_package_spec() for
  2-step and 3-step qualified indentifiers.
  It's used in field_type_all_with_typedefs which covers cases:
  - Variabe list type        : DECLARE var pkg1.type1;
  - RETURN type              : CREATE FUNCTION .. RETURN pkg1.type1 ...
  - Parameter type          : PROCEDURE p1(param1 pkg1.type1);
  - Assoc array element type : TYPE assoc1_t IS TABLE OF pkg1.type1 ...

- Adding a method LEX::declare_type_ref_cursor_return_typedef().
  It handles cases when a new TYPE REF CURSOR RETURN is declared,
  for both for qualified RETURN types and non-qualified RETURN types:
  - TYPE cur0_t IS REF CURSOR RETURN rec1_t;
  - TYPE cur0_t IS REF CURSOR RETURN pkg1.rec1_t;
  - TYPE cur0_t IS REF CURSOR RETURN db1.pkg1.rec1_t;

  The code was moved from LEX::declare_type_ref_cursor() into
  LEX::declare_type_ref_cursor_return_typedef() and extended
  to cover qualified RETURN types.

- Adding a method Sql_path::find_package_spec_type().
  It iterates through all schemas specified in @@path and searches
  for the given type in the given package.

- Adding a helper method sp_pcontext::type_defs_add_ref_cursor()
  to reuse the code.

- Adding a new method sp_package::get_typedef() to search
  for TYPE definitions in PACKAGE specifications.

- Adding a new method sp_head::get_typedef_package_spec()
  to search for TYPE definitions used by a PROCEDURE or FUNCTION.

- Adding a helper method
    Sp_handler::sp_cache_routine_reentrant_suppress_errors
  Adding a method Sp_handler::find_package_spec().