Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Categories: connectors experimental galera main
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

connectors experimental galera main
f6f2d6492166...
f6f2d64921662c73e82cce4486184796cc394235
Aleksey Midenkov
MDEV-30281 MTR: add --strip-hints to omit crash-report hints

A crash report contains, besides the actual server log and backtrace,
several explanatory "hint" paragraphs - bug-reporting boilerplate,
instructions for producing a better stack trace, "Attempting backtrace",
"Output from gdb follows", etc. They are useful once but noise for
someone who reads these reports all day.

Add --strip-hints (off by default). When given, these hints are omitted
and only the real content is kept.

Three sources are handled:

- The paragraphs the server writes to its error log (via the crash
  signal handler) are dropped from the "Server log from this test"
  excerpt by strip_crash_hints(). Each hint is a paragraph running from
  a recognizable opener line to the next blank line, so trimming is
  robust across server-version wording changes; the surrounding data
  (CURRENT_TEST, the signal line, "Server version", the backtrace,
  thread pointer, ...) is kept.

- My::CoreDump prints its own "Output from gdb/lldb follows ..." header;
  it is suppressed too, leaving just the debugger output.

- mariadb-test-run.pl's own end-of-run boilerplate in mtr_report_stats()
  ("The log files in var/log may give you some hint ..." and the
  MariaDB bug-tracker URL) is skipped as well.

$opt_strip_hints is declared with "our" so My::CoreDump and
mtr_report.pm can read it as $::opt_strip_hints.
f35b302b2845...
f35b302b28450e9595a1252f507af6da4a790112
Aleksey Midenkov
MDEV-39384 Use index in TR_table::query

TR_table::query() used table scan. Now utilize the index if possible,
with minimum validity detection. Getting trx_id by commit_ts is still
suboptimal is it does two index accesses (as limited by fields in
commit_ts index).

When the index detection fails TR_table::query() falls back to
original table scanning method.
f0ade4a4acc8...
f0ade4a4acc8bc2c356d4ccb56c9b6ab4f469228
Aleksey Midenkov
MDEV-30281 MTR: add --strip-backtrace and the --strip-log alias

On a crash the server writes its own stack backtrace (my_print_stacktrace)
to the error log, which MTR echoes in the crash report. When a core file
is available the gdb/lldb backtrace from My::CoreDump is more useful, so
this server-side report is often just noise.

Add --strip-backtrace (off by default). When given, it is dropped from
the "Server log from this test" excerpt by strip_backtrace(). The
gdb/lldb backtrace from a core file is a separate thing and is not
affected.

strip_backtrace() treats the backtrace as a bounded block rather than
matching lines everywhere, so a line outside a backtrace that merely ends
in an [0x...] address is kept:

  - a "Thread pointer:" or "Attempting backtrace" line starts a block;
  - inside a block every recognisable backtrace line is dropped: the
    intro prose, the thread pointer, blank lines, a "stack_bottom" or
    "Stack range" header, frame lines (both "...[0x...]" symbol frames
    and the bare "0x..." addresses printed by the frame-pointer walker)
    and the interleaved addr2line / my_addr_resolve diagnostics;
  - the first line that is not recognisable backtrace content ends the
    block and is kept, so pass-through resumes and a later backtrace (or
    the rest of the log) survives.

Because anything unrecognised ends the block rather than being dropped, a
backtrace with no "stack_bottom" line and no bracketed frames - as the
frame-pointer-walker and aborted-backtrace builds produce - does not
swallow the remainder of the log.

Also add --strip-log, a convenience alias that turns on all three of
--strip-hints, --strip-limits and --strip-backtrace at once.

All stripping runs before the --head-log/--tail-log trimming, so those
line counts apply to the already-stripped log.
ebdf6c43b41c...
ebdf6c43b41cd9acb043c96921fbcf51fc85a4a6
Aleksey Midenkov
MDEV-30281 MTR: add --list-combinations to list a test's combinations

Add --list-combinations (alias --lc) which, for the specified test(s),
prints the available combinations in the selectable "test,combination"
form and exits without running anything.

For each collected test it prints a summary line followed by one line
per combination, e.g.:

  $ mtr --list-combinations versioning.foreign
  Combinations for versioning.foreign: timestamp,trx_id
  versioning.foreign,timestamp
  versioning.foreign,trx_id

Each printed line is exactly what you pass back to mtr to run that one
combination (it round-trips). When a test draws combinations from more
than one .combinations file (several dimensions), the lines list the
full product, e.g. "test,dim1_value,dim2_value".
ead91a9e2c0e...
ead91a9e2c0e0e8bb05cb90bc0a6f41542efbb50
Aleksey Midenkov
MDEV-30281 MTR conf: options from .cnf file

MTR options are read from standard MariaDB conf file from [mtr]
section. Command-line options take precedence over configuration file.

There are two variants of config file location:

1. Default behaviour to look for global, server and user configs as
  described in

  https://mariadb.com/kb/en/configuring-mariadb-with-option-files/

2. Custom config location specified by MTR_CONFIG environment
  variable. None of other config files are read in this case.

If there is wrong setting in configuration file MTR fails and displays
the error message. It is clear from the error message from what source
the wrong option comes: config file or command-line. The config file
name and the line number is not printed which is the subject for
further improvement.

Implementation reuses the in-tree My::Config parser instead of adding a
third-party module. A new load_defaults() in mariadb-test-run.pl parses
the config files with My::Config and merges the [mtr] options into the
@ARGV array for further validation in Getopt::Long:

- Config options are merged into the beginning of the array so that
  command-line options take precedence. They are separated from the
  command-line options by ---end-of-config--- which is used as a marker
  for the source of failed options after Getopt::Long;

- a value-less [mtr] option that actually requires a value would let
  Getopt::Long swallow that marker (or the following option) as its
  value; this is detected and reported instead of failing obscurely, and
  the invalid-option message prints the bare option name;

- Global, server and user config files are searched; only the first
  existing among the global and server locations is used, then the
  user's ~/.my.cnf. A config file can also be specified explicitly via
  MTR_CONFIG, in which case no other file is read;

- Argument-less options are passed as is (My::Config::Option::option()
  renders them without a value);

- My::Config already honours !include; it was extended to also support
  !includedir, to skip whitespace-only lines and to ignore any other
  unknown ! directive instead of failing. A line beginning with '#' is
  treated as a comment, as in any my.cnf, not as an option named '#...'.
  (An earlier version of this patch was based on the CPAN MySQL::Config
  module, which ignored all ! directives.)
d60fe43150e1...
d60fe43150e10c2c1eee187362f4279ff9274150
Aleksey Midenkov
MDEV-30281 MTR: speed up test collection by caching result-dir listings

collect_one_test_case() located each test's .result/.rdiff files with a
per-test glob:

  for (<{$resdirglob}/$tname*.{rdiff,result}>) { ... }

Because of the "$tname*" wildcard, glob had to opendir and read the whole
result directory on every call, and the "{rdiff,result}" brace made it do
so twice (once per extension).  Run once per test, this is O(tests *
dirsize) - the suite's result directory is re-read ~2x for every test in
the suite.  Profiling the collection of all default suites showed this
single line accounting for 8.3s of the 13.1s phase (63%), and the phase
issuing 408k stat() calls (339k of them ENOENT).

Read each result directory once instead, indexing its .result/.rdiff
files by base test name (the leading run of characters before the first
'.' or ','), and look a test's files up in that cache.  The "$tname*"
glob was only a coarse prefilter anyway - the existing regex
"^$tname((?:,combo)*)\.(rdiff|result)$" does the real matching - so
indexing by base name yields exactly the same candidate set.

Stats (collecting all default suites, 6548 tests, warm cache):

  collection time        13.1s  ->  4.5s  (~2.9x)
  directory-open syscalls  23026 ->  5284

The collected test list (result_file, base_result, skip and comment for
every test) is byte-for-byte identical before and after.
c0d8b951b626...
c0d8b951b6265c0b5b52b28a6dec83c69db373cd
Aleksey Midenkov
MDEV-30281 MTR: fix --server-arg typo in embedded-server comments

Two comments in the embedded-server setup said "--sever-arg=" instead of
"--server-arg=" (the surrounding code uses the correct spelling). Comment
only; no behavior change. The typo is pre-existing (upstream), so it is
fixed here as a separate commit rather than folded into upstream history.
a77359a7dadc...
a77359a7dadc9e53529afc1885f59387fb8a447b
Dave Gosselin
Clarify NULL handling comment in next_min()
a1a9cc23df18...
a1a9cc23df18c6baf9df3e51651fbcda250ca644
Aleksey Midenkov
MDEV-30281 MTR: allow excluding suites with --suites=!NAME

Extend the --suite[s] option so a suite name prefixed with '!' excludes
that suite instead of adding it. This makes "run everything except X"
easy and lets a config-file suite list be trimmed on the command line.

Semantics:

  --suites=A,B        run suites A and B (as before)
  --suites=!A        run the default set minus A
  --suites=A,B,!B    run A (a '!' name is removed from the positive names)
  [mtr] suites=A,B
    + --suites=!B      run A (command-line exclusion trims the file set)

Rules:

  - If any plain (positive) names are given, they are the base set;
    otherwise the base is the default suite set.
  - Every '!'-prefixed name is then removed from the base.
  - Names accumulate across the [mtr] config file and the command line,
    so a '!' exclusion on the command line applies to the set configured
    in the file (see below).

To make the cross-source case work, --suites now accumulates its values
into @opt_suites instead of a last-wins scalar: the option handler pushes
each comma-separated value, so the [mtr] config file value and the
command-line value are both collected. The positive/negative resolution
then runs once in main(), producing the final suite list.

Exclusion matching ignores the "-<overlay>" suffix used in the default
list (e.g. "rpl-"), so --suites=!rpl removes "rpl-", and slashes are
preserved (--suites=!compat/oracle works).
9772870e1468...
9772870e1468e3e66976ee9b8c4899d2e0f4aed5
Aleksey Midenkov
MDEV-30281 MTR: make the debugger terminal configurable

The terminal emulator for interactive debuggers (--gdb, --exec-gdb, ...)
was hard-coded to xterm.  It is now configurable via the --terminal
option or the MTR_TERM environment variable (the option takes
precedence).  The template understands two placeholders, {title} (window
title) and {command} (the debugger invocation, expanded as separate argv
words); the default "xterm -title {title} -e {command}" reproduces the
previous behaviour.

My::Debugger::term_argv() expands the template into an argv list for the
mysqld/client/boot debuggers; the same template drives the --exec-<dbg>
shell-string wrapper, where {command} must be last.

For example, debug the bootstrap server for main.1st under gdb in a
KDE Konsole window:

  mtr --boot-gdb --terminal='konsole -p tabtitle={title} -e {command}' main.1st
94138faa470c...
94138faa470c3a47eece229eb40eb33c3d619851
Aleksey Midenkov
MDEV-30281 MTR: add --strip-limits to omit the resource-limits table

On a crash the server writes a "Resource Limits" table (read from
/proc/self/limits) to its error log, which MTR echoes in the crash
report. Like the bug-reporting hints it is rarely useful when reading
these reports routinely.

Add --strip-limits (off by default). When given, the table is dropped
from the "Server log from this test" excerpt by strip_resource_limits().

Unlike a hint paragraph the table has no trailing blank line - it runs
straight into the next section ("Core pattern:", ...). So the filter
removes the opener line (matching both "Resource Limits:" and the newer
"Resource Limits (excludes unlimited resources):"), the
"Limit ... Soft Limit ..." header and the "Max ..." rows, and stops at
the first line that is neither, keeping the surrounding data.

This is independent of --strip-hints.
905d29755804...
905d297558046c9f5997ae5b4fdc489322825dd0
Aleksey Midenkov
MDEV-30281 MTR: add --exec-rr / --exec-gdb to wrap every --exec

New debugger options run every mysqltest '--exec' command line under a
wrapper (rr record, or gdb --args in an xterm), so external tools invoked
by tests (myisampack, myisamchk, ...) can be traced or debugged without
naming them.

My::Debugger: each debugger with an 'exec' template gets an --exec-<dbg>
option auto-registered.  pre_setup() detects the requested one, runs its
one-time 'pre' hook, and exports MYSQLTEST_EXEC_WRAP (plus _RR_TRACE_DIR
for rr, and an xterm wrapper for terminal debuggers like gdb).

do_exec(): when MYSQLTEST_EXEC_WRAP is set it is injected into the command
line, after any leading shell NAME=VALUE assignments (quotes and
backslashes honoured), so the tool - not the assignment word - is what
gets wrapped.  The implicit /bin/sh -c that popen() supplies provides the
shell layer.
89d690768896...
89d69076889608d706a5c4ddb60fa2ed29a2dfea
Aleksey Midenkov
MDEV-30281 MTR: speed up test collection by caching result-dir listings

collect_one_test_case() located each test's .result/.rdiff files with a
per-test glob:

  for (<{$resdirglob}/$tname*.{rdiff,result}>) { ... }

Because of the "$tname*" wildcard, glob had to opendir and read the whole
result directory on every call, and the "{rdiff,result}" brace made it do
so twice (once per extension).  Run once per test, this is O(tests *
dirsize) - the suite's result directory is re-read ~2x for every test in
the suite.  Profiling the collection of all default suites showed this
single line accounting for 8.3s of the 13.1s phase (63%), and the phase
issuing 408k stat() calls (339k of them ENOENT).

Read each result directory once instead, indexing its .result/.rdiff
files by base test name (the leading run of characters before the first
'.' or ','), and look a test's files up in that cache.  The "$tname*"
glob was only a coarse prefilter anyway - the existing regex
"^$tname((?:,combo)*)\.(rdiff|result)$" does the real matching - so
indexing by base name yields exactly the same candidate set.

Stats (collecting all default suites, 6548 tests, warm cache):

  collection time        13.1s  ->  4.5s  (~2.9x)
  directory-open syscalls  23026 ->  5284

The collected test list (result_file, base_result, skip and comment for
every test) is byte-for-byte identical before and after.
830e31f4a9e2...
830e31f4a9e2f7697a45ad31437bb00e09b5b6c8
Aleksey Midenkov
MDEV-30281 MTR conf: add --mtr-config-only (-M)

--mtr-config-only (-M) makes --defaults-file / --defaults-extra-file be
read for MTR's own [mtr] options only, without also being applied as the
server config template.

Normally those two options serve two consumers of the same file: the
[mtr] group configures MTR itself (load_defaults), while the remaining
groups - [mysqld]/[client]/... - become the test servers' config template
(collect_option). -M opts out of the second use: once load_defaults has
read [mtr], it removes --defaults-file / --defaults-extra-file from @ARGV,
so the main GetOptions/collect_option never turns them into a template.

The switch may be given on the command line, or set inside the [mtr]
section of the config file itself. In the latter case load_defaults
notices mtr-config-only while reading [mtr] and drops the file options on
the fly, so

    [mtr]
    mtr-config-only

turns any file passed via --defaults-file into an MTR-only config.

get_defaults_options() consumes -M out of @ARGV like the other MTR-only
defaults options; it is a directive, not passed through to GetOptions.

-M is only relevant for a command-line --defaults-file /
--defaults-extra-file (the dual-use options). The MTR_CONFIG /
MTR_CONFIG_EXTRA environment variables are read only for [mtr] and never
reach collect_option, so they are mtr-config-only already and -M is a no-op
for them.
76a3922df0b4...
76a3922df0b4ee4bcdd8536aa80cd8faebc74856
Aleksey Midenkov
MDEV-39384 Wrong result when selecting from precise-versioned table

SELECT ... FOR SYSTEM_TIME AS OF <timestamp> on a trx-precise table
scans transaction_registry via TR_table::query(MYSQL_TIME&, bool) to
translate the timestamp into a trx_id, comparing each row:

    Item_func_le/ge(Item_field(commit_timestamp), Item_datetime_literal)

transaction_registry is opened directly, bypassing setup_tables(), so
TABLE::map stays 0. Item_field::used_tables() then returns 0, and
Item::const_item() (used_tables() == 0) wrongly reports the field as
const. That makes Arg_comparator::cache_converted_constant() wrap the
field in an Item_cache (its handler timestamp differs from the
aggregated datetime handler), snapshotting record[0] once. The scan then
compares every row against the first row's commit_timestamp -> wrong
result.

Fix: in TR_table::query(MYSQL_TIME&, bool) temporarily set table->map to
a nonzero value for the duration of the scan so used_tables() != 0 and
const_item() is false, disabling the caching. A
DBUG_ASSERT(!field->const_item()) guards the invariant.
70bdec4dbdb2...
70bdec4dbdb21c02b17a835d2127a3f3c98fc048
Aleksey Midenkov
MDEV-39384 Comments on update_virtual_fields() modes
6d381ea177fb...
6d381ea177fba2933648f8970357ef5fe390bf40
Aleksey Midenkov
MDEV-30281 MTR: add --exit-line to stop a test at a given line

--exit-line=N stops a test before the command at line N of the test
file, exactly as if an --exit directive were placed there.  Handy for
bisecting where a test starts to misbehave, and for debugging a test
without extracting a standalone test case: record the run under rr
(--rr), then reverse-replay from the end of the trace straight to the
SQL command of interest.

mariadb-test-run.pl gains --exit-line (-l) and forwards it to mysqltest.
The option is global: it applies to every test in the run, each stopping
at line N of its own test file (so N is per-file).  A test whose file has
fewer than N lines is unaffected and runs to completion.

mysqltest gains the --exit-line (-l) option: after reading each command,
if we are in the top-level test file (cur_file == file_stack) and the
command starts at or past the requested line, it aborts like Q_EXIT.
Gating to the main file keeps line numbers of sourced includes from
triggering it.

suite/mtr/feat exercises it by running a child mtr with --exit-line=5 on
main.1st (line 4 "show databases;", line 5 "show tables in mysql;"): the
run stops before line 5, so the recorded result's tables section is
missing and the test fails with a length mismatch - proving the exit
landed exactly at the requested line.  The child's whole output is kept
(no grep); volatile banner/footer lines are cut by prefix via $NORM_RUN.
6c26db4ec675...
6c26db4ec675f1f682b74b73d855ba6c1ac1c8e2
Aleksey Midenkov
MDEV-30281 MTR: add --strip-limits to omit the resource-limits table

On a crash the server writes a "Resource Limits" table (read from
/proc/self/limits) to its error log, which MTR echoes in the crash
report. Like the bug-reporting hints it is rarely useful when reading
these reports routinely.

Add --strip-limits (off by default). When given, the table is dropped
from the "Server log from this test" excerpt by strip_resource_limits().

Unlike a hint paragraph the table has no trailing blank line - it runs
straight into the next section ("Core pattern:", ...). So the filter
removes the opener line (matching both "Resource Limits:" and the newer
"Resource Limits (excludes unlimited resources):"), the
"Limit ... Soft Limit ..." header and the "Max ..." rows, and stops at
the first line that is neither, keeping the surrounding data.

This is independent of --strip-hints.
638bd56ae1f8...
638bd56ae1f82f237fd5087caf27b4c030fb635a
Aleksey Midenkov
MDEV-30281 MTR: add --exec-rr / --exec-gdb to wrap every --exec

New debugger options run every mysqltest '--exec' command line under a
wrapper (rr record, or gdb --args in an xterm), so external tools invoked
by tests (myisampack, myisamchk, ...) can be traced or debugged without
naming them.

My::Debugger: each debugger with an 'exec' template gets an --exec-<dbg>
option auto-registered.  pre_setup() detects the requested one, runs its
one-time 'pre' hook, and exports MYSQLTEST_EXEC_WRAP (plus _RR_TRACE_DIR
for rr, and an xterm wrapper for terminal debuggers like gdb).

do_exec(): when MYSQLTEST_EXEC_WRAP is set it is injected into the command
line, after any leading shell NAME=VALUE assignments (quotes and
backslashes honoured), so the tool - not the assignment word - is what
gets wrapped.  The implicit /bin/sh -c that popen() supplies provides the
shell layer.
636f154bb49b...
636f154bb49b301d5c5272870ac7eba439864b90
Arcadiy Ivanov
MDEV-40376 `Protocol::end_statement` assertion when window function fills HEAP tmp table

`save_window_function_values()` did not take into account that one can
get `HA_ERR_RECORD_FILE_FULL` on `ha_update_row()`. With blob support in
heap, it can now happen more easily (expressions wider than 512
characters are promoted to TEXT in tmp tables, and each update of a blob
result column allocates a continuation record). However, it could also
happen with Aria tables, which is apparently not tested. The error was
returned as a plain `true` with an empty diagnostics area: debug builds
hit `Assertion '0'` in `Protocol::end_statement()`, release builds send
a bare OK packet after the result set metadata, which the client
mis-parses (appears as a hang / lost connection).

Errors of the computation are now exposed:
`save_window_function_values()` calls `handler::print_error()` for any
failure it does not recover from, the previously ignored `ha_rnd_pos()`
return values are checked and reported, and `compute_window_func()`
distinguishes a read error from EOF and stops the row scan as soon as
the per-row loop fails.

The overflow itself is recovered from by converting the tmp table to
the disk-based tmp engine in place: the rows keep their identity and
their contents, only their positions change, and those are translated
in the rowid sequence that the computation is built on.

The filesort result of a window sort is a sequence of row positions
that covers every row of the table exactly once (the sort is set up
without a limit) and is the only place where the positions are kept:
the row scan and all frame cursors read the rows through it. The
conversion (`Window_rowid_remapper`) therefore copies the rows in the
order of that sequence, which makes the new position of a row known as
soon as the row has been written, and stores it back into the very slot
the old position was read from.

For the new position to always fit into the slot, a window sort stores
the row positions in slots of a fixed 8 bytes
(`Filesort::min_ref_length`, set to the new `TMP_TABLE_MAX_REF_LENGTH`),
which holds the position of any engine an internal tmp table can use: a
pointer into memory or a data file offset. `make_sortkey()` zero-pads a
shorter position to its slot, and `SORT_INFO::ref_length` carries the
slot width to the readers, so `init_read_record()` no longer re-derives
it from `handler::ref_length`, which changes when the table is
converted. Sorts that do not set a minimum, and engines whose positions
are wider than 8 bytes, are unchanged byte for byte. A sequence held in
memory is thereby rewritten in place, keeping its layout, so the frame
cursors' places in it stay valid.

A sequence held in a temporary file is written and read through an
`IO_CACHE`, which encrypts the file when tmp file encryption is
enabled, so the file can not be rewritten in place: the old sequence is
streamed out of its cache and the translated sequence into a new cached
temporary file, which then replaces the old one under the cache. The
frame cursors' slave caches stay linked to the master through
`next_file_user` across the replacement, and are re-created from the
new master afterwards (`Frame_cursor::rowids_rewritten()`). Should
re-creating one fail, the cursor forgets its already-released cache, so
that it is not released a second time when the cursor is destroyed, and
reports the failure.

The row whose update did not fit is written from `record[0]`, which
holds its new image, instead of being read from the table, so the
conversion applies the update that failed. Blob values of `record[0]`
that were read from the HEAP table can point into the handler's shared
blob reassembly buffer (`hp_read_blobs()`, blob values that span
multiple allocation blocks), which reading the other rows overwrites,
so they are first given memory of their own.

When the window sort was set up with a deferred filter (a HAVING clause
deferred to the final ORDER BY sort), the sequence omits the rows the
filter rejected and the conversion drops them: every later reader of
the table applies the same filter, either directly or by reading the
table through a sort that does, so such rows can never reach the
result.

`create_internal_tmp_table_from_heap()` gains a `Tmp_table_row_copier`
hook for the copy from the HEAP table to the Aria or MyISAM table that
replaces it. The plain copy loop and the append of the pending
`record[0]` become `Tmp_table_default_copier`, used when the caller
passes no copier, so both cases of the conversion run through the same
code, defined together with `Window_rowid_remapper` just before the
function. `ha_end_bulk_insert()` is called also when the copy fails, so
the new handler does not keep bulk insert state when its table is
dropped; the pending row is thereby written while the bulk insert is
still active, which does not delay its duplicate detection, as bulk
insert does not cache unique keys (`maria_init_bulk_insert()` skips
`HA_NOSAME` keys). An ignored duplicate of the pending row is passed to
the caller in `Tmp_table_row_copier::duplicate_row_error`, which sets
`*is_duplicate`.

Carrying the running computation over the conversion, instead of
re-running it, has two visible consequences:

- Compound expressions containing window functions (`items_to_copy`)
  are evaluated exactly once per row, as they are when the table does
  not overflow, so the conversion does not have to be refused when
  such an expression is non-deterministic (`RAND_TABLE_BIT`: a user
  variable assignment, a non-deterministic stored function); those
  statements produce their result.
- The statement sorts once, so sort related aggregate warnings, e.g.
  the `max_sort_length` truncation counts, are reported for one pass.

The rows are copied in the order of the sequence, so the converted
table does not hold them in the order they were inserted in. This is
safe even when that order is what the table was built for
(`TABLE::keep_row_order`, set for `ROWNUM()` with GROUP BY or ORDER
BY): before HEAP supported blobs, such a tmp table was created in the
disk engine from the start and these statements simply ran, so refusing
here would be a regression against that behavior.

1. `ROWNUM()` values are materialized into the tmp table rows during
  the fill, before the window computation begins; the copy moves them
  verbatim, so their pairing with the rows can not change.
2. The rowid sequence is translated in place, so the running window
  computation continues over exactly the same row order, and
  tie-sensitive window function values (`ROW_NUMBER`, frames over tied
  keys) keep the values the interrupted pass had already produced.
3. Consumers that scan the converted table afterwards see the rows in
  the copy order. Among rows with equal sort keys that order differs
  from the insertion order, but such tie order is unspecified and
  already differs between the memory and disk tmp engines.

The converted table keeps `keep_row_order= true` so the copy order is
also the order the disk engine preserves from then on.

Tests, in `heap.blob_window_overflow` unless noted: a single window
over all rows and a partitioned window overflow mid-computation and
must transparently convert; the side-effecting window expression test
verifies that the user variable is assigned exactly once per row,
against the same query that does not overflow; a fourth test covers the
sequence being an array in memory instead of a merge file; a fifth test
covers the `keep_row_order` conversion, verifying that every `ROWNUM()`
value keeps its insertion-order pairing with its row across the
conversion; a sixth test covers blob values larger than a HEAP
allocation block surviving the conversion of the pending row;
`heap.blob_window_overflow_encrypt` runs the conversion with encrypted
temporary files; `heap.blob_window_overflow_debug` covers the failure
to re-create a slave cache of the sequence
(`simulate_window_seq_slave_oom`), which previously hung the server in
the cursor destructor.
636189be0c53...
636189be0c537ab0ba6a70f54db75a9ef659fb92
forkfun
Merge branch '11.8' into '12.3'
5e30f3b2f3e4...
5e30f3b2f3e4e11ebb151ba1c59fe540e9af7f16
Aleksey Midenkov
MDEV-30281 MTR: add --strip-hints to omit crash-report hints

A crash report contains, besides the actual server log and backtrace,
several explanatory "hint" paragraphs - bug-reporting boilerplate,
instructions for producing a better stack trace, "Attempting backtrace",
"Output from gdb follows", etc. They are useful once but noise for
someone who reads these reports all day.

Add --strip-hints (off by default). When given, these hints are omitted
and only the real content is kept.

Three sources are handled:

- The paragraphs the server writes to its error log (via the crash
  signal handler) are dropped from the "Server log from this test"
  excerpt by strip_crash_hints(). Each hint is a paragraph running from
  a recognizable opener line to the next blank line, so trimming is
  robust across server-version wording changes; the surrounding data
  (CURRENT_TEST, the signal line, "Server version", the backtrace,
  thread pointer, ...) is kept.

- My::CoreDump prints its own "Output from gdb/lldb follows ..." header;
  it is suppressed too, leaving just the debugger output.

- mariadb-test-run.pl's own end-of-run boilerplate in mtr_report_stats()
  ("The log files in var/log may give you some hint ..." and the
  MariaDB bug-tracker URL) is skipped as well.

$opt_strip_hints is declared with "our" so My::CoreDump and
mtr_report.pm can read it as $::opt_strip_hints.
584abff30387...
584abff3038718b1aa893c7310f0196d953244c1
Dave Gosselin
MDEV-25964:  Unexpected bypass of lock

When an uncommitted transaction inserts rows into a table and
another statement locks rows in the same table (SELECT ... FOR UPDATE)
while computing a MIN or MAX, then:
  1. In a Debug build, the server aborts on an assertion
  2. In a Release build, the server returns wrong results
These errors occur because, while reading a group of rows for computing
a MAX, the transaction timeout error was swallowed.

Under the scenario described above and captured in the new test at this
commit, QUICK_GROUP_MIN_MAX_SELECT::next_max() emits a lock timeout error
during QUICK_GROUP_MIN_MAX_SELECT::get_next() but the error was suppressed
if we computed a MIN.

The InnoDB storage engine has an unwritten convention that after it has
returned a fatal error (which is any error except HA_ERR_END_OF_FILE
or HA_ERR_KEY_NOT_FOUND), then the SQL layer should not try to make
any further reads.  This is because InnoDB might have rolled back
the current transaction already.  So in the case of an error, return
immediately from QUICK_GROUP_MIN_MAX_SELECT::get_next().
57600aa67350...
57600aa673503872af5f378a84521612b10db52d
Rex Johnston
MDEV-36610 Subquery wrongly eliminated by table elimination

When equality propagation (build_equal_items()) merges an equality that
contains a subquery, such as "t1.a = (SELECT ...)", with an outer join's
ON equality, it can inject a reference to that subquery into the ON
expression. If the join columns have compatible types the subquery ends
up as the constant of a multiple equality (which Item::walk() skips); if
they differ (e.g. BIGINT vs INT) the field cannot be merged and the
subquery is substituted in as a plain "tbl.col = (SELECT ...)" argument.

In the latter case, if that outer join is removed by table elimination,
mark_as_eliminated() walks the ON expression and flags the shared
Item_subselect as eliminated. The subquery, however, still lives in
another part of the query and has to be executed, tripping
DBUG_ASSERT(!eliminated) in Item_subselect::exec() (and, in release
builds, disabling the subquery cache and hiding it from EXPLAIN).

The surviving reference can be:
- a WHERE/HAVING/select-list/ORDER/GROUP expression (subquery written
  there and pushed down into the eliminated ON), or
- the ON expression of an outer join that was not eliminated (subquery
  written in a surviving outer ON and pushed down into an eliminated
  inner one).

Fix: after table elimination, walk the expressions that survive into
execution (WHERE, HAVING, select list, ORDER/GROUP BY and the ON
expressions of outer joins that were not eliminated) and clear the
"eliminated" flag on any subquery still reachable from them.

Because a subquery can also be the constant of a multiple equality, and
Item::walk() does not visit an Item_equal's constant, Item_equal gets an
unmark_as_eliminated_processor() override that descends into its constant
explicitly.

Assisted by Claude Opus
55ebf1673cd3...
55ebf1673cd319876859a614dc6b652746e16f6b
Aleksey Midenkov
MDEV-30281 MTR: make the debugger terminal configurable

The terminal emulator for interactive debuggers (--gdb, --exec-gdb, ...)
was hard-coded to xterm.  It is now configurable via the --terminal
option or the MTR_TERM environment variable (the option takes
precedence).  The template understands two placeholders, {title} (window
title) and {command} (the debugger invocation, expanded as separate argv
words); the default "xterm -title {title} -e {command}" reproduces the
previous behaviour.

My::Debugger::term_argv() expands the template into an argv list for the
mysqld/client/boot debuggers; the same template drives the --exec-<dbg>
shell-string wrapper, where {command} must be last.

For example, debug the bootstrap server for main.1st under gdb in a
KDE Konsole window:

  mtr --boot-gdb --terminal='konsole -p tabtitle={title} -e {command}' main.1st
547d3c64dfb1...
547d3c64dfb106ea8ce14cca35a265aed8b1b9a2
Aleksey Midenkov
MDEV-30281 MTR: test --tail-log / --tail-warnings in suite/mtr

Run a child mariadb-test-run.pl on misc.crash (SIGSEGV) and misc.warn
(system-versioned SYSTEM_TIME LIMIT overflow) and check that the crash and
shutdown-warnings reports are trimmed by --head-log / --tail-log / --strip-log
and --tail-warnings.  Also exercise --list-combinations and --suite negation.

The child's volatile output (pids, timestamps, addresses, an unbounded
backtrace, ...) is reduced to a deterministic skeleton with include/grep.inc
and normalized with --replace_regex: the two anchor backtrace frames
(my_print_stacktrace, main) are kept while the variable middle frames and the
per-line values are cut down to "<...cut...>".  The test is gated behind
--big-test.
51fe3921a859...
51fe3921a8591ac2136d1217d3d3ee38768f7b1b
Aleksey Midenkov
MDEV-30281 MTR: add --exit-line to stop a test at a given line

--exit-line=N stops a test before the command at line N of the test
file, exactly as if an --exit directive were placed there.  Handy for
bisecting where a test starts to misbehave, and for debugging a test
without extracting a standalone test case: record the run under rr
(--rr), then reverse-replay from the end of the trace straight to the
SQL command of interest.

mariadb-test-run.pl gains --exit-line (-l) and forwards it to mysqltest.
The option is global: it applies to every test in the run, each stopping
at line N of its own test file (so N is per-file).  A test whose file has
fewer than N lines is unaffected and runs to completion.

mysqltest gains the --exit-line (-l) option: after reading each command,
if we are in the top-level test file (cur_file == file_stack) and the
command starts at or past the requested line, it aborts like Q_EXIT.
Gating to the main file keeps line numbers of sourced includes from
triggering it.

suite/mtr/feat exercises it by running a child mtr with --exit-line=5 on
main.1st (line 4 "show databases;", line 5 "show tables in mysql;"): the
run stops before line 5, so the recorded result's tables section is
missing and the test fails with a length mismatch - proving the exit
landed exactly at the requested line.  The child's whole output is kept
(no grep); volatile banner/footer lines are cut by prefix via $NORM_RUN.
4a986b13ecbb...
4a986b13ecbb843ba2b7f1b763d46f8d52d215ae
Aleksey Midenkov
MDEV-30281 MTR: fix --server-arg typo in embedded-server comments

Two comments in the embedded-server setup said "--sever-arg=" instead of
"--server-arg=" (the surrounding code uses the correct spelling). Comment
only; no behavior change. The typo is pre-existing (upstream), so it is
fixed here as a separate commit rather than folded into upstream history.
493f50513392...
493f50513392ba0dd7003b2b1f413f5dcf80fffb
Aleksey Midenkov
MDEV-30281 MTR: add --strip-backtrace and the --strip-log alias

On a crash the server writes its own stack backtrace (my_print_stacktrace)
to the error log, which MTR echoes in the crash report. When a core file
is available the gdb/lldb backtrace from My::CoreDump is more useful, so
this server-side report is often just noise.

Add --strip-backtrace (off by default). When given, it is dropped from
the "Server log from this test" excerpt by strip_backtrace(). The
gdb/lldb backtrace from a core file is a separate thing and is not
affected.

strip_backtrace() treats the backtrace as a bounded block rather than
matching lines everywhere, so a line outside a backtrace that merely ends
in an [0x...] address is kept:

  - a "Thread pointer:" or "Attempting backtrace" line starts a block;
  - inside a block every recognisable backtrace line is dropped: the
    intro prose, the thread pointer, blank lines, a "stack_bottom" or
    "Stack range" header, frame lines (both "...[0x...]" symbol frames
    and the bare "0x..." addresses printed by the frame-pointer walker)
    and the interleaved addr2line / my_addr_resolve diagnostics;
  - the first line that is not recognisable backtrace content ends the
    block and is kept, so pass-through resumes and a later backtrace (or
    the rest of the log) survives.

Because anything unrecognised ends the block rather than being dropped, a
backtrace with no "stack_bottom" line and no bracketed frames - as the
frame-pointer-walker and aborted-backtrace builds produce - does not
swallow the remainder of the log.

Also add --strip-log, a convenience alias that turns on all three of
--strip-hints, --strip-limits and --strip-backtrace at once.

All stripping runs before the --head-log/--tail-log trimming, so those
line counts apply to the already-stripped log.
311af368d833...
311af368d833e02a06a56dd22230116a93fbe66c
Dave Gosselin
MDEV-25964:  Unexpected bypass of lock

When an uncommitted transaction inserts rows into a table and
another statement locks rows in the same table (SELECT ... FOR UPDATE)
while computing a MIN or MAX, then:
  1. In a Debug build, the server aborts on an assertion
  2. In a Release build, the server returns wrong results
These errors occur because, while reading a group of rows for computing
a MAX, the transaction timeout error was swallowed.

Under the scenario described above and captured in the new test at this
commit, QUICK_GROUP_MIN_MAX_SELECT::next_max() emits a lock timeout error
during QUICK_GROUP_MIN_MAX_SELECT::get_next() but the error was suppressed
if we computed a MIN.

The InnoDB storage engine has an unwritten convention that after it has
returned a fatal error (which is any error except HA_ERR_END_OF_FILE
or HA_ERR_KEY_NOT_FOUND), then the SQL layer should not try to make
any further reads.  This is because InnoDB might have rolled back
the current transaction already.  So in the case of an error, return
immediately from QUICK_GROUP_MIN_MAX_SELECT::get_next().
29a423772c4a...
29a423772c4a893408297a03a28704827ea2f7f8
Aleksey Midenkov
MDEV-30281 MTR conf: add --mtr-config-only (-M)

--mtr-config-only (-M) makes --defaults-file / --defaults-extra-file be
read for MTR's own [mtr] options only, without also being applied as the
server config template.

Normally those two options serve two consumers of the same file: the
[mtr] group configures MTR itself (load_defaults), while the remaining
groups - [mysqld]/[client]/... - become the test servers' config template
(collect_option). -M opts out of the second use: once load_defaults has
read [mtr], it removes --defaults-file / --defaults-extra-file from @ARGV,
so the main GetOptions/collect_option never turns them into a template.

The switch may be given on the command line, or set inside the [mtr]
section of the config file itself. In the latter case load_defaults
notices mtr-config-only while reading [mtr] and drops the file options on
the fly, so

    [mtr]
    mtr-config-only

turns any file passed via --defaults-file into an MTR-only config.

get_defaults_options() consumes -M out of @ARGV like the other MTR-only
defaults options; it is a directive, not passed through to GetOptions.

-M is only relevant for a command-line --defaults-file /
--defaults-extra-file (the dual-use options). The MTR_CONFIG /
MTR_CONFIG_EXTRA environment variables are read only for [mtr] and never
reach collect_option, so they are mtr-config-only already and -M is a no-op
for them.
27cf87f0fd6c...
27cf87f0fd6c0ac1f6eecdb0fcaee40d4680503f
Aleksey Midenkov
MDEV-30281 MTR: allow excluding suites with --suites=!NAME

Extend the --suite[s] option so a suite name prefixed with '!' excludes
that suite instead of adding it. This makes "run everything except X"
easy and lets a config-file suite list be trimmed on the command line.

Semantics:

  --suites=A,B        run suites A and B (as before)
  --suites=!A        run the default set minus A
  --suites=A,B,!B    run A (a '!' name is removed from the positive names)
  [mtr] suites=A,B
    + --suites=!B      run A (command-line exclusion trims the file set)

Rules:

  - If any plain (positive) names are given, they are the base set;
    otherwise the base is the default suite set.
  - Every '!'-prefixed name is then removed from the base.
  - Names accumulate across the [mtr] config file and the command line,
    so a '!' exclusion on the command line applies to the set configured
    in the file (see below).

To make the cross-source case work, --suites now accumulates its values
into @opt_suites instead of a last-wins scalar: the option handler pushes
each comma-separated value, so the [mtr] config file value and the
command-line value are both collected. The positive/negative resolution
then runs once in main(), producing the final suite list.

Exclusion matching ignores the "-<overlay>" suffix used in the default
list (e.g. "rpl-"), so --suites=!rpl removes "rpl-", and slashes are
preserved (--suites=!compat/oracle works).
27999f8a1c64...
27999f8a1c64f31f206f55791e02d6645f145795
Aleksey Midenkov
MDEV-30281 MTR: test --tail-log / --tail-warnings in suite/mtr

Run a child mariadb-test-run.pl on misc.crash (SIGSEGV) and misc.warn
(system-versioned SYSTEM_TIME LIMIT overflow) and check that the crash and
shutdown-warnings reports are trimmed by --head-log / --tail-log / --strip-log
and --tail-warnings.  Also exercise --list-combinations and --suite negation.

The child's volatile output (pids, timestamps, addresses, an unbounded
backtrace, ...) is reduced to a deterministic skeleton with include/grep.inc
and normalized with --replace_regex: the two anchor backtrace frames
(my_print_stacktrace, main) are kept while the variable middle frames and the
per-line values are cut down to "<...cut...>".  The test is gated behind
--big-test.
24d1f52066c3...
24d1f52066c3177dc89a6e973130d315ef2030ec
Aleksey Midenkov
MDEV-39384 Debug trace

mtrr --mysqld=--debug=d,vers_trx_id,query:i:o,/tmp/good.log bug/v.trx_id,debug
1b10b34d0fb6...
1b10b34d0fb6d7770297d50ac51ca3df41ea18b4
Aleksey Midenkov
MDEV-30281 MTR: add --list-combinations to list a test's combinations

Add --list-combinations (alias --lc) which, for the specified test(s),
prints the available combinations in the selectable "test,combination"
form and exits without running anything.

For each collected test it prints a summary line followed by one line
per combination, e.g.:

  $ mtr --list-combinations versioning.foreign
  Combinations for versioning.foreign: timestamp,trx_id
  versioning.foreign,timestamp
  versioning.foreign,trx_id

Each printed line is exactly what you pass back to mtr to run that one
combination (it round-trips). When a test draws combinations from more
than one .combinations file (several dimensions), the lines list the
full product, e.g. "test,dim1_value,dim2_value".
127c8aa00066...
127c8aa0006640952119f677ae593780d04b683b
Aleksey Midenkov
MDEV-30281 MTR conf: report config file errors as file:line without croak

Make every wrong setting in a server config file fail with a
human-friendly message and no Perl stack trace or source-code line.

My::Config now treats parse errors as data errors instead of caller
bugs: each parse failure dies with a clean "file:line: reason" message
naming the config file and the offending line number, and without the
"at FILE line N." Perl location. This also delivers the file name and
line number that were previously missing. The internal misuse/invariant
assertions keep using croak, which is the right tool there.

load_defaults() (in mariadb-test-run.pl) catches that die and re-throws
it through mtr_error(), which flushes output, prints the uniform mtr
error format and exits in a controlled way. This replaces the earlier
fragile approach of stripping the location suffix off a croak with a
regexp.

mariadb-test-run.pl: validate the parallel option value before the
numeric comparison, so a non-numeric or negative value reports a clean
error instead of an "Argument isn't numeric" warning with a code line.
This path is shared with the command line, so it is fixed there too.
11745b6f865a...
11745b6f865a9f0fc46f3dc0bb6d47f8b1ae7626
Aleksey Midenkov
MDEV-30281 MTR conf: honor standard --defaults-* options

Wire the standard MariaDB defaults options into MTR's own option loading,
mirroring libmariadb's get_defaults_options()/my_load_defaults():

  --no-defaults            do not read any option file
  --defaults-file          read [mtr] from this file only
  --defaults-extra-file    read this file in addition (extra-file slot)
  --defaults-group-suffix  also read [mtr<suffix>]
  --print-defaults        print the [mtr] options and exit

--defaults-file and --defaults-extra-file keep their existing MTR meaning
as the server config *template* too: the two consumers read different
groups of the same file - [mysqld]/[client]/... for the template (via
collect_option), [mtr] here. A command-line --defaults-file /
--defaults-extra-file overrides the MTR_CONFIG / MTR_CONFIG_EXTRA
environment variable; --defaults-group-suffix overrides
MARIADB_GROUP_SUFFIX / MYSQL_GROUP_SUFFIX.

An explicitly named file (--defaults-file or $MTR_CONFIG) that does not
exist is an error, matching libmariadb; a missing file in the standard
search order is still skipped silently.

Order of execution in command_line_setup():

  1. get_defaults_options() runs first (before the main GetOptions). It
    consumes the MTR-only options (--no-defaults, --defaults-group-suffix,
    --print-defaults) out of @ARGV, and *peeks* --defaults-file /
    --defaults-extra-file off a copy of @ARGV so they stay in @ARGV for
    the second consumer.

  2. load_defaults() reads the [mtr] group from the selected files and
    prepends the resulting options into @ARGV, ahead of the
    ---end-of-config--- marker.

  3. My::Debugger::fix_options() adjusts optional-argument options.

  4. The main GetOptions(%options) parses everything - the config options
    (before the marker) and the command line (after it). Command-line
    options come later and therefore take precedence. --defaults-file is
    handed to collect_option for the template. Options still unrecognized
    afterwards are validated in a manual loop that uses the marker to
    report whether a bad option came from the config file or the command
    line.

Getopt::Long is configured "pass_through", so a parse leaves any option it
does not declare in @ARGV instead of erroring. Because pass_through is on,
step 1 skims off only the defaults options without touching the rest, and
step 4 does the real parse; neither errors on options it does not declare.
0d6a52d15233...
0d6a52d152334aeaf7bde5356f22c3c35da15bbf
Aleksey Midenkov
MDEV-40480 LOAD DATA leaves a stale STORED generated column after a BEFORE INSERT trigger changes its base column

On the LOAD DATA path the base columns are filled directly from the
input file and fill_record_n_invoke_before_triggers() is then called
with an empty field list (there is no SET clause). After the BEFORE
INSERT trigger changed a base column, the stored generated columns were
not recomputed, so they kept the value derived from the pre-trigger
input (e.g. g=24 instead of 40 for g=v*2 with v set to 20 by the
trigger). A regular INSERT was unaffected.

The recompute was guarded by "fields.elements". That condition is a
leftover from the original computed-columns implementation
(f7a75b999b4), where fill_record_n_invoke_before_triggers() had no
TABLE* argument and had to reverse-derive the table from the first
item of the field list:

    if (fields.elements)
    {
      fld= (Item_field*)f++;
      item_field= fld->field_for_view_update();
      table= item_field->field->table;
      ...
    }

With an empty field list there was no way to obtain the table, so the
recompute was silently skipped. Since bc4a456758c (MDEV-452) the
function receives TABLE* explicitly, which made the whole derivation
dead code (as the in-place DBUG_ASSERT(table == item_field->field->table)
confirmed). Recompute the virtual fields unconditionally on
table->vfield, the same way the Field** overload of
fill_record_n_invoke_before_triggers() already does.
099e15c02d4f...
099e15c02d4ff2e34f3e7c341ff63b8426825ab2
Aleksey Midenkov
MDEV-30281 MTR: add --combination-select to run one combination

Add --combination-select=N (short alias -c) to run only a single
combination from a .combinations file instead of every one.

N is the position of the combination in the file, counting the []
sections in file order:

  1  first combination      -1  last combination
  2  second combination    -2  last but one
  ...                        ...

So --combination-select=-1 runs only the last combination, -c 2 runs
the second one, etc. N must be a non-zero integer; 0, a non-integer, or
a value outside the range of a given file is an error.

The integer format is validated up front in command_line_setup(), so a
bad value is rejected even for tests that have no .combinations file. The
range and the selection itself are applied in combinations_from_file(),
which is the single point through which every .combinations file is read
(both the suite-level "combinations" file and per-test
"<test>.combinations"), so it always follows the [] section order of the
file. When a test draws combinations from more than one file, N selects
within each file.

It is ignored when --combination is given (that already replaces the
.combinations files with command-line combinations).
030b30419fbd...
030b30419fbd4b25f1b29e22042c19cf190b27d4
Aleksey Midenkov
MDEV-30281 MTR: add --combination-select to run one combination

Add --combination-select=N (short alias -c) to run only a single
combination from a .combinations file instead of every one.

N is the position of the combination in the file, counting the []
sections in file order:

  1  first combination      -1  last combination
  2  second combination    -2  last but one
  ...                        ...

So --combination-select=-1 runs only the last combination, -c 2 runs
the second one, etc. N must be a non-zero integer; 0, a non-integer, or
a value outside the range of a given file is an error.

The integer format is validated up front in command_line_setup(), so a
bad value is rejected even for tests that have no .combinations file. The
range and the selection itself are applied in combinations_from_file(),
which is the single point through which every .combinations file is read
(both the suite-level "combinations" file and per-test
"<test>.combinations"), so it always follows the [] section order of the
file. When a test draws combinations from more than one file, N selects
within each file.

It is ignored when --combination is given (that already replaces the
.combinations files with command-line combinations).
BuildBot (0.8.9) working for the MariaDB project.
Page built: Wed 29 Jul 2026 23:46:35 (UTC)