Console View
|
Categories: connectors experimental galera main |
|
| connectors | experimental | galera | main | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Merge branch '11.4' into 11.8 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alexander Barkov
bar@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38698 mysql_upgrade does not fix charset and collation for mysql.user In case if the view mysql.user was created (e.g. in 10.6) with a pre- 18edb0959fe5384673ebd36d72029fa8003ff5d9 server with an unexpected character_set_client or collation_connection, e.g. utf8mb3 and utf8mb3_general_ci, mysql_upgrade did not fix it to the expected latin1 and latin1_swedish_ci. Since 11.8 this could often lead to "Illegax mix of collations" errors when querying mysql.user, because since 11.8 the default collation for utf8mb3 is utf8mb3_uca1400_ai_ci, according to the default @@character_set_collations. For example: MariaDB [test]> select user,host,is_role from -> mysql.user where is_role='N'; ERROR 1267 (HY000): Illegal mix of collations (utf8mb3_general_ci,COERCIBLE) and (utf8mb3_uca1400_ai_ci,COERCIBLE) for operation '=' Fixing mariadb_system_tables_fix.sql to drop the view if it has non-standard character set or collations, so it gets recreated again correctly by the CREATE statement in mariadb_system_tables.sql |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38595: Simplify InnoDB doublewrite buffer creation buf_dblwr_t::create(): Create the doublewrite buffer in a single atomic mini-transaction. Do not write any log records for initializing any doublewrite buffer pages, in order to avoid recovery failure with innodb_log_archive=ON starting from the very beginning. The mtr.commit() in buf_dblwr_t::create() was observed to comprise 295 mtr_t::m_memo entries: 1 entry for the fil_system.sys_space and the rest split between page 5 (TRX_SYS) and page 0 (allocation metadata). We are nowhere near the sux_lock::RECURSIVE_MAX limit of 65535 per page descriptor. Reviewed by: Thirunarayanan Balathandayuthapani Tested by: Saahil Alam (cherry picked from commit e2c63a7062e3acbe44b8f7122c347183b87cbd89) |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension After hash chain is unlocked and buffer pool mutex is released in buf_page_init_for_read(), the chain list can be changed by concurent thread, that's why we need zero out extended buffer page pointer. Make ext_bp_size to be atomic. Process asinc errors in os_aio() correctly. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension Some code review fixes. Needs RQG testing as some new assertions were added and some conditions were changed. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension In one of the practical cloud MariaDB setups, a server node accesses its datadir over the network, but also has a fast local SSD storage for temporary data. The content of such temporary storage is lost when the server container is destroyed. The commit uses this ephemeral fast local storage (SSD) as an extension of the portion of InnoDB buffer pool (DRAM) that caches persistent data pages. This cache is separated from the persistent storage of data files and ib_logfile0 and ignored during backup. The following system variables were introduced: innodb_extended_buffer_pool_size - the size of external buffer pool file, if it equals to 0, external buffer pool will not be used; innodb_extended_buffer_pool_path - the path to external buffer pool file. If innodb_extended_buffer_pool_size is not equal to 0, external buffer pool file will be created on startup. Only clean pages will be flushed to external buffer pool file. There is no need to flush dirty pages, as such pages will become clean after flushing, and then will be evicted when they reach the tail of LRU list. The general idea of this commit is to flush clean pages to external buffer pool file when they are evicted. A page can be evicted either by transaction thread or by background thread of page cleaner. In some cases transaction thread is waiting for page cleaner thread to finish its job. We can't do flushing in external buffer pool file when transaction threads are waithing for eviction, that would heart performance. That's why the only case for flushing is when page cleaner thread evicts pages in background and there are no waiters. For this purprose buf_pool_t::done_flush_list_waiters_count variable was introduced, we flush evicted clean pages only if the variable is zeroed. Clean pages are evicted in buf_flush_LRU_list_batch() to keep some amount of pages in buffer pool's free list. That's why we flush every second page to external buffer pool file, otherwise there could be not enought amount of pages in free list to let transaction threads to allocate buffer pool pages without page cleaner waiting. This might be not a good solution, but this is enought for prototyping. External buffer pool page is introduced to store information in buffer pool page hash about the certain page can be read from external buffer pool file. The first several members of such page must be the same as the members of internal page. External page frame must be equal to the certain value to disthinguish external page from internal one. External buffer pages are preallocated on startup in external pages array. We could get rid of the frame in external page, and check if the page's address belongs to the array to distinguish external and internal pages. There are also external pages free and LRU lists. When some internal page is decided to be flushed in external buffer pool file, a new external page is allocated eighter from the head of external free list, or from the tail of external LRU list. Both lists are protected with buf_pool.mutex. It makes sense, because a page is removed from internal LRU list during eviction under buf_pool.mutex. Then internal page is locked and the allocated external page is attached to io request for external buffer pool file, and when write request is completed, the internal page is replaced with external one in page hash, external page is pushed to the head of external LRU list and internal page is unlocked. After internal page was removed from external free list, it was not placed in external LRU, and placed there only after write completion, so the page can't be used by the other threads until write is completed. Page hash chain get element function has additional template parameter, which notifies the function if external pages must be ignored or not. We don't ignore external pages in page hash in two cases, when some page is initialized for read and when one is reinitialized for new page creating. When an internal page is initialized for read and external page with the same page id is found in page hash, the internal page is locked, the external page in replaced with newly initialized internal page in the page hash chain, the external page is removed from external LRU list and attached to io request to external buffer pool file. When the io request is completed, external page is returned to external free list, internal page is unlocked. So during read external page absents in both external LRU and free lists and can't be reused. When an internal page is initialized for new page creating and external pages with the same page id is found in page hash, we just remove external page from the page hash chain and external LRU list and push it to the head of external free list. So the external page can be used for future flushing. The pages are flushed to and read from external buffer pool file with the same manner as they are flushed to their spaces, i.e. compressed and encrypted pages stay compressed and encrypted in external buffer pool file. Reviewed by: Marko Mäkelä |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension RocksDB test fix. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Atomic doublewrite buffer creation buf_dblwr_t::create(): Create the doublewrite buffer in a single atomic mini-transaction. Do not write any log records for initializing any doublewrite buffer pages, in order to avoid recovery failure with innodb_log_archive=ON starting from the very beginning. Cherry-picking from 1d1699eef3dfca20a075de7a282ec92ccf03166e |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension Fix some tests. Make ext_buf_pool test more stable avoiding race conditions for read/write counters. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
KhaledR57
khaled57.dev@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-37072: Implement IS JSON predicate Add support for the SQL standard IS JSON predicate with the syntax: expr IS [ NOT ] JSON [ { VALUE | ARRAY | OBJECT | SCALAR } ] [ { WITH | WITHOUT } UNIQUE [ KEYS ] ] The predicate allows checking if an expression is valid JSON and optionally constrains the JSON type (VALUE, ARRAY, OBJECT, SCALAR) and whether object keys are unique. The implementation includes: - Basic IS JSON validation - Support for NOT operator - Type constraints (VALUE, ARRAY, OBJECT, SCALAR) - Unique keys constraint (WITH/WITHOUT UNIQUE KEYS) |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension Get rid of ext_buf_page_t::frame. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38595: Simplify InnoDB doublewrite buffer creation buf_dblwr_t::create(): Create the doublewrite buffer in a single atomic mini-transaction. Do not write any log records for initializing any doublewrite buffer pages, in order to avoid recovery failure with innodb_log_archive=ON starting from the very beginning. The mtr.commit() in buf_dblwr_t::create() was observed to comprise 295 mtr_t::m_memo entries: 1 entry for the fil_system.sys_space and the rest split between page 5 (TRX_SYS) and page 0 (allocation metadata). We are nowhere near the sux_lock::RECURSIVE_MAX limit of 65535 per page descriptor. Reviewed by: Thirunarayanan Balathandayuthapani Tested by: Saahil Alam (cherry picked from commit e2c63a7062e3acbe44b8f7122c347183b87cbd89) |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alexander Barkov
bar@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38698 mysql_upgrade does not fix charset and collation for mysql.user In case if the view mysql.user was created (e.g. in 10.6) with a pre- 18edb0959fe5384673ebd36d72029fa8003ff5d9 server with an unexpected character_set_client and collation_connection, e.g. utf8mb3 and utf8mb3_general_ci, mysql_upgrade did not fix it to the expected latin1 and latin1_swedish_ci. Since 11.8 this could often lead to "Illegax mix of collations" errors when querying mysql.user, because since 11.8 the default collation for utf8mb3 is utf8mb3_uca1400_ai_ci, according to the default @@character_set_collations. For example: MariaDB [test]> select user,host,is_role from -> mysql.user where is_role='N'; ERROR 1267 (HY000): Illegal mix of collations (utf8mb3_general_ci,COERCIBLE) and (utf8mb3_uca1400_ai_ci,COERCIBLE) for operation '=' Fixing mariadb_system_tables.sql to drop the view if it has non-standard character set or collations, so it gets recreated again correctly. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Marko Mäkelä
marko.makela@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Multi-file checkpoint parsing in the memory-mapped code path recv_sys_t::find_checkpoint_archived(): Terminate the loop if the log_t::archived_mmap_switch_recovery_complete() had been invoked as a result of a checkpoint spanning two files. FIXME: Before invoking recv_recovery_from_checkpoint_start(), restore the first log file. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alexander Barkov
bar@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38698 mysql_upgrade does not fix charset and collation for mysql.user In case if the view mysql.user was created (e.g. in 10.6) with a pre- 18edb0959fe5384673ebd36d72029fa8003ff5d9 server with an unexpected character_set_client and collation_connection, e.g. utf8mb3 and utf8mb3_general_ci, mysql_upgrade did not fix it to the expected latin1 and latin1_swedish_ci. Since 11.8 this could often lead to "Illegax mix of collations" errors when querying mysql.user, because since 11.8 the default collation for utf8mb3 is utf8mb3_uca1400_ai_ci, according to the default @@character_set_collations. For example: MariaDB [test]> select user,host,is_role from -> mysql.user where is_role='N'; ERROR 1267 (HY000): Illegal mix of collations (utf8mb3_general_ci,COERCIBLE) and (utf8mb3_uca1400_ai_ci,COERCIBLE) for operation '=' Fixing mariadb_system_tables.sql to drop the view if it has non-standard character set or collations, so it gets recreated again correctly. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension Squash it. Fix for the following RQG test failures: 2. Scenario: The server is under load (9 concurrent sessions). At some point of time he crashes with mariadbd: 11.8-MDEV-31956-ext_buf_pool/storage/innobase/buf/buf0flu.cc:294: void buf_page_t::write_complete(buf_page_t::space_type, bool, uint32_t): Assertion `persistent == (om > 2)' failed. 4. Scenario: The server was some time under load (one connection). Intentional SIGKILL DB server followed by restart and running certain checks. All that did not show some error. But the shutdown hang like Fragment of rqg.log: # 2026-01-16T13:15:57 [1467965] INFO: DBServer_e::MySQL::MySQLd::stopServer: server[1]: Stopping server on port 25140 ... # 2026-01-16T13:28:22 [1467965] ERROR: DBServer_e::MySQL::MySQLd::stopServer: server[1]: Did not shut down properly. Terminate it == RQG loses the "patience" and sends finally SIGABRT to the process of the DB server. The server error log shows 2026-01-16 13:15:58 0 [Note] /data/Server_bin/11.8-MDEV-31956-ext_buf_pool_debug_Og/bin/mariadbd (initiated by: root[root] @ localhost [127.0.0.1]): Normal shutdown ... 2026-01-16 13:15:58 0 [Note] InnoDB: FTS optimize thread exiting. 2026-01-16 13:16:01 0 [Note] InnoDB: Starting shutdown... .... 2026-01-16 13:16:01 0 [Note] InnoDB: Buffer pool(s) dump completed at 260116 13:16:01 2026-01-16 13:18:37 0 [Note] InnoDB: Waiting for page cleaner thread to exit .... 2026-01-16 13:26:24 0 [Note] InnoDB: Waiting for page cleaner thread to exit |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Michael Widenius
monty@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-19683 Add support for Oracle TO_DATE() Syntax: TO_DATE(string_expression [DEFAULT string_expression ON CONVERSION ERROR], format_string [,NLS_FORMAT_STRING]) The format_string has the same format elements as TO_CHAR(), except a few elements that are not supported/usable for TO_DATE(). TO_DATE() returns a datetime or date value, depending on if the format element FF is used. Allowed separators, same as TO_CHAR(): space, tab and any of !#%'()*+,-./:;<=> '&' can also be used if next character is not a character a-z or A-Z "text' indicates a text string that is verbatim in the format. One cannot use " as a separator. Format elements supported by TO_DATE(): AD Anno Domini ("in the year of the Lord") AD_DOT Anno Domini ("in the year of the Lord") AM Meridian indicator (Before midday) AM_DOT Meridian indicator (Before midday) DAY Name of day DD Day (1-31) DDD Day of year (1-336) DY Abbreviated name of day FF[1-6] Fractional seconds HH Hour (1-12) HH12 Hour (1-12) HH24 Hour (0-23) MI Minutes (0-59) MM Month (1-12) MON Abbreviated name of month MONTH Name of Month PM Meridian indicator (After midday) PM_DOT Meridian indicator (After midday) RR 20th century dates in the 21st century. 2 digits 50-99 is assumed from 2000, 0-49 is assumed from 1900. RRRR 20th century dates in the 21st century. 4 digits SS Seconds SYYYY Signed 4 digit year; MariaDB only supports positive years Y 1 digit year YY 2 digits year YYY 3 digits year YYYY 4 digits year Note that if there is a missing part of the date, the current date is used! For example if 'MM-DD HH-MM-SS' then the current year will be used. (Oracle behaviour) Not supported options: - BC, D, DL, DS, E, EE, FM, FX, RM, SSSSS, TS, TZD, TZH, TZR, X,SY BC is not supported by MariaDB datetime. - Most of the other are exotic formats does not make sence in MariaDB as we return datetime or datetime with fractions, not string. - D (day-of-week) is not supported as it is not clear exactly how it would map to MariaDB. This element depends on the NLS territory of the session. - RR only works with 2 digit years (In Oracle RR can also work with 4 digit years in some context but the rules are not clear). Extensions / differences compared to Oracle; - MariaDB supports FF (fractional seconds). If FF[#] is used, then TO_DATE will return a datetime with # of subseconds. If FF is not used a datetime will be returned. There is warning (no error) if string contains more digts than what is specified with F(#] - Names can be shortened to it's unique prefix. For example January and Ja works fine. - No error if the date string is shorter format_string and the next not used character is not a number.. This is useful to get a date from a mixed set of strings in date or datetime format. Oracle gives an error if date string is too short. - MariaDB supports short locales as language names - NLS_DATE_FORMAT can use both " and ' for quoting. - NLS_DATE_FORMAT must be a constant string. - This is to ensure that the server knows which locale to use when executing the function. New formats handled by TO_CHAR(): FF[1-6] Fractional seconds DDD Daynumber 1-366 IW Week 1-53 according to ISO 8601 I 1 digit year according to ISO 8601 IY 2 digit year according to ISO 8601 IYY 3 digit year according to ISO 8601 IYYY 4 digit year according to ISO 8601 SYYY 4 digit year according to ISO 8601 (Oracle can do signed) Supported NLS_FORMAT_STRING options are: NLS_CALENDAR=GREGORIAN NLS_DATE_LANGUAGE=language Support languages are: - All MariaDB short locales, like en_AU. - The following Oracle language names: ALBANIAN, AMERICAN, ARABIC, BASQUE, BELARUSIAN, BRAZILIAN PORTUGUESE BULGARIAN, CANADIAN FRENCH, CATALAN, CROATIAN, CYRILLIC SERBIAN CZECH, DANISH, DUTCH, ENGLISH, ESTONIAN, FINNISH, FRENCH, GERMAN, GREEK, HEBREW, HINDI, HUNGARIAN, ICELANDIC, INDONESIAN ITALIAN, JAPANESE, KANNADA, KOREAN, LATIN AMERICAN SPANISH, LATVIAN, LITHUANIAN, MACEDONIAN, MALAY, MEXICAN SPANISH, NORWEGIAN, POLISH, PORTUGUESE, ROMANIAN, RUSSIAN, SIMPLIFIED CHINESE, SLOVAK, SLOVENIAN, SPANISH, SWAHILI, SWEDISH, TAMIL, THAI, TRADITIONAL CHINESE, TURKISH, UKRAINIAN, VIETNAMESE Development bugs fixed: MDEV-38403 Server crashes in Item_func_to_date::fix_length_and_dec upon using an invalid argument MDEV-38400 compat/oracle.func_to_date fails with PS protocol and cursor protocol (Fixed by Serg) MDEV-38404 TO_DATE: MTR coverage omissions, round 1 MDEV-38509 TO_DATE: AD_DOT does not appear to be supported MDEV-38513 TO_DATE: NULL value for format string causes assertion failure MDEV-38521 TO_DATE: Date strings with non-ASCII symbols cause warnings and wrong results MDEV-38578 TO_DATE: Possibly unexpected results upon wrong input MDEV-38582 TO_DATE: NLS_DATE_LANGUAGE=JAPANESE does not parse values which work in Oracle MDEV-38584 TO_DATE: NLS_DATE_LANGUAGE=VIETNAMESE does not parse values which work in Oracle MDEV-38703 TO_DATE: Quotation for multi-word NLS_DATE_LANGUAGE leads to syntax error in view definition MDEV-38675 TO_DATE: MSAN/Valgrind/UBSAN errors in extract_oracle_date_time Known issues: - Format string character matches inside quotes are done one-letter-to-one-letter, like in LIKE predicate. That means things like expansions and contractions do not work. For example 'ss' does not match 'ß' in collations which treat them as equal for the comparison operator. Match is done taking into account case and accent sensitivity of the subject argument collation, so for example this now works: MariaDB [test]> SELECT TO_DATE('1920á12','YYYY"a"MM') AS c; +---------------------+ | c | +---------------------+ | 1920-12-17 00:00:00 | +---------------------+ Co-author and reviewer: Alexander Barkov <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension Fix Windows and liburing issues. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension Evict page on write completion if it's space was removed. Lock external buffer pool file on Linux. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Monty
monty@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38246 aria_read index failed on encrypted database during backup The backup of encrypted Aria tables was not supported. Added support for this. One complication is that the page checksum is for the not encrypted page. To be able to verify the checksum I have to temporarly decrypt the page. In the backup we store the encrypted pages. Other things: - Fixed some (not critical) memory leaks in mariabackup |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alexander Barkov
bar@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38698 mysql_upgrade does not fix charset and collation for mysql.user In case if the view mysql.user was created (e.g. in 10.6) with a pre- 18edb0959fe5384673ebd36d72029fa8003ff5d9 server with an unexpected character_set_client or collation_connection, e.g. utf8mb3 and utf8mb3_general_ci, mysql_upgrade did not fix it to the expected latin1 and latin1_swedish_ci. Since 11.8 this could often lead to "Illegax mix of collations" errors when querying mysql.user, because since 11.8 the default collation for utf8mb3 is utf8mb3_uca1400_ai_ci, according to the default @@character_set_collations. For example: MariaDB [test]> select user,host,is_role from -> mysql.user where is_role='N'; ERROR 1267 (HY000): Illegal mix of collations (utf8mb3_general_ci,COERCIBLE) and (utf8mb3_uca1400_ai_ci,COERCIBLE) for operation '=' Fixing mariadb_system_tables_fix.sql to drop the view if it has non-standard character set or collations, so it gets recreated again correctly by the CREATE statement in mariadb_system_tables.sql |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
KhaledR57
khaled57.dev@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-37072: Implement IS JSON predicate Add support for the SQL standard IS JSON predicate with the syntax: expr IS [ NOT ] JSON [ { VALUE | ARRAY | OBJECT | SCALAR } ] [ { WITH | WITHOUT } UNIQUE [ KEYS ] ] The predicate allows checking if an expression is valid JSON and optionally constrains the JSON type (VALUE, ARRAY, OBJECT, SCALAR) and whether object keys are unique. The implementation includes: - Basic IS JSON validation - Support for NOT operator - Type constraints (VALUE, ARRAY, OBJECT, SCALAR) - Unique keys constraint (WITH/WITHOUT UNIQUE KEYS) |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Michael Widenius
monty@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-19683 Add support for Oracle TO_DATE() Syntax: TO_DATE(string_expression [DEFAULT string_expression ON CONVERSION ERROR], format_string [,NLS_FORMAT_STRING]) The format_string has the same format elements as TO_CHAR(), except a few elements that are not supported/usable for TO_DATE(). TO_DATE() returns a datetime or date value, depending on if the format element FF is used. Allowed separators, same as TO_CHAR(): space, tab and any of !#%'()*+,-./:;<=> '&' can also be used if next character is not a character a-z or A-Z "text' indicates a text string that is verbatim in the format. One cannot use " as a separator. Format elements supported by TO_DATE(): AD Anno Domini ("in the year of the Lord") AD_DOT Anno Domini ("in the year of the Lord") AM Meridian indicator (Before midday) AM_DOT Meridian indicator (Before midday) DAY Name of day DD Day (1-31) DDD Day of year (1-336) DY Abbreviated name of day FF[1-6] Fractional seconds HH Hour (1-12) HH12 Hour (1-12) HH24 Hour (0-23) MI Minutes (0-59) MM Month (1-12) MON Abbreviated name of month MONTH Name of Month PM Meridian indicator (After midday) PM_DOT Meridian indicator (After midday) RR 20th century dates in the 21st century. 2 digits 50-99 is assumed from 2000, 0-49 is assumed from 1900. RRRR 20th century dates in the 21st century. 4 digits SS Seconds SYYYY Signed 4 digit year; MariaDB only supports positive years Y 1 digit year YY 2 digits year YYY 3 digits year YYYY 4 digits year Note that if there is a missing part of the date, the current date is used! For example if 'MM-DD HH-MM-SS' then the current year will be used. (Oracle behaviour) Not supported options: - BC, D, DL, DS, E, EE, FM, FX, RM, SSSSS, TS, TZD, TZH, TZR, X,SY BC is not supported by MariaDB datetime. - Most of the other are exotic formats does not make sence in MariaDB as we return datetime or datetime with fractions, not string. - D (day-of-week) is not supported as it is not clear exactly how it would map to MariaDB. This element depends on the NLS territory of the session. - RR only works with 2 digit years (In Oracle RR can also work with 4 digit years in some context but the rules are not clear). Extensions / differences compared to Oracle; - MariaDB supports FF (fractional seconds). If FF[#] is used, then TO_DATE will return a datetime with # of subseconds. If FF is not used a datetime will be returned. There is warning (no error) if string contains more digts than what is specified with F(#] - Names can be shortened to it's unique prefix. For example January and Ja works fine. - No error if the date string is shorter format_string and the next not used character is not a number.. This is useful to get a date from a mixed set of strings in date or datetime format. Oracle gives an error if date string is too short. - MariaDB supports short locales as language names - NLS_DATE_FORMAT can use both " and ' for quoting. - NLS_DATE_FORMAT must be a constant string. - This is to ensure that the server knows which locale to use when executing the function. New formats handled by TO_CHAR(): FF[1-6] Fractional seconds DDD Daynumber 1-366 IW Week 1-53 according to ISO 8601 I 1 digit year according to ISO 8601 IY 2 digit year according to ISO 8601 IYY 3 digit year according to ISO 8601 IYYY 4 digit year according to ISO 8601 SYYY 4 digit year according to ISO 8601 (Oracle can do signed) Supported NLS_FORMAT_STRING options are: NLS_CALENDAR=GREGORIAN NLS_DATE_LANGUAGE=language Support languages are: - All MariaDB short locales, like en_AU. - The following Oracle language names: ALBANIAN, AMERICAN, ARABIC, BASQUE, BELARUSIAN, BRAZILIAN PORTUGUESE BULGARIAN, CANADIAN FRENCH, CATALAN, CROATIAN, CYRILLIC SERBIAN CZECH, DANISH, DUTCH, ENGLISH, ESTONIAN, FINNISH, FRENCH, GERMAN, GREEK, HEBREW, HINDI, HUNGARIAN, ICELANDIC, INDONESIAN ITALIAN, JAPANESE, KANNADA, KOREAN, LATIN AMERICAN SPANISH, LATVIAN, LITHUANIAN, MACEDONIAN, MALAY, MEXICAN SPANISH, NORWEGIAN, POLISH, PORTUGUESE, ROMANIAN, RUSSIAN, SIMPLIFIED CHINESE, SLOVAK, SLOVENIAN, SPANISH, SWAHILI, SWEDISH, TAMIL, THAI, TRADITIONAL CHINESE, TURKISH, UKRAINIAN, VIETNAMESE Development bugs fixed: MDEV-38403 Server crashes in Item_func_to_date::fix_length_and_dec upon using an invalid argument MDEV-38400 compat/oracle.func_to_date fails with PS protocol and cursor protocol (Fixed by Serg) MDEV-38404 TO_DATE: MTR coverage omissions, round 1 MDEV-38509 TO_DATE: AD_DOT does not appear to be supported MDEV-38513 TO_DATE: NULL value for format string causes assertion failure MDEV-38521 TO_DATE: Date strings with non-ASCII symbols cause warnings and wrong results MDEV-38578 TO_DATE: Possibly unexpected results upon wrong input MDEV-38582 TO_DATE: NLS_DATE_LANGUAGE=JAPANESE does not parse values which work in Oracle MDEV-38584 TO_DATE: NLS_DATE_LANGUAGE=VIETNAMESE does not parse values which work in Oracle MDEV-38703 TO_DATE: Quotation for multi-word NLS_DATE_LANGUAGE leads to syntax error in view definition MDEV-38675 TO_DATE: MSAN/Valgrind/UBSAN errors in extract_oracle_date_time Known issues: - Format string character matches inside quotes are done one-letter-to-one-letter, like in LIKE predicate. That means things like expansions and contractions do not work. For example 'ss' does not match 'ß' in collations which treat them as equal for the comparison operator. Match is done taking into account case and accent sensitivity of the subject argument collation, so for example this now works: MariaDB [test]> SELECT TO_DATE('1920á12','YYYY"a"MM') AS c; +---------------------+ | c | +---------------------+ | 1920-12-17 00:00:00 | +---------------------+ Co-author and reviewer: Alexander Barkov <[email protected]> |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension Added testing for: 1. Async read 2. Space removing during async io 3. IO error during async write The case of io error during sync and async io should also be added. Squash it. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
KhaledR57
khaled57.dev@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-37072: Implement IS JSON predicate Add support for the SQL standard IS JSON predicate with the syntax: expr IS [ NOT ] JSON [ { VALUE | ARRAY | OBJECT | SCALAR } ] [ { WITH | WITHOUT } UNIQUE [ KEYS ] ] The predicate allows checking if an expression is valid JSON and optionally constrains the JSON type (VALUE, ARRAY, OBJECT, SCALAR) and whether object keys are unique. The implementation includes: - Basic IS JSON validation - Support for NOT operator - Type constraints (VALUE, ARRAY, OBJECT, SCALAR) - Unique keys constraint (WITH/WITHOUT UNIQUE KEYS) |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sergei Golubchik
serg@mariadb.org |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38709 ASAN heap-buffer-overflow in my_convert_using_func Don't forget up to update stored_rec_length when extending temp table reclength. Followup for 4f9a13e9ecf2 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension Use persistent named files for external buffer pool instead of temporary one. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Aleksey Midenkov
midenok@gmail.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38710 Assertion is_lock_owner on error returning from auto-create in mysql_admin_table After returning error from check_vers_constants() the stack immediately returns error status up to mysql_admin_table() where it does close_thread_tables(). The latter expects MDL acquired. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension Some code review fixes. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Varun Deep Saini
varun.23bcs10048@ms.sst.scaler.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-16335 Include deadlock information in SHOW WARNINGS When innodb_print_all_deadlocks=ON, push the deadlock report to the session warning list so it can be retrieved via SHOW WARNINGS |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension Remove double-write buffer workaround. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension In one of the practical cloud MariaDB setups, a server node accesses its datadir over the network, but also has a fast local SSD storage for temporary data. The content of such temporary storage is lost when the server container is destroyed. The commit uses this ephemeral fast local storage (SSD) as an extension of the portion of InnoDB buffer pool (DRAM) that caches persistent data pages. This cache is separated from the persistent storage of data files and ib_logfile0 and ignored during backup. The following system variables were introduced: innodb_extended_buffer_pool_size - the size of external buffer pool file, if it equals to 0, external buffer pool will not be used; innodb_extended_buffer_pool_path - the path to external buffer pool file. If innodb_extended_buffer_pool_size is not equal to 0, external buffer pool file will be created on startup. Only clean pages will be flushed to external buffer pool file. There is no need to flush dirty pages, as such pages will become clean after flushing, and then will be evicted when they reach the tail of LRU list. The general idea of this commit is to flush clean pages to external buffer pool file when they are evicted. A page can be evicted either by transaction thread or by background thread of page cleaner. In some cases transaction thread is waiting for page cleaner thread to finish its job. We can't do flushing in external buffer pool file when transaction threads are waithing for eviction, that would heart performance. That's why the only case for flushing is when page cleaner thread evicts pages in background and there are no waiters. For this purprose buf_pool_t::done_flush_list_waiters_count variable was introduced, we flush evicted clean pages only if the variable is zeroed. Clean pages are evicted in buf_flush_LRU_list_batch() to keep some amount of pages in buffer pool's free list. That's why we flush every second page to external buffer pool file, otherwise there could be not enought amount of pages in free list to let transaction threads to allocate buffer pool pages without page cleaner waiting. This might be not a good solution, but this is enought for prototyping. External buffer pool page is introduced to store information in buffer pool page hash about the certain page can be read from external buffer pool file. The first several members of such page must be the same as the members of internal page. External page frame must be equal to the certain value to disthinguish external page from internal one. External buffer pages are preallocated on startup in external pages array. We could get rid of the frame in external page, and check if the page's address belongs to the array to distinguish external and internal pages. There are also external pages free and LRU lists. When some internal page is decided to be flushed in external buffer pool file, a new external page is allocated eighter from the head of external free list, or from the tail of external LRU list. Both lists are protected with buf_pool.mutex. It makes sense, because a page is removed from internal LRU list during eviction under buf_pool.mutex. Then internal page is locked and the allocated external page is attached to io request for external buffer pool file, and when write request is completed, the internal page is replaced with external one in page hash, external page is pushed to the head of external LRU list and internal page is unlocked. After internal page was removed from external free list, it was not placed in external LRU, and placed there only after write completion, so the page can't be used by the other threads until write is completed. Page hash chain get element function has additional template parameter, which notifies the function if external pages must be ignored or not. We don't ignore external pages in page hash in two cases, when some page is initialized for read and when one is reinitialized for new page creating. When an internal page is initialized for read and external page with the same page id is found in page hash, the internal page is locked, the external page in replaced with newly initialized internal page in the page hash chain, the external page is removed from external LRU list and attached to io request to external buffer pool file. When the io request is completed, external page is returned to external free list, internal page is unlocked. So during read external page absents in both external LRU and free lists and can't be reused. When an internal page is initialized for new page creating and external pages with the same page id is found in page hash, we just remove external page from the page hash chain and external LRU list and push it to the head of external free list. So the external page can be used for future flushing. The pages are flushed to and read from external buffer pool file with the same manner as they are flushed to their spaces, i.e. compressed and encrypted pages stay compressed and encrypted in external buffer pool file. Reviewed by: Marko Mäkelä |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension In one of the practical cloud MariaDB setups, a server node accesses its datadir over the network, but also has a fast local SSD storage for temporary data. The content of such temporary storage is lost when the server container is destroyed. The commit uses this ephemeral fast local storage (SSD) as an extension of the portion of InnoDB buffer pool (DRAM) that caches persistent data pages. This cache is separated from the persistent storage of data files and ib_logfile0 and ignored during backup. The following system variables were introduced: innodb_extended_buffer_pool_size - the size of external buffer pool file, if it equals to 0, external buffer pool will not be used; innodb_extended_buffer_pool_path - the path to external buffer pool file. If innodb_extended_buffer_pool_size is not equal to 0, external buffer pool file will be created on startup. Only clean pages will be flushed to external buffer pool file. There is no need to flush dirty pages, as such pages will become clean after flushing, and then will be evicted when they reach the tail of LRU list. The general idea of this commit is to flush clean pages to external buffer pool file when they are evicted. A page can be evicted either by transaction thread or by background thread of page cleaner. In some cases transaction thread is waiting for page cleaner thread to finish its job. We can't do flushing in external buffer pool file when transaction threads are waithing for eviction, that would heart performance. That's why the only case for flushing is when page cleaner thread evicts pages in background and there are no waiters. For this purprose buf_pool_t::done_flush_list_waiters_count variable was introduced, we flush evicted clean pages only if the variable is zeroed. Clean pages are evicted in buf_flush_LRU_list_batch() to keep some amount of pages in buffer pool's free list. That's why we flush every second page to external buffer pool file, otherwise there could be not enought amount of pages in free list to let transaction threads to allocate buffer pool pages without page cleaner waiting. This might be not a good solution, but this is enought for prototyping. External buffer pool page is introduced to store information in buffer pool page hash about the certain page can be read from external buffer pool file. The first several members of such page must be the same as the members of internal page. External page frame must be equal to the certain value to disthinguish external page from internal one. External buffer pages are preallocated on startup in external pages array. We could get rid of the frame in external page, and check if the page's address belongs to the array to distinguish external and internal pages. There are also external pages free and LRU lists. When some internal page is decided to be flushed in external buffer pool file, a new external page is allocated eighter from the head of external free list, or from the tail of external LRU list. Both lists are protected with buf_pool.mutex. It makes sense, because a page is removed from internal LRU list during eviction under buf_pool.mutex. Then internal page is locked and the allocated external page is attached to io request for external buffer pool file, and when write request is completed, the internal page is replaced with external one in page hash, external page is pushed to the head of external LRU list and internal page is unlocked. After internal page was removed from external free list, it was not placed in external LRU, and placed there only after write completion, so the page can't be used by the other threads until write is completed. Page hash chain get element function has additional template parameter, which notifies the function if external pages must be ignored or not. We don't ignore external pages in page hash in two cases, when some page is initialized for read and when one is reinitialized for new page creating. When an internal page is initialized for read and external page with the same page id is found in page hash, the internal page is locked, the external page in replaced with newly initialized internal page in the page hash chain, the external page is removed from external LRU list and attached to io request to external buffer pool file. When the io request is completed, external page is returned to external free list, internal page is unlocked. So during read external page absents in both external LRU and free lists and can't be reused. When an internal page is initialized for new page creating and external pages with the same page id is found in page hash, we just remove external page from the page hash chain and external LRU list and push it to the head of external free list. So the external page can be used for future flushing. The pages are flushed to and read from external buffer pool file with the same manner as they are flushed to their spaces, i.e. compressed and encrypted pages stay compressed and encrypted in external buffer pool file. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension In one of the practical cloud MariaDB setups, a server node accesses its datadir over the network, but also has a fast local SSD storage for temporary data. The content of such temporary storage is lost when the server container is destroyed. The commit uses this ephemeral fast local storage (SSD) as an extension of the portion of InnoDB buffer pool (DRAM) that caches persistent data pages. This cache is separated from the persistent storage of data files and ib_logfile0 and ignored during backup. The following system variables were introduced: innodb_extended_buffer_pool_size - the size of external buffer pool file, if it equals to 0, external buffer pool will not be used; innodb_extended_buffer_pool_path - the path to external buffer pool file. If innodb_extended_buffer_pool_size is not equal to 0, external buffer pool file will be created on startup. Only clean pages will be flushed to external buffer pool file. There is no need to flush dirty pages, as such pages will become clean after flushing, and then will be evicted when they reach the tail of LRU list. The general idea of this commit is to flush clean pages to external buffer pool file when they are evicted. A page can be evicted either by transaction thread or by background thread of page cleaner. In some cases transaction thread is waiting for page cleaner thread to finish its job. We can't do flushing in external buffer pool file when transaction threads are waithing for eviction, that would heart performance. That's why the only case for flushing is when page cleaner thread evicts pages in background and there are no waiters. For this purprose buf_pool_t::done_flush_list_waiters_count variable was introduced, we flush evicted clean pages only if the variable is zeroed. Clean pages are evicted in buf_flush_LRU_list_batch() to keep some amount of pages in buffer pool's free list. That's why we flush every second page to external buffer pool file, otherwise there could be not enought amount of pages in free list to let transaction threads to allocate buffer pool pages without page cleaner waiting. This might be not a good solution, but this is enought for prototyping. External buffer pool page is introduced to store information in buffer pool page hash about the certain page can be read from external buffer pool file. The first several members of such page must be the same as the members of internal page. External page frame must be equal to the certain value to disthinguish external page from internal one. External buffer pages are preallocated on startup in external pages array. We could get rid of the frame in external page, and check if the page's address belongs to the array to distinguish external and internal pages. There are also external pages free and LRU lists. When some internal page is decided to be flushed in external buffer pool file, a new external page is allocated eighter from the head of external free list, or from the tail of external LRU list. Both lists are protected with buf_pool.mutex. It makes sense, because a page is removed from internal LRU list during eviction under buf_pool.mutex. Then internal page is locked and the allocated external page is attached to io request for external buffer pool file, and when write request is completed, the internal page is replaced with external one in page hash, external page is pushed to the head of external LRU list and internal page is unlocked. After internal page was removed from external free list, it was not placed in external LRU, and placed there only after write completion, so the page can't be used by the other threads until write is completed. Page hash chain get element function has additional template parameter, which notifies the function if external pages must be ignored or not. We don't ignore external pages in page hash in two cases, when some page is initialized for read and when one is reinitialized for new page creating. When an internal page is initialized for read and external page with the same page id is found in page hash, the internal page is locked, the external page in replaced with newly initialized internal page in the page hash chain, the external page is removed from external LRU list and attached to io request to external buffer pool file. When the io request is completed, external page is returned to external free list, internal page is unlocked. So during read external page absents in both external LRU and free lists and can't be reused. When an internal page is initialized for new page creating and external pages with the same page id is found in page hash, we just remove external page from the page hash chain and external LRU list and push it to the head of external free list. So the external page can be used for future flushing. The pages are flushed to and read from external buffer pool file with the same manner as they are flushed to their spaces, i.e. compressed and encrypted pages stay compressed and encrypted in external buffer pool file. Reviewed by: Marko Mäkelä |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension Some code review fixes. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thirunarayanan Balathandayuthapani
thiru@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-32067 InnoDB linear read ahead had better be logical - Ported MDEV-32067 branch into main. Added a few interface for multi-range read limit and InnoDB also aware of number of pages to be read. Yet to integrate with buf_read_ahead_pages() |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thirunarayanan Balathandayuthapani
thiru@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-32067 InnoDB linear read ahead had better be logical - Ported MDEV-32067 branch into main. Added a few interface for multi-range read limit and InnoDB also aware of number of pages to be read. Yet to integrate with buf_read_ahead_pages() |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Alexander Barkov
bar@mariadb.com |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-38698 mysql_upgrade does not fix charset and collation for mysql.user In case if the view mysql.user was created (e.g. in 10.6) with a pre- 18edb0959fe5384673ebd36d72029fa8003ff5d9 server with an unexpected character_set_client or collation_connection, e.g. utf8mb3 and utf8mb3_general_ci, mysql_upgrade did not fix it to the expected latin1 and latin1_swedish_ci. Since 11.8 this could often lead to "Illegax mix of collations" errors when querying mysql.user, because since 11.8 the default collation for utf8mb3 is utf8mb3_uca1400_ai_ci, according to the default @@character_set_collations. For consistency, it's better to fix the problem starting from 11.4. For example: MariaDB [test]> select user,host,is_role from -> mysql.user where is_role='N'; ERROR 1267 (HY000): Illegal mix of collations (utf8mb3_general_ci,COERCIBLE) and (utf8mb3_uca1400_ai_ci,COERCIBLE) for operation '=' Fixing mariadb_system_tables_fix.sql to drop the view if it has non-standard character set or collations, so it gets recreated again correctly by the CREATE statement in mariadb_system_tables.sql |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Vlad Lesin
vlad_lesin@mail.ru |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MDEV-31956 SSD based InnoDB buffer pool extension Some code review fixes. Needs RQG testing as some new assertions were added and some conditions were changed. The previous commit passed RQG testing. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||