Fun with Bugs #93 - On MySQL Bug Reports I am Subscribed to, Part XXVII

No matter what I write and present about dynamic tracing, blog posts about MySQL bugs are more popular based on statistics. So, to make more readers happy, I'd like to continue my review of interesting bugs reported in November with this post on bugs reported during December, 2019.

As usual, I'll try to concentrate on bug reports related to InnoDB, replication and optimizer, but some other categories also got my attention:

• Bug #97911 - "[FATAL] Semaphore wait has lasted > 600 seconds. We intentionally crash the serv...". This bug got marked as a duplicate of other, older long semaphore wait bug (in "Can't repeat" status!) without much analysis. I think all Oracle engineers who added comments to that bug missed one interesting point:
  

  ... [ERROR] [MY-012872] [InnoDB] Semaphore wait has lasted > 39052544 seconds.

  even though bug reporter highlighted it in a comment. Reported wait time is a problem and is surely a bug, no matter how to reproduce the long wait itself and what is its root cause!
• Bug #97913 - "Undo logs growing during partitioning ALTER queries". This bug (affecting only MySQL 5.7.x) was reported by Przemyslaw Malkowski from Percona, who also presented useful examples of monitoring queries to the information_schema.innodb_metrics and performance_schema. Check also comments that may explain why 8.0 is not affected in a similar way.
• Bug #97935 - "Memory leak in client connection using information_schema". It took some efforts (starting from but not limited to Valgrind Massif profiling of heap memory usage) and time for Daniel Nichter to prove the point and get this bug "Verified". It is also not clear if MySQL 8 is also affected.
• Bug #97950 - "buf_read_page_handle_error can trigger assert failure". Bug reporter, Shu Lin, tried his best to make the point. It's clear enough how to repeat this, and one could use one of documented test synchonisation methods if gdb is too much for bug verification. I do not think this bug was handled properly or got the level of attention it truly deserved.
• Bug #97966 - "XA COMMIT in another session will not write binlog event". This bug was reported by Lujun Wang and immediately verified, but again with no documented check if MySQL 8 is affected. This happens too often, unfortunately.
• Bug #97971 - "Roles not handling column level privileges correctly; Can SELECT, but not UPDATE". Clear and simple bug report with a workaround from Travis Bement. It was immediately verified.
• Bug #98014 - "Lossy implicit conversion in conditional breaks ONLY_FULL_GROUP_BY". Yet another case of (IMHO) improper bug processing. The argument presented (from the manual):
  

  "MySQL 5.7.5 and later also permits a nonaggregate column not named in a GROUP BY clause when ONLY_FULL_GROUP_BY SQL mode is enabled, provided that this column is limited to a single value"

  does not apply, as "single value" for = 0 is NOT selected, we have multiple Host values matching it due to conversion. This is how proper version (guess what it is) works:
  

  mysql> SELECT User, Host, COUNT(*) FROM mysql.user WHERE Host = 0 GROUP BY 1;
  ERROR 1055 (42000): 'mysql.user.Host' isn't in GROUP BY
  mysql> select @@sql_mode;
  +--------------------+
  | @@sql_mode         |
  +--------------------+
  | ONLY_FULL_GROUP_BY |
  +--------------------+
  1 row in set (0.001 sec)
  
  mysql> set session sql_mode='';
  Query OK, 0 rows affected (0.029 sec)
  
  mysql> SELECT User, Host, COUNT(*) FROM mysql.user WHERE Host = 0 GROUP BY 1;+---------------+-----------+----------+
  | User          | Host      | COUNT(*) |
  +---------------+-----------+----------+
  | data_engineer |           |        1 |
  | en            | localhost |        1 |
  | ro1           |           |        1 |
  | ro2           |           |        1 |
  | role-1        |           |        1 |
  | root          | ::1       |        3 |
  ...
  | user1         | %         |        1 |
  +---------------+-----------+----------+
  13 rows in set, 17 warnings (0.003 sec)

  I think this bug reported by Joshua Varner must be verified.
• Bug #98046 - "Inconsistent behavior while logging a killed query in the slow query log". Bug reporter, Pranay Motupalli, provided a clear test case and a detailed analysis, including the gdb debugging session that proves the point. Nice bug report.
• Bug #98055 - "MySQL Optimizer Bug not picking right index". Both the bug reporter (Sudheer Gadipathi) and engineer who verified the bug stated that MySQL 8.0.x is similarly affected (UNIQUE key is preferred for the partitioned table, even though there is a better non-unique index). But 8.0.x is NOT listed in the "Version:" filed. Weird.
• Bug #98068 - "SELECT FOR UPDATE not-exist table in PERFORMANCE SCHEMA reports confusing error". This is a funny (but still a regression) bug report by William ZHANG. Proper versions work like this:
  

  mysql> select database();+--------------------+
  | database()         |
  +--------------------+
  | performance_schema |
  +--------------------+
  1 row in set (0.001 sec)
  
  mysql> select * from not_exist_table;
  ERROR 1146 (42S02): Table 'performance_schema.not_exist_table' doesn't exist
  mysql> select * from not_exist_table for update;
  ERROR 1146 (42S02): Table 'performance_schema.not_exist_table' doesn't exist

• Bug #98072 - "innochecksum summary shows blob pages as other type of page for 8.0 tables". The bug was reported by SERGEY KUZMICHEV. This time "regression" tag is missing, even though it's clearly stated that MySQL 5.7 worked differently. This is from the proper version:
  

  ...
  File::..\data\test\blob_test.ibd
  ================PAGE TYPE SUMMARY==============
  #PAGE_COUNT     PAGE_TYPE
  ===============================================
         1        Index page
         0        Undo log page
         1        Inode page
         0        Insert buffer free list page
       508        Freshly allocated page
         1        Insert buffer bitmap
         0        System page
         0        Transaction system page
         1        File Space Header
         0        Extent descriptor page
        64        BLOB page
         0        Compressed BLOB page
         0        Page compressed page
         0        Page compressed encrypted page
         0        Other type of page
  ...
  

• Bug #98083 - "Restarting the computer when deleting the database will cause directory residues". One would expect that MySQL 8 with a data dictionary should have some means to figure out the remaining database directory for a dropped database upon startup (as it stores information about databases elsewhere) and do proper cleanup. I think this bug reported by Jinming Liao must be verified and fixed. There is no "... manual creation or deletion of tables or databases..." involved in this case.
• Bug #98091 - "InnoDB does not initialize raw disk partitions". As simple as that and both 5.7.29 and 8.0.219 are surely affected. It was not always the case, I've used raw devices myself with older MySQL versions, so this bug reported by Saverio M is a regression. Still, "regression" tag is missing.

That's all bugs reported in December, 2019 that I cared to subscribe to  and mention here. Next time I'll check bugs reported in January, 2020. There are at least 16 in my list already, so stay tuned.

Follow the links in this post to get more details about profiling and creating off-CPU FlameGraphs for MySQL. This post is devoted to bugs, though :)

To summarize:

1. I am happy to see bug reports from people whom I never noticed before. MySQL Community is alive.
2. Some flexibility in following common sense based bugs verification procedures is still visible. Bugs reported for 5.7 are not checked on 8.0 (or the results of this check are not documented in public), nobody cares to read what bug reporter says carefully or go extra mile, "regression" tag not added, and so on. 
3. Probably at this stage my writings are mostly ignored by Oracle's decision makers. But I keep watching them all anyway.

Fun with Bugs #81 - On MySQL Bug Reports I am Subscribed to, Part XVII

Two weeks passed since my previous review of public MySQL bug reports I consider interesting enough to subscribe to them. Over this period I picked up a dozen or so new public bug reports that I'd like to briefly review today.

Here is my recent subscriptions list, starting from the oldest bug reports:

• Bug #94431 - "Can't upgrade from 5.7 to 8.0 if any database have a hyphen in their name". It seems one actually needs a database like that created in MySQL 5.6 with at least one InnoDB table having FULLTEXT index to hit the problem. Great finding by Phil Murray. Note that after several unsuccessful attempts by others the bug was eventually reproduced and verified by Jesper Wisborg Krogh. Let's hope we'll see it fixed in MySQL 8.0.16.
• Bug #94435 - "mysql command hangs up and cosume CPU almost 100%". It was reported by Masaaki HIROSE, whose previous related/similar Bug #94219 - "libmysqlclient enters and infinite loop and consume CPU usage 100%" ended up as "Not a bug" (wrongly, IMHO, as nobody cared enough to reproduce the steps instead of commenting on their correctness and checking something else). Bug reporter had not only insisted and provided all the details, but also tried to analyze the reasons of the bug and provided links to other potentially related bug reports (Bug #88428 - "mysql_real_query hangs with EINTR errno (using YASSL)" and Bug #92394 - "libmysqlclient enters infinite loop after signal (race condition)"). Great job and nice to see the bug "Verified" eventually.
• Bug #94441 - "empty ibuf aio reads in innodb status". This regression vs MySQL 5.6 was noted by Nikolai Ikhalainen from Percona. MariaDB 10.3.7 is also affected, unfortunately:
  

  ...
  I/O thread 9 state: native aio handle (write thread)
  Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
   ibuf aio reads:, log i/o's:, sync i/o's:Pending flushes (fsync) log: 0; buffer pool: 0
  1344 OS file reads, 133 OS file writes, 2 OS fsyncs
  ...

• Bug #94448 - "Rewrite LOG_BLOCK_FIRST_REC_GROUP during recovery may be dangerous.". Yet another MySQL 8 regression (not marked with "regression" tag) was found by Kang Wang.
• Bug #94476 - "mysql semisync replication stuck with master in Waiting to finalize termination". It has "Need feedback" status at the moment. I've subscribed to this report from Shirish Keshava Murthy mostly to find out how a report that may look like a free support request will be processed by Oracle engineers. Pure curiosity, for now.
• Bug #94504 - "AIO::s_log seems useless". This problem was reported by Yuhui Wang. It's a regression in a sense that part of the code is no longer needed (and seems not to be used) in MySQL 8, but still remains.
• Bug #94541 - "Assertion on import via Transportable Tablespace". This bug reported by  Daniël van Eeden was verified based on code review and some internal discussion. We do not know if any other version besides 5.7.25 is affected, though. The assertion itself:
  

  InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) == btr_pcur_get_block(cursor)->page.id.page_no()

  does not seem to be unique. We can find it in MDEV-18455 also (in other context).
• Bug #94543 - "MySQL does not compile with protobuf 3.7.0". I care about build/compiling bugs historically, as I mostly use MySQL binaries that I built myself from GitHub source. So, I've immediately subscribed to this bug report from Laurynas Biveinis.
• Bug #94548 - "Optimizer error evaluating JSON_Extract". This bug was reported by Dave Pullin. From my quick test it seems MariaDB 10.3.7 is also affected. Error message is different in the failing case, but the point is the same - the function is not evaluated if the column from derived table that is built using the function is not referenced in the SELECT list. This optimization is questionable and may lead to hidden "bombs" in the application code.
• Bug #94550 - "generated columns referring to current_timestamp fail". I tried to check simple test case in this bug report by Mario Beck on MariaDB 10.3.7, but it does not seem to accept NOT NULL constraint for generated stored columns at all:

  MariaDB [test]> CREATE TABLE `t2` (
      ->   `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
      ->   `content` varchar(42) DEFAULT NULL,
      ->   `bucket` tinyint(4) GENERATED ALWAYS AS ((floor((to_seconds(`created_at
  `) / 10)) % 3)) STORED NOT NULL);
  ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that
  corresponds to your MariaDB server version for the right syntax to use near 'NOT
   NULL)' at line 4

  I do not see this option in formal syntax described here as well. But in case of MariaDB we can actually make sure the generated column is never NULL by adding CHECK constraint like this:
  

  MariaDB [test]> CREATE TABLE `t2` (    ->   `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
      ->   `content` varchar(42) DEFAULT NULL,
      ->   `bucket` tinyint(4) GENERATED ALWAYS AS ((floor((to_seconds(`created_at`) / 10)) % 3)) STORED);
  Query OK, 0 rows affected (0.434 sec)
  
  MariaDB [test]> INSERT INTO t2 (content) VALUES ("taraaaa");
  Query OK, 1 row affected (0.070 sec)
  
  MariaDB [test]> alter table t2 add constraint cnn CHECK (`bucket` is NOT NULL);
  Query OK, 1 row affected (1.159 sec)
  Records: 1  Duplicates: 0  Warnings: 0
  
  MariaDB [test]> INSERT INTO t2 (content) VALUES ("tarabbb");
  Query OK, 1 row affected (0.029 sec)
  
  MariaDB [test]> INSERT INTO t2 (content) VALUES ("");
  Query OK, 1 row affected (0.043 sec)
  
  MariaDB [test]> select * from t2;
  +---------------------+---------+--------+
  | created_at          | content | bucket |
  +---------------------+---------+--------+
  | 2019-03-09 17:28:03 | taraaaa |      0 |
  | 2019-03-09 17:29:43 | tarabbb |      1 |
  | 2019-03-09 17:29:50 |         |      2 |
  +---------------------+---------+--------+
  3 rows in set (0.002 sec)
  
  MariaDB [test]> show create table t2\G*************************** 1. row ***************************
         Table: t2
  Create Table: CREATE TABLE `t2` (
    `created_at` timestamp NOT NULL DEFAULT current_timestamp(),
    `content` varchar(42) DEFAULT NULL,
    `bucket` tinyint(4) GENERATED ALWAYS AS (floor(to_seconds(`created_at`) / 10)
  MOD 3) STORED,
    CONSTRAINT `cnn` CHECK (`bucket` is not null)
  ) ENGINE=InnoDB DEFAULT CHARSET=latin1
  1 row in set (0.011 sec)

  So, maybe after all we can state that MariaDB is NOT affected.
• Bug #94552 - "innodb.virtual_basic fails when valgrind is enabled". I still wonder if anyone in Oracle runs MTR test suite on Valgrind-enabled (-DWITH_VALGRIND=1 cmake option) at least in the process of official release (and if they check the failures). It seems not to be the case based on this bug report from Manuel Ung.
• Bug #94553 - "Crash in trx_undo_rec_copy". Bernardo Perez noted that as a side effect of still "Verified" Bug #82734 - "trx_undo_rec_copy needlessly relies on buffer pool page alignment" (that affects both MySQL 5.7 and 8.0) we may get crashes while working with generated columns. I hope to see them both fixed soon, but for now Bug #94553 has status "Need Feedback", probably in a hope to get a repeatable test case. I'll watch it carefully.
• Bug #94560 - "record comparison in spatial index non-leaf rtree node seems incorrect". I doubt spatial indexes of InnoDB are widely used, and I have no doubts there are many bugs waiting to be discovered in this area. This specific bug was reported by Jie Zhou who had also suggested a fix.
• Bug #94610 - "Server stalls because ALTER TABLE on partitioned table holds dict mutex". My former colleague Justin Swanhart reported this bug just yesterday, so no wonder it is not verified yet. It refers to a well known verified old Bug #83435 - "ALTER TABLE is very slow when using PARTITIONED table"  (that I've also subscribed to immediately) from Roel Van de Paar, affecting both MySQL 5.6 and 5.7. I hope to see this bug verified and fixed soon, as recently I see this kind of state for main thread:
  

  Main thread process no. 3185, id 140434206619392, state: enforcing dict cache limit

  too often in INNODB STATUS outputs to my liking...

As you could note, I still try to check (at least in some cases) if MariaDB is also affected by the same problem. I think it's a useful check both for me (as I work mostly with MariaDB as a support engineer) and for the reader (to know if switching to MariaDB may help in any way or if there are any chances for MariaDB engineers to contribute anything useful, like a fix).

"Hove, actually". For years residents of Hove used this humorous reply when they live in Brighton... "Regression, actually" is what I want to say (seriously) about every other MySQL bug report I subscribe to... So, you see Hove and many regression bugs above!

To summarize:

1. Sometimes Oracle engineers demonstrate proper collective effort to understand and carefully verify public bug reports. Good to know they are not ready to give up fast!
2. I have to copy-paste this item from my previous post. As the list above proves, Oracle engineers still do not use "regression" tag when setting "Verified" status for obviously regression bugs. I think bug reporters should care then to always set it when they report regression of any kind.
3. It seems there no regular MTR test runs for Valgrind builds performed by Oracle engineers, or maybe they just ignore failures.

Fun with Bugs #77 - On MySQL Bug Reports I am Subscribed to, Part XIV

Slides for my talk about MySQL bugs at FOSDEM 2019 MySQL, MariaDB and Friends Devroon are ready, support customers decided not to break anything badly on weekend, so I have some free time for blogging.  As usual, when I do not have any better idea or useful recent real life experience to share I write about MySQL bugs.

Today I'd like to continue my review of interesting MySQL bug reports added by Community members in December, 2018. I'll review them starting from the oldest:

• Bug #93701 - "Assertion `maybe_null' failed |Item_func_concat::val_str(String*)". It's debug assertion, not a big deal (S6 bug), so why should anyone care about this report from Ramesh Sivaraman? Because last time same assertion failure ended up as serious enough Bug #83115 fixed in 8.0.11.
• Bug #93708 - "Page Cleaner will sleep for long time if clock changes". In this bug report from Marcelo Altmann I was mostly impressed by arguing started by Oracle engineer on bug severity (documentation request, feature request vs real bug). IMHO it was a waste of time.
• Bug #93728 - "mysqld crash after using alter table with SPATIAL key". Nice finding by Rui Xu. Unfortunately MariaDB 10.3.7 is also affected. At least in the error log I also see this error message for each SPATIAL index of the table:
  

  2019-01-20 19:26:15 9 [ERROR] InnoDB: Record in index `idx2` of table `test`.`tab` was not found on update: TUPLE (info_bits=0, 2 fields): {[32]      $@      $@
        $@      $@(0x0000000000000400000000000000040000000000000004000000000000000
  400),[4]    (0x00000001)} at: COMPACT RECORD(info_bits=0, 1 fields): {[8]infimum
   (0x090E06090D050D00)}

• Bug #93734 - "MySQL 8.0 is 36 times slower than MySQL 5.7". I like everything about this bug reported by Vadim Tkachenko, CTO of Percona. I like synopsis that sounds cool, missing exact MySQL 8.0.x version (but I suspect 8.0.13), impact of innodb_flush_log_at_trx_commit, suggested fix and the fact that the bug is still "Open" and nobody from Oracle paid any attention to the report from CTO of one of key Oracle partners and supporters of MySQL 8. Wonderful!
• Bug #93737 - "mysqlbinlog mermory used grows unstoped". I had not tried to reproduce it (and nobody had it seems, as the bug is still "Open"), but this report by Chandler Bong sounded interesting. Let's see what it may end up with...
• Bug #93746 - "MySQL Crashes with deadlock at Mutex DICT_SYS created dict0dict.cc:1172". Long semaphore waits involving DICT_SYS mutex and leading to watchdog thread killing MySQL are common. Bug processing is still in progress, but I subscribed to this bug by Anton Ravich mostly to follow another bug he refers to as a possibly related, Bug #80919. That bug is declared a duplicate of some bug in Oracle's internal bugs database, without bug number mentioned. This is not the first time I see such am artistic way to process community bugs (that I consider totally wrong and unacceptable).
• Bug #93760 - "InnoDB got assert failure while figuring out space id at startup". Fungo Wang found this crash while running MTR test case innodb.innodb_redo_debug_1 with debug binaries. Now I wonder if Oracle QA runs MTR tests with debug binaries at all and do they care to check failures (or rely on Percona for this activity)? Any comments?
• Bug #93761 - "optimize table cause the slave MTS deadlock!". Sounds interesting and serious, but nobody cared to check this bug report. It's "Open" without comments for 3 weeks already.
• Bug #93767 - "INSTALL COMPONENT fails - handled segfault during installation nothing in log". I am surprised that somebody outside of Oracle already uses new MySQL 8.0 server components feature. My former colleague, Justin Swanhart, tried but failed so far. Let's see how this report may end up. For now Oracle engineer stated that Justin tried to do something this framework was not designed for, see nice last comment there.
• Bug #93777 - "the Originator of mysql.event is not correct". Pay attention to this bug found by Dennis Gao if you use events in replication environment.
• Bug #93779 - "dml in table with trigger to other Table map in binlog lead to slave sql stopped". Work on this bug report from Mohamed Atef is still in progress. Bug reported cared to add a lot of details in recent comments, so I hope to see this bug processed soon.

I've also included a couple of bugs reported in January, 2019 in the list above, but I hope you don't mind.

Today I'd like to remember those sunny days in Venice in September and those pesky MySQL bugs reported this winter...

To summarize:

1. Some bug reports stay "Open" for weeks without any good reason recently.
2. The shameful practice of referring to bugs in Oracle's internal bugs database without giving a number (that we see in release notes when the bug is fixed and can look for in commit messages at GitHub) should be stopped! Next time I'll blame engineer(s) who this by name, here and in social media. Check how this is properly done by Shane Bester.
3. I wonder if Oracle QA runs MTR tests cases on debug builds on a regula basis and checks the results, or this is now a job for Percona mostly?
4. I should write a separate blog post about bugs related to SPATIAL indexes implementation in InnoDB. There should be dragons...

P.S. Don't forget to click on names of bug reporters to see the entire list of still active MySQL bug reports each of them created.

Fun with Bugs #39 - Known Bugs in MySQL 5.7.9 GA

These days everybody is excited with recent announcement of MySQL 5.7.9 GA release. If you are not aware of this event yet (I've noted it from numerous posts even during my short vacation), wait for the Oracle Open World 2015 to begin tomorrow to announce it even wider and louder!

I already have 5.7.9 built from source, up and running, so it's time to check what else we can expect from this new GA release besides new great features (this is a topic for a separate post or two) and usual excitement. Yes, I mean known, verified bugs in MySQL 5.7.9.

Let me start with a quick summary and then present the details. So, even though MySQL Community tried hard to check 5.7.x at early stages and report bugs to Oracle, MySQL 5.7.9 GA has a number of known installation and upgrade problems, code is not always clean, not all community suggested patches are included. As usual with new releases, there are some performance issues (one should expect single thread to run slower than on older versions in many cases), including those on replication slaves. Some new features are not completely documented, some implementation details are missing here and there, and transparent page compression, for example, is hardly can be widely used at the moment. All these are expected for users who had seen previous GA releases of MySQL. What's a bit worse, I suspect there are known crashing bugs (sequence of SQL statements that, when executed by a user with enough privileges, crashes non-debug binaries) in MySQL 5.7.9. Also, MySQL 5.7.9 inherits some regression bugs from MySQL 5.6.x.

Quick search for active bugs in version 5.7 reported over last 6 months gives me a list of 157 bugs that probably affect MySQL 5.7.9. I've excluded those that are not yet verified formally, those affecting older versions (5.6.x and 5.5.x), most of documentation and test problems and ended up with the following list that I want to present to those who plan to upgrade to 5.7 GA really soon:

• Bug #78936 - "Installing MySQL on Linux Using Debian Packages missing dependency". This is mostly a documentation request. Please, check comments there for the correct order of .deb packages installation, if you plan to use dpkg.
• Bug #78591 - "mysqld --initialize fails on empty data directory". Some more installation problems on Ubuntu...
• Bug #78895 - "Lack information about generated columns in mysql_fetch_fields function". Now, this is typical, unfortunately, for new MySQL features. They are rarely completely implemented in first few GA releases. Oracle still has to rely on MySQL Community to find these small missing implementation details (or design problems) here and there.
• Bug #78848 - "Docs still claim Performance Schema does not allocate memory after startup". While formally it's just a documentation bug, it is about a notable change in behavior. In 5.7 additional memory may be allocated after startup to support performance_schema. This had never been the case before. My report that is based on real customer case, Bug #78808, "Manual is wrong when explaining the default UMASK value", is of a similar kind - some clear changes in behavior are NOT documented properly before making GA announcement. MySQL 5.7 follows the same bad tradition of 5.0, 5.1, 5.5, and 5.6 GA releases here...
• Bug #78778 - "mysqlclient does not return correct mysql_insert_id via database handle". It seems to be a regression bug (even though it is not tagged so) and as a result your DBD::mysql Perl module may not work with 5.7 the way you expect.
• Bug #78774 - "old-password subcommand is still in mysqladmin --help in 5.7". Minor problem, but I wonder why it could not be fixed in the code before GA release. It seems the build/QA/release process still takes a lot of time in Oracle, so even those bugs noted by Community in RC releases were reported "too late" to influence the GA release. This is unfortunate.
• Bug #78758 - "Inserting (invalid) utf8 data into 5.6.25 master breaks 5.7 slave". There is a workaround documented, but this bug report shows that upgrade from 5.6 may not be seamless and simple (are you surprised?) and that upgrade was NOT really tested really well by the internal QA process. I am not surprised, it had always been the case.
• Bug #78672 - "assert fails in fil_io during linkbench with transparent innodb compression". Facebook engineers are trying hard, but transparent page compression mostly does NOT work well (if only on NVMFS, see Bug #78277 also). Just remember that for now, before you became too excited with the feature.
• Bug #78495 - "Table 'mysql.gtid_executed' cannot be opened." Don't be scared with this warning, it's "Ok"...
• Bug #78374 - "CREATE USER IF NOT EXISTS" reports an error". New feature that just does not work as documented. Well done.
• Bug #78352 - "Slow startup of 5.7.x slave with relay_log_recovery = ON and many relay logs". This my report is also based on a real life customer case. Take care when upgrading, really.
• Bug #78254 - "After running mysql_upgrade proxies_priv user columns are not updated to 32". mysql_upgrade is not able to fix all incompatibilities for further seamless use of MySQL 5.7.
• Bug #78251 - "handle_fatal_signal (sig=11) in mach_double_read". No further comments for now.
• Bug #78236 - "InnoDB: Failing assertion: holder != requestor". Again, no comments.
• Bug #78219 - "compression can be defined on a compressed table if innodb_strict_mode is OFF"
• Bug #78068 - "replication_* tables miss information about binary log position and other". I should not comment on performance_schema any more probably.
• Bug #77740 - "silent failure to start if mysql.gtids_executed gets HA_ERR_LOCK_WAIT_TIMEOUT". You can end up with failures and error log does not help to find out why...
• This bug was actually reported back in February, so it was not in the list of 157 recent reports, and there is nothing unexpected here, but still. Note that single thread performance of MySQL 5.7.x GA may be worse than for older versions, see Bug #75981, "MySQL 5.7.5 30% slower than 5.6.23 for 1 thread write", for example.

Some potential performance issues, mistakes in the code and debug-only assertions (that may show huge internal corruptions) are never noted until somebody starts to test a lot or read the code carefully:

• Bug #78894 - ''buf_pool_resize can lock less in checking whether AHI is on or off".
• Bug #78637 - "Incorrect for loop condition in mysql_crawler.cc"
• Bug #78732 - "InnoDB: Failing assertion: *mbmaxlen < 5 in file ha_innodb.cc line 1803".
• Bug #78728 - "InnoDB: Failing assertion: 0 in file ha_innopart.cc line 3526" 
• ... check the list yourself, there are many more assertion failures and code review problems reported  ...
• Bug #77094 - "Reduce log_sys->mutex contention by allowing concurrent mtr commit and log write". This is just one of many patches contributed by Zhai Weixiang that remain just "Verified"

I expect many more reports of this kind from Roel, Laurynas and my other colleagues now, when Percona is actively working on its own Percona Server 5.7 GA release.

The last but not the least, if you are upgrading to 5.7 GA from 5.5.x or older versions, check known regressions in MySQL 5.6. Many of them are affecting 5.7.9 as well, unfortunately.

On responsible bugs reporting

Let me start with questions related to responsible MySQL bugs reporting that I'd like to be discussed and then present a history behind them.

Assuming that you, my dear reader from MySQL Community, noted or found some simple sequence of SQL statements that, when executed by authenticated MySQL user explicitly having all the privileges needed to execute these statements, crashes some version of your favorite MySQL fork, please, answer the following questions:

1. Do you consider this kind of a bug a "security vulnerability"?
2. Should you share complete test case at any public site (MySQL bugs database, Facebook, your personal blog, any)?
3. Should you share just a description of possible "attack vector", as Oracle does when they publish security bug fixes?
4. Should you share just a stack trace or failed assertion information, without any details on how to get it?
5. Should you inform vendor of your MySQL fork about it in a private email officially (secalert_us@oracle.com in case of Oracle)?
6. Should you try to have it added to CVE database and, if you should, what is the best way to do this?
7. Should you try to use any informal, but still private ways share this information with MySQL vendors and other interested parties?
8. What is the best way to do "responsible bugs reporting" in cases like this?

Now, the history behind these questions. Usually when I have to deal with any MySQL issue not obviously caused by a user mistake or misconfiguration, I suspect that the problem may be related to some (known or yet unknown, fixed or still active) MySQL bug. I search MySQL bugs database for error messages and codes, keywords, stack trace entries or assertions user noted, at the earliest stage of investigation. This approach works well enough for me for more than 8 years, as even if the problem was not related to any bug, similar reports may give explantions, workarounds or other hints on possible reasons.

To be able to search bugs faster and have more things readily available "from the top of my head" I try to read new public bug reports whenever they appear. Sometimes I send comments or hints. Often enough I see simple test cases there to check. And I do try to run these test cases in a hope to see the problem more clearly or find out that it is not repeatable for me (to try to understand why is that so later). I do this during my free time and while I work, I do this this for fun and when colleagues ask me for some assistance or, specifically, for any bug that may be related to their problem at hand.

I used to share my finding immediately on Facebook by posting URLs to bugs with my comments, but this activity is now stopped at least till PLMCE 2014 in a hope to find some better way of cooperation on bug fixing and as a part of one simple experiment for my talk proposed for that conference (that was not selected by the committee anyway). So, since the end of November, 2013 I share (rarely) only some questions or test cases from time to time, and ask colleagues to try them on their favorite MySQL, MariaDB or Percona Server versions, if they are also curious.

This week during my usual review of new bug reports in the morning I've noted an "Open" bug with a simple test case, just a couple of UPDATEs for some InnoDB table having a single row of data, that, according to the bug reporter, crashed recently released MySQL 5.6.16 (that I blogged about just a day before) and 5.5.36. Surely I've immediately tried this test case, that crashed my test instances because of assertion failure. I've waited for some time to see that the bug is still open, informed few of my colleagues hanging around at IRC chat about new crashing bug found, waited for some more time and... posted test case in a status update on Facebook. This post got some comments that suspected crash as a result, but not so many. I have maybe 200 friends there and most of them are actually current or former colleagues, MySQL engineers or software developers who work on MySQL and other databases in Oracle, HP, SkySQL, Percona and few other companies. My idea was to make them all immediately aware about potential problem. I've noted that very soon after this post the bug report had "disappeared" - it was marked as Private probably and, I hope, verified and got proper attention from Oracle engineers.

Next morning while checking recent emails quickly I've noted message from my former colleague, who now works in Oracle, asking to remove that Facebook post as explicit publishing of a way to crash MySQL is not a "responsible reporting" - I had to give Oracle a chance and time to fix this, as a potential "security vulnerability".

I've immediately deleted the post based just on respect to the person who asked, but internally I had doubts on what was actually a proper thing to do in this case, and I even felt insulted for a while by this (yet another) attempt to shut me up and stop me from discussing MySQL bugs in public (not that this really happens, but I have feelings like that from time to time). So, I tried to discuss this with my colleagues and found out that some of them do think the bug is actually a potential security vulnerability and thus had to be private. Others (like me) do not believe in any "security through obscurity" practice, especially for open source software, where any interested person can just read the code, find assertion there and then just think when and why it can be hit, based on code analysis.


Formal Oracle policy in this case is well known: "If you are not a customer or partner, please email secalert_us@oracle.com with your discovery". Customers should just open a Support Request.

I was explicitly against applying this policy to MySQL (I've suggested to create public bug reports in my New Year 2013 wishes to Oracle customers back in 2012) among other reasons because "security bugs" and "security issues" have more chances internally in Oracle to become forgotten as they get less visibility (at the same time as they get higher priority) and very few people in Oracle can access or escalate or otherwise monitor them.

Surely, this policy can not be enforced on community members and individuals like me. In the worst case if somebody discloses details of the security vulnerability, she will not get a credit, as: "In order to receive credit, security researchers must follow responsible disclosure practices".


That's the whole story so far. In my case it was not me who found the bug, so why should I care even about credit. My action caused immediate attention from Oracle side and thus I can be sure this specific bug is not forgotten and fixed as soon as possible (by the way, MySQL 5.5.35 is also affected by the same assertion failure, so this is NOT a recent regression and those who use older versions in production are potentially affected). I am sure my post was also noted by "security officers" in MariaDB Foundation and Percona, so in case these forks are also affected we now have chances to get fixes there as well, no matter when and how Oracle fixes the problem.

Still, this case and related private discussions led me to the questions at the beginning of this blog post. I'd like to get your opinions on them.