It's all about bugs fixed: MySQL 5.6.14

Most of MySQL gurus and famous users are probably in San Francisco now, getting ready for fun at MySQL Connect. Part of that fun should come from the announcement of great new MySQL 5.6.14 release (that somewhat silently happened yesterday).

I am sitting at home though and I've seen at best 3 sunny days in September. The rest of the time it rains, so hardly I can do anything more funny and useful than review of MySQL bug reports even during my weekend. Let me try to tell you what MySQL 5.6.14 is really about and what you should expect from it based on the list of bugs fixed. Please, do not blame me if my summary would be different from the upcoming keynotes at MySQL Connect. It rains here...


I'll use good old approach of checking my older posts about bugs in MySQL 5.6.13 and comparing it to the release notes of MySQL 5.6.14. You have to read them carefully anyway to check if any of the bugs you cared about are fixed. Chances are real, with 67 or so bugs mentioned, but previous releases of MySQL 5.6 GA contained notably more fixes. Either release before MySQL Connect was a top priority or MySQL 5.6 is getting mature now, with less problems remaining to solve in new releases.

Let's check, starting from InnoDB bugs I had written about back in June. From the bugs I mentioned there I see no new fixes, even Bug #69179 (that I've mentioned many times and that should be already fixed in recent Percona Server 5.5.x) and Bug #69236 (single thread performance regression affecting Facebook) are still just "Verified". Bug #69325 is also still "Verified", even though at least documenting potential memory use by ALTER against a partitioned InnoDB table (as it was suggested by Shane back in May) would help a lot...

MySQL 5.6.13 was released less than 2 months ago, so surely one should not expect that all new bugs are fixed so fast. But I am really disappointed to see Bug #69892 (that questions the possibility of forced recovery if InnoDB persistent statistics was ever used and that I had written about here) still "Verified".

It can not be that bad, so I checked later post, and found one problem I've cared about solved in MySQL 5.6.14. The problem of improper use of InnoDB tables to store replication information (see Bug #69898) is fixed. Related Bug #69907 is still "Verified", even though recent comment claim it is a duplicate of the previous one and must be fixed in MySQL 5.6.14. Let's assume it's a matter of documenting or just a mistake. In any case, if you plan to use crash safe replication in MySQL 5.6, you should upgrade to 5.6.14 ASAP!

Of the InnoDB bugs I had not paid attention to recently that I'd like to mention the following ones fixed (quote from the Release Notes):

• InnoDB; Partitioning: Following any query on the INFORMATION_SCHEMA.PARTITIONS table, InnoDB index statistics as shown in the output of statements such as SELECT * FROM INFORMATION_SCHEMA.STATISTICS were read from the last partition, instead of from the partition containing the greatest number of rows. (Bug #11766851, Bug #60071)
• InnoDB: When logging the delete-marking of a record during online ALTER TABLE...ADD PRIMARY KEY, InnoDB writes the transaction ID to the log as it was before the deletion or delete-marking of the record. When doing this, InnoDB would overwrite the DB_TRX_ID field in the original table, which could result in locking issues. (Bug #17316731)
• InnoDB: The row_sel_sec_rec_is_for_clust_rec function would incorrectly prepare to compare a NULL column prefix in a secondary index with a non-NULL column in a clustered index. (Bug #17312846)
• InnoDB: An incorrect purge would occur when rolling back an update to a delete-marked record. (Bug #17302896)
• InnoDB: An assertion would be raised in fil_node_open_file due to a missing .ibd file. Instead of asserting, InnoDB should return false and the caller of fil_node_open_file should handle the return message. (Bug #17305626, Bug #70007)
• InnoDB: The assertion ut_ad(oldest_lsn <= cur_lsn) in file buf0flu.cc would fail because the current max LSN would be retrieved from the buffer pool before the oldest LSN. (Bug #17252421)

Bug #60071 was waiting since 5.5 GA times and it's great to see it fixed. Bug #70007, on the other hand, was reported just a month ago against 5.6.13. Both fixes should make my colleague Justin happy.

Other bugs sounds serious and show good work of MySQL QA in Oracle probably, but it's hard to say anything more than we see in the Release Notes as all the details are hidden in internal Oracle's bugs database.

Let's move on to replication bugs I mentioned in June. Bug #69444 is still "Verified". Bug #69135 from Giuseppe is till "Open". Even Bug #69097 is still "Verified", since April 30... Read Bug #69095 to find out why you should not switch replication format from STATEMENT to ROW if you use GTIDs and why failures in this case do not indicate any bug... I hope Giuseppe is satisfied (I am just not brave enough to recommend to switch to GTID-based replication in production, yet). Bug #68892 is closed, but it says the fix will go to MySQL 5.6.15...

To summarize, MySQL 5.6.14 had NOT solved any more of my replication-related concerns expressed back in June, comparing to 5.6.13. The only really great improvement is the fix for Bug #69898, as explained above. Your millage may vary though, as I see many fixes to bugs in internal bugs database related to Replication, like these (quote from the Release Notes):

• Replication: When a master with semisynchronous replication enabled was shut down, the master failed to wait for either a semisyncnronous ACK or timeout before completing the shutdown. This prevented semisynchronous replication from reverting to asynchronous replication and allowed open transactions to complete on the master, which resulted in missing events on the slave.
  To fix this problem, dump threads are now stopped last during shutdown, after the client is told to stop, so that, if the dump thread has pending events from active clients, they can be sent to the slave. (Bug #16775543)
  
• Replication: A session attachment error during group commit causes the rollback of the transaction (as intended), but the transaction in which this happened was still written to the binary log and replicated to the slave. Thus, such an error could lead to a mismatched master and slave.
  Now when this error occurs, an incident event is written in the binary log which causes replication to stop, and notifies the user that redundant events may exist in the binary log. An additional error is also now reported to the client, indicating that the ongoing transaction has been rolled back. (Bug #16579083)
  
• Replication: START SLAVE failed when the server was started with the options --master-info-repository=TABLE relay-log-info-repository=TABLE and with autocommit set to 0, together with --skip-slave-start.
  A workaround for previous versions of MySQL is to restart the slave mysqld without the --skip-slave-start option. (Bug #16533802)

Probably I have to stop at this stage and try to get some real-life experience with MySQL 5.6.14...

To summarize: if you use MySQL 5.6 in production setup where replication is a key component, you should definitely upgrade to MySQL 5.6.14. Do not expect it to solve all the problems you cared about though, for me it was not the case.

Fun with Bugs, Issue #3, January 2013

This week in my posts on Facebook I paid attention mostly to bugs in MySQL 5.6.x, as I really expected to see 5.6 GA announcement soon. Maybe some of the bugs were the reason to postpone release a bit. Anyway, let's start with serious bug reports and some trends I've noted.

I've noted that some of "Verified" bug reports somehow silently miss comments about their status in MySQL 5.6. Check Bug #68137, for example. It is repeatable on 5.5.31 (yet to be released, but it means 5.5.30 is cloned off and will be released soon) and 5.7.1. But 5.6.x is not mentioned anywhere. Does it mean 5.6.x is not affected? Surely, no. I hope it does NOT mean any attempt by Oracle to pretend that 5.6 is bugs free. Looks like Sveta had just forgotten to check or report result on this version when following her great bugs verification procedure.

I would say that the above was just a one time mistake, but then look at Bug #68145 that was verified by Sveta in less than 30 minutes after my last comment there (and "verification", at the level I can do it as community member busy with main job actually). It was verified on many versions again, 5.0.97, 5.1.69, 5.5.31, 5.7.1, but NOT on 5.6.x. This looks like a trend now, and I do not like it. There are bugs in 5.6.9, 5.6.10 and even 5.6.11, there is nothing bad about this if they are noted, processed and fixed fast. There is not (good) reason to skip checking on mentioning this version. I am monitoring carefully anyway, so nothing in public database will pass unnoticed and unknown to MySQL community.

Serious bug in InnoDB was reported (and verified on both 5.5.27, 5.6.9-rc and recent 5.6 code) this week, Bug #68148. After executing SET FOREIGN_KEY_CHECKS=0; you can easily drop index unsed by foreign key and then the tables becomes unusable (you can not even drop it with SQL statement) after server restart. As it is not a recent regression, I am not sure it would be easy to fix, so better just avoid DDL with the setting above.

When 5.6 is approaching GA it is really strange to see bugs in new/key features of MySQL 5.6 with unclear status. Check Bug #66916 (replication). We can only guess is it fixed in 5.6.10 or not. Check Bug #66865 (performance regression in 5.6 that is all about performance) reported by my colleague Vadim more than 4 months ago. It is still just "Open". So, is there a regression or not? Or it is just forgotten?

When bugs are forgotten and test cases are not added to the regression test suite (now we, in Community, can not check if they are added), we are doomed to see the same problems again and again. If you are Oracle customer, ask them what Bug #68178 is about (in a support request probably), what versions are affected and had it been previously reported and fixed. If you care what is it about (Oracle will probably just tell you to upgrade to the version with the fix eventually), it is about an easy way to crash MySQL server by sending many smart SELECT convert_tz(...) in a row. I hope 5.6 will NOT be released as GA before this bug is fixed.

Forgotten bug reports had always been a problem, but I tried to at least review all "Open" bug reports once in while. Now we can still see bug reports with clear, repeatable test case finally got from reporter (like Bug #67517), or bug that is just easy to verify by code review (like Bug #67386) open and having no visible attention for months. Some bugs just hang in other intermediate states for ages as well (after verification), like Bug #56267. Some are verified, but they look serious and still remain just in "Verified" status, like Bug #67638 (optimizer).

Chances are high for a casual reporter to never report again when he gets no reply for months. This bad trend continues, but with less bugs open reports there is chance to overcome this trend eventually.

Hardly any of new key features of 5.6 is bug free. For example, PERFORMANCE_SCHEMA, as it was recently found out (see Bug #68180) does not cover all execution paths. Comparing to bugs listed at the beginning of this post it may be not that serious, but I am a bit tired of getting new features in GA that are NOT implemented completely (as it happened to 5.0, 5.1 and 5.5 GA).

The last (but not the least) potential problem with 5.6 is related to the documentation. There are missing details and misleading statements there (see Bug #68097), sections like http://dev.mysql.com/doc/refman/5.5/en/innodb-restrictions.html#innodb-5-5 that are no longer relevand after recent funny bug fixes (see funny fix to Bug #63435 about innodb_version server variable) and misprints in release notes (like Bug #68181). Some sections are still work in progress it seems, like http://dev.mysql.com/doc/refman/5.6/en/online-ddl-partitioning.html. Some fixes (like Bug #67759) are not documented it seems. Maybe manual being not yet ready is the reason why MySQL 5.6 had not been declared GA yet?

Now about good trends. This week I've noted more MySQL engineers working on bugs processing. Long waiting Bug #68023 was finally verified by Matt Lord, who rarely worked on bugs not related to customer issues in the past. Look at his detailed research in Bug #67517 that I've noted (and "escalated") just today. Erlend Dahl had noted and verified awful Bug #68127 soon after my "escalation" and it was closed in 2 days (with the fix in 5.6.10, one of the reasons I though that it will be GA and announced really soon during the week)! Way to go, Oracle, really!

As a result, this week Oracle finally managed to decrease number of "Open" bug reports to the level smaller than during my last day in the company, August 31, 2012. Since December 2012 number of open bug reports is decreased by more than 170! New bug reports are processed fast, see Bug #68144 as one of recent examples. Verified in less that 3 hours (as well as two other bugs from Hartmut reported that day). Looks like it is not a problem now for Oracle to keep up with inflow of at least server bug reports (I'll write about Cluster and Workbench bugs in details in some of the following issues).

Another (IMHO good) trend - Oracle customers, at least those who did that before, do not hesitate to report bugs in public database (as I've suggested to do last year). See Bug #68165 as one of recent examples related to 5.6 and upgrade to it.

Finally, some fun. Make sure you check the oldest and still not closed MySQL bugs, Bug #2. 10 years passed and the problem is still far from solved! This is awful, really.

MariaDB also contributed some fun for us all with their version 10.0. Read Bug #68187 and then try to guess if Oracle is going to fix it any time soon. More fun in the next issue, I hope...