Sunday, July 9, 2017

MySQL Support Engineer's Chronicles, Issue #7

This week in Support was busy enough for me. Among other things I had to study all possible reasons (other than obvious query cache impact) for queries hanging in "query end" status and noted Bug #80652 related to binlog group commit and fixed in MySQL 5.7.17+ and 8.0.1+ only. The case I had to review was related to Galera though, and I suggest you to note that "query end" may be related to Galera replication stall. Studying this path further soon brought lp:1197771 - "Cluster stalls while distributing transaction" to my attention again, so I asked about proper status for it on Facebook. As it happens way too often recently, I've got few 'Likes" but no further comments, neither from Percona nor from Codership. It seems good old times of my efficient bugs escalations via Facebook had gone...

Does it mean I am going to stop writing and sharing posts about bugs on Facebook? Definitely it does NOT! One of the recent reasons is this post about Bug #86902 that, after being shared by me, got a quick comment and clarification by Sveta Smirnova that the problem (performance problem when binary logging is turned OFF and you use more than one transactional storage engine concurrently) is actually know, - it was  reported as Bug #80870 (that is still "Open", what a shame).

So, even if some Oracle or Percona employees will continue to ignore new bugs reports or my Facebook (re-)posts about MySQL bugs, at least bug reporter gets more chances to get help from Community, additional references and clarifications. So, stay tuned - I'll keep writing about MySQL bugs in this blog and, on a more regular basis and probably for a wider audience, - on Facebook. As popular Mahatma Gandhi's quote says:
"First they ignore you, then they laugh at you, then they fight you, then you win."
In my case with public MySQL bugs and the way they are processed by Oracle it was not strictly in this order, and I am yet to see people laughing at me for this reason, but I think I already won as since September 2012 public MySQL bugs database still exists and several Oracle engineers still pay attention to community bug reports there.

I had reported one bug myself this week, Bug #86930 - "Wrong results for queries with row constructors and information_schema". It was quickly verified by Miguel Solorzano. It's a funny bug (with workarounds), but not as scary as few other bugs recently reported and described in this post by Jean-François Gagné.Note, that as he stated, there are more than just that 2 bugs he mentioned (Bug #86926 and Bug #86927). Everybody in Oracle who should know that already knows it, and I know it. I keep stating that the implementation of InnoDB persistent statistics gives us more troubles than benefits. Last time I cared and complained that much it was about Bug #70629 - "InnoDB updated rec_per_key[] statistics not published to the optimizer enough often". But, trust me, this recent set of problems Jean-François found is way more serious.

I'd like to share few references I found extremely useful this week while working on Support issues this week. First of all, if you set up SSL connections for MySQL, take time to review this old post by my former colleague from Percona Roman Vynar, and another one by Matthew Boehm. This may save you some time and debugging efforts. If you care about MariaDB ColumnStore performance (I have to), check this KB article. If you want to replicate data to the table with somewhat different column types on slave, make sure to read the manual. Finally, if you care enough to keep SELinux enabled and use Galera clusters with any non-default ports or directories, make sure to read this blog post.

I've noted a couple of serious decisions made by Oracle in regard to MySQL. First of the is to discontinue work on MySQL Internals manual and integrate it some how into the main manual. You can find related text added recently to some bug reports, like Bug #67989 - "MySQL Internals documentation missing 5.6 binlog protocol parts" by my colleague Andrew Hutchings:
"No more updates are made to the MySQL Internals documentation, because it's in the process of being replaced by"
Sounds promising, isn't it?

The other one, that is going to be accepted way better, IMHO, is to discontinue work on query cache. Check Bug #86046 - "FROM sub query is cached by mistake" by Meiji Kimura, for example:
"[7 Jul 9:09] Erlend Dahl

MySQL will no longer invest in the query cache, see:

So, that were some of the links I've noted during this week. Some of the topics mentioned above may be continued and presented in more details in my further posts.


  1. About bug escalation on Facebook: are you sure those reach the right persons ? Maybe mentioning people in comments would help...

  2. I am sure my Facebook posts about bugs reach enough people who can ping decision makers directly, outside of Facebook, if needed and if they care.

    In the past, in 2013 for example, some MySQL bugs were processed properly minutes after I mentioned them, or even minutes after they were reported in a hope to no attract my attention. I think this was one of the driving forces that made MySQL 5.6.x good release, after all. Then, probably, my followers got tired, so they found a way to shut me up for a long time. After that one year or so pause in "escalations" they are not treated as something to deal with immediately any more.

    As for that specific PXC bug, Laurynas had tagged proper dev. lead in his comment to my previous post about it. Obviously it had not helped.

    Even if this escalation approach is no longer efficient, I plan to use it and let people laugh on me.

  3. I'm all for reporting bugs, in public, like is possible at, and I do it whenever I hit one.
    But I will never use Facebook for anything, be it reading or writing, because of its policy to invade my privacy.

    1. Probably I also should have privacy-related concerns, but for now I just do not use Facebook on any mobile device and try to control what I like and share.

      I still find it a very useful tool to follow what's going on in life of few my real friends without privacy concerns, and many former, current and potential future colleagues related to MySQL. In the past Facebook was really useful as the fastest way of sharing any MySQL related concern to proper audience.

      In any case, I had no plans to use Facebook until I left Oracle. Then I noted numerous invitations from my dear friend Sinisa to join, and gave up... It's all his fault after all.

  4. really really hope that they will continue work on MySQL Internals manual one day

    1. Yes, it does not matter if it's a separate manual or just additional information in the main manual. A lot of details are still not documented.