Showing posts with label savepoint. Show all posts
Showing posts with label savepoint. Show all posts

Wednesday, May 1, 2019

Fun with Bugs #85 - On MySQL Bug Reports I am Subscribed to, Part XX

We have a public holiday here today and it's raining outside for a third day in a row already, so I hardly have anything better to do than writing yet another review of public MySQL bug reports that I've subscribed to recently.

Not sure if these reviews are really considered useful by anyone but few of my readers, but I am still going to try in a hope to end up with some useful conclusions. Last time I've stopped on Bug #94903, so let me continue with the next bug in my list:
  • Bug #94912 - "O_DIRECT_NO_FSYNC possible write hole". In this bug report Janet Campbell shared some concerns related to the way O_DIRECT_NO_FSYNC (and O_DIRECT) settings for innodb_flush_method work. Check comments, including those by Sunny Bains, where he agrees that "...this will cause problems where the redo and data are on separate devices.". Useful reading for anyone interested in InnoDB internals or using  innodb_dedicated_server setting in MySQL 8.0.14+.
  • Bug #94971 - "Incorrect key file error during log apply table stage in online DDL". Monty Solomon reported yet another case when "online' ALTER for InnoDB table fails in a weird way. The bug is still "Open" and there is no clear test case to just copy/paste, but both the problem and potential solutions (make sure you have "big enough" innodb_online_alter_log_max_size or better use pt-online-schema-change or gh-ost tools) were already discussed here.
  • Bug #94973 - "Wrong result with subquery in where clause and order by". Yet another wrong results bug with subquery on MySQL 5.7.25 was reported by Andreas Kohlbecker. We can only guess if MySQL 8 is also affected (MariaDB 10.3.7 is not, based on my test results shared below) as Oracle engineer who verified the bug had NOT card to check or share the results of this check. What can be easier than running this (a bit modified) test case on every MySQL major version and copy pasting the results:
    MariaDB [test]> CREATE TABLE `ReferenceB` (
        ->   `id` int(11) NOT NULL,
        ->   `bitField` bit(1) NOT NULL,
        ->   `refType` varchar(255) NOT NULL,
        ->   `externalLink` longtext,
        ->   PRIMARY KEY (`id`)
        -> ) ENGINE=MyISAM DEFAULT CHARSET=utf8;
    Query OK, 0 rows affected (0.170 sec)

    MariaDB [test]> INSERT INTO ReferenceB (id, bitField, refType, externalLink) VALUES(1, 0, 'JOU', NULL);
    Query OK, 1 row affected (0.027 sec)

    MariaDB [test]> INSERT INTO ReferenceB (id, bitField, refType, externalLink) VALUES(2, 0, 'JOU', NULL);
    Query OK, 1 row affected (0.002 sec)

    MariaDB [test]> SELECT hex(bitField) from ReferenceB  where id in (select id as
    y0_ from ReferenceB  where refType='JOU') order by externalLink asc;
    +---------------+
    | hex(bitField) |
    +---------------+
    | 0             |
    | 0             |
    +---------------+
    2 rows in set (0.028 sec)
    But we do not see anything like that in the bug report... This is sad.
  • Bug #94994 - "Memory leak detect on temptable storage engine". Yet another memory leak (found with ASan) reported by Zhao Jianwei, who had also suggested a patch.
  • Bug #95008 - "applying binary log doesn't work with blackhole engine tables". This bug was reported by Thomas Benkert. It seems there is a problem to apply row-based events to BLACKHOLE table and this prevents some nice recovery tricks from working.
  • Bug #95020 - "select no rows return but check profile process Creating sort index". Interesting finding from cui jacky. I can reproduce this with MariaDB as well. It seems we either have to define some new stage or define "Creating sort index" better than in the current manual. This:
    The thread is processing a SELECT that is resolved using an internal temporary table.
    is plain wrong in the case shown in the bug report IMHO.
  • Bug #95040 - "Duplicately remove locks from lock_sys->prdt_page_hash in btr_compress". One of those rare cases when Zhai Weixiang does not provide the patch, just suggests the fix based on code review :)
  • Bug #95045 - "Data Truncation error occurred on a write of column 0Data was 0 bytes long and". This really weird regression bug in MySQL 8.0.14+ was reported by Adarshdeep Cheema. MariaDB 10.3 is surely not affected.
  • Bug #95049 - "Modified rows are not locked after rolling back to savepoint". Bug reporter, John Lin, found that fine MySQL manual does not describe the real current implementation. Surprise!
  • Bug #95058 - "Index not used for column with IS TRUE or IS FALSE operators". Take extra care when using BOOLEAN columns in MySQL. As it was noted by Monty Solomon, proper index is NOT used when you try to check BOOLEAN values as manual suggests, using IS TRUE or IS FALSE conditions. Roy Lyseng explained how such queries are threated internally, but surely there is a better way. MariaDB 10.3.7 is also affected, unfortunately.
  • Bug #95064 - "slave server may has gaps in Executed_Gtid_Set when a special case happen ". Nice bug report from yoga yoga, who had also contributed a patch. Parallel slave can easily get out of sync with master in case of lock wait timeout and failed retries. Again, we do NOT see any check if MySQL 8 is affected, unfortunately.
  • Bug #95065 - "Strange memory management when using full-text indexes". We all know that InnoDB FULLTEXT indexes implementation is far from perfect. Now, thanks to Yura Sorokin, we know also about a verified memory leak bug there that may lead to OOM killing of MySQL server.
  • Bug #95070 - "INSERT .. VALUES ( .., (SELECT ..), ..) takes shared lock with READ-COMMITTED". Seunguck Lee found yet another case of InnoDB locking behavior that MySQL manual does not explain. The bug is still "Open" for some reason.
  • Bug #95115 - "mysqld deadlock of all client threads originating from 3-way deadlock". It took some efforts for bug reporter, Sandeep Dube, and other community users (mostly Jacek Cencek) to attract proper attention to this bug from proper Oracle developer, Dmitry Lenev, until it ended up "Verified" based on code review. We still can not be sure if MySQL 8 is also affected.
That's all for now. I have few more new bug reports that I monitor, but I do not plan to continue with this kind of reviews in upcoming few months in this blog. I hope I'll get a reason soon to write different kind of posts, with more in depth study of various topics...

In any case you may follow me on Twitter for anything related to recent interesting or wrongly handled MySQL bug reports.

This view of Chelsea from our apartment at Chelsea Cloisters reminds me that last year I spent spring holiday season properly - no time was devoted to MySQL bugs :)
To summarize:
  1. Do not use O_DIRECT_NO_FSYNC value for innodb_flush_method if your redo logs are located on different device than your data files. Just don't.
  2. Some Oracle engineers who process bugs still do not care to check if all supported major versions are affected and/or share the results of such checks in public.
  3. There are still many details of InnoDB locking to study, document properly and maybe fix.
  4. I am really concerned with the state of MySQL optimizer. We see all kinds of weird bugs (including regressions) and very few fixes in each maintenance release.

Thursday, October 19, 2017

Fun with Bugs #56 - On Some Public Bugs Fixed in MySQL 5.7.20

While MySQL 8.0.x hardly has much impact on my regular work, recent MySQL 5.7.20 release is something to check carefully. MySQL 5.7 is widely used in production, as a base for Percona Server 5.7, some features may be merged into MariaDB 10.x etc. So, here is my review of some community reported bugs that were fixed in recently released MySQL 5.7.20, based on the release notes.

Usually I start with InnoDB bug fixes, but in 5.7.20 several related fixes were made only to bugs reported internally. So, this time I have to start with partitioning:
  • Bug #86255 - First one to write about, and the bug report is private... Second one (Bug #76418) is also private. All we have is this:
    "Partitioning: In certain cases when fetching heap records a partition ID could be set to zero. (Bug #86255, Bug #26034430)"

    "Partitioning: Queries involving NULL were not always handled correctly on tables that were partitioned by LIST. (Bug #76418, Bug #20748521)"
That's all, folks. I fail to understand why bugs with such a description can stay private after they are fixed. I have to admit: I do not get it. Moreover, I am not going even to try any longer. Lists with one item look stupid, but hiding such bugs is not much better, IMHO.

Lucky I am, there were several bug fixes related to replication:
  • Bug #85352 - "Replication regression with RBR and partitioned tables". This regression bug (comparing to 5.5.x) was reported by Juan Arruti and immediately verified by Umesh Shastry. I do not know why it is listed as InnoDB in the release notes, as it hardly can be about native InnoDB partitioning, based on versions affected.
  • Bug #86224 - "mysqlrplsync errors out because slave is using --super-read-only option". It was reported by Richard Morris and verified by Bogdan Kecman.
  • Bug #84731 - "Group Replication: mysql client connections hang during group replication start", was reported by Kenny Gryp and verified by Umesh Shastry. Another bug reported by Kenny was also fixed in 5.7.20, Bug #84798 - "Group Replication can use some verbosity in the error log".
  • Bug #86250 - "thd->ha_data[ht_arg->slot].ha_ptr_backup == __null || (thd->is_binlog_applier()". This debug assertion was reported by Roel Van de Paar and verified by Umesh Shastry
  • Bug #85639 - "XA transactions are 'unsafe' for RPL using SBR". It was reported by João Gramacho.
  • Bug #86288 - "mysqlbinlog read-from-remote-server not honoring rewrite_db filtering", was reported by Julie Hergert.
Other bug fixes not to miss are:
  • Bug #85622 - "5.6: wrong results with materialization, views". It was reported by Shane Bester. Even though it was initially stated that only 5.6 is affected, release notes say there was a fix for 5.7.20 and even 8.0.3.
  • Bug #82992 - "Some warnings appear in dump from mysqldump". This funny bug was found by Nong LO and verified by Sinisa Milivojevic.
  • Bug #81714 - "mysqldump get_view_structure does not free MYSQL_RES in one error path". Thisbug was reported by Laurynas Biveinis, but it was noticed and patched by Yura Sorokin, also from Percona.
  • Bug #83950 - "LOAD DATA INFILE fails with an escape character followed by a multi-byte one", was reported by yours truly and verified by Umesh Shastry. Unfortunately the bug report itself does NOT say explicitly what versions had got the fix.
  • Bug #79596 - "If client killed after ROLLBACK TO SAVEPOINT previous stmts committed". This regression and potential data corruption bug was reported by Sveta Smirnova, verified by Umesh Shastry and studied at the source code level by Zhai Weixiang. Nice to see it fixed!
That's all, few build and packaging related bugs aside.

This was my very last attempt to do a detailed review of bug reports from MySQL Community based on official Release Notes. With private bugs and very few fixes for things coming from the public bugs database in general, it seems to make zero sense now to continue these. Authors of patches contributed are properly mentioned by Oracle, and we all know who verify community bug reports... One Twitter message would be enough to fit everything I have to say, and any real details should better be checked in git commits.

It's time to get back to the roots of this series and maybe write about bugs just opened, bugs not properly handled or just funny ones. I think it helped a lot back in 2013 to make MySQL 5.6 a release that was commonly accepted as a "good one". It's not fun any more and not much useful to report Oracle achievements in public bugs fixing, so I'd better switch to their problems.

Saturday, October 3, 2015

Fun with Bugs #37 - Bugs fixed in MySQL 5.6.27

MySQL 5.6.27 was released on September 30 formally. Source code is also available on GitHub, and I have it compiled (some users are less lucky) and running for a couple of days already. In this post I'll comment on some bugs reported by MySQL Community that are fixed there.

I'd like to start with a couple of bugs where patches were also contributed. First of all, the fix suggested by Stewart Smith in Bug #72811, "Set NUMA mempolicy for optimum mysqld performance", helps to allocate memory in a more reasonable way on NUMA-enabled systems. Previously it was like all interleaved or nothing, now there is a way to apply this only to the InnoDB buffer pool. The bug was formally verified by Umesh.

Alexey Kopytov had contributed two important fixes, for Bug #76927, "Duplicate UK values in READ-COMMITTED (again)" that was verified by Shane Bester, and for Bug #74891 (reported by Stewart Smith and verified by Umesh).

As a side note, I like to see fixes for other platforms than x86_64 and other OSes than Linux. Historically I started to work on MySQL by checking compilation or build bugs for unusual platforms, compilers and environments... There is one more bug of this kind fixed in 5.6.27: Bug #76135 , "data corruption on arm64", reported by Daniel Frazier.

Other important InnoDB bug fixes include:
  • Bug #77743, "Auto-increment sequence gets reset", reported by my colleague Marcos Albe and verified by Umesh
  • Bug #75185, "lower_case_table_names=0 on windows leads to problems", reported by Shane Bester and verified by Umesh

A lot of replication bugs were fixed in this release:
  • Bug #78389, "5.6.24: Lost data during master restart if partial transact has been downloaded", reported by Simon Mudd and verified by Shane Bester.
  • Bug #74607, "slave io_thread may get stuck when using GTID and low slave_net_timeouts", reported by Santosh Praneeth Banda and verified by Umesh
  • Bug #76959, "Gaps in Retrieved_Gtid_Set while no gaps in Executed_Gtid_Set", reported by my colleague Sveta Smirnova and verified by Umesh
  • Bug #76727, "Slave assertion in unpack_row with ROLLBACK TO SAVEPOINT in error handler", reported by Philip Stoev and verified by Umesh. In Percona we found out hard way that while one can create a trigger that does ROLLBACK TO SAVEPOINT in the code (see related documentation Bug #77163) and it even seems to work as expected, this leads to problems on slave replicating such a transaction. This fix in 5.6.27 is just a start of the long way to proper fixes in all places, and those interested in that's wrong can check great analysis by Vlad Lesin in Percona Server bug #1483251
  • Bug #76618, "SHOW BINLOG EVENTS completely locks down writes to binlog, hence transactions", reported by Shlomi Noach and verified by Shane Bester
  • Bug #76493, "Binlog statement is not ignored", reported by Dan Lukes and verified by Umesh
  • Bug #76379, "binlog_error_action doesn't handle some failures during binlog rotation", reported by Santosh Praneeth Banda and verified by Umesh
  • Bug #74950, that is still private at the moment (had I written before that I hate when this happens?), is described as follows in release notes:
    "Modifying the master_info_repository or relay_log_info_repository inside a transaction and later rolling back that transaction left the repository in an unusable state. We fix this by preventing any modification of these repositories inside a transaction."
    This is not the last remaining bug related to "crash-safe" replication in 5.6, implemented by storing information in explicit InnoDB tables in mysql database. Changes there are a part of transaction and when changes of transaction are rolled back, we often end up with replication broken in some way or in some otherwise bad state. It seems we are doomed to see these problems until MySQL implements some "micro transactions" for its "data dictionary", that are handled differently than normal transactions to usual InnoDB tables... One day I'll write a separate blog post on all the related bugs already identified
  • Bug #74089, "Report Relay_Log_File and Relay_Log_Pos on relay-log-recovery", is a nice feature request by Jean-François Gagné that was verified by Luis Soares himself, and implemented in 5.6.27. Well done!
  • Bug #73806, still private, and Bug #76746, "Broken replication on SQL thread restart if gtid_mode is enabled", reported by Davi Arnaut who also contributed a patch, and verified by Umesh
  • Bug #68525, "Error "When GTID_NEXT is set to a GTID" ROW based replication", reported by Nogueira Jesus and verified by Sveta Smirnova
I find it important to note that most replication bugs fixed in 5.6.27 were found by MySQL Community users, NOT internally by Oracle. I am also surprised with so many replication bugs still existing in 5.6.x now, 2+ years after GA release. I wonder what's going to happen when new replication features of 5.7 will be tested by MySQL users...

Several optimizer bugs were also fixed:
  • Bug #77135, "Update on varchar and text columns produce incorrect results", reported by Chris Sims and formally verified by Sinisa Milivojevic
  • Bug #76349, "memory leak in add_derived_key()", reported by Vlad Lesin and verified by Shane Bester
  • Bug #75248, "OR conditions on multi-column index may not use all index columns to filter rows", reported by Yoshinori Matsunobu and verified by Umesh

Performance Schema also had got some fixes:
  • Bug #77577, "setup_timers initialization assumes CYCLE timer is always available", reported by Alexey Kopytov who also contributed a patch, and verified by Umesh
  • Bug #74614, "events_statements_history has errors=0 when there are errors", reported by Daniël van Eeden and verified by Umesh 
I'd also want to mention Bug #76480, "mysqlimport --use-threads doesn't use multiple threads", reported by my colleague Miguel Angel Nieto that got a detailed analysis from yours truly and then was verified by Umesh. That's all I contributed to the release :)

The last but not the least, if you use memcached to work with InnoDB tables, check Bug #75199, "MySQL crashed because of flush_all", reported by Zhai Weixiang and verified by Umesh (as most of the community bug reports for a couple of last years).

MySQL 5.6.27 seems to be a really useful and important release for those who rely on InnoDB and replication (that is, for all of us probably). Note that most of the bugs mentioned above also affected 5.7.x and are fixed in upcoming MySQL 5.7.9 and 5.8.0 releases.