Fun with Bugs #67 - On Some Public Bugs Fixed in MySQL 8.0.11 GA

I stopped reviewing MySQL Release Notes for quite a some time, but major GA releases of MySQL do not happen often, so I decided to make an exception and write about some bugs from Community users fixed in MySQL 8.0.11 GA.

I'll start with good news about MySQL 8.0.11 GA! You can get sources at GitHub, and I had no problems to build on Fedora 27 on my good old QuadCore box, using the following cmake command line:

[openxs@fc23 mysql-server]$ cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DWITH_EMBEDDED_SERVER=OFF -DDOWNLOAD_BOOST=1 -DWITH_BOOST=/home/openxs/boost -DENABLE_DOWNLOADS=1 -DWITH_UNIT_TESTS=OFF -DCMAKE_INSTALL_PREFIX=/home/openxs/dbs/8.0
...
[openxs@fc23 mysql-server]$ time make -j 4
...

[100%] Built target mysqld

real    33m52.791s
user    105m47.475s
sys     8m19.018s

Comparing to previous experience, I had minor problem with unit tests, so just skipped them with -DWITH_UNIT_TESTS=OFF option. There is no problem to run the resulting binaries, unless you try to use data directory from older 8.0.x. Then you'll end up with:

2018-04-19T15:36:35.165841Z 1 [ERROR] [MY-011092] [Server] Upgrading the data dictionary from dictionary version '80004' is not supported.
2018-04-19T15:36:35.166239Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2018-04-19T15:36:35.166310Z 0 [ERROR] [MY-010119] [Server] Aborting

I had to remove data directory and initialize it from scratch (it was testing instance anyway, last time used for real while I worked on this presentation):

[openxs@fc23 8.0]$ rm -rf data/*
[openxs@fc23 8.0]$ bin/mysqld --no-defaults --initialize-insecure --port=3308 --socket=/tmp/mysql.sock --basedir=/home/openxs/dbs/8.0 --datadir=/home/openxs/dbs/8.0/data --skip-log-bin
2018-04-19T15:43:55.324606Z 0 [Warning] [MY-010139] [Server] Changed limits: max_open_files: 1024 (requested 8161)
2018-04-19T15:43:55.324726Z 0 [Warning] [MY-010142] [Server] Changed limits: table_open_cache: 431 (requested 4000)
2018-04-19T15:43:55.325147Z 0 [System] [MY-013169] [Server] /home/openxs/dbs/8.0/bin/mysqld (mysqld 8.0.11) initializing of server in progress as process 20034
2018-04-19T15:44:14.438776Z 4 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2018-04-19T15:44:29.625227Z 0 [System] [MY-013170] [Server] /home/openxs/dbs/8.0/bin/mysqld (mysqld 8.0.11) initializing of server has completed
[openxs@fc23 8.0]$ bin/mysqld_safe --no-defaults --port=3308 --socket=/tmp/mysql.sock --basedir=/home/openxs/dbs/8.0 --datadir=/home/openxs/dbs/8.0/data --skip-log-bin &
[1] 20080
[openxs@fc23 8.0]$ 2018-04-19T15:44:58.224816Z mysqld_safe Logging to '/home/openxs/dbs/8.0/data/fc23.err'.
2018-04-19T15:44:58.271255Z mysqld_safe Starting mysqld daemon with databases from /home/openxs/dbs/8.0/data

[openxs@fc23 8.0]$ bin/mysql -uroot --socket=/tmp/mysql.sock
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 7
Server version: 8.0.11 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show variables like '%version%';
+-------------------------+------------------------------+
| Variable_name           | Value                        |
+-------------------------+------------------------------+
| innodb_version          | 8.0.11                       |
| protocol_version        | 10                           |
| slave_type_conversions  |                              |
| tls_version             | TLSv1,TLSv1.1,TLSv1.2        |
| version                 | 8.0.11                       |
| version_comment         | MySQL Community Server (GPL) |
| version_compile_machine | x86_64                       |
| version_compile_os      | Linux                        |
| version_compile_zlib    | 1.2.11                       |
+-------------------------+------------------------------+
9 rows in set (0.00 sec)

So, you can build MySQL 8.0.11 right now and start using it to make your own conclusions about this release.

I still do not care about NoSQL, JSON, new cool features etc. You'll see megabytes of texts about these by the end of 2018. I am going to concentrate mostly on InnoDB, replication bugs and few others:

•  I am happy to start with Bugt #89509 - "Valgrind error on innodb.blob_page_reserve, bundled zlib", reported by Laurynas Biveinis. See also his Bug #89597 - "Valgrind reporting memory leak on MTR test main.validate_password_component" and Bug #89433 - "NULL dereference in dd::tables::DD_properties:unchecked_get". Percona engineers spent a lot of efforts recently testing MySQL 8.0.x and reporting bugs noted. I think Oracle should explicitly admit the impact of Percona's QA effrots for the quality of this GA release.
• Biug #89127 - "Optimize trx_rw_is_active() by tracking the lowest active transaction id". This bug was reported by Zhai Weixiang, who had suggested a patch also.
• Bug #89129 - "create table+DML on innodb_ddl_log table=crash in lock0lock.cc:7414:release_lock". This bug was reported by Ramana Yeruva. Tables were made protected and DDL and DML operations on these tables are no longer permitted.
• Bug #89087 - "Assertion `key->flags & 1' failed". This debug assertion (related to the way PRIMARY key was created based on UNIQUE one) was reported by Roel Van de Paar for 5.7.21, but we see the fix documented only for 8.0.x.
• Bug #87827 - "Performance regression in "create table" speed and scalability in 8.0.3". It was reported by Alexander Rubin from Percona.
• Bug #87812 - "Concurrent DDL operation in progress even after acquiring backup lock". Nice bug report from Debarun Banerjee.
• Bug #87532 - "Replay log record cause mysqld crash during online DDL". I am happy to see impovements in "online ALTER" implementation that covers all GA versions, not just 8.0. I am also happy to see Oracle engineers (Ohm Hong in this case) reporting bugs in public!
• Bug #88272 - "Assertion `new_value >= 0' failed.". Yet another debug assertion found by Roel Van de Paar, this time related to GTIDs and XA transactions. Check also his Bug #88262 - "ERROR 1598 (HY000): Binary logging not possible + abort".
• Bug #84415 - "slave don't report Seconds_Behind_Master when running slave_parallel_workers > 0". Yet another contribution from Percona engineers. This bug was reported by Marcelo Altmann and patches were provided by Robert Golebiowski. This bug is also fixed in MySQL 5.7.22.
• Bug #89793 - "INFORMATION_SCHEMA.STATISTICS field type change". Unexpected change in early 8.0.x versions was noted and reported by Mark Guinness.
• Bug #89584 - "5.7->8.0 upgrade crash with default-time-zone set". Nice to see this bug (reported by Shane Bester) fixed in GA release.
• Bug #89487 - "ALTER TABLE hangs in "Waiting for tablespace metadata lock" state". This regression bug was reported by Sveta Smirnova.
• Bug #89324 - "main.comment_column2 fails with compression". This regression was noted and reported by Manuel Ung.
• Bug #89122 - "Severe performance regression in server bootstrap". I am really happy to see this bug reported by Georgi Kodinov fixed. I noted it as soon as I started testing 8.0.x (see a duplicate by Roel Van de Paar, Bug #89444) and it was very annoying. I've already checked (see above) that the problem is gone!
• Bug #89038 - "Add new column to 'mysql.routines' to accommodate the Polygot project". So, Oracle is planning to support stored programs in different languages! Thank you, Sivert Sørumgård, for reporting this in public! See also his Bug #89035 - "Reject LCTN changing after --initialize".
• Bug #87836 - "XA COMMIT/ROLLBACK rejected by non-autocommit session with no active transaction". It would be sad if this bug is not fixed in MySQL 5.7.x, where it was originally found by Wei Zhao.
• Bug #87708 - "MDL for column statistics is not properly reflected in P_S.METADATA_LOCKS". It was reported by Erik Frøseth.
• Bug #85997 - "inplace alter table with foreign keys causes table definition mismatch". This bug was reported by Magnus Blåudd.
• Bug #85561 - "Users can be assigned non-existing roles as default". Nice to see this bug reported by Giuseppe Maxia fixed in GA release.
• Bug #33004 - "integer constants casted to bigints by unions". This bug was reported by Domas Mituzas more than 10 years ago!

Now I have to stop, as I found private bug in release notes, Bug #89512. Based on description:

"Window function row-buffer handling has been refactored to reduce the number of handler reads by 25%. (Bug #89512, Bug #27484133)"

I truly do not get why it remains private (or why it was reported in public for such a "sensitive" matter), so I better stop.

MySQL 8 is GA, finally! There are a lot more fixes there that I had not mentioned above. I am surely there is even more bugs to find. So, happy hunting!

Fun with Bugs #66 - On MySQL Bug Reports I am Subscribed to, Part VI

I have some free time today, but I am still lazy enough to work on numerous planned and pending "ToDo" kind of posts, so why not to continue review of older MySQL bugs I am subscribed to. Today I am going to list 15 more bugs reported more than a year ago and still not fixed:

• Bug #85805 - "Incorrect ER_BAD_NULL_ERROR after LOAD DATA LOCAL INFILE". This detailed bug report by Tsubasa Tanaka stays "Verified" for more than a year already. It's a great example of gdb use for MySQL troubleshooting. Setting a couple of breakpoints may really help to understand how MySQL works and why some weird errors happen.
• Bug #85536 - "Build error on 5.5.54". It's clear that almost nobody besides Roel Van de Paar cares about build problem of MySQL 5.5.x(!) on Ubuntu 16.10(!). Anyway, it's strange that the bug remains "Verified" and not closed in any way if Oracle really does not intend to support MySQL 5.5 any longer. For now it seems MySQL 5.5 is still under extended support, so I hope to see this build problem fixed with some final 5.5.x release.
• Bug #85501 - "Make all options settable as variables in configuration files". We usually see Umesh Shastry processing bugs reported by other, but this is a rare case when he reports something himself. It's a great feature request.
• Bug #85447 - "Slave SQL thread locking issue on a certain XA workload on master". There are good reasons to think that this bug reported by Laurynas Biveinis may be fixed since MySQL 5.7.18, but no one cares to close it properly.
• Bug #85382 - "Getting semi-sync reply magic number errors when slave_compressed_protocol is 1". This bug was reported by Jaime Sicam. Read also comments from other community members and make your own conclusions. It seems setting slave_compressed_protocol to 1 is a bad idea in general...
• Bug #85191 - "performance regression with HANDLER READ syntax". Zhai Weixiang found clear performance regression in the way MySQL 5.7 uses metadata locking for HANDLER commands.
• Bug #85016 - "better description for: OS error: 71". Clear and simple request from Shane Bester still stays "Verified". I am not that Oracle customer affected anyway, but this seems strange to me.
• Bug #84958 - "InnoDB's MVCC has O(N^2) behaviors". This one bug report from Domas Mituzas could be a topic for a series of blog posts... It clearly states that:
  

  "if there're multiple row versions in InnoDB, reading one row from PK may have O(N) complexity and reading from secondary keys may have O(N^2) complexity"

  There is a patch that partially fixes the problem submitted by Laurynas Biveinis and created by Alexey Midenkov. While this bug is still "Verified" take carer when using secondary indexes in concurrent environments when the same data are often changed.
• Bug #84868 - "Please make it possible to query replication information consistently". Great feature request (or bug report, if you consider inconsistency as a bug) from Simon Mudd.
• Bug #84615 - "More steps in connection processlist state/ events_stages". Sveta Snirnova cared to ask to split some well known statement execution stages like "cleaning up" into more detailed ones. I think this is really important to simplify troubleshooting with performance_schema. Wrong/misleading/too generic stages forces to use other tools and may lead to wrong conclusions. I hit this with "statistics" also, see Bug #84858. Rare case when Sveta's request just stays "Open", for more than a year already.
• Bug #84467 - "ALTERing KEY_BLOCK_SIZE keeps the old kbs in KEYs.". Jean-François Gagné and other well known bug reporters found several problems related to KEY_BLOCK_SIZE. It seems Oracle engineers decided NOT to fix them (see Bug #88220). But then why this bug still stays "Verified"? Consistency in bugs processing is one of my dreams...
• Bug #84439 - "Table of row size of ~800 bytes does not compress with KEY_BLOCK_SIZE=1." Yet another bug report from Jean-François Gagné. Based on lack of activity, those looking for smaller data size, compression etc should look elsewhere and do not expect much from Oracle's InnoDB. Question is, what other engines with data compression will be supported by Oracle's MySQL 8 (or 9) GA? When you get tired wondering, consider MariaDB or Percona Server instead - they do support storage engines that are both transactional and were designed with write efficiency and space efficiency in mind. Hint: they rock...
• Bug #84274 - "READ COMMITTED does not scale after 36 threads (in 5.6 after 16 threads)". Sveta Smirnova had a chance to run benchmarks on 144 cores (the largest box I ever had a chance to use for benchmarking had 12 cores, so what do I know...) and the result is clear - READ COMMITTED transaction isolation level does not scale well (comparing to default REPEATABLE READ). It's counter intuitive for many, but that's what we have. I doubt MySQL 8 is going to change this (unfortunate) situation.
• Bug #84241 - "Potential Race Condition". This was found in MySQL 5.7 by Rui Gu with a little help from Helgrind.
• Bug #84024 - "Optimizer thinks clustered primary key is not covering". This bug was reported by Manuel Ung. Let me quote a comment by Øystein Grøvlen:
  

  "I can agree that the cost model for join buffering is not perfect. If so, I think we should improve this model, not rely on heuristics about covering indexes versus table scan."

  I can not agree more! Let's hope this really happens in MySQL 9 at least.

You probably noted that we see mostly already famous bug reporters mentioned in this list. But names of reporters, their customer or partner status, known achievements, even clear regressions found or patches provided do not force Oracle to fix problems faster these days... They have their own agenda and great plans for MySQL, obviously.

I also have my own agenda, so I'll proceed with this glass of wine...

Fun with Bugs #65 - On MySQL Bug Reports I am Subscribed to, Part V

I think it's time to review some bugs I've subscribed to several months ago, those older than in the first post from this series. There are several really serious bugs in the list of 15 below:

• Bug #87560 - "XA PREPARE log order error in replication and binlog recovery". This bug was reported by Wei Zhao, who also provided patches.
• Bug #87526 - "The output of 'XA recover convert xid' is not useful". This bug reported by Sveta Smirnova is well known and is a real pain for DBAs who have to deal with incomplete XA transactions after some crash or unexpected restart. Check PS-1818 and MariaDB task MDEV-14593. The problem is resolved in MariaDB 10.3.3+ by a new XA RECOVER FORMAT='SQL' option.
• Bug #87164 - "Queries running much slower in version 5.7 versus 5.6". It was reported by  Alok Pathak from Percona and stays "Verified" since August, 2017.
• Bug #87130 - "XA COMMIT not taken as transaction boundary". Yet another XA bug report with a patch contributed by Wei Zhao.
• Bug #87084 - "FK DELETE CASCADE does not honor innodb_lock_wait_timeout". Nice report by Elena Stepanova from MariaDB. As you can find out from MDEV-15219, it's properly fixed in MariaDB 10.2.13+.
• Bug #87065 - "Release lock on table statistics after query plan created". Great feature request by Sveta Smirnova. The actual problem behind this feature request was resolved by Percona in versions 5.7.20-18+. This fix is one of few really good reasons to use recent Percona Server 5.7, so I opened MDEV-15101 for MariaDB also.
• Bug #86926 - "The field table_name (varchar(64)) from mysql.innodb_table_stats can overflow.". I really wonder why this bug report by Jean-François Gagné still remains just "Verified".
• Bug #86865 - "InnoDB does unnecessary work when extending a tablespace". Bug report and patch by Alexey Kopytov.
• Bug #86705 - "Memory leak of Innodb". Great example of a leak (in MySQL 5.5.x only) found with a help of Valgrind/Massif. Qinglin Zhang also suggested a simple patch.
• Bug #86475 - "Error with functions and group by with ONLY_FULL_GROUP_BY". Nice feature request from Arnaud Adant.
• Bug #86462 - "mysql_ugprade: improve handling of upgrade errors". Simon Mudd asked for some better error messages at least, so that running under strace would not be needed.
• Bug #86215 - "MySQL is much slower in 5.7 vs 5.6". This report from Mark Callaghan includes a lot of results and details on performance regressions at low concurrency starting from 5.0 and up to 8.0. For some cases studied the biggest drop in QPS is from 5.6 to 5.7.
• Bug #86163 - "can't update temporary table when joined with table with triggers on read-only". I'd call this bug found by Bret Westenskow funny. The rest of them in this list are serious.
• Bug #85970 - "Memory leak with transactions greater than 10% of the total redo log size". Nice corner case was studied by Joffrey MICHAÏE. Whenever you see
  

  The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size...

  error messages, take care, as memory allocated may not be released.
• Bug #85910 - "Increased Performance_Schema overhead on Sending data". Seems to be a known problem fixed in MySQL 5.7. Still a nice report by Jervin R on the overhead one may expect from MySQL 5.6 on some systems.

So, we traveled back in time for a year. I am really sorry to see XA-related bugs and some reports with patches properly contributed still active.

Fun with Bugs #64 - On MySQL Bug Reports I am Subscribed to, Part IV

I've subscribed to more than 15 new MySQL bug reports since the previous post in this series, so it's time for a new one. I am trying to follow important, funny or hard to process bug reports every day. Here is the list of the most interesting recent ones starting from the latest (with several still not processed properly):

• Bug #90211 - "Various warnings and errors when compiling MySQL 8 with Clang".  Roel Van de Paar and Percona in general continue their QA efforts in a hope to make MySQL 8 better. Current opinion of Oracle engineers on this bug is the following:
  

  "First of all, these issues are in protobuf, not MySQL per se. There are some warnings with Clang 6, but since they're in third-party code, we have simply disabled them when compiling protobuf (will be part of 8.0.11). Optionally, -DUSE_SYSTEM_LIBS=1 will use system protobuf and thus not compile the files in question.
  As for the crash, we don't support prerelease compilers (more generally, we support platforms, not compilers). Given the stack trace, it is highly likely that the issue either is in the prerelease Clang, or in protobuf."

  Let's see how it may end up. Roel rarely gives up easily...
• Bug #90209 - "Performance regression with > 15K tables in MySQL 8.0 (with general tablespaces)". Nice regression bug report from Alexander Rubin. It is still "Open".
• Bug #90190 - "Sig=6 assertion in MYSQL_BIN_LOG::new_file_impl | binlog.cc:6862". Yet another bug report from Percona employee, Ramesh Sivaraman.
• Bug #89994 - "INDEX DIRECTORY shown as valid option for InnoDB table creation". Everybody knows how much I like fine MySQL manual. Even more I like when missing or wrong details are found there, like in this case reported by by colleague from MariaDB, Claudio Nanni.
• Bug #89963  - "Slowdown in creating new SSL connection". Maybe it's comparing apples to oranges, as stated in one of comments, but I am surprised that this (performance regression) bug report by Rene' Cannao' is still "Open". It requires more attention, IMHO. Speed of connections matters a lot for MySQL.
• Bug #89904 - "Can't change innodb_max_dirty_pages_pct to 0 to flush all pages". Good intentions to set better default value (applied a bit later than needed) led to the problem. As Simon Mudd put it:
  

  "innodb_max_dirty_pages_pct_lwm setting has existed since 5.6. This issue only comes up as by changing the default value to 10 those of us who have ignored it until now never noticed it existed. That is a shame as setting this value to a value other than 0 (e.g. 10 which is the new default) should be better and trigger some background flushing of dirty pages avoiding us hitting innodb_max_dirty_pages_pct which would trigger much more aggressive behaviour which is not really desirable."

• Bug #89876 - "mysqladmin flush-hosts is not safe in GTID mode". Yet another bug report from Simon Mudd. See also Bug #88720 that highlights even more problems with various FLUSH statements and GTIDs.
• Bug #89870 - "Group by optimization not used with partitioned tables". For some reason this report from Arnaud Adant is still "Open". As my colleague Richard Stracke stated:
  

  "The only solution would be, that the optimizer is able to check, if the condition in the where clause include the whole table (or partition) and in this case use group by optimization."

• Bug #89860 - "XA may lost prepared transaction and cause different between master and slave." As this (and other, like Bug #88534) bug report from Michael Yang shows, there is still a long way to go until it would be safe to use XA transactions with MySQL.
• Bug #89834 - "Replication will not connect on IPv6 - does not function in an IPv6 only environ". This bug report from Tim St. Pierre is still "Open".
• Bug #89822 - "InnoDB retries open on EINTR error only if innodb_use_native_aio is enabled". We have patch contributed by Laurynas Biveinis from Percona.
• Bug #89758 - "Conversion from ENUM to VARCHAR fails because mysql adds prefix index". This funny bug was found and reported by Monty Solomon.
• Bug #89741 - "Events log Note level messages even for log_warnings=0". Nikolai Ikhalainen found that this problem happens only in versions 5.5.x and 5.6.x, so chances to see it fixed are low. But I still want to know if this ever happens.
• Bug #89696 - "Cyclic dependencies are not resolved properly with cascade removal". Make sure to check nice discussion that my dear friend Sinisa Milivojevic had with a bug reporter, Andrei Anishchenko, before marking the bug as "Verified". This regression was most likely caused by a change in MySQL 5.7.21:
  

  "InnoDB: An iterative approach to processing foreign cascade operations resulted in excessive memory use. (Bug #26191879, Bug #86573)"

• Bug #89625 - "please package the debug symbols *.pdb files!". Shane Bester always cared about having a way to debug on Windows. Recently I also started to care about this...

---

It's April Fools' Day today, so why not to make fool of myself assuming that anyone cares about the series of blog posts.