Showing posts with label security. Show all posts
Showing posts with label security. Show all posts

Saturday, January 18, 2020

Fun with Bugs #92 - On MySQL Bug Reports I am Subscribed to, Part XXVI

I'd like to continue reviewing MySQL bug reports from Community users that I considered interesting and subscribed to. Unlike in the previous post in this series, I am not going to check test cases on any competitor product, but will use only recently released MySQL 5.7.29 and 8.0.19 for checks, if any. This time I'll concentrate on bugs reported in November 2019.

As usual, I mostly care about optimizer, InnoDB and replication related bugs. Here is the list:
  • Bug #97476 - "Range optimizer skips rows". This bug reported by Ilya Raudsepp looks like a clear regression in MySQL 8.0.x comparing to MySQL 5.7.x at least. I get the following correct results with 5.7.29:
    mysql> SELECT t.id
        -> FROM Test t
        -> JOIN (
        ->     SELECT item_id, MAX(created_at) AS created_at
        ->     FROM Test t
        ->     WHERE (platform_id = 2) AND (item_id IN (3,2,111)) AND (type = 'Default')
        ->     GROUP BY item_id
        -> ) t2 ON t.item_id = t2.item_id
        ->   t.item_id = t2.item_id
        ->   AND t.created_at = t2.created_at
        ->   AND t.type = 'Default'
        -> WHERE t.platform_id = 2;
    +----+
    | id |
    +----+
    |  6 |
    |  3 |
    |  5 |
    +----+
    3 rows in set (0,03 sec)

    mysql> select version();
    +-----------+
    | version() |
    +-----------+
    | 5.7.29    |
    +-----------+
    1 row in set (0,02 sec)
  • Bug #97531 - "5.7 replication breakage with syntax error with GRANT management". This tricky bug reported by Simon Mudd applies also to MySQL 8.0.x. It is closed as fixed, but the fix had not made it to recent 5.7.29 and 8.0.19 releases, so you'll have to wait for few more months.
  • Bug #97552 - "Regression: LEFT JOIN with Impossible ON condition performs slowly". Yet another optimizer regression in MySQL 8 (comparing to 5.7.x) that is fixed only in MySQL 8.0.20+. The bug was reported by Fredric Johansson.
  • Bug #97648 - "Bug in order by clause in union clause". Yet another regression (at least from user's point of view) in recent MySQL 5.7.x and 8.0.x comparing to 5.6.x. This time without a "regression" tag. The bug was reported by Andrei Mart.
  • Bug #97662 - "MySQL v8.0.18 FIPS mode is no longer supported". According to Ryan L, MySQL 8.0.18+ is no longer supporting ssl_fips_mode=STRICT, as OpenSSL 1.1.1 is not FIPS-compatible and MySQL Server must be compiled using OpenSSL 1.1.1 or higher. That's interesting. Check also this link.
  • Bug #97682 - "Handler fails to trigger on Error 1049 or SQLSTATE 42000 or plain sqlexception". This regression (comparing to MySQL 5.7) was reported by Jericho Rivera. It is fixed in MySQL 8.0.20. The patch was provided by Kamil Holubicki.
  • Bug #97692 - "Querying information_schema.TABLES issue". I do not see any documented attempt to check on MySQL 8.0, so I had to add a comment to the bug report. From what I see, in MySQL 8.0.19 we still get different (empty) result from the second query, but at least now we have a warning:
    mysql> SELECT ts.TABLE_SCHEMA
        -> FROM information_schema.TABLES ts
        -> WHERE ts.TABLE_TYPE ='VIEW'
        -> AND ts.TABLE_SCHEMA NOT IN ('sys')
        -> AND ts.TABLE_COMMENT LIKE '%invalid%';
    +--------------+
    | TABLE_SCHEMA |
    +--------------+
    | test         |
    +--------------+
    1 row in set, 1 warning (0,00 sec)

    mysql> show warnings\G
    *************************** 1. row ***************************
      Level: Warning
       Code: 1356
    Message: View 'test.v' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them
    1 row in set (0,00 sec)

    mysql> select version();
    +-----------+
    | version() |
    +-----------+
    | 8.0.19    |
    +-----------+
    1 row in set (0,00 sec)
    The bug was reported by Vinicius Malvestio Grippa.
  • Bug #97693 - "ALTER USER user IDENTIFIED BY 'password' broken by invalid authentication_string". The bug was reported by Nikolai Ikhalainen. MySQL 8.0.19 is still affected.
  • Bug #97694 - "MySQL 8.0.18 fails on STOP SLAVE/START SLAVE stress test". For some reason I do not see any documented attempt to verify this on MySQL 5.7 also. The bug was reported by Przemysław Skibiński, who also suggested a fix.
  • Bug #97734 - "Document the correct method to stop slaving with MTS without a warning or error". I can only agree with this request from Buchan Milne. Please. do :)
  • Bug #97735 - "ALTER USER IF EXISTS ... WITH_MAX_USER_CONNECTIONS 9999 not applied correctly". yet another bug report by Simon Mudd in this list. For some reason, again, I do not see any documented attempt to verify the bug on MySQL 8.0.x, while there is no clear reason to think it is not affected.
  • Bug #97742 - "bad item ref from correlated subquery to outer distinct table". This bug was reported by Song Zhibai, who also had contributed a patch. Based on further comments from  Øystein Grøvlen and these results:
    mysql> EXPLAIN SELECT f3 FROM t1 HAVING (SELECT 1 FROM t2 HAVING f2 LIMIT 1);
    +----+--------------------+-------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
    | id | select_type        | table | partitions | type  | possible_keys | key     | key_len | ref  | rows | filtered | Extra       |
    +----+--------------------+-------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
    |  1 | PRIMARY            | t1    | NULL       | ALL   | NULL          | NULL    | NULL    | NULL |    3 |   100.00 | NULL        |
    |  2 | DEPENDENT SUBQUERY | t2    | NULL       | index | NULL          | PRIMARY | 4       | NULL |    1 |   100.00 | Using index |
    +----+--------------------+-------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
    2 rows in set, 2 warnings (0,00 sec)

    mysql> show warnings\G
    *************************** 1. row ***************************
      Level: Note
       Code: 1276
    Message: Field or reference 'f2' of SELECT #2 was resolved in SELECT #1
    *************************** 2. row ***************************
      Level: Note
       Code: 1003
    Message: /* select#1 */ select `test`.`t1`.`f3` AS `f3` from `test`.`t1` having (/* select#2 */ select 1 from `test`.`t2` having `test`.`t1`.`f2` limit 1)
    2 rows in set (0,00 sec)

    mysql> select version();
    +-----------+
    | version() |
    +-----------+
    | 5.7.29    |
    +-----------+
    1 row in set (0,00 sec)
    I'd say that MySQL 5.7.x is also affected, but for some reason nobody documented any attempt to verify it there. So, I've added a comment.
  • Bug #97777 - "separate global variables (from hot variables) using linker script (ELF)". Beautiful bug report from Daniel Black.With a lot of details, perf and readelf outputs and patch contributed. See also his Bug #97822 - "buf_page_get_gen buf_pool->stat.n_page_gets++ is a cpu waste", with perf analysis up to a single assembler instruction level and fix suggested.
  • Bug #97825 - "dd_mdl_acquire in dd_table_open with dict_sys->mutex hold may cause deadlock". Here I am really puzzled by no visible attempt to check the arguments of bug reporter, Dave Do, who tried to perform lock order analysis by code review. All we see as a result is this:
    "Lock order could be different, but it is irrelevant, since these are locks on totally different levels and can't, in themselves, cause any deadlock."
    What a great argument! Not a bug, surely... We trust you.
"What bugs are you talking about? I have no bugs, neither does MySQL 8!"
To summarize:
  1. MySQL 8 introduces some optimizer (and some other) regressions. They seem to be fixed fast enough, but I wonder why only Community users were able to find them not Oracle's QA...
  2. MySQL 8.0.19 is surely great, but I see many serious bugs fixed o0nly in 8.0.20+.
  3. Percona, Booking and Facebook engineers still continue contributing high quality bug reports, comments/verification details and patches. Oracle is lucky to have such nice partners in making MySQL better.
  4. I still see problems with following proper verification procedures and documenting the results. Too often the bug reported for 8.0.x is NOT checked on 5.7.x as well, regression tag is not set, and so on. Sometimes reports are closed as "Not a bug" without any attempt to follow the analysis provided or prove the point. This is sad and wrong.

Sunday, July 22, 2018

Problems with Oracle's Way of MySQL Bugs Database Maintenance

In one of my previous posts I stated that Oracle does not care enough to maintain public MySQL bugs database properly. I think it's time to explain this statement in details.

The fact that https://bugs.mysql.com/ still exists and community bug reports there are still processed on a regular basis by my former colleagues, Miguel Solorzano, Sinisa Milivojevic, Umesh Shastry, Bogdan Kecman and others, is awesome. Some probably had not expected this to still be the case for 8+ years since Oracle took over the software and procedures around it. My former bugs verification team still seems to exist and even get some new members. Moreover, today we have less "Open" bugs than 6 years ago, when I was preparing to leave Oracle to join (and build) the best MySQL support team in the industry elsewhere...

That's all good and beyond the best expectation of many. Now, what's wrong then with the way Oracle engineers process community bug reports these days?

On the photo above that I made last autumn we see the West Pier in Brighton, England. It keeps collapsing after major damages it got in 2002 and 2003 from storms and fires, and this spring I've seen even more ruins happened. Same happens to MySQL public bugs database - we see it is used less than before, and severe damage to the Community is made by some usual and even relatively simple actions and practices. Let me summarize them.
  1. "Security" bugs are handled in such a way that they never becomes public back, even after the problem is fixed in all affected versions and this fix is documented.

    Moreover, often it takes bug reporter to check "security" flag by mistake or for whatever reason for nobody else ever to be able to find out what exactly the bug was about. This is done even in cases when all other vendors keep the information public or open it after the fix is published. Even worse, sometimes when somebody "escalates" some public bug forgotten for a long time (or wrongly handled) to Oracle engineers, in public, the bug immediately becomes private, even though nobody cared about it for months before. I have complained about this many times everywhere.
    I would so much like to publish all the details of mishandling of Bug #91118, for example, but the bug was made private, so you'd have to just trust my words and quotes, while I prefer to provide arguments everyone can verify... Remember the bug number though and this (top) part of the stack trace:
    ...
    /home/openxs/dbs/8.0/bin/
    mysqld(row_search_mvcc(unsigned char*,
    page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long)+0x2b76)
    [0x1dc78a6]
    /home/openxs/dbs/8.0/bin/
    mysqld(ha_innobase::general_fetch(unsigned
    char*, unsigned int, unsigned int)+0x15f) [0x1c98c1f]
    /home/openxs/dbs/8.0/bin/
    mysqld(handler::ha_index_next_same(unsigned
    char*, unsigned char const*, unsigned int)+0x1e4) [0xe909a4]
    /home/openxs/dbs/8.0/bin/
    mysqld() [0xc5636a]
    ...
    The practice of hiding bugs once and forever was advocated by some MySQL engineers well before Oracle's acquisition, but in Oracle it became a rule that only bravest engineers sometimes can afford NOT to follow.

  2. Some older bug reports are never processed or never revisited back by Oracle engineers.

    Consider this simple example, Bug #70237 - "the mysqladmin shutdown hangs". It was reported almost 5 years ago by a famous Community member, Zhai Weixiang, who even suggested a patch. This bug had not got a single public comment from anyone in Oracle.
    The oldest open server bug today is Bug #44411  - "some Unicode text garbled in LOAD DATA INFILE with user variables". It was reported more than 9 years ago. Moreover, it was once "Verified", but then silently became "Not a bug" and then was reopened by the bug reporter. Nobody cares, even though it's clear that many Oracle engineers should get notification emails whenever any change to public bug report happens. This is also fundamentally wrong, no matter what happened to assignee or engineer who worked on the bug in the past.

    This specific problem with bugs handling is not new, we always had a backlog of bugs to verify and some bugs were re-checked maybe once in many years, but Oracle now has all kinds of resources to fix this problem, and not just reduce the number of open reports by closing them by all fair and not that fair means... See the next item also.

  3. Recently some bug reports are handled wrongly, with a trend of wasting bug reporter time on irrelevant clarifications or closing the bug too early when there is a problem to reproduce it.

    If you report some MySQL problem to Oracle, be ready to see your report closed soon with a suggestion to contact Support. Check this recent example, Bug #90375 - "Significantly improve performance of simple functions". OK, this is a known limitation, but user suggested several workarounds that are valid feature requests for optimizer, that can be smart enough to inline the stored function if it just returns some simple expression. Why not to verify this as a valid feature request?
    Another great example is Bug #80919 - "MySQL Crashes when Droping Indexes - Long semaphore wait". It was closed (after wasting more than a year on waiting) as a "Duplicate" with such a nice comment:
    "[16 Jan 15:53] Sinisa Milivojevic
    Hi!

    This is a duplicate bug, because it is very similar to an internal-only bug, that is not present in the public bugs database.


    I will provide the internal bug number in the hidden comment, but will update this page with public comment, once when that bug is fixed.
    "
    No reference to the internal bug number, just nothing. If you are wondering what is the real problem, check MDEV-14637.
    Yet another case of "Not a bug", where the decision is questionable and further statements from the bug reporter are ignored is Bug #89065 - "sync_binlog=1 on a busy server and slow binary log filesystem stalls slaves".
    The bug may be "Verified" finally, but after some time wasted on discussions and clarifications when the problem is obvious and bug report contains everything needed to understand this. Nice example is Bug #91386 - "Index for group-by is not used with primary key for SELECT COUNT(DISTINCT a)". Yet another example is Bug #91010 - "WolfSSL build broken due to cmake typo". Knowing parties involved in that discussions in person, I wish they spent their time on something more useful than arguing on the obvious problems.

    This practice discourage users from reporting bugs to Oracle. Not that bug handling mistakes never happened before Oracle (I did many myself), but recently I see more and more wrongly handled bugs. This trend is scary!

  4. Oracle mostly fixes bugs reported internally.

    Just check any recent Release Notes and make your own conclusion. One may say it means that internal QA and developers find bugs even before Community notices them, but the presence of all kinds of test failures, regression bugs etc in the same recent versions still tells me that Community QA is essential for MySQL quality. But it does not set the agenda for the bug fixing process, for many years.
    One may also say that bug fixing agenda is defined by Oracle customers mostly. I let Oracle customers to comment here if they are happy with what they get. Some of the were not so happy with the speed of resolution even for their real security related bugs.
I can continue with more items, but let's stop for now.

If we want MySQL public bugs database to never reach the state of the West Pier (declared to be beyond repair in 2004), we should force Oracle to do something to fix the problems above. Otherwise we'll see further decline of bugs database and Community activity switched elsewhere (to Percona's and MariaDB's public bugs databases, or somewhere else). It would be sad to not have any central public location for all problem reports about core MySQL server functionality...

Sunday, February 11, 2018

Fun with Bugs #61 - On MySQL Bug Reports I am Subscribed to, Part III

Since my previous post on this topic I've subscribed to 19 more MySQL bugs, so it's time for yet another review of these reports. I am trying to pick up important, funny or hard to process reports every day, and here is the list of the most interesting ones starting from the latest:
  • Bug #89607 - "MySQL crash in debug, PFS thread not handling singals." We have a patch contributed by Robert Golebiowski.
  • Bug #89583 - "no rpm build instructions from source git tree". As Simon Mudd put it:
    "... The sources are supposed to be in the git tree so just tell me how I can use that to produce the files needed to make the rpm SOURCES.

    Without that information only Oracle can build src rpms and no-one else can repeat the process they use and any changes required by people building rpms can't be done in a separate branch of a cloned copy of github.com/mysql/mysql-server. That would be better as from there I can much more easily provide PRs which should improve the MySQL sources."
    I hope one day Oracle will start to share all the code for open source MySQL, properly, including all test cases and detailed, repeatable build instructions. Having proper, up to date and working build instructions that let one end up with binaries built the same way as vendor builds them is not an easy task in general, and KB articles like this is just a step towards the goal...
  • Bug #89559 - "P_S recording wrong digest/digest_text for select statements using views". Everybody knows how much I like Performance Schema, but I like to notice bugs in it even more This is a funny bug that is still "Open" (even though it was noticed by my dear friend Sinisa Milivojevic already) for some reason.
  • Bug #89534 - "Crash during innodb recovery when working with encryption". Yet another bug report (this time "Verified", in 5.7.21) from Robert Golebiowski. Purge thread running during startup may cause problems.
  • Bug #89519 - "Documentation for SSL/TLS and replication is incomplete for 8.0". In this report  Daniël van Eeden noted that --ssl-verify-server-cert option is not documented.
  • Bug #89444 - "8.0.4rc --initialize-insecure is prohibitively slow". I've noted the same immediately while working on tests for my planned presentation at FOSDEM, but I do not care much about MySQL 8. Fortunately,  Roel Van de Paar cares, so we have a "Verified" bug report about this new behavior.
  • Bug #89441 - "Foreign keys constraints ignored after RENAME TABLE". I am sure we'll see many regressions related to the new data dictionary in MySQL 8. This bug report by Carlos Salguero highlights one of them.
  • Bug #89430 - "Release notes are missing important CVE fixes". Oracle and transparency in anything related to security issues seems to be historically incompatible things. Still, Roel Van de Paar has a hope they may improve their release notes... In the meantime check this great document with all the relevant details.
  • Bug #89375 - "Parallel replication always fails with specific workload from sysbench". It's always great to see public bug report from Oracle employee. Thank you, Frederic Descamps, for sharing your findings with community!
  • Bug #89372 - "Using --no-dd-upgrade seems to have no effect". This bug report by Geert Vanderkelen is still "Open" for some reason.
  • Bug #89367 - "Storing result in a variable(UDV) causes query on a view to use derived tables". Nice optimizer bug reported by Jaime Sicam.
  • Bug #89331 - "Changing lock/release order in group commit causes a deadlock". I think there are at most 10 people in the world who understands all the details of this bug report by Aliaksei Sandryhaila (and I am not one of them). The goal they are trying to achieve is interesting:
    "We are implementing START TRANSACTION WITH CONSTISTENT INNODB SNAPSHOT functionality in 8.0, and want to change it to first taking the next stage's lock, then releasing the previous stage's lock. The implementation of this feature for 5.6 is here: https://github.com/facebook/mysql-5.6/commit/c2b8ced ..."
    I see active discussion with Oracle developer in that report, so I am sure they'll figure out how to proceed.
  • Bug #89272 - "Binlog and Engine become inconsistent when binlog cache file gets out of space". Great finding by Yoshinori Matsunobu and Jeff Jiang. If you ever had "Errcode: 28 - No space left on device" on master while doing some large transactions, make sure to check if slave is in sync. It may be NOT.
  • Bug #89267 - "Unable to access 8.0.4 server after starting on top of 5.7.20 database". I am sure we'll get a lot more bug reports when users actively start to upgrade. For now check comments for possible workaround.
  • Bug #89247 - "Deadlock with MTS when slave_preserve_commit_order = ON." There are many problems with parallel replication/multi-threaded slaves, and Jean-François Gagné probably knows more about them than anyone else. Check also his another bug report about this feature, Bug #89141 - "Error in Group Replication caused by bad Write Set tracking."
  • Bug #89126 - "create table panic on innobase_parse_hint_from_comment". Nice bug report and patch from Yan Huang.
  • Bug #89101 - "MySQL server hang when gtid_mode=on and innodb_thread_concurrency>0". This is a second reason for me to think twice from now on before suggesting to limit innodb_thread_concurrency ever again. Great bug report by Seunguck Lee and MTR test case (and argumentation) by Przemyslaw Malkowski! This is a masterpiece of bug reporting art, Sinisa had to give up...
  • Bug #89098 - "Adding an auto_increment column to existing table creates gaps". Really weird behavior noticed by Riccardo Pizzi. I hope it is truly repeatable literally (but I had not checked myself).
That's all new bug reports in my list of subscriptions. Next time I hope to review few older ones that were not presented in this series yet.

Sunday, September 28, 2014

Fun with Bugs #34 - Who has fun to verify your bugs (based on 5.6.21)

This time I decided not just to review all community bugs fixed in recent MySQL 5,6 GA release, 5.6.21, but also to mention who reported them (Morgan does this recently) and, most important for this post - who verified them.

As I've explained long time ago, verification is an important part of a bugs life cycle in MySQL. We need some MySQL engineer to check the bug and make sure there is a repeatable test case for it or it is at least clear what is the real problem behind the report. Bug must be "Verified" (confirmed internally) before developers start to work on the fix.

Usually this engineer is somebody from the bugs verification team where I worked for 7 years. But we later agreed that if a bug is reported by MySQL Support Engineer or Developer, it's accepted to set it to "Verified" immediately, without waiting on somebody else from bugs verification team to pay attention and double check.

So, while reporting and fixing are, surely, important stages, verification is also important as it helps developers to concentrate on real bugs only and isolate them from most false positives and user complains. I've noted (at was surprised) that it seems more than 45% of all bugs recently fixed were actually verified by one my former colleague in Oracle since 2011 (who used to work as a regular support engineer during EMEA hours mostly), Umesh Shastry (that you may know as "Umesh Umesh" in the bugs database for the reason only him and Oracle probably knows about).

This is what we have in MySQL 5.6.21 with regards to bugs from http://bugs.mysql.com:
  • Bug #70641 - 5.6 partitions use much more memory than 5.1. Reported by famous Yoshinori Matsunobu and verified by Umesh (no details shared in public). As a side note: I think it was me who introduced a habit not just to write "verified" in a comment, but also to share all the details and steps, just copy/pasting outputs from mysql command line client or terminal. (I've got this habit while following famous Oracle guru Tom Kyte.) So, I am always sad when details about the exact steps performed and versions used are no shared...
  • Bug #73650 is still private and we can see no details. We can only rely on public description in the release notes: "An ALTER TABLE ... ADD FOREIGN KEY operation could cause a serious error." I know what the bug is about as I've reported a duplicate of it based on Percona Server bug report (that is public as far as I remember). many community users know all the details about the crash (let's use proper words, it was a crash). Bug is fixed. But we see nothing, neither about test case, nor about the fix (even though it's visible when one checks what changed since last version in bzr/source code repository). If you ask me, this Oracle policy is applied in a wrong way to open source software, to put it into polite words. One day I'll write long text about this approach and historical reasons for it...
  • Bug #72453 - InnoDB Memcached Plugin "gets" command. Reported by qixiu xiao and verified by Umesh.
  • Bug #72586 - memcached daemon crashes and restarts on second ADD operation. Reported by David Schwartz and verified by Umesh.
  • Bug #72594 - InnoDB in-place ALTER failures block future ALTERs. Reported by famous code contributor, InnoDB guru and great conference speaker, Jeremy Cole. Recently we see his name in all release notes. This bug was verified by Miguel Solorzano, who was my team manager and trainer since day one in MySQL. Nice to see that he still not only manages a successful (and key!) team in Oracle MySQL Support, but also finds time to do real work.  
  • Bug #72137 - inaccurate handling of srv_activity_count. Reported by famous InnoDB guru Inaam Rana and verified by Sveta Smirnova. There is no much need to explain who is Sveta and what she does - she speaks, writes and works for herself, and her work is recognized and awarded in Oracle. Check her sessions at MySQL Connect 2013, if you wonder, or read her book. I am just happy to find out that she still works on bugs processing.
  • Bug #67718 - InnoDB drastically under-fills pages in certain conditions. Reported by Jeremy Cole and verified by Sinisa Milivojevic. I am happy to call Sinisa my teacher, in diplomatic skills if not in bugs processing or reading the code. If any of my former colleagues from MySQL asked what I am doing in Percona, I'd reply that I play a role of Sinisa here, "ideal" Sinisa as I wanted him to be, not the "real" one. I am not good enough in playing the role of real one, as I lack diplomatic skills, to begin with.
  • Bug #71695 - Concatenation of mysqlbinlog output does not work with GTID. Reported by Sadao Hiratsuka and verified by Umesh.
  • Bug #70327 - Assertion error when setting future binlog file/pos with semisync. Reported by Yoshinori Matsunobu and verified by yet another my former colleague since MySQL AB times, Mikiya Okuno. Nice to see many MySQL engineers contributing to bugs processing, not just members of bugs verification team.
  • Bug #69873 - Replication stop with "Error in Xid_log_event: Commit could not be completed". Reported by Alexander Du and verified by Umesh.
  • Bug #71047 - shutdown hang if semisync is enabled and slave abort. Reported by a great community contributor to MySQL, zhai weixiang. Verified by Umesh.
  • Bug #72788 - HASH_SCAN seems broken: Can't find record in 't1' Error_code: 1032. Reported by Shane Bester, who probably works more on community bugs reports than all other engineers outside bugs verification team together. As I mentioned more than once, Shane is a QA department for MySQL by himself, and it had always been the case, no matter what company he worked at. Bug was verified, surprise surprise, by Umesh.
  • Bug #72313 - stop sql_thread, start sql_thread causes a trx to log with a different GTID. Reported by Santosh Praneeth Banda. Not sure who verified it, but probably the reporter himself.
  • Bug #72578 - Duplicate slave server_uuid in replication forum gives no specific error message. Reported by Hartmut Holzgraefe, former my colleague at MySQL AB and Sun. He helped me a lot during my first days in MySQL and he still reports a lot of bugs. Bug was verified by Umesh.
  • Bug #72901 - Keep CLIENT_REMEMBER_OPTIONS for compressed connections. Bug was reported by Ramil Kalimullin, whom I miss a lot since MySQL developers conference in Riga, 2008 (as many others, it was last team meeting or conference for me till I quite from Oracle). He verified the bug himself it seems.
  • Bug #73324 - Client flags are getting discarded due to incorrect assignment operator. Reported by Sujatha Sivakumar. She then verified and fixed the bug herself it seems. I am always happy to see Oracle employees and customers using public bugs database. You know why, I hope...
  • Bug #72610 - Out of memory replicating using RBR and slave having different data types. Reported by Jean-Francois Gagn and verified by Umesh.
  • Bug #70429 - list handling incorrect in mysql_prune_stmt_list(). Reported and verified by yet another former colleague from MySQL Support, Andrew Dalgleish.
  • Bug #47641 - LIKE doesn't match when cp932_bin/sjis_bin collations are used. Reported and verified by Mikiya Okuno.
    Bug #73507 - On EL7, installation of MySQL from RPM packages could fail if postfix had previously been installed using yum. It is still private, even though I fail to see what "security implications" it may have, based on that description, comparing to any other public bug report about upgrade being screwed up. There are many, public ones...
  • Bug #72066 - mysql_upgrade duplicate key error for mysql.user for 5.5.35+, 5.6.15+, 5.7.3+. Reported and verified by Jesper wisborg Krogh, again, former colleague who still works in Oracle.
  • Bug #67088 - When the general_log is a socket and the reader goes away, mysql quits working. Reported by Rolf Martin-Hoster and verified by Sveta Smirnova.
  • Bug #71433 - recreate+analyze OPTIMIZE TABLE T and online ALTER TABLE T may deadlock. Reported by my colleagues from Percona and great code contributor, Laurynas Biveinis. Recently he reached 100 as a number of public MySQL bugs reported. Time to review and update my list of famous bug reporters, it seems... Bug was verified by Umesh.
  • Bug #72547 - Query cache not invalidated on cascade delete if db name has special symbols. Reported by Elena Stepanova, whom I met first in Riga, 2008. He contribution to QA in both MariaDB and upstream MySQL is outstanding - we can see her name in every issue of community release notes by Morgan. Bug was verified by Umesh.
So, out of 24 community bug reports 11 were surely verified by Umesh Shastry. This is almost 46% of them all. Now you know who seems to play the most important role in community bugs processing. I hope I am not the only one who noted that I am now successfully replaced in my former main job role, by a single person (whom I trained to some extent while we still worked together in Oracle).

One can probably make more conclusions based on above. For example, it looks like that most "community" bugs fixed are actually from former or current MySQL support engineers and developers, not just "users"... But let me stop on highlighting names at the moment. I am sure that some of these names were rarely mentioned in public before.

Tuesday, June 4, 2013

MySQL 5.6 Experiences - .mylogin.cnf and mysql_config_editor

Having  basic ideas of how I am going to describe new features explained, I can proceed with some real (and I hope useful) content. As I read this page about new features from top to bottom, let's start with security improvements...

.mylogin.cnf and mysql_config_editor

 

Details:

  • you can store authentication credentials encrypted in an option file named .mylogin.cnf (in user's home directory or in %APPDATA%\MySQL on Windows)
  • password is no longer stored in plain text (like in .my.cnf) and still is not exposed in the command lines...
  • you have to use mysql_config_editor utility to create the .mylogin.cnf file
  • but if someone can read .mylogin.cnf, they have your MySQL password
  • ~/.mylogin.cnf is not a more secure place to store your password than ~/.my.cnf

 

Links:

(!) http://mysqlblog.fivefarmers.com/2012/08/16/understanding-mysql_config_editors-security-aspects/ - Todd explains that "ease of use" was the goal of this feature, not actually security
(!?) http://serge.frezefond.com/2013/02/mysql-5-6-credentials-securely-stored/ - some explanations on what this new feature is about and, most importantly, code of the mysql_showconfigpwd.cc program from Serge to expose the credentials stored
(!?) http://mysqldump.azundris.com/archives/104-.mylogin.cnf-password-recovery.html - Kristian explains file format and provides PHP code to expose the credentials stored 
(?#) http://www.skysql.com/blogs/kolbe/mysql-56-security-through-complacency - Kolbe explains (among other rambling on MySQL 5.6 security features) how to "hack"  the mask_password_and_print() function from  client/mysql_config_editor.cc  file in the source code to build mysql_config_editor  version that will show stored password in plain text. Bug #68602 mentioned in this post is not related to this file and tool (but related to security, still "Verified" as of today and will be discussed later in this series).

 

Bugs:

New MySQL features always come with related bugs. This simple enough tool and new file caused several related reports:

Bug #68680 - manual had some wrong details on where the .mylogin.cnf file is located on Windows, now fixed
Bug #68034 - while it is declared a "Duplicate", in the bug report you can read how this feature can be used to workaround Bug #66546 that nobody is going to fix (see Todd’s comment)
Bug #68277 - build problem affecting mysql_config_editor, fixed in 5.6.11
Bug #66546 - some people are not happy that there is no way to hardcode password in the script,
without getting warning. But who cares? Only Todd, who explained how to use one new security improvement to workaround problem created by another security improvement...

If you have any new links or bug reports related to the topic of this post, please, send them in comments.

 

Summary

While some more secure approach to store and provide authentication credentials for scripts was requested many times and by many users, I am not sure this new feature adds anything to security. IMHO it just may simplify work for a lazy DBA who does not remember proper permissions for .my.cnf file...

At the same time, this new feature probably caused additional work for developers and QA of MySQL Connectors and Workbench (as I have reasons to think it was probably decided that this time, for 5.6 GA, connectors and tools should really support all new GA features at the moment of GA release). I had found only one related bug in the public bugs database to prove this point (Bug #68356 - problems in Workbench tests that cover both MySQL 5.5 and 5.6), but who knows how many of them were reported internally and then fixed...

I share concerns that this feature may cause wrong feeling of improved security, that's why I talked about it during my presentation and had written this post. I agree with the following comment to Kolbe's post from Sergei Golubchik though:
"There's not much one can do to improve password handling. The mysql client needs to store some bit of information somewhere that allows to identify a user to the server. So if a bad guy would have access to this bit of information - no matter how obfuscated or encrypted it is - he will be able to impersonate the user just as well by repeating whatever mysql client does."