On Upgrades to Percona Server 5.7 GA: Building from Git Source on Fedora

While writing my first post in this series I was bulding Percona Server 5.7 from Git sources on my wife's Fedora Core 23 box based on this manual page, for my collection of test instances. As I build MySQL and related software from sources on different platforms almost every day for 11 years already, the process was completed successfully without much troubles, but there are still some details I'd like to highlight.

Manual says one has to do the following:

$ git clone https://github.com/percona/percona-server.git
$ cd percona-server
$ git checkout 5.7
$ git submodule init
$ git submodule update

I had source code clonned already for regular 5.6.x builds, so I did this instead:

[openxs@fc23 percona-server]$ git pull
Already up-to-date.
[openxs@fc23 percona-server]$ git checkout 5.7
Checking out files: 100% (20321/20321), done.
M       plugin/tokudb-backup-plugin/Percona-TokuBackup
M       storage/tokudb/PerconaFT
Branch 5.7 set up to track remote branch 5.7 from origin.
Switched to a new branch '5.7'
[openxs@fc23 percona-server]$ git submodule init
[openxs@fc23 percona-server]$ git submodule update

On this FC 23 box I've already built MySQL 5.7, MariaDB 10.1, Facebook's MySQL 5.6 and Percona Server 5.6 from Git sources more than once, and I've updated all packages today and even built Facebook's 5.6 code (for testing MyRocks further) after that successfully, so I expect everything should work really well.

I put my test instances into dbs subdirectory usually:

[openxs@fc23 percona-server]$ ls ~/dbs
5.7  fb56  maria10.1  p5.6

So, I've modified cmake command line a bit comparing to the one recommended in the manual:

cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DWITH_EMBEDDED_SERVER=OFF

as you can see below:

[openxs@fc23 percona-server]$ cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DWITH_EMBEDDED_SERVER=OFF -DCMAKE_INSTALL_PREFIX=/home/openxs/dbs/p5.7
-- Running cmake version 3.4.1
-- Configuring with MAX_INDEXES = 64
-- Looking for SHM_HUGETLB
-- Looking for SHM_HUGETLB - found
-- SIZEOF_VOIDP 8
-- MySQL 5.7.10-3
-- Packaging as: percona-server-5.7.10-3-Linux-x86_64
-- Looked for boost/version.hpp in  and
-- BOOST_INCLUDE_DIR BOOST_INCLUDE_DIR-NOTFOUND
-- LOCAL_BOOST_DIR
-- LOCAL_BOOST_ZIP
-- Could not find (the correct version of) boost.
-- MySQL currently requires boost_1_59_0

CMake Error at cmake/boost.cmake:76 (MESSAGE):
  You can download it with -DDOWNLOAD_BOOST=1 -DWITH_BOOST=<directory>

  This CMake script will look for boost in <directory>.  If it is not there,
  it will download and unpack it (in that directory) for you.

  If you are inside a firewall, you may need to use an http proxy:

  export http_proxy=http://example.com:80

Call Stack (most recent call first):
  cmake/boost.cmake:228 (COULD_NOT_FIND_BOOST)
  CMakeLists.txt:439 (INCLUDE)


-- Configuring incomplete, errors occurred!
See also "/home/openxs/git/percona-server/CMakeFiles/CMakeOutput.log".
See also "/home/openxs/git/percona-server/CMakeFiles/CMakeError.log".

As you can see, manual obviously misses some details, like adding options related to Boost library. We see nice hints on what to do, and I've already downloaded Boost once into ~/boost while building MySQL 5.7. If Percona Server 5.7 was based on 5.7.11, we could use -DWITH_BOOST=system, see the release notes:

"A value of system is now permitted for the WITH_BOOST CMake option. If this option is not set or is set to system, it is assumed that the correct version of Boost is installed on the compilation host in the standard location. In this case, the installed version of Boost is used rather than any version included with a MySQL source distribution. (Bug #22224313)"

But it's based on 5.7.10, so I've modified my command like as follows:

[openxs@fc23 percona-server]$ cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DWITH_EMBEDDED_SERVER=OFF -DDOWNLOAD_BOOST=1 -DWITH_BOOST=/home/openxs/boost -DCMAKE_INSTALL_PREFIX=/home/openxs/dbs/p5.7
...
-- Check if the system is big endian - little endian
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
  Could NOT find ZLIB (missing: ZLIB_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake/Modules/FindZLIB.cmake:124 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  libbinlogstandalone/CMakeLists.txt:30 (FIND_PACKAGE)


-- Configuring incomplete, errors occurred!
See also "/home/openxs/git/percona-server/CMakeFiles/CMakeOutput.log".
See also "/home/openxs/git/percona-server/CMakeFiles/CMakeError.log".

to end up with another error. I was surprised a bit, as I do have zlib-devel installed:

[openxs@fc23 percona-server]$ rpm -qa | grep zlib
zlib-1.2.8-9.fc23.x86_64
zlib-devel-1.2.8-9.fc23.x86_64

To not waste time on zlib now I've used the "trick" from Facebook's wiki this time:

[openxs@fc23 percona-server]$ cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DWITH_EMBEDDED_SERVER=OFF -DDOWNLOAD_BOOST=1 -DWITH_BOOST=/home/openxs/boost -DWITH_ZLIB=bundled -DCMAKE_INSTALL_PREFIX=/home/openxs/dbs/p5.7
...
-- MySQL 5.7.10-3
-- Packaging as: percona-server-5.7.10-3-Linux-x86_64
-- Local boost dir /home/openxs/boost/boost_1_59_0
-- Local boost zip /home/openxs/boost/boost_1_59_0.tar.gz
...
-- Library perconaserverclient depends on OSLIBS -lpthread;m;rt;/usr/lib64/libssl.so;/usr/lib64/libcrypto.so;dl
-- Performing Test HAVE_NO_UNUSED_TYPEDEFS
-- Performing Test HAVE_NO_UNUSED_TYPEDEFS - Success
-- Performing Test HAVE_NO_LOGICAL_OP
-- Performing Test HAVE_NO_LOGICAL_OP - Success
-- INSTALL perconaserverclient.pc lib/pkgconfig
-- CMAKE_BUILD_TYPE: RelWithDebInfo
-- COMPILE_DEFINITIONS: _GNU_SOURCE;_FILE_OFFSET_BITS=64;HAVE_CONFIG_H
-- CMAKE_C_FLAGS:  -Wall -Wextra -Wformat-security -Wvla -Wwrite-strings -Wdeclaration-after-statement
-- CMAKE_CXX_FLAGS:  -Wall -Wextra -Wformat-security -Wvla -Woverloaded-virtual -Wno-unused-parameter
-- CMAKE_C_FLAGS_RELWITHDEBINFO: -O3 -D_FORTIFY_SOURCE=2 -g -fabi-version=2 -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF
-- CMAKE_CXX_FLAGS_RELWITHDEBINFO: -O3 -D_FORTIFY_SOURCE=2 -g -fabi-version=2 -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF
-- Configuring done
CMake Warning (dev) at storage/tokudb/PerconaFT/portability/CMakeLists.txt:22 (add_dependencies):
  Policy CMP0046 is not set: Error on non-existent dependency in
  add_dependencies.  Run "cmake --help-policy CMP0046" for policy details.
  Use the cmake_policy command to set the policy and suppress this warning.

  The dependency target "build_jemalloc" of target
  "tokuportability_static_conv" does not exist.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /home/openxs/git/percona-server

to end up with cmake command completed sucessfully. Now, time to make it (my FC23 box I've assembled myself back in 2011 has single QuadCore CPU):

[openxs@fc23 percona-server]$ time make -j 4
...
[100%] Built target mysqld
Scanning dependencies of target udf_example
[100%] Building CXX object sql/CMakeFiles/udf_example.dir/udf_example.cc.o
[100%] Built target pfs_connect_attr-t
[100%] Linking CXX shared module udf_example.so
[100%] Built target udf_example

real    10m40.254s
user    36m18.994s
sys     3m9.644s

[openxs@fc23 percona-server]$ make install && make clean
...
-- Installing: /home/openxs/dbs/p5.7/share/aclocal/mysql.m4
-- Installing: /home/openxs/dbs/p5.7/support-files/mysql.server

It took reasonable time to build and, some warnings aside, it worked well.

Percona manual says nothing on what to do next, but if it's a new test instance we need to initialize the database directory for it. Let's do it new way (mysql_install_db is NOT only in the bin subdirectory now in 5.7 instead of scripts, but is deprecated now):

[openxs@fc23 percona-server]$ cd ~/dbs/p5.7
[openxs@fc23 p5.7]$ bin/mysqld --initialize-insecure
2016-02-25T10:14:19.910929Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
2016-02-25T10:14:19.911033Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
2016-02-25T10:14:19.911197Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-02-25T10:14:19.927376Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
2016-02-25T10:14:19.927448Z 0 [ERROR] Aborting

 At this moment I've suspected that I probably have my.cnf or .my.cnf somewhere pointing to some other datadir that can be non-empty, so I tried to add --no-defaults, as usual:

 [openxs@fc23 p5.7]$ bin/mysqld --no-defaults --initialize-insecure
2016-02-25T10:14:51.318896Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
2016-02-25T10:14:51.319006Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
2016-02-25T10:14:51.319199Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-02-25T10:14:51.321754Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
2016-02-25T10:14:51.321823Z 0 [ERROR] Aborting

[openxs@fc23 p5.7]$ ls -l data
ls: cannot access data: No such file or directory
[openxs@fc23 p5.7]$ ls -l
total 128
drwxrwxr-x.  2 openxs openxs  4096 Feb 25 12:06 bin
-rw-r--r--.  1 openxs openxs 17987 Feb  9 09:44 COPYING
-rw-r--r--.  1 openxs openxs 34520 Feb  9 13:04 COPYING.AGPLv3
-rw-r--r--.  1 openxs openxs 17987 Feb  9 13:04 COPYING.GPLv2
drwxr-xr-x.  2 openxs openxs  4096 Feb 25 12:06 docs
drwxrwxr-x.  3 openxs openxs  4096 Feb 25 12:06 include
-rw-r--r--.  1 openxs openxs   301 Feb 25 11:44 INSTALL-BINARY
drwxrwxr-x.  4 openxs openxs  4096 Feb 25 12:06 lib
drwxrwxr-x.  4 openxs openxs  4096 Feb 25 12:06 man
drwxrwxr-x. 10 openxs openxs  4096 Feb 25 12:07 mysql-test
-rw-r--r--.  1 openxs openxs  2211 Feb  9 13:04 PATENTS
-rw-r--r--.  1 openxs openxs  4442 Feb  9 13:04 README.md
-rw-r--r--.  1 openxs openxs  2478 Feb 25 11:44 README.MySQL
drwxrwxr-x. 28 openxs openxs  4096 Feb 25 12:07 share
drwxrwxr-x.  2 openxs openxs  4096 Feb 25 12:07 support-files

[openxs@fc23 p5.7]$ pwd
/home/openxs/dbs/p5.7

 Usually for the Git source by default the datadir is supposed to be in the basedir (current directory in my case). But surely I can specify it explicitly (and I should, see Bug #73845 and the manual):

[openxs@fc23 p5.7]$ bin/mysqld --no-defaults --datadir=/home/openxs/dbs/p5.7/data --initialize-insecure
2016-02-25T10:19:37.946463Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
2016-02-25T10:19:37.946558Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
2016-02-25T10:19:37.946723Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-02-25T10:19:39.784840Z 0 [Warning] InnoDB: New log files created, LSN=45790
2016-02-25T10:19:40.201625Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2016-02-25T10:19:40.513883Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 47cb6f30-dba9-11e5-9c78-485b3952834f.
2016-02-25T10:19:40.546366Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2016-02-25T10:19:41.073896Z 0 [Warning] CA certificate ca.pem is self signed.
2016-02-25T10:19:41.225334Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
[openxs@fc23 p5.7]$

It seems everything worked well with --datadir specified explicitly, so we can try to use this new instance:

[openxs@fc23 p5.7]$ ps aux | grep mysqld
openxs   12123  0.0  0.0 116996  2316 pts/1    S+   12:21   0:00 grep --color=auto mysqld

[openxs@fc23 p5.7]$ bin/mysqld_safe --no-defaults &
[1] 12125
[openxs@fc23 p5.7]$ 160225 12:25:06 mysqld_safe Logging to '/home/openxs/dbs/p5.7/data/fc23.err'.
160225 12:25:06 mysqld_safe Starting mysqld daemon with databases from /home/openxs/dbs/p5.7/data

[openxs@fc23 p5.7]$ bin/mysql -uroot test
ERROR 1049 (42000): Unknown database 'test'
[openxs@fc23 p5.7]$ bin/mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.10-3 MySQL Community Server (GPL)

Copyright (c) 2009-2015 Percona LLC and/or its affiliates
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show variables like '%version%';
+-------------------------+------------------------------+
| Variable_name           | Value                        |
+-------------------------+------------------------------+
| innodb_version          | 5.7.10-3                     |
| protocol_version        | 10                           |
| slave_type_conversions  |                              |
| tls_version             | TLSv1,TLSv1.1,TLSv1.2        |
| version                 | 5.7.10-3                     |
| version_comment         | MySQL Community Server (GPL) |
| version_compile_machine | x86_64                       |
| version_compile_os      | Linux                        |
+-------------------------+------------------------------+
8 rows in set (0.00 sec)

That's it. MySQL 5.7.x (and thus Percona Server 5.7) is "secure by default", and in frames of that effort test database is not created.

As a last step, let's check the size of the mysqld binary:

[openxs@fc23 p5.7]$ ls -l bin/mysqld
-rwxr-xr-x. 1 openxs openxs 211932312 Feb 25 12:05 bin/mysqld
[openxs@fc23 p5.7]$ file bin/mysqld
bin/mysqld: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=f0ba086eb3639e927529cfdc4699c0ac3e748319, not stripped

 I'll try to build on Ubuntu 14.04 later tonight. I do not expect any problems other than zlib related and --datadir required for mysql_install_db or mysqld --initialize, but who knows...

On Upgrades to Percona Server 5.7 GA: CentOS 6 Case

As you probably noted, Percona had finally released Percona Server 5.7 GA two days ago. I had not paid much attention to this branch during last month or so, but as it's GA now I decided to check what's there, how it works and install/upgrade different boxes and VMs to use it.

For testing purposes and for this series of blog posts I am going to build it from Git source (at least on Fedora Core 23), upgrade to it on 32-bit Ubuntu 15.04 VM, upgrade to it on 64-bit Ubuntu 14.04 laptop, upgrade to it on 64-bit CentOS 6.7 VM and maybe more. If any of the attempts to install/upgrade ends up with something worth sharing, I am going to write about the experience, problems (if any) and solutions.

Yesterday I've spent some time upgrading on Ubuntu 15.04 VM and some observations are discussed on Facebook already. Then I tried to upgrade on my main CentOS 6.7 VM, where I had a "full stack" of Percona software for Percona Server 5.6.28, including TokuDB. The idea was to make sure everything I need works as expected after the upgrade. In this post I am going to share the observations from that experince.

Smart user should definitely start with reading the manual, in this case - "Percona Server In-Place Upgrading Guide: From 5.6 to 5.7"? as this is what I planned to do, in-place upgrade. Reading and following the directions there can save you notable time, but I am goign to show what happens when one tries naive "trial and failure" approach.

So, without reading anything, I just logged in as root, run yum update and then checked what Percona software I had there:

[root@centos ~]# yum update
...
No Packages marked for Update
[root@centos ~]# rpm -qa | grep -i percona
Percona-Server-shared-56-5.6.28-rel76.1.el6.x86_64
Percona-Server-test-56-5.6.28-rel76.1.el6.x86_64
Percona-Server-tokudb-56-5.6.28-rel76.1.el6.x86_64
Percona-Server-56-debuginfo-5.6.28-rel76.1.el6.x86_64
percona-release-0.1-3.noarch
Percona-Server-shared-51-5.1.73-rel14.12.625.rhel6.x86_64
Percona-Server-server-56-5.6.28-rel76.1.el6.x86_64
percona-nagios-plugins-1.1.6-1.noarch
percona-toolkit-2.2.16-1.noarch
percona-xtrabackup-2.3.3-1.el6.x86_64
Percona-Server-client-56-5.6.28-rel76.1.el6.x86_64
Percona-Server-devel-56-5.6.28-rel76.1.el6.x86_64

Great, so automatic upgrade to 5.7 GA doesd NOT happen. First thing to take into account for naive user like me. Manual mentioned above wisely suggests, as a next step, to stop MySQL server and then remove all Percona Server software without dependecies:

$ rpm -qa | grep Percona-Server | xargs rpm -e --nodeps

But I do not read the manuals (as many users) and previously upgrades worked for me without removing everything, so I just proceeded in a straighforward manner:

[root@centos ~]# yum install Percona-Server-server-57
...
--> Finished Dependency Resolution
Error: Percona-Server-client-57 conflicts with Percona-Server-client-56-5.6.28-rel76.1.el6.x86_64
Error: Percona-Server-server-57 conflicts with Percona-Server-server-56-5.6.28-rel76.1.el6.x86_64
Error: Percona-Server-shared-51 conflicts with Percona-Server-shared-57-5.7.10-3.1.el6.x86_64
 You could try using --skip-broken to work around the problem
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
Percona-Server-shared-51-5.1.73-rel14.12.625.rhel6.x86_64 has installed conflicts mysql-libs: Percona-Server-shared-51-5.1.73-rel14.12.625.rhel6.x86_64

So, removing seems unavoidable and I proceeded step by step (surely I had not cared about dependencies, who do, yum will tell us anyway or fix it for us, right?):

[root@centos ~]# yum remove Percona-Server-server-56
...
Removed:
  Percona-Server-server-56.x86_64 0:5.6.28-rel76.1.el6

Dependency Removed:
  Percona-Server-tokudb-56.x86_64 0:5.6.28-rel76.1.el6

Complete!

Let's remember that we need to install TokuDB for 5.7 separately, and proceed:

[root@centos ~]# yum remove Percona-Server-client-56 Percona-Server-shared-51
...
Transaction Summary
================================================================================
Remove        8 Package(s)

Installed size: 369 M
...
Removed:
  Percona-Server-client-56.x86_64 0:5.6.28-rel76.1.el6
  Percona-Server-shared-51.x86_64 0:5.1.73-rel14.12.625.rhel6

Dependency Removed:
  Percona-Server-test-56.x86_64 0:5.6.28-rel76.1.el6
  percona-toolkit.noarch 0:2.2.16-1
  percona-xtrabackup.x86_64 0:2.3.3-1.el6
  perl-DBD-MySQL.x86_64 0:4.013-3.el6
  php-mysql.x86_64 0:5.3.3-46.el6_7.1
  sysbench.x86_64 0:0.5-6.el6

Complete!

As we had not cared much about dependencies, we've got a lot of software removed that depended on old shared libraries. Let's save the list to install it back later. Now, we can try to install Percona Server 5.7 again:

[root@centos ~]# yum install Percona-Server-server-57
...
Dependencies Resolved

================================================================================
 Package                  Arch   Version           Repository              Size
================================================================================
Installing:
 Percona-Server-server-57 x86_64 5.7.10-3.1.el6    percona-release-x86_64  40 M
Installing for dependencies:
 Percona-Server-client-57 x86_64 5.7.10-3.1.el6    percona-release-x86_64 8.7 M
 Percona-Server-shared-57 x86_64 5.7.10-3.1.el6    percona-release-x86_64 902 k

Transaction Summary
================================================================================
Install       3 Package(s)
...

Transaction Check Error:
  file /usr/bin/mysql_config from install of Percona-Server-client-57-5.7.10-3.1.el6.x86_64 conflicts with file from package Percona-Server-devel-56-5.6.28-rel76.1.el6.x86_64

Error Summary
-------------

Great, so we also have a conflict with Percona-Server-devel-56 package that yum had not informed us about initially (Percona manual was probably right, remove them all, Percona-Srerver-* pacvkages first!). Let's remember to install it back for 5.7, remove it and install Percona Server 5.7 finally:

[root@centos ~]# yum remove Percona-Server-devel-56
...
Removed:
  Percona-Server-devel-56.x86_64 0:5.6.28-rel76.1.el6

Complete!
[root@centos ~]# yum install Percona-Server-server-57
...
Dependencies Resolved

================================================================================
 Package                  Arch   Version           Repository              Size
================================================================================
Installing:
 Percona-Server-server-57 x86_64 5.7.10-3.1.el6    percona-release-x86_64  40 M
Installing for dependencies:
 Percona-Server-client-57 x86_64 5.7.10-3.1.el6    percona-release-x86_64 8.7 M
 Percona-Server-shared-57 x86_64 5.7.10-3.1.el6    percona-release-x86_64 902 k

Transaction Summary
================================================================================
Install       3 Package(s)
...
Installed:
  Percona-Server-server-57.x86_64 0:5.7.10-3.1.el6

Dependency Installed:
  Percona-Server-client-57.x86_64 0:5.7.10-3.1.el6
  Percona-Server-shared-57.x86_64 0:5.7.10-3.1.el6

Complete!

Now, let's try to use it:

[root@centos ~]# mysql -uroot test
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
[root@centos ~]# ps aux | grep mysql
root      2665  0.0  0.0 103304   884 pts/0    S+   21:09   0:00 grep mysql

So, it is NOT started, as we had NOT upgraded it, we removed Percona Server 5.6.x first! Let's start the server now (at this step on recent Ubuntu etc you may find yourself in troubles):

[root@centos ~]# service mysql start
Starting mysqld:                                           [  OK  ]
[root@centos ~]# mysql -uroot  test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.10-3-log Percona Server (GPL), Release 3, Revision 63dafaf

Copyright (c) 2009-2015 Percona LLC and/or its affiliates
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

So, I've got it up and running. Now let me install some software back to replace those older packages I've removed:

[root@centos ~]# yum install Percona-Server-devel-57 Percona-Server-test-57 Percona-Server-tokudb-57
...

Installed:
  Percona-Server-devel-57.x86_64 0:5.7.10-3.1.el6
  Percona-Server-test-57.x86_64 0:5.7.10-3.1.el6
  Percona-Server-tokudb-57.x86_64 0:5.7.10-3.1.el6

Complete!

So, we have missing Percona Server packages for 5.7, now let me try to install dependecies back:

[root@centos ~]# yum install percona-toolkit percona-xtrabackup perl-DBD-MySQL php-mysql sysbench
...
Installed:
  percona-toolkit.noarch 0:2.2.16-1    percona-xtrabackup.x86_64 0:2.3.3-1.el6
  perl-DBD-MySQL.x86_64 0:4.013-3.el6  php-mysql.x86_64 0:5.3.3-46.el6_7.1
  sysbench.x86_64 0:0.5-6.el6

Dependency Installed:
  Percona-Server-shared-51.x86_64 0:5.1.73-rel14.12.625.rhel6

Complete!

Works well! Now, let me check if "full stack" works as expected. I had xtrabackup and toolkit installed, so do we have xtrabackup capable to work with 5.7 GA as result? Let's check:

[root@centos ~]# xtrabackup --version
xtrabackup version 2.3.3 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 525ca7d)
[root@centos ~]# xtrabackup --backup
160224 21:28:08  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock' as 'mysql'  (using password: NO).
160224 21:28:08  version_check Connected to MySQL server
160224 21:28:08  version_check Executing a version check against the server...
DBD::mysql::db selectall_hashref failed: Table 'performance_schema.session_variables' doesn't exist at - line 1221.
160224 21:28:10  version_check Done.
160224 21:28:10 Connecting to MySQL server host: localhost, user: mysql, password: not set, port: 0, socket: /var/lib/mysql/mysql.sock
Error: failed to execute query SHOW VARIABLES: Table 'performance_schema.session_variables' doesn't exist

Well, I was not sure (let's assume) if xtrabackup 2.3.3 is supposed to work with 5.7 (it was released recently, no upgrade happened, why not?), so I tried to use it. But it does not work as we do not have that performance_schema.session_variables table. Surely it's not there, it does not exist in MySQL 5.6. We missed important step that manual clearly mentiones, running mysql_upgrade and restart of service:

[root@centos ~]# mysql_upgrade
Checking if update is needed.
Checking server version.
Running queries to upgrade MySQL server.
Checking system database.
mysql.columns_priv                                 OK
...
Upgrade process completed successfully.
Checking if update is needed.
[root@centos ~]# service mysql restart
Stopping mysqld:                                           [  OK  ]
Starting mysqld:                                           [  OK  ]

Let's try again now:

[root@centos ~]# xtrabackup --backup
160224 21:31:00  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock' as 'mysql'  (using password: NO).
160224 21:31:00  version_check Connected to MySQL server
160224 21:31:00  version_check Executing a version check against the server...
160224 21:31:00  version_check Done.
160224 21:31:00 Connecting to MySQL server host: localhost, user: mysql, password: not set, port: 0, socket: /var/lib/mysql/mysql.sock
Error: Unsupported server version: '5.7.10-3-log'. Please report a bug at https://bugs.launchpad.net/percona-xtrabackup

No luck. Indeed, as anyone can find out after some reading, only Percona Xtrabackup 2.4 (that was also released as GA on February 23) is supposed to work with Percona Server or MySQL 5.7. We have to install it explicitly as "it was decided" that Percona does not want it to be used by default, yet:

[root@centos ~]# yum install percona-xtrabackup-24
...
--> Processing Conflict: percona-xtrabackup-24-2.4.1-1.el6.x86_64 conflicts percona-xtrabackup
--> Finished Dependency Resolution
Error: percona-xtrabackup-24 conflicts with percona-xtrabackup-2.3.3-1.el6.x86_64
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
[root@centos ~]# yum remove percona-xtrabackup
...
Removed:
  percona-xtrabackup.x86_64 0:2.3.3-1.el6

Complete!
[root@centos ~]# yum install percona-xtrabackup-24
...
Running Transaction
  Installing : percona-xtrabackup-24-2.4.1-1.el6.x86_64                     1/1
  Verifying  : percona-xtrabackup-24-2.4.1-1.el6.x86_64                     1/1

Installed:
  percona-xtrabackup-24.x86_64 0:2.4.1-1.el6

Complete!

So, one has to explicitly remove percona-xtrabackup package and then install percona-xtrabackup-24. Now we can finally backup our new Percona Server 5.7 online (after exp;licitly specifying root user for backup, I wonder why, and doing some cleanup from the previous failed attempt):

[root@centos ~]# xtrabackup --backup
160224 21:35:46  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock' as 'mysql'  (using password: NO).
160224 21:35:46  version_check Connected to MySQL server
160224 21:35:46  version_check Executing a version check against the server...
160224 21:35:46  version_check Done.
160224 21:35:46 Connecting to MySQL server host: localhost, user: mysql, password: not set, port: 0, socket: /var/lib/mysql/mysql.sock
Using server version 5.7.10-3-log
xtrabackup version 2.4.1 based on MySQL server 5.7.10 Linux (x86_64) (revision id: a2dc9d4)
xtrabackup: uses posix_fadvise().
...
160224 21:35:53 Executing LOCK TABLES FOR BACKUP...
Error: failed to execute query LOCK TABLES FOR BACKUP: Access denied; you need (at least one of) the RELOAD privilege(s) for this operation
[root@centos ~]# xtrabackup --user=root --backup
...
xtrabackup: Can't create/write to file '/root/xtrabackup_backupfiles/xtrabackup_logfile' (Errcode: 17 - File exists)
xtrabackup: error: failed to open the target stream for 'xtrabackup_logfile'.
[root@centos ~]# rm -rf xtrabackup_backupfiles/
[root@centos ~]# xtrabackup --user=root --backup
...
160224 21:37:40 Backup created in directory '/root/xtrabackup_backupfiles/'
160224 21:37:40 [00] Writing backup-my.cnf
160224 21:37:40 [00]        ...done
160224 21:37:40 [00] Writing xtrabackup_info
160224 21:37:40 [00]        ...done
xtrabackup: Transaction log of lsn (1018105090) to (1018105141) was copied.
160224 21:37:40 completed OK!

Now the last step for this post, let's try if magic tools from Percona Toolkit works with 5.7, at least pt-mysql-summary is something to test for sure:

[root@centos ~]# pt-mysql-summary
# Percona Toolkit MySQL Summary Report #######################
              System time | 2016-02-24 19:38:09 UTC (local TZ: EET +0200)
# Instances ##################################################
  Port  Data Directory             Nice OOM Socket
  ===== ========================== ==== === ======
        /var/lib/mysql             0    0   /var/lib/mysql/mysql.sock
# MySQL Executable ###########################################
       Path to executable | /usr/sbin/mysqld
              Has symbols | No
# Report On Port 3306 ########################################
                     User | root@localhost
                     Time | 2016-02-24 21:38:09 (EET)
                 Hostname | centos
                  Version | 5.7.10-3-log Percona Server (GPL), Release 3, Revision 63dafaf
                 Built On | Linux x86_64
                  Started | 2016-02-24 21:30 (up 0+00:07:22)
                Databases | 6
                  Datadir | /var/lib/mysql/
                Processes | 1 connected, 1 running
              Replication | Is not a slave, has 0 slaves connected
                  Pidfile | /var/run/mysqld/mysqld.pid (exists)
...

So, it works and recongnizes the version properly, but at this moment I've noted I miss debug symbols for 5.7 (and I'll need them as soon as I attach gdb to running server, something I do on a regular basis). So, I have to install them:

[root@centos ~]# yum install Percona-Server-57-debuginfo
...
  file /usr/lib/debug/usr/share/mysql-test/lib/My/SafeProcess/my_safe_process.debug from install of Percona-Server-57-debuginfo-5.7.10-3.1.el6.x86_64 conflicts with file from package Percona-Server-56-debuginfo-5.6.28-rel76.1.el6.x86_64

Error Summary
-------------
[root@centos ~]# yum remove Percona-Server-56-debuginfo
Removed:
  Percona-Server-56-debuginfo.x86_64 0:5.6.28-rel76.1.el6

Complete!
[root@centos ~]# yum install Percona-Server-57-debuginfo
...
Installed:
  Percona-Server-57-debuginfo.x86_64 0:5.7.10-3.1.el6

Complete!
[root@centos ~]# rpm -qa | grep -i percona
Percona-Server-shared-56-5.6.28-rel76.1.el6.x86_64
Percona-Server-shared-57-5.7.10-3.1.el6.x86_64
Percona-Server-test-57-5.7.10-3.1.el6.x86_64
percona-toolkit-2.2.16-1.noarch
percona-xtrabackup-24-2.4.1-1.el6.x86_64
Percona-Server-tokudb-57-5.7.10-3.1.el6.x86_64
percona-release-0.1-3.noarch
percona-nagios-plugins-1.1.6-1.noarch
Percona-Server-server-57-5.7.10-3.1.el6.x86_64
Percona-Server-shared-51-5.1.73-rel14.12.625.rhel6.x86_64
Percona-Server-client-57-5.7.10-3.1.el6.x86_64
Percona-Server-devel-57-5.7.10-3.1.el6.x86_64
Percona-Server-57-debuginfo-5.7.10-3.1.el6.x86_64

So, after dealing with one more conflict I seem to have a "full stack" of Percona tools that are ready to work with Percona Server 5.7 GA.

To summarize my experience of a naive user, one should note the following while upgrading to Percona Server 5.7 on CentOS 6.x:

1. Read the manual about upgrade carefully and don't ignore it. You'll have to remove all Percona-Server-* packages first, deal with dependencies somehow and then install all Percona-Server-57-* packages you need.
2. mysql_upgrade is NOT executed automatically upon installation when there is some older datadir from the previous version. You have to execute it explicitly, check the outputs and then restart mysql service before any further use.
3. If you need to use xtrabackup with Percona Server 5.7, you have to remove percona-xtrabackup package and then install percona-xtrabackup-24 explicitly.

I hope this step by step guide will be useful to some of you. I'll try to upgrade "full stack" of Percona software on 64-bit Ubuntu 14.04 laptop tonight and carefully note all steps. I may end up with the next blog post in this series after that, depending on the results, so stay tuned!

MySQL Support People - Directors, Managers and Coordinators

This post is devoted to my colleagues whose contribution to MySQL Support was really important, but mostly non-technical, that is, pure Directors, Managers and Coordinators, who hardly ever reported any bug in MySQL in public and had to step into the technical issues from customers mostly when we, engineers, failed to some extent:

• Tom Basil - he hired me twice, first as a Director of Support in MySQL back in 2005 and then as a COO in Percona in 2012. Tom was a Director of Support and Development (thanks Salle for clarification) in MySQL since 2001 and he defined, to large extent, the Support culture I am proud to be a part and defender of till today. The approach to providing Support 24x7 by the international team of the best engineers working remotely from home and coordinated by one of them, playing the role of Support Shift Captain (SSC), working based on Standard Operation Procedures (SOP) written as a set of wiki pages, using emails (properly, "the burden should be on the writer" etc!) and IRC to communicate with colleagues - all these, for me, were created either by or under the influence of Tom.
  He left us in Sun in September, 2008, somewhat unexpectedly and silently, to join Percona, but he left us in a very good state. MySQL Support, in general, was really a privileged organization inside MySQL and Sun, thanks mostly to Tom, I think. Tom was a really great Director of Support for MySQL! Everything was in order for me, I became Senior in MySQL fast, I've got great employment conditions from Sun and felt really safe for few more years ahead.
  The entire story of my work for Percona also started with his question some day in 2011 (or was it 2010?) when we met in Kiev (they had administrative team meeting iin the hotel some 500 meters or so from my flat), got some sushi and he asked me would I like to join Percona... I answered that I am not invited, to begin with. Then, some day in 2012, I was invited and got a job offer that I eventually accepted. As a result I worked for 3 more years in the best Support team in the industry. I was surprised that Tom was not involved in anything related to Support directly in Percona and I am still not sure why this was the case.
  Tom is still in Percona now, he is a Chief of Staff. In this role he tried his best to help me to still work in Percona, and the fact that I am not there any more is by no means his fault. Thank you, Tom, for your continued efforts that help entire MySQL Community!
• Peter Farkas - he was my Director of Support in Percona, a leader of Support organization that I've already nominated as the best in the industry! He had that position starting from December, 2011 and more or less till December, 2015. Peter had spent a lot of time first to get me into his team (he agreed to wait for 3 months for me to complete hand over of all my work in Oracle) and then to keep me happy and operational while I worked for Percona (I became desperate few times there). We worked a lot together to make sure Percona Support is really the best, and implemented some unique procedures (inspired and even requested by Peter Zaitsev, to be fair) of internal quality assurance. Unfortunately some of the work in progress we still had to complete was not completed before we left Percona, first Peter and then, recently, me.
  In private writings to some colleagues I've already named him as the best Director of Support of those whom I worked for, with Tom Basil being only second. I am sure not everybody in Percona agrees with me. My statement above does not mean that Peter Farkas had never made mistakes or could not do many things better while managing the team. But his level of responsibility and care about the team and each engineer there was outstanding, he really sacrificed 4 years of his life to the team. Let me give just one example. Peter was the only Director of Support who took SSC shifts, during any time zone, when he or his reporting team managers failed in scheduling or were not able to step in. He was never scared to talk to anyone, from (few) demanding and angry customers to CEO, to defend his team members when they actually did everything right. Only enormous pressure could make him fail in these efforts.
  Now Peter is a Co-Founder and CEO of Altinity, a company that has something to do with SQLite. I hope he'll tell us all more about that during his talk at upcoming Percona Live conference. I'd like to work with him again one day.
• Lorna Ward - she joined us in Sun, early 2008, as an Assistant Support Engineer and eventually became a Support manager in Oracle. She was a reliable SSC, capable engineer and good colleague for us, and I am really sorry that my direct way of communication about problems influenced her badly once.
• Erzsebet Olsovszky -  she joined Percona Support in August, 2013, as an Executive Assistant, to help Peter Farkas to manage all the tasks we had in Support. Based on her notable corporate experience with IBM, she helped us a lot to set up proper procedures around meetings, internal quality assurance initiatives, and, most importantly, setting priorities for the bug fixes in Percona software taking into account the real needs of our customers and usual goals of Development.
  She made real miracles to get both me, Director of Support, managers, Director of Development, CTO and even CEO (when it was badly needed), sitting around one "virtual" table and discussing the best approaches to process, fix or workaround known bugs and deficiencies affecting Percona customers. Moreover, she managed to document the results of these discussions properly.
  Erzsebet is a Senior Coordinator in Percona since January, 2015. We worked closely together for more than 2 years and I really appreciate all her help, efforts and "can do" attitudes. She never gives up!

Again, this list is not complete (I remember at least 3 more names to put there) and I am going to correct it soon. Your feedback is appreciated, I could really forget someone without any good reason.

So, now you know also about those who mostly helped us, MySQL Support Engineers, to work properly and efficiently. They are also "MySQL Support People" to me. Let me remind you about the previous posts in this series that listed great engineers I had an honor to work with while supporting MySQL:

• MySQL Support People - Those Who Were There First
• MySQL Support People - The Next Generation
• MySQL Support People - Those Who Joined Oracle to Work on MySQL
• MySQL Support People - Percona Support

As this is a final post in the "MySQL Support People" series, I'd like to conclude with the following statement:

From pure practical point of view, I value opinion on how to do Support (and services in general) for MySQL from anyone mentioned in the posts from this series as much more important in decision making comparing to the opinion from anyone who is not mentioned in them. To be considered of the same league, they should start with providing some links explaining their contribution to MySQL Community...

MySQL Support People - Those Who Joined Oracle to Work on MySQL

In this post I'd like to mention my colleagues from MySQL Support who had joined the team when Oracle already acquired Sun (and MySQL as a part of it), in 2010-2012. There were several really great engineers among them:

• Jesper Krogh - with his 31 public bug reports, numerous blog posts (check also this one), ps_tools and presentations, Jesper is well known to MySQL community. He had joined Oracle as Senior MySQL Support Engineer in March, 2011 and is a Senior Principal Technical Support Engineer at Oracle now. Check one of his "Verified" bug reports for SYS schema, Bug #77853. His tutorial on Performance Schema from 2013 was a really great one!
• Santo Leto - he had joined Oracle in December 2010 (as contractor initially) and made a good career there as Principal Support Engineer and then, since April 2013, a team manager of EMEA MySQL Support team in Oracle. We worked closely together during his first year(s) in Oracle. Santo was a reliable SSC, good MySQL Cluster Support engineer, Windows guru of a kind, and a contributor to the MySQL Community, as one can conclude from his 54 public bug reports. Check the oldest his bug report that is still "Verified", Bug #33287. I'd never forget a terrible weekend shift he took in 2012 and managed to survive, when MySQL Cluster was failing for one major European telecom provider... I had to step in to help with all customer issues not related to that terrible incident (that made me think this software is really not supportable unless you can involve its developers, the idea I still deeply believe in). I can not tell you how good Santo was as a team manger, as I quit in August, 2012, well before he switched to management. Recently Santo also quit from Oracle and now he is a Global Manager, Customer Support at Neo Technology, creators of Neo4j.
• Arnaud Adant - he is one of the most brilliant MySQL Support engineers I've ever worked with. Really good skill set for the job and proper attitude, since day one. Arnaud joined Oracle as Senior Support Engineer in January, 2011, became Principal in 2012 and worked there till May, 2014. Then he left to do some Java development for some time, but now he is back to MySQL/MariaDB world as a DBA at Jump Trading LLC. In Oracle Arnaud was one of the core support providers for EMEA customers. He had always contributed to MySQL Community, and I see 17 public bug reports from him (probably he had several accounts and reported more), to say nothing about his great presentation and tutorial from MySQL Connect@Oracle OpenWorld conference. He also did some bugs processing, check my Bug #68079 that is still assigned to him formally :) I am really proud that I worked with him for more than 1.5 years and consider him my good friend in real life.
• Umesh Shastry - he had joined Oracle in May, 2011, when everything in MySQL was already done "Oracle way". We worked closely together since his first days and till my last day in Oracle, as he had always been interested in my work on bugs, experience and approaches to the job. Umesh had worked a lot as a key Support engineer for MySQL Cluster during EMEA hours and soon gained a well deserved respect from MySQL Cluster developers, customers and many "old-timer" colleagues in Support. After I left Oracle he started to work more and more on public bugs processing and eventually, in 2013, he ended up in my Bugs Verification Team, doing large part of my job. When Sveta Smirnova joined me in Percona, Umesh (who now is a Principal Technical Support Engineer) took over some of her job duties as well, and at the moment he is the most important person in the processes of community bugs verification and bug fixing in GA MySQL versions. Just trust me, he is the key to success for MySQL Community in everything related to bug fixing in Oracle. That's why I mention his name many times in each and every post in my "Fun with Bugs" series. Umesh had reported 13 public bugs for MySQL, but you should understand that his job is mostly to have as few non-processed bugs as possible.
• Anand (Ananadakumar S) - he had also joined Oracle in May, 2011 and now he is a Principal Support Engineer there. I can not find any public bug reports by him (as he mostly worked on usual MySQL customer issues in Oracle, based his previous production MySQL DBA experience), but check Bug #50285 for his comment. We started to work closely together probably only during my last months in Oracle, in 2012, so I had few time to work on proper habits towards bugs and MySQL Community with him. Still, I remember him as a reliable and useful tam member.
• Uma Bhat (Uma Hoysala) - she had also joined Oracle in May, 2011 and worked in MySQL Support as a Senior Support Engineer more or less till October 2015, when she switched to management role. She is now a Product Support Manager at Oracle. I can not find public bug reports, but check Bug #11696 for her comment.As with Anand, we started to work closely together with Uma on bugs, compiling from source and similar boring matters only during my last months in 2012. But bugs processing aside, Uma is well known for her community contributions to MySQL (check her blog posts here).
• Bhupinder Singh - he is a Principal Technical Support Analyst in Oracle now, probably still working mostly on MySQL Cluster customer issues. I think he joined MySQL Support some time before May, 2011. I see one public bug report from him here and one more from this account, so he seems to be interested in NDB cluster for a long time. We had not so many chances to work together as I tried my best to keep myself as far from MySQL Cluster as possible, and it was possible for me during weekdays and EMEA hours (when we both worked), thanks to hard work of my colleagues named here and in the previous post.
• Staffan Flink - he joined in January, 2012 as a Senior MySQL Support Engineer and was a very reliable colleague in that role. As far as I understand he left Oracle soon after me, in November, 2012, Now Staffan is a MySQL DBA / Operations Engineer at eTRAVELi. Staffan is a member of Sweden MySQL User Group, but otherwise I do not see any his public contributions to MySQL Community recently.

That's all whom I remember at the moment of those who did Support engineer's job on a regular basis and contributed to MySQL Community in a way I can find. Please, correct me if I missed somebody (it happens all the time, but I have to rely on my memory only, and it's selective and faulty at times).

Next post will be devoted to pure MySQL Support managers, those who had rarely (if ever) worked on technical issues from customers, unless we, Support Engineers, made terrible mistakes... Stay tuned!

My First Steps in Exploring RocksDB

RocksDB and storage engine for MySQL based on it (so called "MyRocks") is widely discussed in my circles since August 2015 at least, so I decided to spend some time checking it. The easy way to get it running is to use Facebook's MySQL 5.6, so I just clonned it and built from source with minor customization based on instructions (that just work in case of Fedora Core 23):

33       mkdir git
34       cd git
35       git clone https://github.com/facebook/mysql-5.6.git
36       cd mysql-5.6/
37       git submodule init
38       git submodule update
39       cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DWITH_SSL=system -DWITH_ZLIB=bundled -DMYSQL_MAINTAINER_MODE=0 -DENABLED_LOCAL_INFILE=1 -DCMAKE_INSTALL_PREFIX=/home/openxs/dbs/fb56
...
41       time make
42       make install && make clean
...
44       cd
45       vi fb56.cnf
46       cd dbs/fb56/
...
48       scripts/mysql_install_db --defaults-file=/home/openxs/fb56.cnf
49       ls data/mysql/
50       bin/mysqld_safe --defaults-file=/home/openxs/fb56.cnf &
51       bin/mysql -uroot test

On Ubuntu 14.04.3 last week I was affected by the problem similar to Issue #147, but with today's code (commit e9d85381d22a2c3a2f8cea614baa70f7e0cef7b7) there was no problem to build there as well.

The fb56.cnf file is quite simple:

[openxs@fc23 ~]$ cat fb56.cnf
[mysqld]
rocksdb
default-storage-engine=rocksdb
skip-innodb
default-tmp-storage-engine=MyISAM

log-bin
binlog-format=ROW

 and the reasons for the settings there are explained on their wiki (here and there).

As a result, I've got my first RocksDB table(s) created in a matter of minutes (spent mostly on building from source), and my first bug reports filed almost immediately:

• Issue #159 - "Indexes on RocksDB table are listed as BTREE ones in SHOW INDEXES". It was closed very soon.
• Issue #160 - "ANALYZE TABLE does not seem to update statistics for the RocksDB table". It was closed less than a day ago.Unfortunately it seems now ANALYZE TABLE updates some statistics but does not set data size properly, see my Issue #164 reported today.
• Issue #163 - "Strange EXPLAIN output for UPDATE ("Using temporary")". I've reported this today and it seems something in optimizer (maybe just a feature that I am not aware about) that produce the result I consider strange. In the process I've also noted that ICP (index condition pushdown) is NOT sued for the PRIMARY key of RocksDB tables (unlike for MyISAM ones), but the same limitation is known and documented for InnoDB tables.

Besides some testing, I've surely executed the command I expected to be there, SHOW ENGINE ROCKSDB STATUS:

mysql> show engine rocksdb status\G
*************************** 1. row ***************************
  Type: DBSTATS
  Name: rocksdb
Status:
** DB Stats **
Uptime(secs): 106.0 total, 106.0 interval
Cumulative writes: 6 writes, 524K keys, 6 batches, 0.9 writes per batch, ingest: 0.01 GB, 0.08 MB/s
Cumulative WAL: 6 writes, 4 syncs, 1.20 writes per sync, written: 0.01 GB, 0.08 MB/s
Cumulative compaction: 0.01 GB write, 0.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.6 seconds
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 6 writes, 524K keys, 6 batches, 0.9 writes per batch, ingest: 8.00 MB, 0.08 MB/s
Interval WAL: 6 writes, 4 syncs, 1.20 writes per sync, written: 0.01 MB, 0.08 MB/s
Interval compaction: 0.01 GB write, 0.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.6 seconds
Interval stall: 00:00:0.000 H:M:S, 0.0 percent
** Level 0 read latency histogram (micros):
Count: 5  Average: 10.2000  StdDev: 8.70
Min: 1.0000  Median: 6.5000  Max: 22.0000
Percentiles: P50: 6.50 P75: 19.50 P99: 22.00 P99.9: 22.00 P99.99: 22.00
------------------------------------------------------
[       0,       1 )        1  20.000%  20.000% ####
[       1,       2 )        1  20.000%  40.000% ####
[       6,       7 )        1  20.000%  60.000% ####
[      18,      20 )        1  20.000%  80.000% ####
[      20,      25 )        1  20.000% 100.000% ####
...

and checked the content of the datadir related to RocksDB:

[openxs@fc23 ~]$ ls -la dbs/fb56/data/
total 1228
drwxrwxr-x.  6 openxs openxs    4096 Feb  9 11:27 .
drwxrwxr-x. 13 openxs openxs    4096 Feb  4 12:36 ..
-rw-rw----.  1 openxs openxs      56 Feb  4 12:37 auto.cnf
-rw-rw----.  1 openxs openxs   24872 Feb  4 12:36 fc23-bin.000001
-rw-rw----.  1 openxs openxs 1148080 Feb  4 12:36 fc23-bin.000002
-rw-rw----.  1 openxs openxs     488 Feb  4 13:26 fc23-bin.000003
-rw-rw----.  1 openxs openxs     139 Feb  4 13:26 fc23-bin.000004
-rw-rw----.  1 openxs openxs    1990 Feb  9 10:55 fc23-bin.000005
-rw-rw----.  1 openxs openxs     684 Feb  9 11:30 fc23-bin.000006
-rw-rw----.  1 openxs openxs     108 Feb  9 11:27 fc23-bin.index
-rw-r-----.  1 openxs openxs   24184 Feb  9 11:30 fc23.err
-rw-rw----.  1 openxs openxs       6 Feb  9 11:27 fc23.pid
drwx------.  2 openxs openxs    4096 Feb  4 12:36 mysql
drwx------.  2 openxs openxs    4096 Feb  4 12:36 performance_schema
drwxr-x--x.  2 openxs openxs    4096 Feb  9 11:30 .rocksdb
drwxrwxr-x.  2 openxs openxs    4096 Feb  9 11:30 test
[openxs@fc23 ~]$ ls -la dbs/fb56/data/.rocksdb/
total 176
drwxr-x--x. 2 openxs openxs  4096 Feb  9 11:30 .
drwxrwxr-x. 6 openxs openxs  4096 Feb  9 11:27 ..
-rw-r-----. 1 openxs openxs   702 Feb  9 10:35 000040.sst
-rw-r-----. 1 openxs openxs   219 Feb  9 11:30 000047.log
-rw-r-----. 1 openxs openxs  1622 Feb  9 11:27 000054.sst
-rw-r-----. 1 openxs openxs    16 Feb  9 11:27 CURRENT
-rw-r-----. 1 openxs openxs    37 Feb  4 12:36 IDENTITY
-rw-r-----. 1 openxs openxs     0 Feb  4 12:36 LOCK
-rw-rw----. 1 openxs openxs 28361 Feb  9 11:30 LOG
-rw-rw----. 1 openxs openxs 19212 Feb  4 12:36 LOG.old.1454582184760374
-rw-rw----. 1 openxs openxs 19438 Feb  4 12:36 LOG.old.1454582234660587
-rw-rw----. 1 openxs openxs 20023 Feb  4 13:26 LOG.old.1455004201986761
-rw-rw----. 1 openxs openxs 38739 Feb  9 10:55 LOG.old.1455010078294084
-rw-r-----. 1 openxs openxs   556 Feb  9 11:30 MANIFEST-000045
-rw-r-----. 1 openxs openxs  5414 Feb  9 11:27 OPTIONS-000051
-rw-r-----. 1 openxs openxs  5415 Feb  9 11:27 OPTIONS-000053

I've also checked some messages in the error log related to RocksDB:

2016-02-09 11:27:58 14773 [Warning] The option innodb (skip-innodb) is deprecated and will be removed in a future release2016-02-09 11:27:58 14773 [Note] Plugin 'InnoDB' is disabled.
2016-02-09 11:27:58 14773 [Note] Plugin 'FEDERATED' is disabled.
2016-02-09 11:27:58 14773 [Note] RocksDB: 2 column families found
2016-02-09 11:27:58 14773 [Note] RocksDB: Column Families at start:
2016-02-09 11:27:58 14773 [Note]   cf=default
2016-02-09 11:27:58 14773 [Note]     write_buffer_size=4194304
2016-02-09 11:27:58 14773 [Note]     target_file_size_base=2097152
2016-02-09 11:27:58 14773 [Note]   cf=__system__
2016-02-09 11:27:58 14773 [Note]     write_buffer_size=4194304
2016-02-09 11:27:58 14773 [Note]     target_file_size_base=2097152
2016-02-09 11:27:58 14773 [Note] RocksDB: Table_store: loaded DDL data for 6 tables
2016-02-09 11:27:58 14773 [Note] RocksDB instance opened
2016-02-09 11:27:58 14773 [Note] Starting crash recovery...
RocksDB: Last binlog file position 1772, file name fc23-bin.000005
2016-02-09 11:27:58 14773 [Note] Crash recovery finished.

All these details will be discussed in the next blog posts eventually.

So, these were my very first steps with RocksDB as a storage engine for MySQL. I am really impressed by the speed of bug fixing and adding missing features.

The real fun started when I've attached gdb to mysqld and set some proper breakpoints to find out how RocksDB locks the data accessed. Stay tuned - it's a topic for next (maybe several) posts about RocksDB.

Fun with Bugs #41 - Bugs Fixed in MySQL 5.7.11

I've just noted that Oracle had released new versions of MySQL on February 5, 2016 formally, so while these days I am mostly thinking about the ways to do support properly, remembering my colleagues and trying to understand some of RocksDB internals, it's time to postpone all these and write about bugs again. This time about some of the public bug reports from MySQL Community and Oracle engineers that were fixed by Oracle in MySQL 5.7.11.

As usual, I'll try to mention who had reported a bug and who verified it, as I think that names matter in MySQL world. I'll concentrate mostly on InnoDB, replication and optimizer bug reports, trying to highlight regressions clearly.

As usual, I prefer to start with InnoDB bugs:

• Bug #75112 - "Optimize table removes the data directory in partitions". This bug was reported by my former colleague Nilnandan Joshi and verified by Umesh. The fix should appear in MySQL 5.6.30 also.
• Bug #79775 - "Innodb doesn't update index stats when dropping virtual column in virtual index". It was reported by Shaohua Wang and probably was verified and fixed by him. I am always happy to see Oracle engineers reporting MySQL bugs in public! Check also his Bug #79773, "handler::my_eval_gcolumn_expr doesn't set correct null bit" fixed in 5.7.11.
• Bug #79691 - "Block Memcached from modifying tables that contain indexed virtual columns". One more  case of public bug reporting by Oracle engineer. This time it was Thirunarayanan Balathandayuthapani.
• Bug #79516 - "Server crashes on invalid value for variable innodb_saved_page_number_debug". When I hit third public report by Oracle engineer, this time even about a crash, I started to think that there is a new generation of engineers there that just do right things from the very beginning. Thank you, Naga Satyanarayana Bodapati, for reporting this bug in public!
• Bug #79030 - "REORGANIZE PARTITION MOVES PARTITION TO TABLE-LEVEL TABLESPACE BY DEFAULT". When I see all caps in the bug synopsis I suspect it's from Oracle engineer (and from Oracle's internal bugs database, originally). This bug was reported by Darshan M n.
• Bug #79201 - "InnoDB: Too small buffer pool for innodb_page_size=64k". Marko Mäkelä had never hesitated to report bugs in public. Check also his Bug #79150, "InnoDB: Remove runtime checks for 32-bit file offsets". Thank you for noting and fixing these! 
• Bug #79434 - "Incorrect NUMA-specific code in buf_chunk_init()". It was reported by my former colleague Alexey Kopytov and verified by Sinisa Milivojevic formally. Check also his Bug #79354, "Incomplete NUMA support on buffer pool resizing".
• Bug #78960 - "Unclear note "InnoDB: not started" in the error log". I've reported this bug back in October, 2015, based on the log from some customer issue. The message that was produced during loading of the buffer pool looked scary and misleading for customer, so I reported the bug immediately. It was verified by Umesh and is also fixed in 5.8.0.
• Bug #78955 - "Import Dump with InnoDB Fulltext Index and Constraint fails". This regression bug was reported by Nico Niepraschk and verified by Umesh.
• Bug #78831 - "InnoDB looks for buffer pool file name in '/' if innodb_data_home_dir is empty". It was reported by someone named NOT_FOUND NOT_FOUND. I'd like to find out one day who is it, as I see other useful bug reports under the same name, all fixed.
• Bug #78714 - "InnoDB: Failing assertion: !(&dict_sys->mutex)->is_owned() in dict0stats.cc 3049", was reported by my former colleague Roel Van de Paar and verified by Umesh. 
• Bug #78647 - "POTENTIAL RUNNING OUT OF SPACE IN REDO - INVALID MARGIN CALCULATIONS". This bug was reported by Pawel Olchawa and probably verified by Marko Mäkelä. The fix was mostly to increase the minimum value of innodb_log_file_size from 1MB to 4MB.
• Bug #78262 - "Adding indexes on empty table is slow with large innodb_sort_buffer_size.". It was reported by Jean-François Gagné (who had made a great presentation at FOSDEM 2016, by the way) and verified by Umesh.
• Bug #74832 - "ut_delay missing compiler barrier, UT_RELAX_CPU isn't at all relaxing". This bug was reported long time ago by Stewart Smith, who had provided the patch also. It was formally processed by Umesh.

The only public bug in partitioning that is listed in the release notes as fixed is Bug #79145, by Debarun Banerjee, who is probably working for Oracle.

Now, let's move on to replication (and XA transactions) bugs:

• Bug #79416 - "InnoDB: Failing assertion: total_trx >= trx_sys->n_prepared_trx". This (debug builds only) bug was reported by Roel Van de Paar and verified by Umesh. See also Bug #78695, "Assertion `static_cast<Sql_cmd_xa_commit*>(thd->lex->m_sql_cmd)-> get_xa_opt() =".
• Bug #77740 - "silent failure to start if mysql.gtids_executed gets HA_ERR_LOCK_WAIT_TIMEOUT". It was reported and verified by Shane Bester. The fix is to NOT allow XA transactions to modify the mysql.gtid_executed table.
• Bug #77417 - "Applying create temporary table SQL on a slave with replicate-rewrite-db fails". This bug was reported by JAMES HETHERINGTON (who seems to be a customer of Oracle) and verified by Umesh. Good to see Oracle customers following my old advice and reporting bugs in public!
• Bug #77393 - "Failure to generate GTID leads to inconsistency". It was reported by Davi Arnaut, who had provided a patch, and verified by Umesh.
• Bug #77354 - "multi drop temp table statements in UDF cause replication fails using 5.6 GTID". This bug was reported by Fungo Wang (check his other replication-related bug reports, still "Verified") and verified by Umesh (no wonder, he seems to verify up to 70% of all bugs recently...)
• Bug #77288 - "Variables in alter event not replicated properly". It was reported by Trey Raymond and verified by Umesh.
• Bug #77249 - "Drop table may cause slaves to break". This bug was reported by  Santosh Praneeth Banda (who is a very active bug reporter, I see 7 more replication bugs) and verified by Umesh. It is also fixed in MySQL 5.6.29.
• Bug #71859 - "Replication broken after creation of scheduled events". It was reported by Shahid Hussain and verified by Sveta Smirnova long time ago. The bug is also fixed in MySQL 5.6.29.

Let me conclude with several optimizer-related bugs:

• Bug #79590 - "ERROR 1093 (HY000): You can't specify target table '.' for update in FROM clause". It was reported by Zhang Jinzhu and verified by Miguel Solorzano.
• Bug #79549 - "Regression: “unknown column” for outer computed values used inside a subquery". This bug was reported by Chris Adams and verified by Miguel Solorzano.
• Bug #79398 - "view using hexadecimal or bit literal gives wrong results". It was reported, verified and probably fixed by Guilhem Bichot.
• Bug #79333 - "Unexpected error 1093 on nested subquery for update". This regression bug was reported by Danielle McLean and verified by Miguel Solorzano. 
• Bug #79294 - "Query with double nested subquery much slower in 5.7". This regression bug was reported by Jonathan Corwin and verified by Umesh. Nice to see it fixed.
• Bug #79194 - "Wrong result on outer join with uncorrelated subquery, derived_merge". This regression bug was reported by Elena Stepanova and verified by Miguel Solorzano.
• Bug #79131 - "Get negative FOUND_ROWS() for UNION stmt". It was reported by Su Dylan and verified by Umesh. Based on the comments, it's also a regression bug in 5.7 comparing to 5.6, even though it is not tagged like that.
• Bug #78946 - "WRONG RESULT FOR SELECT NULL IN (<SUBQUERY>) ". One more bug report from an Oracle developer, Dag Wanvik. Good that he had found, reported and probably fixed this regression, but in general I am negatively surprised by the number of recent optimizer regression bugs in MySQL 5.7 that are found only after they are introduced into public releases.

There are many more bug fixes, changes in behavior and new features in MySQL 5.7.11. So, read the release notes carefully and prepare to upgrade.

MySQL Support People - Percona Support

I planned to continue this series of posts with the one about MySQL Support engineers who joined us in Oracle while I was working there, but based on recent events in my life I'd prefer to postpone it and move directly to the team I worked for during last 3+ years, Percona Support Team.

Disclamer: In the list below I still try to pay attention to public contribution by each of engineers mentioned to MySQL Community, mostly in a form of bug reports. This is not the only way they contributed, but I have to stick to this format in current series of posts. I also base my comments mostly on my memory and quick search of public sources of information, so I may be mistaking about the roles, time periods and other details. Please, inform me if you think there is something to fix.

So, here is the list of my former colleagues who formed the best MySQL Support team in the industry in 2013-2015 while working with me in Percona:

• Miguel Angel Nieto - Miguel was my mentor during my first days in Percona, so I have to start with him. He joined Percona in 2011 and was a team manager of EMEA Support team for a long time. We always managed to work together in a productive and efficient way. Miguel is a real expert in Percona XtraDB Cluster and almost every new technology we had to support, up to MongoDB recently. He used to act as a great consultant as well when Support engineers worked together with Consulting on the same pool of issues, and he still takes SSC (Support Shift Captain) shifts, so he can be the first person from Percona whom you deal with in case of problems even today. I see 13 public bug reports from Miguel and suggest to check his Bug #77654. See also his 12 bug reports for Percona software at  https://bugs.launchpad.net/~miguelangelnieto/+reportedbugs.
• Ovais Tariq - when I joined Percona Ovais was the only Principal Support Engineer in the team. He was really great in everything technical he did, from complex data recovery consulting cases to query optimization, and on top of that he was a really good writer. It was a pleasure to read every email in issues he worked on, it was like reading a good technical blog post if not a book. I see 16 public bug reports for MySQL by Ovais. Check his replication-related Bug #70923, for example. He had also reported 20 bugs for Percona software at https://bugs.launchpad.net/~ovais-tariq/+reportedbug. In 2013 Ovais moved to Consulting and a year later, in 2014, he quit from Percona to become a Lead MySQL DBA - Automation Engineer and later a Lead Reliability Engineer at Lithium. He is also active at TwinDB site.
• Fernando Ipar - As far as I know, he was the first full time Support engineer and dedicated Support Shift Captain in Percona since September, 2009. For two years later he was a Director of Global Support. When I joined he was working in Consulting and recently he seems to do Development. But during all this time Fernando was interested in Support-related discussions and helped my colleagues promptly when needed. I see 11 bugs reported by him for MySQL, including funny optimizer regression since 5.1.x, Bug #66825. Fernando also reported 10 bugs for Percona software at https://bugs.launchpad.net/~fipar/+reportedbugs.
• Michael Rikmas (a.k.a. Mixa) - he moved to Support from Administrative team in 2011, and there he was one of the first Percona employees. He always cared about Percona operations and customers, covering endless SSC shifts any time a day, sometimes for 12+ hours in a row. He was always ready to step into anything any time a day to help, if needed. While working in Support he improved his technical skills a lot in all MySQL-related areas, so he ended up as a good problem solver and eventually quit from Percona (in 2014) to join PSCE as a Consultant. I see only one bug he reported for MySQL, Bug #62426. See also https://bugs.launchpad.net/~michael.rikmas/+reportedbugs.
• Martin Arrieta - Martin joined Percona in January, 2012, but was already a well known Support engineer when I joined. In 2013 he moved to Consulting and two years later quit from Percona in September, 2015, to join Pythian as Database Consultant. Martin had reported 6 MySQL bugs in public (he was really interested in MySQL Fabric), but he is better known as a blog posts author. I think we always cooperated really well with him. 
• Marcos Albe (a.k.a. Markus) - he is the most hard working and successful Support engineer of all times in Percona. Marcos was also my team manager after I switched to AMER Support Team. Marcos joined Percona in 2010 and by 2012 he was not only a hard working SSC, but also a full stack expert. Marcos created many useful hints for the internal knowledge base and is also know as a great speaker at Percona Live conferences (I've met him for the first time at Percona Live New York, in 2012, where he had talks on several topics). The way he works (he is ready to call you using any media, join any chat, log in promptly and spend entire night helping any customer interactively) defines famous "Percona style" of providing Support (whatever I may think about it) and gave him huge experience with all technologies even remotely related to MySQL. Marcos had reported 4 public MySQL bugs, including a 5.6.x regression one, Bug #71616, that is still "Verified". I also see 5 bugs reported by him at https://bugs.launchpad.net/~markus-albe/+reportedbugs.
• Nickolay Ihalainen - Nickolay is a classical Percona Consultant (who still works in Support for a long time), with huge real life experience of managing MySQL at scale and full application stack troubleshooting. He knows a lot about Linux, hardware used with MySQL and, as any other great consultant in Percona, is always ready to step into the issue of any complexity and length, call customer, write code, recover InnoDB data or fix a bug quickly. He works in Percona since January 2010, reported 3 MySQL bugs in public, including Bug #78051 that is still "Verified". I also see 7 bugs reported by Nickolay for Percona software at https://bugs.launchpad.net/~ihanick/+reportedbugs. I studied a lot while working with and talking to him.
• Przemyslaw Malkowski - he joined Percona in August, 2012, just before me. We were at the same "bootcamp" session in famous "Hotel Percona" and worked closely together since that time. Przemek is a Principal Support Engineer since July, 2015. He is the only one who was promoted to that level while I was working in Percona, not just joined like that. He works hard on all kinds of complex issues and is one of key experts in the industry in Galera and Percona XtraDB Cluster (He can easily parse and understand obscure, huge logs from multiple nodes, and explain the interactions and problems  that happened, quickly. This is the skill I always missed). Moreover, he is a well known bug reporter, with 22 bugs reported for upstream MySQL, including infamous Bug #78777 that mislead numerous MySQL users and customers for months, and 37 (!) bugs reported for Percona software at https://bugs.launchpad.net/~pmalkowski/+reportedbugs. Przemek is able to work with any kind of annoying customer, on issues of any complexity, and still remain helpful. I really enjoyed working with him in one team, a dream team actually!
• Nilnandan Joshi (a.k.a. Nil) - he joined Percona in May, 2012 and successfully worked there as Support Engineer till January, 2016. He managed to master all key tools and technologies in MySQL, while working hard on boring duties like SSC and bugs processing. Now Nil is a Big Data Engineer at zData Inc and is probably working with Hadoop or something of that kind. He also was visiting "Hotel Percona" with me in 2012 for a "bootcamp", where we spent a lot of time talking about the way InnoDB works (and drinking some beer). Since that time we were working in a close cooperation till his last day in Percona. Nil helped me a lot with bugs processing when I've got a task to manage it from Support side, and was one of few engineers in Support who were ready to report or properly process/verify any bug, build any Percona software, test it on all kinds of weird Linux versions and distros, using all kinds of virtual and real machines. Nil reported 13 public MySQL bugs, including Bug #79469. I also see 9 bugs reported by him for Percona software at https://bugs.launchpad.net/~nilnandan-joshi/+reportedbugs. I miss our useful work together on bugs and Percona Live sessions already...
• Jervin Real - he worked in Percona Support since early 2010 and until summer 2013, when he moved to Consulting (he worked great there as well, always ready to step in when we needed help in Support). When I joined he was a key Support engineer for APAC customers. Jervin is a well known blogger, who writes a lot about HA and other technologies for MySQL. I finally met him in real life few days ago, at FOSDEM conference, where he was speaking about TokuDB. Now Jervin is a Technical Service Manager in Percona. He had contributed a lot to MySQL Community also with his 20 public bug reports, including Bug #77715 that is still "Open". I also see 34 bugs reported by Jervin for Percona software at https://bugs.launchpad.net/~revin/+reportedbugs.
• Jaime Sicam - Jaime joined Percona Support in June, 2011 and later became a good manager of APAC team in Support. He always worked hard and helped colleagues with anything, from covering SSC shift to complex troubleshooting. He is a great engineer capable to deal with any kind of issues, but he was our key expert in anything related to PAM and authentication in general. Jaime reported 7 bugs for MySQL software, including infamous Bug #77344, and 11 bugs for Percona software at https://bugs.launchpad.net/~jssicam/+reportedbugs. We rarely were both online and working together, but I already miss him.
• Muhammad Irfan - he joined Percona Support soon after me, in December 2012. Muhammad actively started to work and cooperate with me and other senior colleagues from the beginning, and soon had become a high skilled engineer in everything related to MySQL, including XtraDB Cluster and bugs processing. He is a quite famous blogger, who was happily writing about new features, tools and best practices (while I prefer to write about bugs and problems, that is, worst practices). I am proud to get a change to help him with some posts as a reviewer. Muhammad spent fair amount of time on bugs processing and reporting as well, as we can conclude from his 8 MySQL bug reports including "Verified" Bug #73094 and 9 bugs reported for Percona software at https://bugs.launchpad.net/~muhammad-irfan/+reportedbugs.
• Fernando Laudares (a.k.a. Nando) - he joined Percona in January, 2013 and worked in AMER Support Team. I'd call him a student of Marcos, as he quickly worked out similar habits and approaches to Support, "Percona style". Fernando always tried his best to help both customers and colleagues, without any limits, and this quickly made him one of the key team members. He works hard, on many issues, day after day without becoming tired in a visible way. Fernando had written many great blog posts and is great in all kinds of automation/testing environments setups. He was fast in studying, accepting and following new technologies we were supposed to support in Percona. His community contributions also include recent enough Bug #77684 and 2 bugs reported at https://bugs.launchpad.net/~fernando-laudares/+reportedbugs.
• Jericho Rivera - with mostly sysadmin and developer background, Jericho had to work really hard since October, 2013 to reach the level of MySQL expertise we required in Percona Support. But he did that and became a key support provider in 2015, that can deal efficiently with all kinds of customer issues. He helped us to manage Support servers all this time and became a local expert in Docker and other technologies (as you can figure out from his blog posts). He had reported Bug #77073 for MySQL server (still "Verified") and 3 bugs for Percona software at https://bugs.launchpad.net/~jericho-rivera/+reportedbugs. It was really great to see how he becomes better every month, until he ended up just doing an awesome job one day.
• Peiran Song - I was really happy when she joined us in 2014. By May 2014, just in a couple of months, she already did a great job in Support, working on many complex issues. She was really good with troubleshooting, InnoDB and query optimization. We discussed not only technical details, but also procedures and values of Support, a lot. Peiran reported 6 upstream MySQL bugs, including incredible Bug #73369 that is still waiting for explanation and fix, and 8 bugs for Percona software at https://bugs.launchpad.net/~peiran-song/+reportedbugs. It was really sad to meet her in 2015 during Percona Live just to find out that she can not stay with us any more (and not because of any her faults). Peiran is a Data Architect at Smule, Inc now.
• Justin Swanhart - Justin worked for Percona at different positions (mostly as a Consultant and Instructor), since 2010. In August 2013 he joined Support as a Principal Support Engineer, and since that time we often worked together, in the same AMER team. He had got the Community Contributor of the year 2015 award, for his great contributions including (but not limited to) his software. Justin created 40 MySQL community bug reports, including Bug #76210, and I see 12 bugs reported by him for Percona software at https://bugs.launchpad.net/~greenlion/+reportedbugs. Justin worked well, if you ask me, and helped customers a lot, while being a source of unique expertise and great insights for colleagues and customers (based on his trainer and software developer experience, among other things), but one sad day in June 2015 he found himself fired in the middle of the shift. I tried to find out what was really that much wrong with his work, and found literally nothing myself, so I let others to comment on what happened and why. I think firing Justin was a very bad mistake, even though he had better NOT to comment on it in public in the way he did... I hope we'll work together one day again, maybe even soon.
• Bill Karwin - Bill worked for Percona since 2010, mostly as a Consultant and Trainer. We were really lucky that he joined Support in 2014 full time (as a Principal Support Engineer and Senior Knowledge Manager). Using his huge experience in SQL and everything MySQL, he started to play a key role in AMER team, working a lot and providing great service to customers. He is very famous for his answers at Quora and StackOverflow. He is a great writer and speaker on MySQL topics, and his book on SQL anti-patters is well known. Bill is literally second to nobody in helping MySQL Community. Check also his 9 MySQL bugs including Bug #73283 (still "Verified"), and 2 bugs at https://bugs.launchpad.net/~bill-karwin/+reportedbugs. At the end of 2014 Bill left us, probably because it was boring for him to resolve the same problems again and again for customers, instead of doing things right from the very beginning. He is now a Senior Database Architect and Software Engineer at SchoolMessenger.
• Agustin Gallego (a.k.a. Guli) - I met him first in 2012, at Percona Live New York. He worked for Percona since February, 2012, but joined Support later, in December, 2013. Since that time Agustin worked as SSC and Support Engineer in AMER Support team and played an important role in making it operational and efficient. He had reported Bug #77186 for MySQL and 6 bugs for Percona software at https://bugs.launchpad.net/~agustin-gallego/+reportedbugs. He is gaining experience every day, cooperates well with colleagues and now is a very reliable, respected and useful team member.
• Roben Paul Namuag - Paul joined Percona Support in July, 2013. In January, 2015 he left us to continue working in Percona as Remote DBA, but we still discuss technical details and, sometimes, life after that. Paul had good and harder times in Support, but he always tried to do his best and played an important role in APAC team, both as SSC and as an experienced engineer working on complex enough issues, even when there was nobody around to help. He had reported 3 MySQL bugs, including Bug #77180 that is till "Open", and 2 bugs for Percona software at https://bugs.launchpad.net/~paul-namuag/+reportedbugs. I'll remember him as a kind, helpful and friendly person.
• Akshay Suryavanshi - we chatted with with Akshay a lot before he joined Percona, he was interested in a lot of in-depth details on how MySQL works. Eventually he joined Percona Support, in March 2013, but immediately switched to Remote DBA role and had a great success there, moving later to management positions, up to DBA Team Lead and Senior position in Technical Operations recently. Then, in December 2015, he quit from Percona, to become a MySQL Engineer at Tumblr. He helped community with his blog posts and webinars, and also reported Bug #70818
• Pablo Padua - he joined Percona in 2014 as a Jr. Support Engineer, and made a lot of progress since that. Unfortunately I can not find any public MySQL bug reports by him, but he worked a lot as SSC in AMER Support Team, and he gets more experience every day. We often worked together on technical issues, when I just provided some background help, checks and comments, while he did real time work for customer. It had been a pleasure to cooperate that way, and, as a result, for many areas in MySQL Pablo now is ready to work without any assistance.
• Abdelhak Errami - he joined us as Senior Support Engineer in April 2015, when Tokutek was acquired. Since that time he played a key role in TokuDB Support, but also often helped with pure MySQL issues as well. We spent a good time together during Percona Live. Abdel often works on TokuDB bugs, check DB-922 as one of recent examples.
• Joe Laflamme - I think he also joined us some time in 2015 in the process of Tokutek acquisition. Check DB-809 for one of his TokuDB related bug reports. He coordinated TokuDB bug fixing and support during the period when I describe. Now he is a manager at Percona Technical Services.

I had not mentioned Sveta Smirnova, who joined Percona in March 2015, because she was already mentioned in the previous post. I still plan to write about managers and coordinators of Support who had never reported MySQL bugs in public separately, so stay tuned Tom Basil, Peter Farkas and Erzsebet Olsovzsky. I remember about you and I value your contribution a lot, but engineers first...

I had not named colleagues in Percona who joined after November 1, 2015 explicitly here, and this is the date of the beginning of the entirely new epoch in the history of Percona that should be described separately, if ever.