On Upgrades to Percona Server 5.7 GA: Ubuntu 14.04 Case

After upgrading Percona Server 5.6.28 (and "full stack" of Percona software) to 5.7.10-3 GA on CentOS 6 and building it from Git Source on Fedora Core 23 yesterday, I've decided to spend some time upgrading Percona Server on my netbook that plays a role of "home build/test/VMs server" and runs under Ubuntu 14.04:

openxs@ao756:~$ uname -a
Linux ao756 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
openxs@ao756:~$ cat /etc/issue
Ubuntu 14.04.4 LTS \n \l

It had the following set of Percona packages (as a result of some tests with Percona XtraDB Cluster I had to use sometime while working there) and mysqld process running after OS startup:

openxs@ao756:~$ dpkg -l | grep -i percona | grep ii
ii  libmysqlclient18                                      1:5.5.35-rel33.0-611.raring                         amd64        Percona Server database client library
ii  libmysqlclient18-dev                                  1:5.5.35-rel33.0-611.raring                         amd64        Percona Server database development files
ii  libperconaserverclient18                              5.5.47-rel37.7-1.trusty                             amd64        Percona Server database client library
ii  libperconaserverclient18.1                            5.6.28-76.1-1.trusty                                amd64        Percona Server database client library
ii  percona-server-5.6-dbg                                5.6.28-76.1-1.trusty                                amd64        Debugging package for Percona Server
ii  percona-server-client-5.6                             5.6.28-76.1-1.trusty                                amd64        Percona Server database client binaries
ii  percona-server-common-5.5                             5.5.47-rel37.7-1.trusty                             amd64        Percona Server database common files (e.g. /etc/mysql/my.cnf)
ii  percona-server-common-5.6                             5.6.28-76.1-1.trusty                                amd64        Percona Server database common files (e.g. /etc/mysql/my.cnf)
ii  percona-server-server-5.6                             5.6.28-76.1-1.trusty                                amd64        Percona Server database server binaries
ii  percona-toolkit                                       2.2.16-1                                            all          Advanced MySQL and system command-line tools
ii  percona-xtrabackup                                    2.3.3-1.trusty                                      amd64        Open source backup tool for InnoDB and XtraDB
ii  percona-xtradb-cluster-common-5.6                     5.6.28-25.14-1.trusty                               amd64        Percona XtraDB Cluster database common files (e.g. /etc/mysql/my.cnf)
ii  percona-xtradb-cluster-galera-3.x                     3.14-1.trusty                                       amd64        Galera components of Percona XtraDB Cluster

openxs@ao756:~$ ps aux | grep mysqld
root      1283  0.0  0.0   4440   756 ?        S    10:35   0:00 /bin/sh /usr/bin/mysqld_safe
mysql     1505  0.1 13.0 895592 505516 ?       Sl   10:35   0:01 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/lib/mysql/ao756.err --pid-file=/var/lib/mysql/ao756.pid
openxs    2938  0.0  0.0  14640   972 pts/4    S+   10:55   0:00 grep --color=auto mysqld

 This time I decided to follow the manual closely, so I checked for all usual/possible locations of configuration files, stopped MySQL server first and then installed the packages for 5.7 (I've noted TokuDB package was NOT installed, so I've added it to the list, as TokuDB storage engine is one of the reasons to use Percona Server 5.7 instead of well tested and known upstream MySQL 5.7):

openxs@ao756:~$ ls -l /etc/mysql/
total 16
drwxr-xr-x 2 root root 4096 тра 22  2014 conf.d
-rw------- 1 root root  333 тра 22  2014 debian.cnf
-rwxr-xr-x 1 root root 1285 січ  9 02:24 debian-start
-rw-r--r-- 1 root root  120 січ 12 13:03 my.cnf

openxs@ao756:~$ ls -l /etc/mysql/conf.d/
total 0

openxs@ao756:~$ ls -l /etc/my.cnf
-rw-r--r-- 1 root root 180 лис 17 14:22 /etc/my.cnf

openxs@ao756:~$ sudo service mysql stop
[sudo] password for openxs:
 * Stopping MySQL (Percona Server) mysqld                                [ OK ]
openxs@ao756:~$ ps aux | grep mysqld
openxs    3102  0.0  0.0  14636   968 pts/4    S+   11:01   0:00 grep --color=auto mysqld

openxs@ao756:~$ sudo apt-get install percona-server-server-5.7 percona-server-tokudb-5.7Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libperconaserverclient18.1 percona-server-common-5.6
  percona-xtradb-cluster-galera-3.x socat
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  libmecab2 percona-server-client-5.7 percona-server-common-5.7
The following packages will be REMOVED:
  percona-server-5.6-dbg percona-server-client-5.6 percona-server-server-5.6
The following NEW packages will be installed:
  libmecab2 percona-server-client-5.7 percona-server-common-5.7
  percona-server-server-5.7 percona-server-tokudb-5.7
0 upgraded, 5 newly installed, 3 to remove and 3 not upgraded.
Need to get 21,6 MB of archives.
After this operation, 166 MB disk space will be freed.
...
Setting up libmecab2 (0.996-1.1) ...
Setting up percona-server-server-5.7 (5.7.10-3-1.trusty) ...
Installing new version of config file /etc/default/mysql ...
Installing new version of config file /etc/init.d/mysql ...
...
.................... * Percona Server 5.7.10-3 did not start. Please check logs for more details.
Processing triggers for ureadahead (0.100.0-16) ...
Setting up percona-server-tokudb-5.7 (5.7.10-3-1.trusty) ...

 * This release of Percona Server is distributed with TokuDB storage engine.
 * Run the following script to enable the TokuDB storage engine in Percona Server:

        ps_tokudb_admin --enable -u <mysql_admin_user> -p[mysql_admin_pass] [-S <socket>] [-h <host> -P <port>]

 * See http://www.percona.com/doc/percona-server/5.7/tokudb/tokudb_installation.html for more installation details

 * See http://www.percona.com/doc/percona-server/5.7/tokudb/tokudb_intro.html for an introduction to TokuDB


Processing triggers for libc-bin (2.19-0ubuntu6.7) ...
openxs@ao756:~$ ps aux | grep mysqldopenxs   17729  0.0  0.0  14636   972 pts/4    S+   11:20   0:00 grep --color=auto mysqld

So, even though the server was stopped (by the manual) at the moment of installation, there was an attempt to start server in the process that failed after painful 900 seconds of waiting... I've noted that new /etc/init.d/mysql script was installed and decided to check it (starting it manually under bash -x etc, reading and so on). While 900 seconds startup and 300 second shutdown default timeouts were visible in the code, it was not clear what's wrong. I suspected Percona Server Bug #1490897 again (it was a problem for me on Ubuntu 15.04), but here usual init.d approach to startup was used and none of the configuration files I checked had a single pid_file setting. Attempts to start using the script directly also ended up with 900 dots printed:

openxs@ao756:~$ sudo /etc/init.d/mysql start
No directory, logging in with HOME=/
...

Getting a bit tired of all these, I've checked that at least one way to start server up works, without defaults:

openxs@ao756:~$ sudo mysqld_safe --no-defaults &
[1] 21048
openxs@ao756:~$ 160226 11:51:32 mysqld_safe Adding '/usr/lib/x86_64-linux-gnu/libjemalloc.so.1' to LD_PRELOAD for mysqld
160226 11:51:32 mysqld_safe Logging to '/var/lib/mysql/ao756.err'.
160226 11:51:32 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

openxs@ao756:~$ ps aux | grep mysqld
root     21048  0.0  0.0  74060  2204 pts/4    S    11:51   0:00 sudo mysqld_safe --no-defaults
root     21049  0.0  0.0   4440   784 pts/4    S    11:51   0:00 /bin/sh /usr/bin/mysqld_safe --no-defaults
mysql    21124  1.8 14.8 1042020 573708 pts/4  Sl   11:51   0:00 /usr/sbin/mysqld --no-defaults --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/lib/mysql/ao756.err --pid-file=/var/lib/mysql/ao756.pid
openxs   21154  0.0  0.0  14636   972 pts/4    S+   11:51   0:00 grep --color=auto mysqld

openxs@ao756:~$ mysql -uroot -proot test
mysql: [Warning] Using a password on the command line interface can be insecure.
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.10-3 Percona Server (GPL), Release '3', Revision '63dafaf'

Copyright (c) 2009-2015 Percona LLC and/or its affiliates
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

 So, the problem was surely with some settings in the configuration files and/or init.d script. One of the problems was actually something to expect from 5.7, based on the messages in the error log:

2016-02-26T09:54:47.927969Z 0 [ERROR] You have enabled the binary log, but you haven't provided the mandatory server-id. Please refer to the proper server start-up parameters documentation
2016-02-26T09:54:47.928010Z 0 [ERROR] Aborting

2016-02-26T09:54:47.928052Z 0 [Note] Binlog end
2016-02-26T09:54:47.928131Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

and taking into account the content of /etc/my.cnf at the moment:

openxs@ao756:~$ cat /etc/my.cnf[mysqld_safe]
pid_file=/var/lib/mysql/ao756.pid
[mysql]
connect_timeout=2
[mysqld]
innodb_file_format=Barracuda
table_open_cache=1156
table_open_cache_instances=1
log_bin
gtid_mode=ON
enforce_gtid_consistency
log_slave_updates

 So, I've added mandatory (for my case) server_id=N setting to /etc/my.cnf and tried to start again, to find out error messages that I've seen too many times recently and this year:

2016-02-26T09:56:47.576196Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2016-02-26T09:56:47.666859Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2016-02-26T09:56:47.668162Z 0 [Warning] InnoDB: Retry attempts for reading partial data failed.
2016-02-26T09:56:47.668218Z 0 [ERROR] InnoDB: Tried to read 2048 bytes at offset 0 was only able to read02016-02-26T09:56:47.668238Z 0 [ERROR] InnoDB: Operating system error number 22 in a file operation.
2016-02-26T09:56:47.668263Z 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
2016-02-26T09:56:47.668274Z 0 [Note] InnoDB: Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html
2016-02-26T09:56:47.668287Z 0 [ERROR] InnoDB: File (unknown): 'read' returned OS error 122. Cannot continue operation2016-02-26T09:56:47.668298Z 0 [ERROR] InnoDB: Cannot continue operation.

I know these messages by heart already thanks to Percona Server Bug #1131949. One of my configuration files had to have innodb_flush_method set to ALL_O_DIRECT:

openxs@ao756:~$ cat /etc/mysql/my.cnf
[mysqld]
innodb_flush_method = ALL_O_DIRECT
innodb_track_changed_pages = 1
innodb_log_file_size                    = 2G

After replacing it with usual O_DIRECT I was able to finally start the server as expected and execute mysql_upgrade:

openxs@ao756:~$ sudo mysqld_safe &
[1] 21711
openxs@ao756:~$ 160226 11:58:33 mysqld_safe Adding '/usr/lib/x86_64-linux-gnu/libjemalloc.so.1' to LD_PRELOAD for mysqld
160226 11:58:33 mysqld_safe Logging to '/var/lib/mysql/ao756.err'.
160226 11:58:33 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

openxs@ao756:~$ sudo mysql_upgrade -uroot -proot
mysql_upgrade: [Warning] Using a password on the command line interface can be insecure.
Checking if update is needed.
Checking server version.
Running queries to upgrade MySQL server.
...
Repairing tables
mysql.proxies_priv                                 OK
Upgrade process completed successfully.
Checking if update is needed.

openxs@ao756:~$ ps aux | grep mysqld
root     21711  0.0  0.0  74060  2200 pts/4    S    11:58   0:00 sudo mysqld_safe
root     21712  0.0  0.0   4440   760 pts/4    S    11:58   0:00 /bin/sh /usr/bin/mysqld_safe
mysql    21938  6.8 15.8 1066860 610576 pts/4  Sl   11:58   0:12 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/lib/mysql/ao756.err --pid-file=/var/lib/mysql/ao756.pid
openxs   22292  0.0  0.0  14636   972 pts/4    S+   12:01   0:00 grep --color=auto mysqld

Moreover, now it can be started as a service also (even though the output looks unusual):

openxs@ao756:~$ sudo service mysql start
No directory, logging in with HOME=/
....
 * Percona Server 5.7.10-3 is started

Next step was to enable TokuDB based on instructions I've already seen:

openxs@ao756:~$ sudo ps_tokudb_admin --enable -uroot -proot
Checking if Percona Server is running with jemalloc enabled...
INFO: Percona Server is running with jemalloc enabled.

Checking transparent huge pages status on the system...
INFO: Transparent huge pages are enabled (should be disabled).

Checking if thp-setting=never option is already set in config file...
INFO: Option thp-setting=never is not set in the config file.
      (needed only if THP is not disabled permanently on the system)

Checking TokuDB engine plugin status...
INFO: TokuDB engine plugin is not installed.

Disabling transparent huge pages for the current session...
INFO: Successfully disabled transparent huge pages for this session.

Adding thp-setting=never option into /etc/my.cnf
INFO: Successfully added thp-setting=never option into /etc/my.cnf

Installing TokuDB engine...
INFO: Successfully installed TokuDB engine plugin.

and create my first TokuDB table on Percona Server 5.7:

openxs@ao756:~$ mysql -uroot -proot testmysql: [Warning] Using a password on the command line interface can be insecure.
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.7.10-3-log Percona Server (GPL), Release '3', Revision '63dafaf'

Copyright (c) 2009-2015 Percona LLC and/or its affiliates
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create table tt(id int primary key) engine=tokudb;
Query OK, 0 rows affected (0.40 sec)

mysql> show create table tt\G
*************************** 1. row ***************************
       Table: tt
Create Table: CREATE TABLE `tt` (
  `id` int(11) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=TokuDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

As I want to check "full stack", I definitely need xtrabackup version 2.4 that is not installed by default (note the unusual package name for .deb packages, server has dot in the name before minor version):

openxs@ao756:~$ sudo apt-get install percona-xtrabackup-24
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libperconaserverclient18.1 percona-server-common-5.6
  percona-xtradb-cluster-galera-3.x socat
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  percona-xtrabackup
The following NEW packages will be installed:
  percona-xtrabackup-240 upgraded, 1 newly installed, 1 to remove and 3 not upgraded.
Need to get 6 084 kB of archives.
After this operation, 10,3 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://repo.percona.com/apt/ trusty/main percona-xtrabackup-24 amd64 2.4.1-1.trusty [6 084 kB]
Fetched 6 084 kB in 8s (691 kB/s)
(Reading database ... 801961 files and directories currently installed.)
Removing percona-xtrabackup (2.3.3-1.trusty) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Selecting previously unselected package percona-xtrabackup-24.
(Reading database ... 801947 files and directories currently installed.)
Preparing to unpack .../percona-xtrabackup-24_2.4.1-1.trusty_amd64.deb ...
Unpacking percona-xtrabackup-24 (2.4.1-1.trusty) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up percona-xtrabackup-24 (2.4.1-1.trusty) ...

openxs@ao756:~$ xtrabackup --version
xtrabackup version 2.4.1 based on MySQL server 5.7.10 Linux (x86_64) (revision id: a2dc9d4)

It seems at least in my case xtrabackup was not able to find the socket by default, but eventually it worked:

openxs@ao756:~$ sudo xtrabackup --user=root --password=root --backup
160226 12:08:45  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup' as 'root'  (using password: YES).
160226 12:08:45  version_check Connected to MySQL server
160226 12:08:45  version_check Executing a version check against the server...
160226 12:08:47  version_check Done.
160226 12:08:47 Connecting to MySQL server host: localhost, user: root, password: set, port: 0, socket: (null)
Failed to connect to MySQL server: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2).

openxs@ao756:~$ mysql -uroot -proot test -e "show variables like 'socket%'"
mysql: [Warning] Using a password on the command line interface can be insecure.
+---------------+-----------------------------+
| Variable_name | Value                       |
+---------------+-----------------------------+
| socket        | /var/run/mysqld/mysqld.sock |
+---------------+-----------------------------+

openxs@ao756:~$ sudo xtrabackup --user=root --password=root --socket=/var/run/mysqld/mysqld.sock  --backup
...
160226 12:11:43 Finished backing up non-InnoDB tables and files
160226 12:11:43 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...
xtrabackup: The latest check point (for incremental): '7481732856'
xtrabackup: Stopping log copying thread.
.160226 12:11:43 >> log scanned up to (7481732865)

160226 12:11:43 Executing UNLOCK TABLES
160226 12:11:43 All tables unlocked
160226 12:11:43 [00] Copying ib_buffer_pool to /home/openxs/xtrabackup_backupfiles/ib_buffer_pool
160226 12:11:43 [00]        ...done
160226 12:11:43 Backup created in directory '/home/openxs/xtrabackup_backupfiles/'
160226 12:11:43 [00] Writing backup-my.cnf
160226 12:11:43 [00]        ...done
160226 12:11:43 [00] Writing xtrabackup_info
160226 12:11:43 [00]        ...done
xtrabackup: Transaction log of lsn (7481732815) to (7481732865) was copied.
160226 12:11:44 completed OK!

I've also installed debug package to have symbols while working with gdb, and checked that pt-mysql-summary works:

openxs@ao756:~$ sudo apt-get install percona-server-5.7-dbg
...
Fetched 160 MB in 59s (2 702 kB/s)
Selecting previously unselected package percona-server-5.7-dbg.
(Reading database ... 801962 files and directories currently installed.)
Preparing to unpack .../percona-server-5.7-dbg_5.7.10-3-1.trusty_amd64.deb ...
Unpacking percona-server-5.7-dbg (5.7.10-3-1.trusty) ...
Setting up percona-server-5.7-dbg (5.7.10-3-1.trusty) ...
openxs@ao756:~$ pt-mysql-summary
# Percona Toolkit MySQL Summary Report #######################
              System time | 2016-02-26 10:17:09 UTC (local TZ: EET +0200)
# Instances ##################################################
  Port  Data Directory             Nice OOM Socket
  ===== ========================== ==== === ======
        /var/lib/mysql             0    0
# MySQL Executable ###########################################
       Path to executable | /usr/sbin/mysqld
              Has symbols | No
# Report On Port 3306 ########################################
                     User | @localhost
                     Time | 2016-02-26 12:17:08 (EET)
                 Hostname | ao756
                  Version | 5.7.10-3-log Percona Server (GPL), Release '3', Revision '63dafaf'
                 Built On | debian-linux-gnu x86_64
                  Started | 2016-02-26 12:02 (up 0+00:14:11)
                Databases | 2
                  Datadir | /var/lib/mysql/
                Processes | 1 connected, 1 running
              Replication | Is not a slave, has 0 slaves connected
                  Pidfile | /var/lib/mysql/ao756.pid (does not exist)...

As a last note, service can not be stopped properly:

openxs@ao756:~$ sudo service mysql stop
 * Percona Server 5.7.10-3 is already stopped
openxs@ao756:~$ ps aux | grep mysqld
mysql    24382  0.1  0.0   4440   760 ?        S    12:21   0:00 /bin/sh /usr/bin/mysqld_safe
mysql    24643  2.9  5.6 1234144 218252 ?      Sl   12:21   0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --log-error=/var/lib/mysql/ao756.err --pid-file=/var/lib/mysql/ao756.pid
openxs   24756  0.0  0.0  14636   972 pts/4    S+   12:21   0:00 grep --color=auto mysqld

so one has to use mysqladmin:

openxs@ao756:~$ sudo mysqladmin -uroot -proot shutdownmysqladmin: [Warning] Using a password on the command line interface can be insecure.

openxs@ao756:~$ ps aux | grep mysqld
openxs   24765  0.0  0.0  14636   972 pts/4    S+   12:21   0:00 grep --color=auto mysqld

To summarize my experience of a less naive user of Ubuntu 14.04 (the one who already upgraded to Percona Server 5.7.10-3 on other Linux versions), one should note the following:

1. Please, check all your configuration files carefully for any possible incompatibility with 5.7 before attempting upgrade.
2. Make sure ALL_O_DIRECT value is NOT used for innodb_flush_method. Check this my Facebook post for more details and discussion with Laurynas about the bugs involved (Percona Server Bug #1529885 and Percona Server Bug #1550404) and manual statements related.
3. If you need to use xtrabackup with Percona Server 5.7, you have to install percona-xtrabackup-24 explicitly and may have to specify --socket option explicitly.
4. I think /etc/init.d/mysql script from Percona Server 5.7.10-3 is somewhat broken on Ubuntu 14.04. Something to study in more details later.

I hope this step by step guide will help some of you to upgrade faster and provide some troubleshooting hints in case of problems.

MySQL Support People - Percona Support

I planned to continue this series of posts with the one about MySQL Support engineers who joined us in Oracle while I was working there, but based on recent events in my life I'd prefer to postpone it and move directly to the team I worked for during last 3+ years, Percona Support Team.

Disclamer: In the list below I still try to pay attention to public contribution by each of engineers mentioned to MySQL Community, mostly in a form of bug reports. This is not the only way they contributed, but I have to stick to this format in current series of posts. I also base my comments mostly on my memory and quick search of public sources of information, so I may be mistaking about the roles, time periods and other details. Please, inform me if you think there is something to fix.

So, here is the list of my former colleagues who formed the best MySQL Support team in the industry in 2013-2015 while working with me in Percona:

• Miguel Angel Nieto - Miguel was my mentor during my first days in Percona, so I have to start with him. He joined Percona in 2011 and was a team manager of EMEA Support team for a long time. We always managed to work together in a productive and efficient way. Miguel is a real expert in Percona XtraDB Cluster and almost every new technology we had to support, up to MongoDB recently. He used to act as a great consultant as well when Support engineers worked together with Consulting on the same pool of issues, and he still takes SSC (Support Shift Captain) shifts, so he can be the first person from Percona whom you deal with in case of problems even today. I see 13 public bug reports from Miguel and suggest to check his Bug #77654. See also his 12 bug reports for Percona software at  https://bugs.launchpad.net/~miguelangelnieto/+reportedbugs.
• Ovais Tariq - when I joined Percona Ovais was the only Principal Support Engineer in the team. He was really great in everything technical he did, from complex data recovery consulting cases to query optimization, and on top of that he was a really good writer. It was a pleasure to read every email in issues he worked on, it was like reading a good technical blog post if not a book. I see 16 public bug reports for MySQL by Ovais. Check his replication-related Bug #70923, for example. He had also reported 20 bugs for Percona software at https://bugs.launchpad.net/~ovais-tariq/+reportedbug. In 2013 Ovais moved to Consulting and a year later, in 2014, he quit from Percona to become a Lead MySQL DBA - Automation Engineer and later a Lead Reliability Engineer at Lithium. He is also active at TwinDB site.
• Fernando Ipar - As far as I know, he was the first full time Support engineer and dedicated Support Shift Captain in Percona since September, 2009. For two years later he was a Director of Global Support. When I joined he was working in Consulting and recently he seems to do Development. But during all this time Fernando was interested in Support-related discussions and helped my colleagues promptly when needed. I see 11 bugs reported by him for MySQL, including funny optimizer regression since 5.1.x, Bug #66825. Fernando also reported 10 bugs for Percona software at https://bugs.launchpad.net/~fipar/+reportedbugs.
• Michael Rikmas (a.k.a. Mixa) - he moved to Support from Administrative team in 2011, and there he was one of the first Percona employees. He always cared about Percona operations and customers, covering endless SSC shifts any time a day, sometimes for 12+ hours in a row. He was always ready to step into anything any time a day to help, if needed. While working in Support he improved his technical skills a lot in all MySQL-related areas, so he ended up as a good problem solver and eventually quit from Percona (in 2014) to join PSCE as a Consultant. I see only one bug he reported for MySQL, Bug #62426. See also https://bugs.launchpad.net/~michael.rikmas/+reportedbugs.
• Martin Arrieta - Martin joined Percona in January, 2012, but was already a well known Support engineer when I joined. In 2013 he moved to Consulting and two years later quit from Percona in September, 2015, to join Pythian as Database Consultant. Martin had reported 6 MySQL bugs in public (he was really interested in MySQL Fabric), but he is better known as a blog posts author. I think we always cooperated really well with him. 
• Marcos Albe (a.k.a. Markus) - he is the most hard working and successful Support engineer of all times in Percona. Marcos was also my team manager after I switched to AMER Support Team. Marcos joined Percona in 2010 and by 2012 he was not only a hard working SSC, but also a full stack expert. Marcos created many useful hints for the internal knowledge base and is also know as a great speaker at Percona Live conferences (I've met him for the first time at Percona Live New York, in 2012, where he had talks on several topics). The way he works (he is ready to call you using any media, join any chat, log in promptly and spend entire night helping any customer interactively) defines famous "Percona style" of providing Support (whatever I may think about it) and gave him huge experience with all technologies even remotely related to MySQL. Marcos had reported 4 public MySQL bugs, including a 5.6.x regression one, Bug #71616, that is still "Verified". I also see 5 bugs reported by him at https://bugs.launchpad.net/~markus-albe/+reportedbugs.
• Nickolay Ihalainen - Nickolay is a classical Percona Consultant (who still works in Support for a long time), with huge real life experience of managing MySQL at scale and full application stack troubleshooting. He knows a lot about Linux, hardware used with MySQL and, as any other great consultant in Percona, is always ready to step into the issue of any complexity and length, call customer, write code, recover InnoDB data or fix a bug quickly. He works in Percona since January 2010, reported 3 MySQL bugs in public, including Bug #78051 that is still "Verified". I also see 7 bugs reported by Nickolay for Percona software at https://bugs.launchpad.net/~ihanick/+reportedbugs. I studied a lot while working with and talking to him.
• Przemyslaw Malkowski - he joined Percona in August, 2012, just before me. We were at the same "bootcamp" session in famous "Hotel Percona" and worked closely together since that time. Przemek is a Principal Support Engineer since July, 2015. He is the only one who was promoted to that level while I was working in Percona, not just joined like that. He works hard on all kinds of complex issues and is one of key experts in the industry in Galera and Percona XtraDB Cluster (He can easily parse and understand obscure, huge logs from multiple nodes, and explain the interactions and problems  that happened, quickly. This is the skill I always missed). Moreover, he is a well known bug reporter, with 22 bugs reported for upstream MySQL, including infamous Bug #78777 that mislead numerous MySQL users and customers for months, and 37 (!) bugs reported for Percona software at https://bugs.launchpad.net/~pmalkowski/+reportedbugs. Przemek is able to work with any kind of annoying customer, on issues of any complexity, and still remain helpful. I really enjoyed working with him in one team, a dream team actually!
• Nilnandan Joshi (a.k.a. Nil) - he joined Percona in May, 2012 and successfully worked there as Support Engineer till January, 2016. He managed to master all key tools and technologies in MySQL, while working hard on boring duties like SSC and bugs processing. Now Nil is a Big Data Engineer at zData Inc and is probably working with Hadoop or something of that kind. He also was visiting "Hotel Percona" with me in 2012 for a "bootcamp", where we spent a lot of time talking about the way InnoDB works (and drinking some beer). Since that time we were working in a close cooperation till his last day in Percona. Nil helped me a lot with bugs processing when I've got a task to manage it from Support side, and was one of few engineers in Support who were ready to report or properly process/verify any bug, build any Percona software, test it on all kinds of weird Linux versions and distros, using all kinds of virtual and real machines. Nil reported 13 public MySQL bugs, including Bug #79469. I also see 9 bugs reported by him for Percona software at https://bugs.launchpad.net/~nilnandan-joshi/+reportedbugs. I miss our useful work together on bugs and Percona Live sessions already...
• Jervin Real - he worked in Percona Support since early 2010 and until summer 2013, when he moved to Consulting (he worked great there as well, always ready to step in when we needed help in Support). When I joined he was a key Support engineer for APAC customers. Jervin is a well known blogger, who writes a lot about HA and other technologies for MySQL. I finally met him in real life few days ago, at FOSDEM conference, where he was speaking about TokuDB. Now Jervin is a Technical Service Manager in Percona. He had contributed a lot to MySQL Community also with his 20 public bug reports, including Bug #77715 that is still "Open". I also see 34 bugs reported by Jervin for Percona software at https://bugs.launchpad.net/~revin/+reportedbugs.
• Jaime Sicam - Jaime joined Percona Support in June, 2011 and later became a good manager of APAC team in Support. He always worked hard and helped colleagues with anything, from covering SSC shift to complex troubleshooting. He is a great engineer capable to deal with any kind of issues, but he was our key expert in anything related to PAM and authentication in general. Jaime reported 7 bugs for MySQL software, including infamous Bug #77344, and 11 bugs for Percona software at https://bugs.launchpad.net/~jssicam/+reportedbugs. We rarely were both online and working together, but I already miss him.
• Muhammad Irfan - he joined Percona Support soon after me, in December 2012. Muhammad actively started to work and cooperate with me and other senior colleagues from the beginning, and soon had become a high skilled engineer in everything related to MySQL, including XtraDB Cluster and bugs processing. He is a quite famous blogger, who was happily writing about new features, tools and best practices (while I prefer to write about bugs and problems, that is, worst practices). I am proud to get a change to help him with some posts as a reviewer. Muhammad spent fair amount of time on bugs processing and reporting as well, as we can conclude from his 8 MySQL bug reports including "Verified" Bug #73094 and 9 bugs reported for Percona software at https://bugs.launchpad.net/~muhammad-irfan/+reportedbugs.
• Fernando Laudares (a.k.a. Nando) - he joined Percona in January, 2013 and worked in AMER Support Team. I'd call him a student of Marcos, as he quickly worked out similar habits and approaches to Support, "Percona style". Fernando always tried his best to help both customers and colleagues, without any limits, and this quickly made him one of the key team members. He works hard, on many issues, day after day without becoming tired in a visible way. Fernando had written many great blog posts and is great in all kinds of automation/testing environments setups. He was fast in studying, accepting and following new technologies we were supposed to support in Percona. His community contributions also include recent enough Bug #77684 and 2 bugs reported at https://bugs.launchpad.net/~fernando-laudares/+reportedbugs.
• Jericho Rivera - with mostly sysadmin and developer background, Jericho had to work really hard since October, 2013 to reach the level of MySQL expertise we required in Percona Support. But he did that and became a key support provider in 2015, that can deal efficiently with all kinds of customer issues. He helped us to manage Support servers all this time and became a local expert in Docker and other technologies (as you can figure out from his blog posts). He had reported Bug #77073 for MySQL server (still "Verified") and 3 bugs for Percona software at https://bugs.launchpad.net/~jericho-rivera/+reportedbugs. It was really great to see how he becomes better every month, until he ended up just doing an awesome job one day.
• Peiran Song - I was really happy when she joined us in 2014. By May 2014, just in a couple of months, she already did a great job in Support, working on many complex issues. She was really good with troubleshooting, InnoDB and query optimization. We discussed not only technical details, but also procedures and values of Support, a lot. Peiran reported 6 upstream MySQL bugs, including incredible Bug #73369 that is still waiting for explanation and fix, and 8 bugs for Percona software at https://bugs.launchpad.net/~peiran-song/+reportedbugs. It was really sad to meet her in 2015 during Percona Live just to find out that she can not stay with us any more (and not because of any her faults). Peiran is a Data Architect at Smule, Inc now.
• Justin Swanhart - Justin worked for Percona at different positions (mostly as a Consultant and Instructor), since 2010. In August 2013 he joined Support as a Principal Support Engineer, and since that time we often worked together, in the same AMER team. He had got the Community Contributor of the year 2015 award, for his great contributions including (but not limited to) his software. Justin created 40 MySQL community bug reports, including Bug #76210, and I see 12 bugs reported by him for Percona software at https://bugs.launchpad.net/~greenlion/+reportedbugs. Justin worked well, if you ask me, and helped customers a lot, while being a source of unique expertise and great insights for colleagues and customers (based on his trainer and software developer experience, among other things), but one sad day in June 2015 he found himself fired in the middle of the shift. I tried to find out what was really that much wrong with his work, and found literally nothing myself, so I let others to comment on what happened and why. I think firing Justin was a very bad mistake, even though he had better NOT to comment on it in public in the way he did... I hope we'll work together one day again, maybe even soon.
• Bill Karwin - Bill worked for Percona since 2010, mostly as a Consultant and Trainer. We were really lucky that he joined Support in 2014 full time (as a Principal Support Engineer and Senior Knowledge Manager). Using his huge experience in SQL and everything MySQL, he started to play a key role in AMER team, working a lot and providing great service to customers. He is very famous for his answers at Quora and StackOverflow. He is a great writer and speaker on MySQL topics, and his book on SQL anti-patters is well known. Bill is literally second to nobody in helping MySQL Community. Check also his 9 MySQL bugs including Bug #73283 (still "Verified"), and 2 bugs at https://bugs.launchpad.net/~bill-karwin/+reportedbugs. At the end of 2014 Bill left us, probably because it was boring for him to resolve the same problems again and again for customers, instead of doing things right from the very beginning. He is now a Senior Database Architect and Software Engineer at SchoolMessenger.
• Agustin Gallego (a.k.a. Guli) - I met him first in 2012, at Percona Live New York. He worked for Percona since February, 2012, but joined Support later, in December, 2013. Since that time Agustin worked as SSC and Support Engineer in AMER Support team and played an important role in making it operational and efficient. He had reported Bug #77186 for MySQL and 6 bugs for Percona software at https://bugs.launchpad.net/~agustin-gallego/+reportedbugs. He is gaining experience every day, cooperates well with colleagues and now is a very reliable, respected and useful team member.
• Roben Paul Namuag - Paul joined Percona Support in July, 2013. In January, 2015 he left us to continue working in Percona as Remote DBA, but we still discuss technical details and, sometimes, life after that. Paul had good and harder times in Support, but he always tried to do his best and played an important role in APAC team, both as SSC and as an experienced engineer working on complex enough issues, even when there was nobody around to help. He had reported 3 MySQL bugs, including Bug #77180 that is till "Open", and 2 bugs for Percona software at https://bugs.launchpad.net/~paul-namuag/+reportedbugs. I'll remember him as a kind, helpful and friendly person.
• Akshay Suryavanshi - we chatted with with Akshay a lot before he joined Percona, he was interested in a lot of in-depth details on how MySQL works. Eventually he joined Percona Support, in March 2013, but immediately switched to Remote DBA role and had a great success there, moving later to management positions, up to DBA Team Lead and Senior position in Technical Operations recently. Then, in December 2015, he quit from Percona, to become a MySQL Engineer at Tumblr. He helped community with his blog posts and webinars, and also reported Bug #70818
• Pablo Padua - he joined Percona in 2014 as a Jr. Support Engineer, and made a lot of progress since that. Unfortunately I can not find any public MySQL bug reports by him, but he worked a lot as SSC in AMER Support Team, and he gets more experience every day. We often worked together on technical issues, when I just provided some background help, checks and comments, while he did real time work for customer. It had been a pleasure to cooperate that way, and, as a result, for many areas in MySQL Pablo now is ready to work without any assistance.
• Abdelhak Errami - he joined us as Senior Support Engineer in April 2015, when Tokutek was acquired. Since that time he played a key role in TokuDB Support, but also often helped with pure MySQL issues as well. We spent a good time together during Percona Live. Abdel often works on TokuDB bugs, check DB-922 as one of recent examples.
• Joe Laflamme - I think he also joined us some time in 2015 in the process of Tokutek acquisition. Check DB-809 for one of his TokuDB related bug reports. He coordinated TokuDB bug fixing and support during the period when I describe. Now he is a manager at Percona Technical Services.

I had not mentioned Sveta Smirnova, who joined Percona in March 2015, because she was already mentioned in the previous post. I still plan to write about managers and coordinators of Support who had never reported MySQL bugs in public separately, so stay tuned Tom Basil, Peter Farkas and Erzsebet Olsovzsky. I remember about you and I value your contribution a lot, but engineers first...

I had not named colleagues in Percona who joined after November 1, 2015 explicitly here, and this is the date of the beginning of the entirely new epoch in the history of Percona that should be described separately, if ever.

My Guidebook for Percona Live Amsterdam, 2015 - Part II

I see on Facebook that many of my friends are already on their way to Amsterdam for Percona Live Amsterdam 2015 that starts on Monday. As I explained, I am not going there, but I'd like to share my ideas on what presentations are worth attending.

So, the program for the last day, September 23, also looks really interesting and I'd have real problems on where to go for most of the slots. As usual, I'd try to listen to all keynotes, but I am especially interested in The Virtues of Boring Technology as I had not seen my former MySQL AB's colleague Kristian Köhntopp (now from Booking.com) for a long time already. The panel discussion after it, on The next disruptive technology, should be also interesting.

After short break I'd probably go to one of these sessions:

• Massively Distributed Backup at Facebook Scale - I like to listen to Shlomo Priymak at the conferences and it's just interesting to find out how Facebook does MySQL backups these days
• ProxySQL : High Availability and High Performance Proxy for MySQL - the more HA solutions to choose from, the better, so I'd like to see what my former colleague René Cannaò came up with

But I would not be surprised to find myself listening about MariaDB 10.1 as well...

Next slot is a real problem. I never miss a chance to find out what new performance records Dimitrii Kravtchuk managed to get from poor old MySQL, so
MySQL 5.7 Performance: Scalability & Benchmarks is a must. But I also try to visit all presentations on replication by Luis Soares from Oracle as well, so I'd hardly could afford to miss
The Latest and Greatest MySQL 5.7 Replication Features and More! But these sessions happen at the same time. I'd get a real problem if I'd go to the conference, so lucky I am that I stay at home and can spend time cutting good oak wood. Winter is coming, you know...

After lunch I'd go to listen to my colleague Vlad Lesin, who speaks about TokuDB internals. I've already attended a couple of presentations on fractal trees and TokuDB performance achievements, and surely I can always ask a question to my colleagues in Support who previously worked in Tokutek, but I really wonder what Vlad thinks about the code. He is one of the best developers in Percona and I am impressed by his analysis and work on several MySQL bugs, so I'd consider this session a real opportunity to find out what we can expect from TokuDB in production.

Next slot is also not a problem for me. I'd like to be at the Non Blocking DDL in Galera Cluster session by Seppo Jaakola from Codership. "Online" DDL for a single MySQL instance is already a problem, so doing it in a truly non-blocking manner on Galera/PXC cluster requires something new. I'd like to know more about this feature of Galera 4.0.

After that I'd either got to listen to LeFred explaining his hacks at Undelete rows from the binary log a hacking session (as far as I remember he presented something similar on FOSDEM 2015) or, if I'd like to speak more than to listen, I'd just go and tell Oracle engineers what MySQL still misses! 
MySQL 5.8 Dreaming and Brainstorming is the ideal session to do this. Personally I want constraints checks at transaction boundaries (not for each row), fully cost-based optimizer, ALTER that is really online/non-blocking, and war in Ukraine stopped. Guess what of these will happen at the moment of MySQL 5.8 GA...



Lightning Talks are always fun, so I'd go to listen to them next.

For the last slot I'd have to choose one of the following sessions:

• MySQL at Wikipedia: How we do relational data at the Wikimedia Foundation - yet another case of web-scale MySQL usage presented by my former colleague from Percona (Jaime Crespo), and MySQL AB (Sean Pringle). I miss working with Sean a lot, he was a key member of "dream team" that provided MySQL support during APAC hours in MySQL AB. I stayed online and working more than once to cover APAC hours, as it was a real pleasure to work with this team!
• At the same time, for the very last slot, we have Jeremy and Davi speaking about InnoDB data storage structures again at the session InnoDB: A hands-on exploration of on-disk storage with innodb_ruby. Surely I can just try myself and real great blog posts, but I never miss their talks... What shell I do? 

I shell keep cutting oak wood and help Percona customers, but you, those who plan to attend Percona Live Amsterdam, are in real troubles!