MySQL Support Engineer's Chronicles, Issue #4

This week I had to deal with some unusual problems. But let me start with Percona's xtrabackup, software that I consider a key component of many current production MySQL setups and use regularly. Recently new minor versions of XtraBackup were released, check the details on 2.4.5, for example. It made a step towards support of MariaDB 10.2, but it's still a long way to go, see this pull request #200.

My main problem with xtrabackup, though, is not with lack of support of MariaDB 10,2-specific features. Why should they care, after all... The problem is that old well known bugs and problems are not resolved, those that may affect all MySQL versions, forks and environments. Check lp:1272329 , "innobackupex dies if a directory is not readable. i.e.: lost+found", for example. Why not to read and take into account ignore_db_dir option (as server does) and let those poor souls who used mount point as a datadir to make backups? Check even older problem, passwords that are not hidden in the command lines, see lp:907280, "innobackupex script shows the password in the ps output, when its passed as a command line argument". My colleague Hartmut even suggested the fix recently, see pull request #289.

Because of these old, known problems (some of them being low hanging fruits) that are not fixed users still suffer while using xtrabackup way more often than they would like to. One day, as a result, they'll have to switch to some other online backup tools or approaches. One may dream about extended myrocks_hotbackup to cover InnoDB one day (when MyRocks and InnoDB will work together in one instance), or just use Percona TokuBackup (after adding script to go SST for Galera with it, maybe), or try something totally different. Anyway, I feel that if more bugs (including low hanging fruits) in xtrabackup are not getting fixed and pull requests are not actively accepted, the tool may become much less relevant and used soon.

I had to deal with MaxScale-related issues this week, so I'd like to remind those who use Docker for testing about https://github.com/asosso/maxscale-docker/blob/master/Dockerfile. Personally I prefer to build from source. In any case, I'd like us all to remember that in older versions one may have to set strip_db_esc option explicitly for service to deal with database names containing underscore (_). Recent 2.0.x versions have it enabled by default (see MXS-801).

I also had to explain how online ALTER TABLE works, specifically, when it sets exclusive metadata locks in the process. I still do not see this topic properly explained in the manual, so I had to report Bug #84004, "Manual misses details on MDL locks set and released for online ALTER TABLE".


By no means I am a developer for 11 years already, even less one should expect writing Java code from me. Anyway, I had to explain how to replace Oracle's ref_cursors (a.k.a cursor variables) in MySQL, both in stored procedures and in Java code that calls them. If you are wondering what is this about, check this fine manual. Note that this feature is missing in MySQL, even though it was suggested to implement it here. In general, MySQL allows just to run SELECTs in stored procedures and then in Java you can process each of the result sets returned any way you want. Things may get more complicated when more than one result set is produced, and they are even more complicated in Oracle with nested cursor expressions. So, I plan to devote a separate blog post to this topic one day. Stay tuned.

I answer questions coming not only from customers. Old friends, community users out of nowhere and, even more, colleagues are welcomed to discuss whatever MySQL- or MariaDB-related  problem they may have. If I know how to help, I'll do this, otherwise I'll quickly explain that I am of no good use. This is how I ended up testing MariaDB's CONNECT storage engine quickly as a replacement for the Federated engine, that is, to link table to a remote MySQL table. Basic instructions on how to set it up and use MySQL type looked simple, but when I tried to test on Fedora 23 and hit a problem of missing libodbc.so.1:

MariaDB [(none)]> INSTALL SONAME 'ha_connect';
ERROR 1126 (HY000): Can't open shared library '/home/openxs/dbs/mariadb10.1/lib/plugin/ha_connect.so'
  (errno: 2, libodbc.so.1: cannot open shared object file: No such file or directory)

the solution was not really straightforward. First of all I had to install unixODBC.x86_64 2.3.4-1.fc23 RPM, but it also does not provide libodbc.so.1:

[openxs@fc23 node2]$ find / -name libodbc.* 2>/dev/null
/usr/lib64/libodbc.so.2
/usr/lib64/libodbc.so
/usr/lib64/libodbc.so.2.0.0

So, I had to apply a quick and dirty hack:

[openxs@fc23 node2]$ sudo ln -s /usr/lib64/libodbc.so.2.0.0  /usr/lib64/libodbc.so.1

As a result CONNECT engine worked as expected, as long as proper account and IP-address where used:

MariaDB [test]> INSTALL SONAME 'ha_connect';
Query OK, 0 rows affected (0.27 sec)

MariaDB [test]> create table r_t2(id int primary key, c1 int) engine=connect table_type=mysql connection='mysql://msandbox:msandbox@127.0.0.1:23532/test/t';
Query OK, 0 rows affected (0.04 sec)

MariaDB [test]> select * from r_t2;                     

+----+------+
| id | c1   |
+----+------+
|  1 |    2 |
|  2 |    3 |
+----+------+
2 rows in set (0.00 sec)

From configuring MaxScale to work with database having underscore in the name to re-writing Java code that used to work with Oracle RDBMS for MySQL, with many missing details in the manuals and software bugs identified or reported in between, and all that with ongoing studies of performance problems and lost quorums, rolling upgrades and failed SSTs in Galera clusters - this is what support engineers here in MariaDB have to deal with during a typical working week.

Upstart Basics for Automatic Restarts of MaxScale After Crash

Historically I do not care much about MariaDB's MaxScale, at least since I know how to build it from source when needed. But, as a support engineer who work at MariaDB, sometimes I have to deal with problems related to MaxScale, and this week it happened so that I had to explain what to do to implement automatic restarts of MaxScale "daemon" in case of crashes on RHEL 6.x.

In the process I had found out that two of my most often used Linux distributions, CentOS 6.x and Ubuntu 14.04, actually use Upstart, so good old System V's init tricks and scripts work there only partially and only because somebody cared to integrate them into this "new" approach to starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running. I had to do this years ago, but customer's question finally forced me to check and study some details on how this system actually works.

So, unfortunately, there is no script like mysqld_safe to start and restart MaxScale after installing official RPM from MariaDB (in this case it was maxscale-2.0.1-2.x86_64). My first idea was to write one, but then I asked myself why it is not there yet, decided to check what's already provided and take a closer look at /etc/init.d/maxscale I have on my CentOS 6.8 VM (the closest to what customer used). It's a typical classic shell script to start service and it starts MaxScale like this:

start() {
    echo -n $"Starting MaxScale: "
...
    ulimit -HSn 65535
    daemon --pidfile $MAXSCALE_PIDFILE /usr/bin/maxscale --user=maxscale $MAXSCALE_OPTIONS >& /dev/null

    RETVAL=$?
    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$servicename
...
    # Return right code
    if [ $RETVAL -ne 0 ]; then
       failure
       RETVAL=$_RETVAL_NOT_RUNNING
    fi

    echo

    return $RETVAL
}

Basically, it runs /usr/bin/maxscale --user=maxscale, the rest are details (like location of PID file the rest of the script then relies on). There is no advanced script to monitor status of this process with this PID (like mysqld_safe) or anything to care about automatic restarts. It is supposed that I just execute chkconfig maxscale on and then service starts when system enters proper runlevel.

Simple test shows that when maxscale process is killed it's gone and is not restarted. At good old times I'd just add something like this:

mx:2345:respawn:/usr/bin/maxscale --user=maxscale

to /etc/inittab (as nice articles like this suggests), but quick check and then further reading proves that it's not going to work on CentOS 6.8, as it uses Upstart.

So, either I had to write something similar to mysqld_safe for MaxScale, or (having in mind size of code and number of bugs we had in the past in that script) I had better to find out what is the supposed way to respawn processes in Upstart. Basic ideas are, again, simple. One has to create /etc/init/service_name.conf file and put something like this there (real code quote from one of Ubuntu files for MySQL):

start on runlevel [2345]
stop on starting rc RUNLEVEL=[016]

respawn
respawn limit 2 5
...
pre-start script
...
end script

exec /usr/sbin/mysqld

post-start script
...
end script

The file is easy to understand even without reading the manual. One has to set when service starts and stops, add respawn clause if we want to restart it in case of unexpected crashes or kills of the process, optionally limit the number of restarts and intervals between restarts etc, and, optionally, do something before and after start.

I quickly created /etc/init/maxscale.conf based on the above and it did a great job in starting it automatically upon system startup. I've just used exec /usr/bin/maxscale --user=maxscale basically and decided to deal with options and other details later if needed. But what was strange for the very beginning is that in /var/log/messages I've seen what looked like repeated attempts to start maxscale process, that failed:

Nov 26 17:44:45 centos maxscale[20229]: MaxScale started with 1 server threads.
Nov 26 17:44:45 centos init: maxscale main process ended, respawning
Nov 26 17:44:45 centos maxscale[20229]: Started MaxScale log flusher.
Nov 26 17:44:45 centos maxscale[20235]: Working directory: /var/log/maxscale
Nov 26 17:44:45 centos maxscale[20235]: MariaDB MaxScale 2.0.1 started
Nov 26 17:44:45 centos maxscale[20235]: MaxScale is running in process 20235
...

Nov 26 17:44:45 centos maxscale[20235]: Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so
Nov 26 17:44:45 centos maxscale[20235]: MaxScale is already running. Process id: 20229. Use another location for the PID file to run multiple instances of MaxScale on the same machine.
Nov 26 17:44:45 centos init: maxscale main process (20234) terminated with status 4
Nov 26 17:44:45 centos init: maxscale main process ended, respawning

Moreover, when proper maxscale process was killed, it was NOT respawned as expected.

It was just a proper time to read the manual more carefully, this part about expect stanza (that I noted in some of official Upstart scripts):

"To allow Upstart to determine the final process ID for a job, it needs to know how many times that process will call fork(2). Upstart itself cannot know the answer to this question since once a daemon is running, it could then fork a number of "worker" processes which could themselves fork any number of times. Upstart cannot be expected to know which PID is the "master" in this case, considering it does not know if worker processes will be created at all, let alone how many times, or how many times the process will fork initially. As such, it is necessary to tell Upstart which PID is the "master" or parent PID. This is achieved using the expect stanza.
The syntax is simple, but you do need to know how many times your service forks."

Let's check quickly how many times fork() is called in maxscale (I'd know it better if I ever cared to study the source code in details, but I had not checked most part of it yet). Test based on that cookbook gives unexpected results:

[root@centos ~]# strace -o /tmp/strace.log -fFv maxscale &
[1] 2369
[root@centos ~]# sleep 10
[root@centos ~]# ps aux | grep strace
root      2369  2.6  0.0   4476   888 pts/0    S    20:28   0:00 strace -o /tmp/strace.log -fFv maxscale
root      2382  0.0  0.0 103312   868 pts/0    S+   20:28   0:00 grep strace
[root@centos ~]# pkill -9 strace
[1]+  Killed                  strace -o /tmp/strace.log -fFv maxscale
[root@centos ~]# ps aux | grep maxscale
root      2375  1.3  0.2 276168  3896 ?        Ssl  20:28   0:00 maxscale
root      2385  0.0  0.0 103312   868 pts/0    S+   20:28   0:00 grep maxscale
[root@centos ~]# egrep "\<(fork|clone)\>\(" /tmp/strace.log | wc | awk '{print $1}'
5

How comes we have 5 fork calls? Here they are:

[root@centos ~]# egrep "\<(fork|clone)\>\(" /tmp/strace.log
2374  clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb024c08ab0) = 2375
2375  clone(child_stack=0x7fb01f819f10, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7fb01f81a9d0, tls=0x7fb01f81a700, child_tidptr=0x7fb01f81a9d0) = 2376
2375  clone(child_stack=0x7fb01e118f10, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7fb01e1199d0, tls=0x7fb01e119700, child_tidptr=0x7fb01e1199d0) = 2377
2375  clone(child_stack=0x7fb01d10af10, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7fb01d10b9d0, tls=0x7fb01d10b700, child_tidptr=0x7fb01d10b9d0) = 2378
2375  clone(child_stack=0x7fb017ffef10, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7fb017fff9d0, tls=0x7fb017fff700, child_tidptr=0x7fb017fff9d0) = 2379

It seems process with PID 2374 actually forked just once, to produce PID 2375, and then several threads were started by that process. We can see them:

[root@centos ~]# ps -T -p `pidof maxscale`
  PID  SPID TTY          TIME CMD
 2375  2375 ?        00:00:00 maxscale
 2375  2376 ?        00:00:00 maxscale
 2375  2377 ?        00:00:00 maxscale
 2375  2378 ?        00:00:00 maxscale
 2375  2379 ?        00:00:00 maxscale
[root@centos ~]#

So, it was really one fork() (and I could notify that even from studying /var/log/messages before) and I have to add expect fork stanza to my Upstart configuration file:

[root@centos ~]# cat /etc/init/maxscale.conf
# MaxScale service

description "MaxScale"

start on stopped rc RUNLEVEL=[2345]
stop on starting rc runlevel [!2345]

respawn
respawn limit 2 5

expect fork

exec /usr/bin/maxscale --user=maxscale

This way it works as expected, as one may easily check:

[root@centos ~]# initctl status maxscale
maxscale start/running, process 6600
[root@centos ~]# maxadmin
MaxScale> show servers
Server 0x19e47a0 (server1)
        Server:                              127.0.0.1
        Status:                              Master, Running
        Protocol:                            MySQLBackend
        Port:                                3306
        Server Version:                      5.7.15-9-log
        Node Id:                             1
        Master Id:                           -1
        Slave Ids:
        Repl Depth:                          0
        Number of connections:               0
        Current no. of conns:                0
        Current no. of operations:           0
MaxScale> quit
[root@centos ~]# kill -9 6600
[root@centos ~]# ps aux | grep maxscale
maxscale  6627  2.0  0.2 276168  3884 ?        Ssl  20:41   0:00 /usr/bin/maxscale --user=maxscale
root      6633  0.0  0.0 103312   872 pts/0    S+   20:41   0:00 grep maxscale
[root@centos ~]# initctl status maxscale
maxscale start/running, process 6627

In the /var/log/messages we clearly see that the process is respawned by init:

...
Nov 26 20:38:15 centos maxscale[6600]: Started MaxScale log flusher.
Nov 26 20:40:33 centos maxscale[6600]: Loaded module MaxAdminAuth: V2.0.0 from /usr/lib64/maxscale/libMaxAdminAuth.so
Nov 26 20:41:52 centos init: maxscale main process (6600) killed by KILL signal
Nov 26 20:41:52 centos init: maxscale main process ended, respawning
Nov 26 20:41:52 centos maxscale[6627]: Working directory: /var/log/maxscale
Nov 26 20:42:07 centos maxscale[6627]: Loaded module MaxAdminAuth: V2.0.0 from /usr/lib64/maxscale/libMaxAdminAuth.so
...

After few more checks I asked to implement this officially in packages for the Linux distributions that use Upstart, see MXS-1027.

To summarize, I wish I care more to find out how Upstart works long time ago. Now it's time to study systemd probably :) Anyway, after some reading and testing one can use it efficiently to provide automated service starts and restarts for MySQL server and services used with it.

On Upgrades to Percona Server 5.7 GA: Building from Git Source on Fedora

While writing my first post in this series I was bulding Percona Server 5.7 from Git sources on my wife's Fedora Core 23 box based on this manual page, for my collection of test instances. As I build MySQL and related software from sources on different platforms almost every day for 11 years already, the process was completed successfully without much troubles, but there are still some details I'd like to highlight.

Manual says one has to do the following:

$ git clone https://github.com/percona/percona-server.git
$ cd percona-server
$ git checkout 5.7
$ git submodule init
$ git submodule update

I had source code clonned already for regular 5.6.x builds, so I did this instead:

[openxs@fc23 percona-server]$ git pull
Already up-to-date.
[openxs@fc23 percona-server]$ git checkout 5.7
Checking out files: 100% (20321/20321), done.
M       plugin/tokudb-backup-plugin/Percona-TokuBackup
M       storage/tokudb/PerconaFT
Branch 5.7 set up to track remote branch 5.7 from origin.
Switched to a new branch '5.7'
[openxs@fc23 percona-server]$ git submodule init
[openxs@fc23 percona-server]$ git submodule update

On this FC 23 box I've already built MySQL 5.7, MariaDB 10.1, Facebook's MySQL 5.6 and Percona Server 5.6 from Git sources more than once, and I've updated all packages today and even built Facebook's 5.6 code (for testing MyRocks further) after that successfully, so I expect everything should work really well.

I put my test instances into dbs subdirectory usually:

[openxs@fc23 percona-server]$ ls ~/dbs
5.7  fb56  maria10.1  p5.6

So, I've modified cmake command line a bit comparing to the one recommended in the manual:

cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DWITH_EMBEDDED_SERVER=OFF

as you can see below:

[openxs@fc23 percona-server]$ cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DWITH_EMBEDDED_SERVER=OFF -DCMAKE_INSTALL_PREFIX=/home/openxs/dbs/p5.7
-- Running cmake version 3.4.1
-- Configuring with MAX_INDEXES = 64
-- Looking for SHM_HUGETLB
-- Looking for SHM_HUGETLB - found
-- SIZEOF_VOIDP 8
-- MySQL 5.7.10-3
-- Packaging as: percona-server-5.7.10-3-Linux-x86_64
-- Looked for boost/version.hpp in  and
-- BOOST_INCLUDE_DIR BOOST_INCLUDE_DIR-NOTFOUND
-- LOCAL_BOOST_DIR
-- LOCAL_BOOST_ZIP
-- Could not find (the correct version of) boost.
-- MySQL currently requires boost_1_59_0

CMake Error at cmake/boost.cmake:76 (MESSAGE):
  You can download it with -DDOWNLOAD_BOOST=1 -DWITH_BOOST=<directory>

  This CMake script will look for boost in <directory>.  If it is not there,
  it will download and unpack it (in that directory) for you.

  If you are inside a firewall, you may need to use an http proxy:

  export http_proxy=http://example.com:80

Call Stack (most recent call first):
  cmake/boost.cmake:228 (COULD_NOT_FIND_BOOST)
  CMakeLists.txt:439 (INCLUDE)


-- Configuring incomplete, errors occurred!
See also "/home/openxs/git/percona-server/CMakeFiles/CMakeOutput.log".
See also "/home/openxs/git/percona-server/CMakeFiles/CMakeError.log".

As you can see, manual obviously misses some details, like adding options related to Boost library. We see nice hints on what to do, and I've already downloaded Boost once into ~/boost while building MySQL 5.7. If Percona Server 5.7 was based on 5.7.11, we could use -DWITH_BOOST=system, see the release notes:

"A value of system is now permitted for the WITH_BOOST CMake option. If this option is not set or is set to system, it is assumed that the correct version of Boost is installed on the compilation host in the standard location. In this case, the installed version of Boost is used rather than any version included with a MySQL source distribution. (Bug #22224313)"

But it's based on 5.7.10, so I've modified my command like as follows:

[openxs@fc23 percona-server]$ cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DWITH_EMBEDDED_SERVER=OFF -DDOWNLOAD_BOOST=1 -DWITH_BOOST=/home/openxs/boost -DCMAKE_INSTALL_PREFIX=/home/openxs/dbs/p5.7
...
-- Check if the system is big endian - little endian
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
  Could NOT find ZLIB (missing: ZLIB_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake/Modules/FindZLIB.cmake:124 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  libbinlogstandalone/CMakeLists.txt:30 (FIND_PACKAGE)


-- Configuring incomplete, errors occurred!
See also "/home/openxs/git/percona-server/CMakeFiles/CMakeOutput.log".
See also "/home/openxs/git/percona-server/CMakeFiles/CMakeError.log".

to end up with another error. I was surprised a bit, as I do have zlib-devel installed:

[openxs@fc23 percona-server]$ rpm -qa | grep zlib
zlib-1.2.8-9.fc23.x86_64
zlib-devel-1.2.8-9.fc23.x86_64

To not waste time on zlib now I've used the "trick" from Facebook's wiki this time:

[openxs@fc23 percona-server]$ cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DWITH_EMBEDDED_SERVER=OFF -DDOWNLOAD_BOOST=1 -DWITH_BOOST=/home/openxs/boost -DWITH_ZLIB=bundled -DCMAKE_INSTALL_PREFIX=/home/openxs/dbs/p5.7
...
-- MySQL 5.7.10-3
-- Packaging as: percona-server-5.7.10-3-Linux-x86_64
-- Local boost dir /home/openxs/boost/boost_1_59_0
-- Local boost zip /home/openxs/boost/boost_1_59_0.tar.gz
...
-- Library perconaserverclient depends on OSLIBS -lpthread;m;rt;/usr/lib64/libssl.so;/usr/lib64/libcrypto.so;dl
-- Performing Test HAVE_NO_UNUSED_TYPEDEFS
-- Performing Test HAVE_NO_UNUSED_TYPEDEFS - Success
-- Performing Test HAVE_NO_LOGICAL_OP
-- Performing Test HAVE_NO_LOGICAL_OP - Success
-- INSTALL perconaserverclient.pc lib/pkgconfig
-- CMAKE_BUILD_TYPE: RelWithDebInfo
-- COMPILE_DEFINITIONS: _GNU_SOURCE;_FILE_OFFSET_BITS=64;HAVE_CONFIG_H
-- CMAKE_C_FLAGS:  -Wall -Wextra -Wformat-security -Wvla -Wwrite-strings -Wdeclaration-after-statement
-- CMAKE_CXX_FLAGS:  -Wall -Wextra -Wformat-security -Wvla -Woverloaded-virtual -Wno-unused-parameter
-- CMAKE_C_FLAGS_RELWITHDEBINFO: -O3 -D_FORTIFY_SOURCE=2 -g -fabi-version=2 -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF
-- CMAKE_CXX_FLAGS_RELWITHDEBINFO: -O3 -D_FORTIFY_SOURCE=2 -g -fabi-version=2 -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF
-- Configuring done
CMake Warning (dev) at storage/tokudb/PerconaFT/portability/CMakeLists.txt:22 (add_dependencies):
  Policy CMP0046 is not set: Error on non-existent dependency in
  add_dependencies.  Run "cmake --help-policy CMP0046" for policy details.
  Use the cmake_policy command to set the policy and suppress this warning.

  The dependency target "build_jemalloc" of target
  "tokuportability_static_conv" does not exist.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /home/openxs/git/percona-server

to end up with cmake command completed sucessfully. Now, time to make it (my FC23 box I've assembled myself back in 2011 has single QuadCore CPU):

[openxs@fc23 percona-server]$ time make -j 4
...
[100%] Built target mysqld
Scanning dependencies of target udf_example
[100%] Building CXX object sql/CMakeFiles/udf_example.dir/udf_example.cc.o
[100%] Built target pfs_connect_attr-t
[100%] Linking CXX shared module udf_example.so
[100%] Built target udf_example

real    10m40.254s
user    36m18.994s
sys     3m9.644s

[openxs@fc23 percona-server]$ make install && make clean
...
-- Installing: /home/openxs/dbs/p5.7/share/aclocal/mysql.m4
-- Installing: /home/openxs/dbs/p5.7/support-files/mysql.server

It took reasonable time to build and, some warnings aside, it worked well.

Percona manual says nothing on what to do next, but if it's a new test instance we need to initialize the database directory for it. Let's do it new way (mysql_install_db is NOT only in the bin subdirectory now in 5.7 instead of scripts, but is deprecated now):

[openxs@fc23 percona-server]$ cd ~/dbs/p5.7
[openxs@fc23 p5.7]$ bin/mysqld --initialize-insecure
2016-02-25T10:14:19.910929Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
2016-02-25T10:14:19.911033Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
2016-02-25T10:14:19.911197Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-02-25T10:14:19.927376Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
2016-02-25T10:14:19.927448Z 0 [ERROR] Aborting

 At this moment I've suspected that I probably have my.cnf or .my.cnf somewhere pointing to some other datadir that can be non-empty, so I tried to add --no-defaults, as usual:

 [openxs@fc23 p5.7]$ bin/mysqld --no-defaults --initialize-insecure
2016-02-25T10:14:51.318896Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
2016-02-25T10:14:51.319006Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
2016-02-25T10:14:51.319199Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-02-25T10:14:51.321754Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
2016-02-25T10:14:51.321823Z 0 [ERROR] Aborting

[openxs@fc23 p5.7]$ ls -l data
ls: cannot access data: No such file or directory
[openxs@fc23 p5.7]$ ls -l
total 128
drwxrwxr-x.  2 openxs openxs  4096 Feb 25 12:06 bin
-rw-r--r--.  1 openxs openxs 17987 Feb  9 09:44 COPYING
-rw-r--r--.  1 openxs openxs 34520 Feb  9 13:04 COPYING.AGPLv3
-rw-r--r--.  1 openxs openxs 17987 Feb  9 13:04 COPYING.GPLv2
drwxr-xr-x.  2 openxs openxs  4096 Feb 25 12:06 docs
drwxrwxr-x.  3 openxs openxs  4096 Feb 25 12:06 include
-rw-r--r--.  1 openxs openxs   301 Feb 25 11:44 INSTALL-BINARY
drwxrwxr-x.  4 openxs openxs  4096 Feb 25 12:06 lib
drwxrwxr-x.  4 openxs openxs  4096 Feb 25 12:06 man
drwxrwxr-x. 10 openxs openxs  4096 Feb 25 12:07 mysql-test
-rw-r--r--.  1 openxs openxs  2211 Feb  9 13:04 PATENTS
-rw-r--r--.  1 openxs openxs  4442 Feb  9 13:04 README.md
-rw-r--r--.  1 openxs openxs  2478 Feb 25 11:44 README.MySQL
drwxrwxr-x. 28 openxs openxs  4096 Feb 25 12:07 share
drwxrwxr-x.  2 openxs openxs  4096 Feb 25 12:07 support-files

[openxs@fc23 p5.7]$ pwd
/home/openxs/dbs/p5.7

 Usually for the Git source by default the datadir is supposed to be in the basedir (current directory in my case). But surely I can specify it explicitly (and I should, see Bug #73845 and the manual):

[openxs@fc23 p5.7]$ bin/mysqld --no-defaults --datadir=/home/openxs/dbs/p5.7/data --initialize-insecure
2016-02-25T10:19:37.946463Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
2016-02-25T10:19:37.946558Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
2016-02-25T10:19:37.946723Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-02-25T10:19:39.784840Z 0 [Warning] InnoDB: New log files created, LSN=45790
2016-02-25T10:19:40.201625Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2016-02-25T10:19:40.513883Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 47cb6f30-dba9-11e5-9c78-485b3952834f.
2016-02-25T10:19:40.546366Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2016-02-25T10:19:41.073896Z 0 [Warning] CA certificate ca.pem is self signed.
2016-02-25T10:19:41.225334Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
[openxs@fc23 p5.7]$

It seems everything worked well with --datadir specified explicitly, so we can try to use this new instance:

[openxs@fc23 p5.7]$ ps aux | grep mysqld
openxs   12123  0.0  0.0 116996  2316 pts/1    S+   12:21   0:00 grep --color=auto mysqld

[openxs@fc23 p5.7]$ bin/mysqld_safe --no-defaults &
[1] 12125
[openxs@fc23 p5.7]$ 160225 12:25:06 mysqld_safe Logging to '/home/openxs/dbs/p5.7/data/fc23.err'.
160225 12:25:06 mysqld_safe Starting mysqld daemon with databases from /home/openxs/dbs/p5.7/data

[openxs@fc23 p5.7]$ bin/mysql -uroot test
ERROR 1049 (42000): Unknown database 'test'
[openxs@fc23 p5.7]$ bin/mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.10-3 MySQL Community Server (GPL)

Copyright (c) 2009-2015 Percona LLC and/or its affiliates
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show variables like '%version%';
+-------------------------+------------------------------+
| Variable_name           | Value                        |
+-------------------------+------------------------------+
| innodb_version          | 5.7.10-3                     |
| protocol_version        | 10                           |
| slave_type_conversions  |                              |
| tls_version             | TLSv1,TLSv1.1,TLSv1.2        |
| version                 | 5.7.10-3                     |
| version_comment         | MySQL Community Server (GPL) |
| version_compile_machine | x86_64                       |
| version_compile_os      | Linux                        |
+-------------------------+------------------------------+
8 rows in set (0.00 sec)

That's it. MySQL 5.7.x (and thus Percona Server 5.7) is "secure by default", and in frames of that effort test database is not created.

As a last step, let's check the size of the mysqld binary:

[openxs@fc23 p5.7]$ ls -l bin/mysqld
-rwxr-xr-x. 1 openxs openxs 211932312 Feb 25 12:05 bin/mysqld
[openxs@fc23 p5.7]$ file bin/mysqld
bin/mysqld: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=f0ba086eb3639e927529cfdc4699c0ac3e748319, not stripped

 I'll try to build on Ubuntu 14.04 later tonight. I do not expect any problems other than zlib related and --datadir required for mysql_install_db or mysqld --initialize, but who knows...

On Upgrades to Percona Server 5.7 GA: CentOS 6 Case

As you probably noted, Percona had finally released Percona Server 5.7 GA two days ago. I had not paid much attention to this branch during last month or so, but as it's GA now I decided to check what's there, how it works and install/upgrade different boxes and VMs to use it.

For testing purposes and for this series of blog posts I am going to build it from Git source (at least on Fedora Core 23), upgrade to it on 32-bit Ubuntu 15.04 VM, upgrade to it on 64-bit Ubuntu 14.04 laptop, upgrade to it on 64-bit CentOS 6.7 VM and maybe more. If any of the attempts to install/upgrade ends up with something worth sharing, I am going to write about the experience, problems (if any) and solutions.

Yesterday I've spent some time upgrading on Ubuntu 15.04 VM and some observations are discussed on Facebook already. Then I tried to upgrade on my main CentOS 6.7 VM, where I had a "full stack" of Percona software for Percona Server 5.6.28, including TokuDB. The idea was to make sure everything I need works as expected after the upgrade. In this post I am going to share the observations from that experince.

Smart user should definitely start with reading the manual, in this case - "Percona Server In-Place Upgrading Guide: From 5.6 to 5.7"? as this is what I planned to do, in-place upgrade. Reading and following the directions there can save you notable time, but I am goign to show what happens when one tries naive "trial and failure" approach.

So, without reading anything, I just logged in as root, run yum update and then checked what Percona software I had there:

[root@centos ~]# yum update
...
No Packages marked for Update
[root@centos ~]# rpm -qa | grep -i percona
Percona-Server-shared-56-5.6.28-rel76.1.el6.x86_64
Percona-Server-test-56-5.6.28-rel76.1.el6.x86_64
Percona-Server-tokudb-56-5.6.28-rel76.1.el6.x86_64
Percona-Server-56-debuginfo-5.6.28-rel76.1.el6.x86_64
percona-release-0.1-3.noarch
Percona-Server-shared-51-5.1.73-rel14.12.625.rhel6.x86_64
Percona-Server-server-56-5.6.28-rel76.1.el6.x86_64
percona-nagios-plugins-1.1.6-1.noarch
percona-toolkit-2.2.16-1.noarch
percona-xtrabackup-2.3.3-1.el6.x86_64
Percona-Server-client-56-5.6.28-rel76.1.el6.x86_64
Percona-Server-devel-56-5.6.28-rel76.1.el6.x86_64

Great, so automatic upgrade to 5.7 GA doesd NOT happen. First thing to take into account for naive user like me. Manual mentioned above wisely suggests, as a next step, to stop MySQL server and then remove all Percona Server software without dependecies:

$ rpm -qa | grep Percona-Server | xargs rpm -e --nodeps

But I do not read the manuals (as many users) and previously upgrades worked for me without removing everything, so I just proceeded in a straighforward manner:

[root@centos ~]# yum install Percona-Server-server-57
...
--> Finished Dependency Resolution
Error: Percona-Server-client-57 conflicts with Percona-Server-client-56-5.6.28-rel76.1.el6.x86_64
Error: Percona-Server-server-57 conflicts with Percona-Server-server-56-5.6.28-rel76.1.el6.x86_64
Error: Percona-Server-shared-51 conflicts with Percona-Server-shared-57-5.7.10-3.1.el6.x86_64
 You could try using --skip-broken to work around the problem
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
Percona-Server-shared-51-5.1.73-rel14.12.625.rhel6.x86_64 has installed conflicts mysql-libs: Percona-Server-shared-51-5.1.73-rel14.12.625.rhel6.x86_64

So, removing seems unavoidable and I proceeded step by step (surely I had not cared about dependencies, who do, yum will tell us anyway or fix it for us, right?):

[root@centos ~]# yum remove Percona-Server-server-56
...
Removed:
  Percona-Server-server-56.x86_64 0:5.6.28-rel76.1.el6

Dependency Removed:
  Percona-Server-tokudb-56.x86_64 0:5.6.28-rel76.1.el6

Complete!

Let's remember that we need to install TokuDB for 5.7 separately, and proceed:

[root@centos ~]# yum remove Percona-Server-client-56 Percona-Server-shared-51
...
Transaction Summary
================================================================================
Remove        8 Package(s)

Installed size: 369 M
...
Removed:
  Percona-Server-client-56.x86_64 0:5.6.28-rel76.1.el6
  Percona-Server-shared-51.x86_64 0:5.1.73-rel14.12.625.rhel6

Dependency Removed:
  Percona-Server-test-56.x86_64 0:5.6.28-rel76.1.el6
  percona-toolkit.noarch 0:2.2.16-1
  percona-xtrabackup.x86_64 0:2.3.3-1.el6
  perl-DBD-MySQL.x86_64 0:4.013-3.el6
  php-mysql.x86_64 0:5.3.3-46.el6_7.1
  sysbench.x86_64 0:0.5-6.el6

Complete!

As we had not cared much about dependencies, we've got a lot of software removed that depended on old shared libraries. Let's save the list to install it back later. Now, we can try to install Percona Server 5.7 again:

[root@centos ~]# yum install Percona-Server-server-57
...
Dependencies Resolved

================================================================================
 Package                  Arch   Version           Repository              Size
================================================================================
Installing:
 Percona-Server-server-57 x86_64 5.7.10-3.1.el6    percona-release-x86_64  40 M
Installing for dependencies:
 Percona-Server-client-57 x86_64 5.7.10-3.1.el6    percona-release-x86_64 8.7 M
 Percona-Server-shared-57 x86_64 5.7.10-3.1.el6    percona-release-x86_64 902 k

Transaction Summary
================================================================================
Install       3 Package(s)
...

Transaction Check Error:
  file /usr/bin/mysql_config from install of Percona-Server-client-57-5.7.10-3.1.el6.x86_64 conflicts with file from package Percona-Server-devel-56-5.6.28-rel76.1.el6.x86_64

Error Summary
-------------

Great, so we also have a conflict with Percona-Server-devel-56 package that yum had not informed us about initially (Percona manual was probably right, remove them all, Percona-Srerver-* pacvkages first!). Let's remember to install it back for 5.7, remove it and install Percona Server 5.7 finally:

[root@centos ~]# yum remove Percona-Server-devel-56
...
Removed:
  Percona-Server-devel-56.x86_64 0:5.6.28-rel76.1.el6

Complete!
[root@centos ~]# yum install Percona-Server-server-57
...
Dependencies Resolved

================================================================================
 Package                  Arch   Version           Repository              Size
================================================================================
Installing:
 Percona-Server-server-57 x86_64 5.7.10-3.1.el6    percona-release-x86_64  40 M
Installing for dependencies:
 Percona-Server-client-57 x86_64 5.7.10-3.1.el6    percona-release-x86_64 8.7 M
 Percona-Server-shared-57 x86_64 5.7.10-3.1.el6    percona-release-x86_64 902 k

Transaction Summary
================================================================================
Install       3 Package(s)
...
Installed:
  Percona-Server-server-57.x86_64 0:5.7.10-3.1.el6

Dependency Installed:
  Percona-Server-client-57.x86_64 0:5.7.10-3.1.el6
  Percona-Server-shared-57.x86_64 0:5.7.10-3.1.el6

Complete!

Now, let's try to use it:

[root@centos ~]# mysql -uroot test
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
[root@centos ~]# ps aux | grep mysql
root      2665  0.0  0.0 103304   884 pts/0    S+   21:09   0:00 grep mysql

So, it is NOT started, as we had NOT upgraded it, we removed Percona Server 5.6.x first! Let's start the server now (at this step on recent Ubuntu etc you may find yourself in troubles):

[root@centos ~]# service mysql start
Starting mysqld:                                           [  OK  ]
[root@centos ~]# mysql -uroot  test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.10-3-log Percona Server (GPL), Release 3, Revision 63dafaf

Copyright (c) 2009-2015 Percona LLC and/or its affiliates
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

So, I've got it up and running. Now let me install some software back to replace those older packages I've removed:

[root@centos ~]# yum install Percona-Server-devel-57 Percona-Server-test-57 Percona-Server-tokudb-57
...

Installed:
  Percona-Server-devel-57.x86_64 0:5.7.10-3.1.el6
  Percona-Server-test-57.x86_64 0:5.7.10-3.1.el6
  Percona-Server-tokudb-57.x86_64 0:5.7.10-3.1.el6

Complete!

So, we have missing Percona Server packages for 5.7, now let me try to install dependecies back:

[root@centos ~]# yum install percona-toolkit percona-xtrabackup perl-DBD-MySQL php-mysql sysbench
...
Installed:
  percona-toolkit.noarch 0:2.2.16-1    percona-xtrabackup.x86_64 0:2.3.3-1.el6
  perl-DBD-MySQL.x86_64 0:4.013-3.el6  php-mysql.x86_64 0:5.3.3-46.el6_7.1
  sysbench.x86_64 0:0.5-6.el6

Dependency Installed:
  Percona-Server-shared-51.x86_64 0:5.1.73-rel14.12.625.rhel6

Complete!

Works well! Now, let me check if "full stack" works as expected. I had xtrabackup and toolkit installed, so do we have xtrabackup capable to work with 5.7 GA as result? Let's check:

[root@centos ~]# xtrabackup --version
xtrabackup version 2.3.3 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 525ca7d)
[root@centos ~]# xtrabackup --backup
160224 21:28:08  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock' as 'mysql'  (using password: NO).
160224 21:28:08  version_check Connected to MySQL server
160224 21:28:08  version_check Executing a version check against the server...
DBD::mysql::db selectall_hashref failed: Table 'performance_schema.session_variables' doesn't exist at - line 1221.
160224 21:28:10  version_check Done.
160224 21:28:10 Connecting to MySQL server host: localhost, user: mysql, password: not set, port: 0, socket: /var/lib/mysql/mysql.sock
Error: failed to execute query SHOW VARIABLES: Table 'performance_schema.session_variables' doesn't exist

Well, I was not sure (let's assume) if xtrabackup 2.3.3 is supposed to work with 5.7 (it was released recently, no upgrade happened, why not?), so I tried to use it. But it does not work as we do not have that performance_schema.session_variables table. Surely it's not there, it does not exist in MySQL 5.6. We missed important step that manual clearly mentiones, running mysql_upgrade and restart of service:

[root@centos ~]# mysql_upgrade
Checking if update is needed.
Checking server version.
Running queries to upgrade MySQL server.
Checking system database.
mysql.columns_priv                                 OK
...
Upgrade process completed successfully.
Checking if update is needed.
[root@centos ~]# service mysql restart
Stopping mysqld:                                           [  OK  ]
Starting mysqld:                                           [  OK  ]

Let's try again now:

[root@centos ~]# xtrabackup --backup
160224 21:31:00  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock' as 'mysql'  (using password: NO).
160224 21:31:00  version_check Connected to MySQL server
160224 21:31:00  version_check Executing a version check against the server...
160224 21:31:00  version_check Done.
160224 21:31:00 Connecting to MySQL server host: localhost, user: mysql, password: not set, port: 0, socket: /var/lib/mysql/mysql.sock
Error: Unsupported server version: '5.7.10-3-log'. Please report a bug at https://bugs.launchpad.net/percona-xtrabackup

No luck. Indeed, as anyone can find out after some reading, only Percona Xtrabackup 2.4 (that was also released as GA on February 23) is supposed to work with Percona Server or MySQL 5.7. We have to install it explicitly as "it was decided" that Percona does not want it to be used by default, yet:

[root@centos ~]# yum install percona-xtrabackup-24
...
--> Processing Conflict: percona-xtrabackup-24-2.4.1-1.el6.x86_64 conflicts percona-xtrabackup
--> Finished Dependency Resolution
Error: percona-xtrabackup-24 conflicts with percona-xtrabackup-2.3.3-1.el6.x86_64
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
[root@centos ~]# yum remove percona-xtrabackup
...
Removed:
  percona-xtrabackup.x86_64 0:2.3.3-1.el6

Complete!
[root@centos ~]# yum install percona-xtrabackup-24
...
Running Transaction
  Installing : percona-xtrabackup-24-2.4.1-1.el6.x86_64                     1/1
  Verifying  : percona-xtrabackup-24-2.4.1-1.el6.x86_64                     1/1

Installed:
  percona-xtrabackup-24.x86_64 0:2.4.1-1.el6

Complete!

So, one has to explicitly remove percona-xtrabackup package and then install percona-xtrabackup-24. Now we can finally backup our new Percona Server 5.7 online (after exp;licitly specifying root user for backup, I wonder why, and doing some cleanup from the previous failed attempt):

[root@centos ~]# xtrabackup --backup
160224 21:35:46  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock' as 'mysql'  (using password: NO).
160224 21:35:46  version_check Connected to MySQL server
160224 21:35:46  version_check Executing a version check against the server...
160224 21:35:46  version_check Done.
160224 21:35:46 Connecting to MySQL server host: localhost, user: mysql, password: not set, port: 0, socket: /var/lib/mysql/mysql.sock
Using server version 5.7.10-3-log
xtrabackup version 2.4.1 based on MySQL server 5.7.10 Linux (x86_64) (revision id: a2dc9d4)
xtrabackup: uses posix_fadvise().
...
160224 21:35:53 Executing LOCK TABLES FOR BACKUP...
Error: failed to execute query LOCK TABLES FOR BACKUP: Access denied; you need (at least one of) the RELOAD privilege(s) for this operation
[root@centos ~]# xtrabackup --user=root --backup
...
xtrabackup: Can't create/write to file '/root/xtrabackup_backupfiles/xtrabackup_logfile' (Errcode: 17 - File exists)
xtrabackup: error: failed to open the target stream for 'xtrabackup_logfile'.
[root@centos ~]# rm -rf xtrabackup_backupfiles/
[root@centos ~]# xtrabackup --user=root --backup
...
160224 21:37:40 Backup created in directory '/root/xtrabackup_backupfiles/'
160224 21:37:40 [00] Writing backup-my.cnf
160224 21:37:40 [00]        ...done
160224 21:37:40 [00] Writing xtrabackup_info
160224 21:37:40 [00]        ...done
xtrabackup: Transaction log of lsn (1018105090) to (1018105141) was copied.
160224 21:37:40 completed OK!

Now the last step for this post, let's try if magic tools from Percona Toolkit works with 5.7, at least pt-mysql-summary is something to test for sure:

[root@centos ~]# pt-mysql-summary
# Percona Toolkit MySQL Summary Report #######################
              System time | 2016-02-24 19:38:09 UTC (local TZ: EET +0200)
# Instances ##################################################
  Port  Data Directory             Nice OOM Socket
  ===== ========================== ==== === ======
        /var/lib/mysql             0    0   /var/lib/mysql/mysql.sock
# MySQL Executable ###########################################
       Path to executable | /usr/sbin/mysqld
              Has symbols | No
# Report On Port 3306 ########################################
                     User | root@localhost
                     Time | 2016-02-24 21:38:09 (EET)
                 Hostname | centos
                  Version | 5.7.10-3-log Percona Server (GPL), Release 3, Revision 63dafaf
                 Built On | Linux x86_64
                  Started | 2016-02-24 21:30 (up 0+00:07:22)
                Databases | 6
                  Datadir | /var/lib/mysql/
                Processes | 1 connected, 1 running
              Replication | Is not a slave, has 0 slaves connected
                  Pidfile | /var/run/mysqld/mysqld.pid (exists)
...

So, it works and recongnizes the version properly, but at this moment I've noted I miss debug symbols for 5.7 (and I'll need them as soon as I attach gdb to running server, something I do on a regular basis). So, I have to install them:

[root@centos ~]# yum install Percona-Server-57-debuginfo
...
  file /usr/lib/debug/usr/share/mysql-test/lib/My/SafeProcess/my_safe_process.debug from install of Percona-Server-57-debuginfo-5.7.10-3.1.el6.x86_64 conflicts with file from package Percona-Server-56-debuginfo-5.6.28-rel76.1.el6.x86_64

Error Summary
-------------
[root@centos ~]# yum remove Percona-Server-56-debuginfo
Removed:
  Percona-Server-56-debuginfo.x86_64 0:5.6.28-rel76.1.el6

Complete!
[root@centos ~]# yum install Percona-Server-57-debuginfo
...
Installed:
  Percona-Server-57-debuginfo.x86_64 0:5.7.10-3.1.el6

Complete!
[root@centos ~]# rpm -qa | grep -i percona
Percona-Server-shared-56-5.6.28-rel76.1.el6.x86_64
Percona-Server-shared-57-5.7.10-3.1.el6.x86_64
Percona-Server-test-57-5.7.10-3.1.el6.x86_64
percona-toolkit-2.2.16-1.noarch
percona-xtrabackup-24-2.4.1-1.el6.x86_64
Percona-Server-tokudb-57-5.7.10-3.1.el6.x86_64
percona-release-0.1-3.noarch
percona-nagios-plugins-1.1.6-1.noarch
Percona-Server-server-57-5.7.10-3.1.el6.x86_64
Percona-Server-shared-51-5.1.73-rel14.12.625.rhel6.x86_64
Percona-Server-client-57-5.7.10-3.1.el6.x86_64
Percona-Server-devel-57-5.7.10-3.1.el6.x86_64
Percona-Server-57-debuginfo-5.7.10-3.1.el6.x86_64

So, after dealing with one more conflict I seem to have a "full stack" of Percona tools that are ready to work with Percona Server 5.7 GA.

To summarize my experience of a naive user, one should note the following while upgrading to Percona Server 5.7 on CentOS 6.x:

1. Read the manual about upgrade carefully and don't ignore it. You'll have to remove all Percona-Server-* packages first, deal with dependencies somehow and then install all Percona-Server-57-* packages you need.
2. mysql_upgrade is NOT executed automatically upon installation when there is some older datadir from the previous version. You have to execute it explicitly, check the outputs and then restart mysql service before any further use.
3. If you need to use xtrabackup with Percona Server 5.7, you have to remove percona-xtrabackup package and then install percona-xtrabackup-24 explicitly.

I hope this step by step guide will be useful to some of you. I'll try to upgrade "full stack" of Percona software on 64-bit Ubuntu 14.04 laptop tonight and carefully note all steps. I may end up with the next blog post in this series after that, depending on the results, so stay tuned!