On Upgrades to Percona Server 5.7 GA: Building from Git Source on Fedora

While writing my first post in this series I was bulding Percona Server 5.7 from Git sources on my wife's Fedora Core 23 box based on this manual page, for my collection of test instances. As I build MySQL and related software from sources on different platforms almost every day for 11 years already, the process was completed successfully without much troubles, but there are still some details I'd like to highlight.

Manual says one has to do the following:

$ git clone https://github.com/percona/percona-server.git
$ cd percona-server
$ git checkout 5.7
$ git submodule init
$ git submodule update

I had source code clonned already for regular 5.6.x builds, so I did this instead:

[openxs@fc23 percona-server]$ git pull
Already up-to-date.
[openxs@fc23 percona-server]$ git checkout 5.7
Checking out files: 100% (20321/20321), done.
M       plugin/tokudb-backup-plugin/Percona-TokuBackup
M       storage/tokudb/PerconaFT
Branch 5.7 set up to track remote branch 5.7 from origin.
Switched to a new branch '5.7'
[openxs@fc23 percona-server]$ git submodule init
[openxs@fc23 percona-server]$ git submodule update

On this FC 23 box I've already built MySQL 5.7, MariaDB 10.1, Facebook's MySQL 5.6 and Percona Server 5.6 from Git sources more than once, and I've updated all packages today and even built Facebook's 5.6 code (for testing MyRocks further) after that successfully, so I expect everything should work really well.

I put my test instances into dbs subdirectory usually:

[openxs@fc23 percona-server]$ ls ~/dbs
5.7  fb56  maria10.1  p5.6

So, I've modified cmake command line a bit comparing to the one recommended in the manual:

cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DWITH_EMBEDDED_SERVER=OFF

as you can see below:

[openxs@fc23 percona-server]$ cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DWITH_EMBEDDED_SERVER=OFF -DCMAKE_INSTALL_PREFIX=/home/openxs/dbs/p5.7
-- Running cmake version 3.4.1
-- Configuring with MAX_INDEXES = 64
-- Looking for SHM_HUGETLB
-- Looking for SHM_HUGETLB - found
-- SIZEOF_VOIDP 8
-- MySQL 5.7.10-3
-- Packaging as: percona-server-5.7.10-3-Linux-x86_64
-- Looked for boost/version.hpp in  and
-- BOOST_INCLUDE_DIR BOOST_INCLUDE_DIR-NOTFOUND
-- LOCAL_BOOST_DIR
-- LOCAL_BOOST_ZIP
-- Could not find (the correct version of) boost.
-- MySQL currently requires boost_1_59_0

CMake Error at cmake/boost.cmake:76 (MESSAGE):
  You can download it with -DDOWNLOAD_BOOST=1 -DWITH_BOOST=<directory>

  This CMake script will look for boost in <directory>.  If it is not there,
  it will download and unpack it (in that directory) for you.

  If you are inside a firewall, you may need to use an http proxy:

  export http_proxy=http://example.com:80

Call Stack (most recent call first):
  cmake/boost.cmake:228 (COULD_NOT_FIND_BOOST)
  CMakeLists.txt:439 (INCLUDE)


-- Configuring incomplete, errors occurred!
See also "/home/openxs/git/percona-server/CMakeFiles/CMakeOutput.log".
See also "/home/openxs/git/percona-server/CMakeFiles/CMakeError.log".

As you can see, manual obviously misses some details, like adding options related to Boost library. We see nice hints on what to do, and I've already downloaded Boost once into ~/boost while building MySQL 5.7. If Percona Server 5.7 was based on 5.7.11, we could use -DWITH_BOOST=system, see the release notes:

"A value of system is now permitted for the WITH_BOOST CMake option. If this option is not set or is set to system, it is assumed that the correct version of Boost is installed on the compilation host in the standard location. In this case, the installed version of Boost is used rather than any version included with a MySQL source distribution. (Bug #22224313)"

But it's based on 5.7.10, so I've modified my command like as follows:

[openxs@fc23 percona-server]$ cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DWITH_EMBEDDED_SERVER=OFF -DDOWNLOAD_BOOST=1 -DWITH_BOOST=/home/openxs/boost -DCMAKE_INSTALL_PREFIX=/home/openxs/dbs/p5.7
...
-- Check if the system is big endian - little endian
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
  Could NOT find ZLIB (missing: ZLIB_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake/Modules/FindZLIB.cmake:124 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  libbinlogstandalone/CMakeLists.txt:30 (FIND_PACKAGE)


-- Configuring incomplete, errors occurred!
See also "/home/openxs/git/percona-server/CMakeFiles/CMakeOutput.log".
See also "/home/openxs/git/percona-server/CMakeFiles/CMakeError.log".

to end up with another error. I was surprised a bit, as I do have zlib-devel installed:

[openxs@fc23 percona-server]$ rpm -qa | grep zlib
zlib-1.2.8-9.fc23.x86_64
zlib-devel-1.2.8-9.fc23.x86_64

To not waste time on zlib now I've used the "trick" from Facebook's wiki this time:

[openxs@fc23 percona-server]$ cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DWITH_EMBEDDED_SERVER=OFF -DDOWNLOAD_BOOST=1 -DWITH_BOOST=/home/openxs/boost -DWITH_ZLIB=bundled -DCMAKE_INSTALL_PREFIX=/home/openxs/dbs/p5.7
...
-- MySQL 5.7.10-3
-- Packaging as: percona-server-5.7.10-3-Linux-x86_64
-- Local boost dir /home/openxs/boost/boost_1_59_0
-- Local boost zip /home/openxs/boost/boost_1_59_0.tar.gz
...
-- Library perconaserverclient depends on OSLIBS -lpthread;m;rt;/usr/lib64/libssl.so;/usr/lib64/libcrypto.so;dl
-- Performing Test HAVE_NO_UNUSED_TYPEDEFS
-- Performing Test HAVE_NO_UNUSED_TYPEDEFS - Success
-- Performing Test HAVE_NO_LOGICAL_OP
-- Performing Test HAVE_NO_LOGICAL_OP - Success
-- INSTALL perconaserverclient.pc lib/pkgconfig
-- CMAKE_BUILD_TYPE: RelWithDebInfo
-- COMPILE_DEFINITIONS: _GNU_SOURCE;_FILE_OFFSET_BITS=64;HAVE_CONFIG_H
-- CMAKE_C_FLAGS:  -Wall -Wextra -Wformat-security -Wvla -Wwrite-strings -Wdeclaration-after-statement
-- CMAKE_CXX_FLAGS:  -Wall -Wextra -Wformat-security -Wvla -Woverloaded-virtual -Wno-unused-parameter
-- CMAKE_C_FLAGS_RELWITHDEBINFO: -O3 -D_FORTIFY_SOURCE=2 -g -fabi-version=2 -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF
-- CMAKE_CXX_FLAGS_RELWITHDEBINFO: -O3 -D_FORTIFY_SOURCE=2 -g -fabi-version=2 -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF
-- Configuring done
CMake Warning (dev) at storage/tokudb/PerconaFT/portability/CMakeLists.txt:22 (add_dependencies):
  Policy CMP0046 is not set: Error on non-existent dependency in
  add_dependencies.  Run "cmake --help-policy CMP0046" for policy details.
  Use the cmake_policy command to set the policy and suppress this warning.

  The dependency target "build_jemalloc" of target
  "tokuportability_static_conv" does not exist.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /home/openxs/git/percona-server

to end up with cmake command completed sucessfully. Now, time to make it (my FC23 box I've assembled myself back in 2011 has single QuadCore CPU):

[openxs@fc23 percona-server]$ time make -j 4
...
[100%] Built target mysqld
Scanning dependencies of target udf_example
[100%] Building CXX object sql/CMakeFiles/udf_example.dir/udf_example.cc.o
[100%] Built target pfs_connect_attr-t
[100%] Linking CXX shared module udf_example.so
[100%] Built target udf_example

real    10m40.254s
user    36m18.994s
sys     3m9.644s

[openxs@fc23 percona-server]$ make install && make clean
...
-- Installing: /home/openxs/dbs/p5.7/share/aclocal/mysql.m4
-- Installing: /home/openxs/dbs/p5.7/support-files/mysql.server

It took reasonable time to build and, some warnings aside, it worked well.

Percona manual says nothing on what to do next, but if it's a new test instance we need to initialize the database directory for it. Let's do it new way (mysql_install_db is NOT only in the bin subdirectory now in 5.7 instead of scripts, but is deprecated now):

[openxs@fc23 percona-server]$ cd ~/dbs/p5.7
[openxs@fc23 p5.7]$ bin/mysqld --initialize-insecure
2016-02-25T10:14:19.910929Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
2016-02-25T10:14:19.911033Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
2016-02-25T10:14:19.911197Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-02-25T10:14:19.927376Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
2016-02-25T10:14:19.927448Z 0 [ERROR] Aborting

 At this moment I've suspected that I probably have my.cnf or .my.cnf somewhere pointing to some other datadir that can be non-empty, so I tried to add --no-defaults, as usual:

 [openxs@fc23 p5.7]$ bin/mysqld --no-defaults --initialize-insecure
2016-02-25T10:14:51.318896Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
2016-02-25T10:14:51.319006Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
2016-02-25T10:14:51.319199Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-02-25T10:14:51.321754Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
2016-02-25T10:14:51.321823Z 0 [ERROR] Aborting

[openxs@fc23 p5.7]$ ls -l data
ls: cannot access data: No such file or directory
[openxs@fc23 p5.7]$ ls -l
total 128
drwxrwxr-x.  2 openxs openxs  4096 Feb 25 12:06 bin
-rw-r--r--.  1 openxs openxs 17987 Feb  9 09:44 COPYING
-rw-r--r--.  1 openxs openxs 34520 Feb  9 13:04 COPYING.AGPLv3
-rw-r--r--.  1 openxs openxs 17987 Feb  9 13:04 COPYING.GPLv2
drwxr-xr-x.  2 openxs openxs  4096 Feb 25 12:06 docs
drwxrwxr-x.  3 openxs openxs  4096 Feb 25 12:06 include
-rw-r--r--.  1 openxs openxs   301 Feb 25 11:44 INSTALL-BINARY
drwxrwxr-x.  4 openxs openxs  4096 Feb 25 12:06 lib
drwxrwxr-x.  4 openxs openxs  4096 Feb 25 12:06 man
drwxrwxr-x. 10 openxs openxs  4096 Feb 25 12:07 mysql-test
-rw-r--r--.  1 openxs openxs  2211 Feb  9 13:04 PATENTS
-rw-r--r--.  1 openxs openxs  4442 Feb  9 13:04 README.md
-rw-r--r--.  1 openxs openxs  2478 Feb 25 11:44 README.MySQL
drwxrwxr-x. 28 openxs openxs  4096 Feb 25 12:07 share
drwxrwxr-x.  2 openxs openxs  4096 Feb 25 12:07 support-files

[openxs@fc23 p5.7]$ pwd
/home/openxs/dbs/p5.7

 Usually for the Git source by default the datadir is supposed to be in the basedir (current directory in my case). But surely I can specify it explicitly (and I should, see Bug #73845 and the manual):

[openxs@fc23 p5.7]$ bin/mysqld --no-defaults --datadir=/home/openxs/dbs/p5.7/data --initialize-insecure
2016-02-25T10:19:37.946463Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
2016-02-25T10:19:37.946558Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
2016-02-25T10:19:37.946723Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-02-25T10:19:39.784840Z 0 [Warning] InnoDB: New log files created, LSN=45790
2016-02-25T10:19:40.201625Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2016-02-25T10:19:40.513883Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 47cb6f30-dba9-11e5-9c78-485b3952834f.
2016-02-25T10:19:40.546366Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2016-02-25T10:19:41.073896Z 0 [Warning] CA certificate ca.pem is self signed.
2016-02-25T10:19:41.225334Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
[openxs@fc23 p5.7]$

It seems everything worked well with --datadir specified explicitly, so we can try to use this new instance:

[openxs@fc23 p5.7]$ ps aux | grep mysqld
openxs   12123  0.0  0.0 116996  2316 pts/1    S+   12:21   0:00 grep --color=auto mysqld

[openxs@fc23 p5.7]$ bin/mysqld_safe --no-defaults &
[1] 12125
[openxs@fc23 p5.7]$ 160225 12:25:06 mysqld_safe Logging to '/home/openxs/dbs/p5.7/data/fc23.err'.
160225 12:25:06 mysqld_safe Starting mysqld daemon with databases from /home/openxs/dbs/p5.7/data

[openxs@fc23 p5.7]$ bin/mysql -uroot test
ERROR 1049 (42000): Unknown database 'test'
[openxs@fc23 p5.7]$ bin/mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.10-3 MySQL Community Server (GPL)

Copyright (c) 2009-2015 Percona LLC and/or its affiliates
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show variables like '%version%';
+-------------------------+------------------------------+
| Variable_name           | Value                        |
+-------------------------+------------------------------+
| innodb_version          | 5.7.10-3                     |
| protocol_version        | 10                           |
| slave_type_conversions  |                              |
| tls_version             | TLSv1,TLSv1.1,TLSv1.2        |
| version                 | 5.7.10-3                     |
| version_comment         | MySQL Community Server (GPL) |
| version_compile_machine | x86_64                       |
| version_compile_os      | Linux                        |
+-------------------------+------------------------------+
8 rows in set (0.00 sec)

That's it. MySQL 5.7.x (and thus Percona Server 5.7) is "secure by default", and in frames of that effort test database is not created.

As a last step, let's check the size of the mysqld binary:

[openxs@fc23 p5.7]$ ls -l bin/mysqld
-rwxr-xr-x. 1 openxs openxs 211932312 Feb 25 12:05 bin/mysqld
[openxs@fc23 p5.7]$ file bin/mysqld
bin/mysqld: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=f0ba086eb3639e927529cfdc4699c0ac3e748319, not stripped

 I'll try to build on Ubuntu 14.04 later tonight. I do not expect any problems other than zlib related and --datadir required for mysql_install_db or mysqld --initialize, but who knows...