Wednesday, April 27, 2016

Building MaxScale 1.4.2 from GitHub on Fedora 23

MariaDB MaxScale is mentioned in many blog posts recently. It's Application of the Year 2016 after all! I'd like to test it, follow posts like this etc, all that on my favorite and readily available testing platforms that are now Ubuntu of all kinds and, surely, Fedora 23 (on my wife's workstation, the most powerful hardware at hand).

My old habits force me to build open source software I test from source, and I do not want to even discuss the topic of "MaxScale binaries availability" that was quite "popular" some time ago. So, after building MaxScale 1.4.1 on CentOS 6.7 back on March 31, 2016 (mostly just following MariaDB KB article on the topic) using libmysqld.a from MariaDB 10.0.23, this morning I decided to check new branch, 1.4.2, and build it on Fedora 23, following that same KB article (that unfortunately does not even mention Fedora after the fix to MXS-248). Thing is, Fedora is not officially supported as a platform for MaxScale 1.4.x, but why should we, those who can build things from source for testing purposes, care about this?

I started with cloning MaxScale:

git clone https://github.com/mariadb-corporation/MaxScale.git
cd MaxScale
and then:

[openxs@fc23 MaxScale]$ git branch -r
...
  origin/HEAD -> origin/develop
...
  origin/release-1.4.2
...
I remember spending enough time fighting with develop branch while building on CentOS 6.7, mostly with sqlite-related things it contained, so this time I proceed immediately to the branch I want to build:

[openxs@fc23 MaxScale]$ git checkout release-1.4.2
Branch release-1.4.2 set up to track remote branch release-1.4.2 from origin.
Switched to a new branch 'release-1.4.2'
[openxs@fc23 MaxScale]$ git branch
  develop
* release-1.4.2

[openxs@fc23 MaxScale]$ mkdir build
[openxs@fc23 MaxScale]$ cd build

Last two steps originate from the KB article. We are almost ready for building, but what about the prerequisites? I've collected all the packages required for CentOS in that article and tried to install them all:
[openxs@fc23 build]$ sudo yum install mariadb-devel mariadb-embedded-devel libedit-devel gcc gcc-c++ ncurses-devel bison flex glibc-devel cmake libgcc perl make libtool openssl-devel libaio libaio-devel librabbitmq-devel libcurl-devel pcre-devel rpm-build[sudo] password for openxs:
Yum command has been deprecated, redirecting to '/usr/bin/dnf install mariadb-devel mariadb-embedded-devel libedit-devel gcc gcc-c++ ncurses-devel bison flex glibc-devel cmake libgcc perl make libtool openssl-devel libaio libaio-devel librabbitmq-devel libcurl-devel pcre-devel rpm-build'.
See 'man dnf' and 'man yum2dnf' for more information.
To transfer transaction metadata from yum to DNF, run:
'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate'

Last metadata expiration check: 0:26:04 ago on Wed Apr 27 10:43:24 2016.
Package gcc-5.3.1-6.fc23.x86_64 is already installed, skipping.
...
Package pcre-devel-8.38-7.fc23.x86_64 is already installed, skipping.
Dependencies resolved.
================================================================================
 Package                  Arch     Version                      Repository
                                                                           Size
================================================================================
Installing:
 autoconf                 noarch   2.69-21.fc23                 fedora    709 k
 automake                 noarch   1.15-4.fc23                  fedora    695 k
 dwz                      x86_64   0.12-1.fc23                  fedora    106 k
 flex                     x86_64   2.5.39-2.fc23                fedora    328 k
 ghc-srpm-macros          noarch   1.4.2-2.fc23                 fedora    8.2 k
 gnat-srpm-macros         noarch   2-1.fc23                     fedora    8.4 k
 go-srpm-macros           noarch   2-3.fc23                     fedora    8.0 k
 libcurl-devel            x86_64   7.43.0-6.fc23                updates   590 k
 libedit-devel            x86_64   3.1-13.20150325cvs.fc23      fedora     34 k
 librabbitmq              x86_64   0.8.0-1.fc23                 updates    43 k
 librabbitmq-devel        x86_64   0.8.0-1.fc23                 updates    52 k
 libtool                  x86_64   2.4.6-8.fc23                 updates   707 k
 mariadb-common           x86_64   1:10.0.23-1.fc23             updates    74 k
 mariadb-config           x86_64   1:10.0.23-1.fc23             updates    25 k
 mariadb-devel            x86_64   1:10.0.23-1.fc23             updates   869 k
 mariadb-embedded         x86_64   1:10.0.23-1.fc23             updates   4.0 M
 mariadb-embedded-devel   x86_64   1:10.0.23-1.fc23             updates   8.3 M
 mariadb-errmsg           x86_64   1:10.0.23-1.fc23             updates   199 k
 mariadb-libs             x86_64   1:10.0.23-1.fc23             updates   637 k
 ocaml-srpm-macros        noarch   2-3.fc23                     fedora    8.1 k
 patch                    x86_64   2.7.5-2.fc23                 fedora    123 k
 perl-Thread-Queue        noarch   3.07-1.fc23                  updates    22 k
 perl-generators          noarch   1.06-1.fc23                  updates    15 k
 perl-srpm-macros         noarch   1-17.fc23                    fedora    9.7 k
 python-srpm-macros       noarch   3-7.fc23                     updates   8.1 k
 redhat-rpm-config        noarch   36-1.fc23.1                  updates    59 k
 rpm-build                x86_64   4.13.0-0.rc1.13.fc23         updates   137 k

Transaction Summary
================================================================================
Install  27 Packages

Total download size: 18 M
Installed size: 64 M
Is this ok [y/N]: Y

...

Complete!
Now, let's try simple approach:

[openxs@fc23 build]$ cmake ..
...
-- MySQL version: 10.0.23
-- MySQL provider: MariaDB
-- Looking for pcre_stack_guard in MYSQL_EMBEDDED_LIBRARIES_STATIC-NOTFOUND
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
MYSQL_EMBEDDED_LIBRARIES_STATIC
    linked by target "cmTC_2494a" in directory /home/openxs/git/MaxScale/build/CMakeFiles/CMakeTmp

CMake Error: Internal CMake error, TryCompile configure of cmake failed
-- Looking for pcre_stack_guard in MYSQL_EMBEDDED_LIBRARIES_STATIC-NOTFOUND - not found
-- PCRE libs: /usr/lib64/libpcre.so
-- PCRE include directory: /usr/include
-- Embedded mysqld does not have pcre_stack_guard, linking with system pcre.
CMake Error at cmake/FindMySQL.cmake:115 (message):
  Library not found: libmysqld.  If your install of MySQL is in a non-default
  location, please provide the location with -DMYSQL_EMBEDDED_LIBRARIES=<path
  to library>
Call Stack (most recent call first):
  CMakeLists.txt:37 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/openxs/git/MaxScale/build/CMakeFiles/CMakeOutput.log".
See also "/home/openxs/git/MaxScale/build/CMakeFiles/CMakeError.log".

Failure, cmake can not find libmysqld.a it seems. Let me try to find it:

[openxs@fc23 build]$ sudo find / -name libmysqld.a 2>/dev/null
/home/openxs/git/percona-xtrabackup/libmysqld/libmysqld.a
/home/openxs/dbs/5.7/lib/libmysqld.a
/home/openxs/dbs/p5.6/lib/libmysqld.a
/home/openxs/dbs/fb56/lib/libmysqld.a
/home/openxs/10.1.12/lib/libmysqld.a
That's all, even though I installed all packages that looked as required based on the article! I have the library in many places (in my own builds and even in sandbox with MariaDB 10.1.12), but it's not installed where expected. Some more desperate tries (installing MariaDB server with sudo yum install mariadb-server, searches for package that provides libmysqld.a etc), chat with engineers of MariaDB and I've ended up with the fact that my packages are from Fedora (not MariaDB) and they just do not include the static library. Looks like a bug in Fedora packaging, if you ask me.

I was not ready to add MariaDB's repository at the moment (to get MariaDB-devel etc, something KB article also suggests for supported platforms), so I decided that it would be fair just to build current MariaDB 10.1.13 from source and use everything needed from there. Last time I built 10.2 branch, so I had to check out 10.1 first:
[openxs@fc23 server]$ git checkout 10.1
Switched to branch '10.1'
Your branch is behind 'origin/10.1' by 2 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
[openxs@fc23 server]$ git pull
Updating 1cf852d..071ae30
Fast-forward
 client/mysqlbinlog.cc                    | 523 ++++++++++++++++++++++---------
 mysql-test/r/mysqlbinlog_raw_mode.result | 274 ++++++++++++++++
 mysql-test/t/mysqlbinlog_raw_mode.test   | 387 +++++++++++++++++++++++
 sql/sql_priv.h                           |   3 +-
 storage/innobase/dict/dict0boot.cc       |  20 +-
 storage/xtradb/dict/dict0boot.cc         |  20 +-
 6 files changed, 1062 insertions(+), 165 deletions(-)
 create mode 100644 mysql-test/r/mysqlbinlog_raw_mode.result
 create mode 100644 mysql-test/t/mysqlbinlog_raw_mode.test
 Then I've executed the following while in server directory:

make clean
rm CMakeCache.txt
cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DWITH_SSL=system -DWITH_ZLIB=bundled -DMYSQL_MAINTAINER_MODE=0 -DENABLED_LOCAL_INFILE=1 -DWITH_JEMALLOC=system -DWITH_WSREP=ON -DWITH_INNODB_DISALLOW_WRITES=ON -DWITH_EMBEDDED_SERVER=ON -DCMAKE_INSTALL_PREFIX=/home/openxs/dbs/maria10.1

make

make install && make clean
Note that I've explicitly asked to build embedded server. I checked that the library is in the location I need:

[openxs@fc23 server]$ sudo find / -name libmysqld.a 2>/dev/null
/home/openxs/git/percona-xtrabackup/libmysqld/libmysqld.a
/home/openxs/dbs/maria10.1/lib/libmysqld.a
/home/openxs/dbs/5.7/lib/libmysqld.a
/home/openxs/dbs/p5.6/lib/libmysqld.a
/home/openxs/dbs/fb56/lib/libmysqld.a
/home/openxs/10.1.12/lib/libmysqld.a
Then I moved back to MaxScale/build directory and explicitly pointed out the location of headers, library and messages that I want to use with MaxScale:

[openxs@fc23 build]$ cmake .. -DMYSQL_EMBEDDED_INCLUDE_DIR=/home/openxs/dbs/maria10.1/include/mysql -DMYSQL_EMBEDDED_LIBRARIES=/home/openxs/dbs/maria10.1/lib/libmysqld.a -DERRMSG=/home/openxs/dbs/maria10.1/share/english/errmsg.sys -DCMAKE_INSTALL_PREFIX=/home/openxs/maxscale -DWITH_MAXSCALE_CNF=N
...
-- Build files have been written to: /home/openxs/git/MaxScale/build

[openxs@fc23 build]$ make
...
[ 95%] [BISON][ruleparser] Building parser with bison 3.0.4
ruleparser.y:34.1-13: warning: deprecated directive, use Б-?%name-prefixБ-? [-Wdeprecated]
 %name-prefix="dbfw_yy"
 ^^^^^^^^^^^^^
[ 96%] Building C object server/modules/filter/dbfwfilter/CMakeFiles/dbfwfilter.dir/ruleparser.c.o
[ 96%] Building C object server/modules/filter/dbfwfilter/CMakeFiles/dbfwfilter.dir/token.c.o
[ 97%] Linking C shared library libdbfwfilter.so
[ 97%] Built target dbfwfilter
Scanning dependencies of target maxadmin
[ 98%] Building C object client/CMakeFiles/maxadmin.dir/maxadmin.c.o
[ 98%] Linking C executable maxadmin
[100%] Built target maxadmin

It seems build completed without problems this time. We can try to test it (some tests do fail):







[openxs@fc23 build]$ make testcore
...
 1/22 Test  #1: Internal-TestQueryClassifier .....***Exception: Other  0.35 sec
      Start  2: Internal-CanonicalQuery
 2/22 Test  #2: Internal-CanonicalQuery ..........***Failed    0.25 sec
      Start  3: Internal-CanonicalQuerySelect
 3/22 Test  #3: Internal-CanonicalQuerySelect ....***Failed    0.04 sec
      Start  4: Internal-CanonicalQueryAlter
 4/22 Test  #4: Internal-CanonicalQueryAlter .....***Failed    0.04 sec
      Start  5: Internal-CanonicalQueryComment
 5/22 Test  #5: Internal-CanonicalQueryComment ...***Failed    0.04 sec
      Start  6: Internal-TestAdminUsers
 6/22 Test  #6: Internal-TestAdminUsers ..........   Passed    0.44 sec
      Start  7: Internal-TestBuffer
 7/22 Test  #7: Internal-TestBuffer ..............   Passed    0.01 sec
      Start  8: Internal-TestDCB
 8/22 Test  #8: Internal-TestDCB .................   Passed    0.01 sec
      Start  9: Internal-TestFilter
 9/22 Test  #9: Internal-TestFilter ..............   Passed    0.03 sec
...
(As a side note, make install in my case had NOT installed anything to /home/openxs/maxscale, something to deal with later, as on CentOS 6.7 it worked...)


In any case, I now have binaries to work with, of version 1.4.2:

[openxs@fc23 build]$ ls bin/
maxadmin  maxbinlogcheck  maxkeys  maxpasswd  maxscale
[openxs@fc23 build]$ bin/maxscale --version


MariaDB Corporation MaxScale 1.4.2      Wed Apr 27 13:24:01 2016
------------------------------------------------------
MaxScale 1.4.2

[openxs@fc23 build]$ bin/maxadmin --version
bin/maxadmin Version 1.4.2
To be continued one day... Stay tuned!




1 comment:

  1. This post saved me about a day of figuring all this out for myself, thanks so much!

    ReplyDelete