Sunday, January 10, 2016

MySQL Support People - Those Who Were There First

I'd like to devote this long weekend post, the first in a new series, to my current and former colleagues who once worked or still work in a company that provided public MySQL Support service and had a job role of MySQL Support engineer. The list of companies include MySQL AB/Inc ("good old MySQL"), Sun, Oracle, Percona, MariaDB, FromDual, maybe more (I named only those that surely provided or provides MySQL Support for customers outside of the company).

This is not the first time that I write about "people of MySQL", in a wider sense "those who contribute to MySQL Community". Last time it was about famous MySQL bug reporters. That's because for me, on personal and professional level, MySQL is about people who work on it, not about companies, services or money we all make while working on it. But today I want to concentrate on "MySQL Support People" mostly, and name many of them, those who still do MySQL Support right now when you read this, to those who are now CEOs, CTOs, Directors, VPs, Product Managers, DBAs and Developers. They all have two things in common: they once provided support for MySQL in the same team with me, and their contribution to MySQL Community is visible (can be found easily at or elsewhere). Click on names to see some of their public contributions to MySQL (bugs reported).

You may want to ask why I highlight "Support" in this post? It happened so that recently I have to emphasize in public the importance of Support as a core service for MySQL. Regular readers probably noted this here and there...

Let me write today about those who started to provide MySQL Support in "good old MySQL" before me:
  • Shane Bester - he started to work in MySQL Support back in 2005 and he still actively provides support in Oracle, as far as I know. Click on his name to see a list of 1029 (at the moment of writing) public bug reports he created for MySQL software! Start with Bug #79591 if you are interested in regression bugs in MySQL 5.7. I wish I'd be able to do the magic he does every day with creating repeatable test case for MySQL bugs.
  • Sinisa Milivojevic - rumors say he probably was the first Support Engineer in MySQL. Maybe not, but he does this job to some extent since 1999 or so. I see 74 bugs reported, started from Bug #4. So, he was already contributing to MySQL Community back in 2002 for sure! Recently he seems to verify public bug reports from time to time while still working in Oracle. Not sure what else he does, but does it really matter? He was one of MySQL people who influenced me a lot and I tried to play "better Sinisa" for a couple of years at my current job.
  • Victoria Reznichenko - a well known Queen of MySQL Support in Oracle now probably enjoys her maternity leave (I hope Oracle does not force her to work during this time). She was already a well known Support provider in EMEA when I joined, as one can clearly conclude from the fact that her first Bug #860 (of 102 I can see) was reported back in July 2103, 2 years before I joined MySQL AB. Surely her opinion would always matter for me.
  • Alexander Keremidarski (a.k.a. Salle) - when I joined he was team manager of EMEA Support team. As far as I know he has a similar role now in MariaDB. A lof of MySQL support practices are related to his name in my mind. Based on Bug #11 he had reported, he contributes to MySQL Community at least since 2002. Maybe it was him, not Sinisa, who was the first employee with MySQL Support Engineer title. I let them argue about that.
  • Timothy Smith - he tested my knowledge when they decided to give me Support Engineer job in MySQL. In 2005 he was an APAC team manager. Probably his activity was the reason why I ended up in Support (they were faster). I've used his approach to testing candidates later. I see 130 bugs reported by Tim, and many are still not fixed. I am not sure if he is still related to MySQL in any way, but he built a great team and I always was happy to work with him.
  • Matt Lord - he was already a key Support engineer in AMER team when I joined. 123 public bug reports till he moved to the dark side in Oracle, where he is a MySQL Product Manager now.
  • Harrison Fisk - he probably did consulting or training when I joined, but I clearly remember him in Support soon after that and until he moved to Facebook (after playing TAM role for them from some time). I see 108 bug reports and Bug #53825, for example, makes me wanting to double check current InnoDB code. I wish I have that gdb functions they discuss with Sunny at hand. Harrison had the highest rank in Support all the time we worked together. I would not be surprised if I am still 5+ years behind Harrison in my understanding of InnoDB internals (or anything related to MySQL)...
  • Arjen Lentz - he was a key Support engineer in APAC team when I joined. Now he is the Exec. Director at Open Query Pty Ltd, yet another MySQL services provider. His first public bug report that I can see in the list of 84 total, Bug #108, is almost 13 years old.
  • James Day - he worked all kind of crazy hours (probably he is the only one besides Shane and me who was SSC for all 3 shifts, EMEA, AMER and APAC more than once) and was already a well known guru in complex issues handling back in 2005. I see just 25 public bug reports from him, but he had processed many bugs and many of his comments in InnoDB-related bug reports are priceless. Read Bug #69842, for example. I'll remember our busy night APAC shift we worked together (with nobody else around) forever... He still does MySQL Support in Oracle, as far as I know.
  • Kolbe Kegel - he joined MySQL several weeks before me and he still provides Support in MariaDB. He had always been a very reliable and smart engineer. Check the latest of his 133 public bug reports, Bug #74517.
  • Hartmut Holzgraefe - Hartmut helped me a lot during my very first days in MySQL. He was ready to answer any question, help with code (he probably pushed the patch to have my Celeron CPU supported by the build scripts), whatever I asked. He provided support for NDB cluster, and people who are able to do this form a separate "super team" in MySQL Support that gain my huge respect. The do magic by supporting software I consider NOT supportable without direct communication channel with its developers even today. He had reported 420 bugs as far as I can see, and Bug #77018 is the last "Verified" one of them reported for NDB cluster.
  • Axel Schwenke - he was a key support provider in EMEA when I joined and he is also from that "super team" of NDB-enabled magicians. By the way, originally I was hired to provide NDB support eventually and went to Bugs Verification Team only for some part of usual 6 months probation period. But I skipped getting back to NDB cluster happily, and was able to mostly ignore it for 10 next years... Axel works in MariaDB and recently tried to fight with Oracle on performance benchmarks several times. Check this, Bug #77586, the last of his 69 public bug reports.
  • Miguel Solorzano - my first and only team manager in MySQL/Sun/Oracle. He still manages the same Bugs Verification Team and still works on bugs processing in a very visible way (Google for his name and count hits). His first bug report out of 115 total, Bug #326, is almost 13 years old, and before devoting himself to bugs processing he worked as developer on porting MySQL to Windows. I've got my first email form him with instructions few days before I started, and this email was enough for me to start doing my job from day one. I've forwarded it to maybe a dozen of colleagues who joined later. To summarize, Miguel was an ideal manager for me at some stage. The amount of time and efforts he devoted to MySQL Community is incredible!
  • Bogdan Degtyariov - Bogdan joined MySQL Support few weeks before me and worked on Connectors (mostly ODBC, but others as well, for some time), both when they were mentioned in customer issues and in public bug reports. He had reported 64 bugs. I am not sure what happened to him recently, as his latest public bug report is 5 years old and we had no chance to get in touch since I left Oracle in 2012. I hope he still works on MySQL in Oracle, no matter what is his current job role.
  • Dean Ellis - he was (or soon became) AMER team manger in Support when I joined. He was also a key Support provider for AMER and defined many proper procedures over years. The last of them that still affects every MySQL user was so called "BPS process" for setting priorities of bug fixes in Oracle MySQL. It was him who set up the grounds of the process that was later implemented (from Support side) by Sinisa, me and Sveta Smirnova. I see 140 bugs reported but I am sure he had more than one account and reports date back to 2003. Some of his famous bug reports include Bug #4291 and Bug #1118 (implemented only in 5.7.5+). He was a Vice President, Technical Support at MariaDB last time I checked.
  • Vasily Kyshkin (a.k.a. WAX) - he was a key engineer in my Bugs Verification Team when I joined. No wonder he reported just few bugs - our task was to get the counter of open and not processed bugs smaller, not bigger. It had been <90 for months, not 528 as i see now (and 500+ for months recently). So, engineers from bugs team, mostly reported bugs when requested by customers. I was really sorry when he decided to leave MySQL, but the habit to not care much about Support engineers (and even less - about those working on community bug reports) was not invented recently. Some had it back during "good old MySQL" days...
  • Aleksey Kishkin (a.k.a. Walrus) - according to many, he was the best SSC (Support Shift Captain) of MySQL AB times. I can not argue with that (even if I'd like to think it was me :). He worked a lot and was always ready to help with bugs processing. He left MySQL AB in 2006 probably, then eventually returned. Not sure what is he doing now. As a side note, those engineers with nicknames in this list surely worked since the times were there were just a handful of people in entire MySQL... In 2005 I was asked about the preferred nickname (and I used "openxs", you can search bugs on that, it's still my main UNIX user name everywhere, but it was not widely used in the company).
  • Michael Widenius (a.k.a. Monty) - yes, creator of MySQL provided Support for it and worked in tight cooperation with first Support engineers to define some of the approaches and practices of doing Support. This is what I've heard and can conclude based on his bug reports. Check his Bug #18 to understand the state of MySQL quality back in 2002 :)
  • Jorge del Conde - when I joined Jorge worked in Bugs Verification Team, a lot, on all kinds of bugs. But essentially he was a developer in MySQL since 1998 (one of the very first employees). More than 7 last years he works as Senior Developer at Microsoft. I was really sorry back in 2006 when he had to quit from MySQL Support. Not the last time when I could do nothing to keep a team member, but it's always painful to me.
  • Peter Zaitsev - yes, founder and CEO of Percona was a leader of High Performance Team in Support back in 2005. this team worked on complex performance-related issues and run all kinds of benchmarks. Moreover, Peter had SSC shifts scheduled sometimes, and at least once was late for his shift so I had to call him. He had reported 133 public bugs. Check Bug #59899 as one of the last examples of my communication with Peter from the other side of public bugs database.
  • Vadim Tkachenko - current CTO and co-founder of Percona was the only person from MySQL AB whom I've seen in the real life before I joined. He met me to evaluate if I am good enough to join support. We had some coffee at talked about transaction isolation levels and probably something else. Vadim worked with Peter Zaitsev in High Performance Team and formally was a Performance Engineer, but he worked on usual support issues as well and took SSC shifts. Vadim managed to report 23 public bugs while working in MySQL, and his Bug #14347 is still just "Verified".
  • Alexey Kopytov (a.k.a. Kaamos) - he also worked in High Performance Team in 2005 when I joined, and is famous for his sysbench tool and numerous contributions to Percona Server and Percona XtraBackup since 2010 when he joined Percona. He left Percona somewhat silently some time ago, so I am not sure where exactly he is working know, but he still writes a lot about MySQL and keeps reporting bugs (I see 90 of them) and contributing patches. Check his last bug report and patch in Bug #79487.
  • Lachlan Mulcahy - I remember him as a great APAC team manager after Timothy Smith moved back to USA, but quick check shows that he was a Support Engineer when I joined. List of bugs he reported (49 in total) move us back to September 2004, that's probably when he started. Check his Bug #36151 that is still "Verified". Now he is a Production Engineering Manager at Facebook.
  • Sasha Pachev was a developer in MySQL, but some time in during 2006 or 2007 he worked in Support as well, part time. Check his last public bug report with a patch suggested (out of 8 that I see) that is still "Open", Bug #60593. There are also still "Verified" optimizer bugs reported by him.
  • Jani Tolonen was mostly a developer, but he played many roles and worked in Support part time. His 23 public bug reports covers everything from mysql command line client to Maria storage engine. I am not sure what he is doing now.
  • Indrek Siitan - when I joined and for some time after that he played mostly a "full time SSC" (Support Coordinator) role mostly in Support. But quick check for bugs gives a list of 15, starting with Bug #138, so he surely did other work as well. Now he seems to be a Full-Stack Web Developer (and probably he had always been a web developer in one of his roles).
I studied a lot from all these people while working with them. They surely know what MySQL Support is about and contributed a lot both to MySQL Community and to my personal ideas on what's good and bad in MySQL services.

UPDATE: my readers on Facebook noted that there were other engineers as well who worked in Support or who had provided Support in MySQL before I joined and before many people of those I've listed started to do that. I tried to work based on my memory and also, just because of format of this blog, I had to list those who has several real bugs reported in public. I did some corrections and additions in the list based on feedback I've got.

If I missed your name in the list or had written something about your work that is not correct, please, forgive me my old memory and correct me.

Next time I'll write about those great engineers who joined MySQL Support after me and became much better than I'd ever be able to become. Stay tuned!

For now just remember the following statement that I've used more than once in different discussions already:
"There are two kinds of people: people of MySQL and all other people. You can not judge them by the same rules."
I had not changed my mind since the first time I stated that while defending one of colleagues in internal discussion.


  1. Fun fact: I went to high school with Lachlan.

  2. Fun fact: I went to high school with Lachlan.

  3. Domas Mituzas should be in that list.

    1. He will be in another list, see next post "MySQL Support People: The Next Generation". He just joined after me, as many other great engineers.

  4. A great trip down memory lane - thanks for that! In addition to Domas, as Giuseppe notes, I'd suggest Tom Basil. Tom is directly responsible for defining and building the MySQL Support Team culture (probably also at Percona!) which continues years (and two acquisitions) later.

    1. Tom Basil probably deserves a separate post in this series. Maybe I'll write a separate post about all managers (who had not reported bugs and maybe even never played support engineer roles on a regular basis) who, from my point of view, defined or contributed in a positive way to MySQL Support.