July 22, 2014

High Availability with MySQL Fabric: Part II

This is the third post in our MySQL Fabric series. If you missed the previous two, we started with an overall introduction, and then a discussion of MySQL Fabric’s high-availability (HA) features. MySQL Fabric was RC when we started this series, but it went GA recently. You can read the press release here, and see this blog post from Oracle’s Mats Kindahl for more details. In our previous post, we showed a simple HA setup managed with MySQL Fabric, including some basic failure scenarios. Today, we’ll present a similar scenario from an application developer’s point of view, using the Python Connector for the examples. If you’re following the examples on these posts, you’ll notice that the UUID for servers will be changing. That’s because we rebuild the environment between runs. Symbolic names stay the same though. That said, here’s our usual 3 node setup:

For our tests, we will be using this simple script:

This simple script requests a MODE_READWRITE connection and then issues selects in a loop. The reason it requests a RW connector is that it makes it easier for us to provoke a failure, as we have two SECONDARY nodes that could be used for queries if we requested a MODE_READONLY connection. The select includes a short sleep to make it easier to catch it in SHOW PROCESSLIST. In order to work, this script needs the test.test table to exist in the mycluster group. Running the following statements in the PRIMARY node will do it:

Dealing with failure

With everything set up, we can start the script and then cause a PRIMARY failure. In this case, we’ll simulate a failure by shutting down mysqld on it:

While this happens, here’s the output from the script:

The ‘sleeping 1 second and reconnecting’ line means the script got an exception while running a query (when the PRIMARY node was stopped, waited one second and then reconnected. The next lines confirm that everything went back to normal after the reconnection. The relevant piece of code that handles the reconnection is this:

If fcnx.reset_cache() is not invoked, then the driver won’t connect to the xml-rpc server again, but will use it’s local cache of the group’s status instead. As the PRIMARY node is offline, this will cause the reconnect attempt to fail. By reseting the cache, we’re forcing the driver to connect to the xml-rpc server and fetch up to date group status information. If more failures happen and there is no PRIMARY (or candidate for promotion) node in the group, the following error is received:

Running without MySQL Fabric

As we have discussed in previous posts, the XML-PRC server can become a single point of failure under certain circumstances. Specifically, there are at least two problem scenarios when this server is down:

  • When a node goes down
  • When new connection attempts are made

The first case is obvious enough. If MySQL Fabric is not running and a node fails, there won’t be any action, and clients will get an error whenever they send a query to the failed node. This is worse if the PRIMARY fails, as failover won’t happen and the cluster will be unavailable for writes. The second case means that while MySQL Fabric is not running, no new connections to the group can be established. This is because when connecting to a group, MySQL Fabric-aware clients first connect to the XML-RPC server to get a list of nodes and roles, and only then use their local cache for decisions. A way to mitigate this is to use connection pooling, which reduces the need for establishing new connections, and therefore minimises the chance of failure due to MySQL Fabric being down. This, of course, is assuming that something is monitoring MySQL Fabric ensuring some host provides the XML-PRC service. If that is not the case, failure will be delayed, but it will eventually happen anyway. Here is an example of what happens when MySQL Fabric is down and the PRIMARY node goes down:

This happens when a new connection attempt is made after resetting the local cache.

Making sure MySQL Fabric stays up

As of this writing, it is the user’s responsibility to make sure MySQL Fabric is up and running. This means you can use whatever you feel comfortable with in terms of HA, like Pacemaker. While it does add some complexity to the setup, the XML-RPC server is very simple to manage and so a simple resource manager should work. For the backend, MySQL Fabric is storage engine agnostic, so an easy way to resolve this could be to use a small MySQL Cluster set up to ensure the backend is available. MySQL’s team blogged about such a set up here. We think the ndb approach is probably the simplest for providing HA at the MySQL Fabric store level, but believe that MySQL Fabric itself should provide or make it easy to achieve HA at the XML-RPC server level. If ndb is used as store, this means any node can take a write, which in turns means multiple XML-PRC instances should be able to write to the store concurrently. This means that in theory, improving this could be as easy as allowing Fabric-aware drivers to get a list of Fabric servers instead of a single IP and port to connect to.

What’s next

In the past two posts, we’ve presented MySQL Fabric’s HA features, seen how it handles failures at the node level, how to use MySQL databases with a MySQL Fabric-aware driver, and what remains unresolved for now. In our next post, we’ll review MySQL Fabric’s Sharding features.

About Fernando Ipar

Fernando is part of Percona's team working as Senior Consultant. Prior to joining Percona, Fernando worked as a consultant for financial services institutions, telcos, and technology providers.


  1. MWM says:

    Dear Fernando,

    Do we have switchover command in MySQL Fabric like we have in mysqlrpladmin utility.

    Once our down server gets up and become part of the Group as Secondary then deliberately down the primary server is not a good option. We have the option of server-status to change the primary one to secondary and secondary to primary. What is the best approach to resolve this issue.


  2. @MWM: I think an analogue in Fabric to mysqlrpladmin’s switchover command would be the group promote command, when specifying an explicit slave id.

    For example, suppose the cluster looks like this:

    > [vagrant@store ~]$ mysqlfabric group health mycluster
    > Command :
    > { success = True
    > return = {‘ae27d4c7-e752-11e3-a747-0800274fb806′: {‘status’: ‘SECONDARY’, ‘is_alive’: True, ‘threads’: {}}, ‘c4a19aad-e752-11e3-a747-0800274fb806′: {‘status’: ‘SECONDARY’, ‘is_alive’: True, ‘threads’: {}}, ‘b8816184-e752-11e3-a747-0800274fb806′: {‘status’: ‘PRIMARY’, ‘is_alive’: True, ‘threads’: {}}}
    > activities =
    > }

    You can change the PRIMARY role to the first node in the list with this command:

    > [vagrant@store ~]$ mysqlfabric group promote mycluster –slave_id=ae27d4c7-e752-11e3-a747-0800274fb806
    > Procedure :
    > { uuid = 911e77e3-fafa-4881-a3a9-2fffa57594fc,
    > finished = True,
    > success = True,
    > return = True,
    > activities =
    > }
    > [vagrant@store ~]$ mysqlfabric group health mycluster
    > Command :
    > { success = True
    > return = {‘ae27d4c7-e752-11e3-a747-0800274fb806′: {‘status’: ‘PRIMARY’, ‘is_alive’: True, ‘threads’: {}}, ‘c4a19aad-e752- 11e3-a747-0800274fb806′: {‘status’: ‘SECONDARY’, ‘is_alive’: True, ‘threads’: {}}, ‘b8816184-e752-11e3-a747-0800274fb806′: {‘status’: ‘SECONDARY’, ‘is_alive’: True, ‘threads’: {}}}
    > activities =
    > }

    Does that answer your question?

  3. MWM says:

    Thank you Fernando. Its the answer.

    Few more questions coming into my mind regarding data replication and data integrity / data loss. Let me first experiment few scenarios then ill come back to blog soon. :)


Speak Your Mind