In my previous post, I blogged about using Percona Server with Docker and have shown you how fast and easy it was to create a virtual environment with just a few commands.
This time I will be showing you how to setup a three-node Percona XtraDB Cluster (PXC) 5.6 on the Docker open-source engine. Just to review Docker… “is an open-source engine that automates the deployment of any application as a lightweight, portable, self-sufficient container that will run virtually anywhere.”
In this case we will make use of a Dockerfile, think of this more like the Vagrantfile, it is a build script with a set of commands automating the creation of a new docker container.
For this case, we will use the following Dockerfile contents and use Ubuntu 12.04 instead of CentOS 6.5 as guest OS:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | FROM ubuntu:precise MAINTAINER Percona.com <info@percona.com> # Upgrade RUN apt-get update && apt-get upgrade -y && apt-get dist-upgrade -y ENV DEBIAN_FRONTEND noninteractive RUN echo "deb http://repo.percona.com/apt precise main" >> /etc/apt/sources.list.d/percona.list RUN echo "deb-src http://repo.percona.com/apt precise main" >> /etc/apt/sources.list.d/percona.list RUN apt-key adv --keyserver keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A RUN apt-get update RUN apt-get install -y percona-xtradb-cluster-56 qpress xtrabackup RUN apt-get install -y python-software-properties vim wget curl netcat |
Create a my.cnf file and add the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | [MYSQLD] user = mysql default_storage_engine = InnoDB basedir = /usr datadir = /var/lib/mysql socket = /var/run/mysqld/mysqld.sock port = 3306 innodb_autoinc_lock_mode = 2 log_queries_not_using_indexes = 1 max_allowed_packet = 128M binlog_format = ROW wsrep_provider = /usr/lib/libgalera_smm.so wsrep_node_address ={node_IP} wsrep_cluster_name="mydockerpxc" wsrep_cluster_address = gcomm://{node1_ip},{node2_ip},{node3_ip} wsrep_node_name = {node_name} wsrep_slave_threads = 4 wsrep_sst_method = xtrabackup-v2 wsrep_sst_auth = pxcuser:pxcpass [sst] streamfmt = xbstream [xtrabackup] compress compact parallel = 2 compress_threads = 2 rebuild_threads = 2 |
Build an image from the Dockerfile we just created.
1 | root@Perconallc-Support /home/jericho.rivera/docker-test # docker build -rm -t ubuntu_1204/percona:galera56 . |
The ‘docker build’ command will create a new image from the Dockerfile build script. This will take a few minutes to complete. You can check if the new image was successfully built:
1 2 3 4 5 6 | root@Perconallc-Support /home/jericho.rivera/docker-test # docker images | grep ubuntu ubuntu_1204/percona galera56 c2adc932aaec 9 minutes ago 669.4 MB ubuntu 13.10 5e019ab7bf6d 13 days ago 180 MB ubuntu saucy 5e019ab7bf6d 13 days ago 180 MB ubuntu 12.04 74fe38d11401 13 days ago 209.6 MB ubuntu precise 74fe38d11401 13 days ago 209.6 MB |
Now we will launch three containers with Percona XtraDB Cluster using the new docker image we have just created.
1 | root@Perconallc-Support /home/jericho.rivera/docker-test # for n in {1..3}; do docker run --name dockerpxc$n -i -t -d ubuntu_1204/percona:galera56 bash; done |
Check if the new containers were created:
1 2 3 4 5 | root@Perconallc-Support /home/jericho.rivera/docker-test # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES dd5bafb99108 ubuntu_1204/percona:galera56 bash 5 minutes ago Up 5 minutes dockerpxc3 01664cfcbeb7 ubuntu_1204/percona:galera56 bash 5 minutes ago Up 5 minutes dockerpxc2 2e44d8baee99 ubuntu_1204/percona:galera56 bash 5 minutes ago Up 5 minutes dockerpxc1 |
Get relevant information from the container using ‘docker inspect’ command which by default will show a JSON-format output on the terminal. Since we only need to get the IP address for each container just run the following commands:
1 2 3 4 5 6 | root@Perconallc-Support /home/jericho.rivera/docker-test # docker inspect dockerpxc1 | grep IPAddress "IPAddress": "172.17.0.2", root@Perconallc-Support /home/jericho.rivera/docker-test # docker inspect dockerpxc2 | grep IPAddress "IPAddress": "172.17.0.3", root@Perconallc-Support /home/jericho.rivera/docker-test # docker inspect dockerpxc3 | grep IPAddress "IPAddress": "172.17.0.4", |
Take note of the IP address because we will need them later.
1 | wsrep_cluster_address=gcomm://172.17.0.2,172.17.0.3,172.17.0.4 |
Do the same for dockerpxc2 and dockerpxc3 nodes. To attach to a container you need to run ‘docker attach {node_name}’ (eg # docker attach dockerpxc1). To exit without stopping the containers you need to hit CTRL+p/CTRL+q, otherwise an explicit ‘exit’ command on the prompt will drop you out of the container and stop the container as well, as much as possible we try to avoid this. Also make sure to edit wsrep_node_name and wsrep_node_address accordingly.
Bootstrapping the Cluster
Next step is to start the first node in the cluster, or bootstrapping.
1 2 3 | root@Perconallc-Support /home/jericho.rivera/docker-test # docker attach dockerpxc1 root@2e44d8baee99:# /etc/init.d/mysql bootstrap-pxc |
After bootstrapping the first node, we can then prepare the first node for SST. That means we need to create the SST auth user, and in this case it is wsrep_ss_auth=pxcuser:pxcpass.
After adding the SST auth user on the first node the next step would be to start dockerpxc2 and dockerpxc3:
1 2 3 | root@Perconallc-Support /home/jericho.rivera/docker-test # docker attach dockerpxc2 root@01664cfcbeb7:/# /etc/init.d/mysql start |
After starting all nodes, check the status of the entire cluster:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | root@2e44d8baee99:/# mysql -e "show global status like 'wsrep%';" +------------------------------+--------------------------------------+ | Variable_name | Value | +------------------------------+--------------------------------------+ | wsrep_local_state_uuid | 7ae2a03e-d71e-11e3-9e00-2f7c2c79edda | | wsrep_protocol_version | 5 | | wsrep_last_committed | 1 | | wsrep_replicated | 1 | | wsrep_replicated_bytes | 270 | | wsrep_repl_keys | 1 | | wsrep_repl_keys_bytes | 31 | | wsrep_repl_data_bytes | 175 | | wsrep_repl_other_bytes | 0 | | wsrep_received | 18 | | wsrep_received_bytes | 2230 | | wsrep_local_commits | 0 | | wsrep_local_cert_failures | 0 | | wsrep_local_replays | 0 | | wsrep_local_send_queue | 0 | | wsrep_local_send_queue_avg | 0.333333 | | wsrep_local_recv_queue | 0 | | wsrep_local_recv_queue_avg | 0.000000 | | wsrep_local_cached_downto | 1 | | wsrep_flow_control_paused_ns | 0 | | wsrep_flow_control_paused | 0.000000 | | wsrep_flow_control_sent | 0 | | wsrep_flow_control_recv | 0 | | wsrep_cert_deps_distance | 1.000000 | | wsrep_apply_oooe | 0.000000 | | wsrep_apply_oool | 0.000000 | | wsrep_apply_window | 1.000000 | | wsrep_commit_oooe | 0.000000 | | wsrep_commit_oool | 0.000000 | | wsrep_commit_window | 1.000000 | | wsrep_local_state | 4 | | wsrep_local_state_comment | Synced | | wsrep_cert_index_size | 1 | | wsrep_causal_reads | 0 | | wsrep_cert_interval | 0.000000 | | wsrep_incoming_addresses | ,, | | wsrep_cluster_conf_id | 11 | | wsrep_cluster_size | 3 | | wsrep_cluster_state_uuid | 7ae2a03e-d71e-11e3-9e00-2f7c2c79edda | | wsrep_cluster_status | Primary | | wsrep_connected | ON | | wsrep_local_bf_aborts | 0 | | wsrep_local_index | 2 | | wsrep_provider_name | Galera | | wsrep_provider_vendor | Codership Oy <info@codership.com> | | wsrep_provider_version | 3.5(r178) | | wsrep_ready | ON | +------------------------------+--------------------------------------+ |
All three nodes are in the cluster!
Install net-tools to verify the default port for Galera.
1 2 3 4 5 6 7 8 | root@dd5bafb99108:/# netstat -ant Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:4567 0.0.0.0:* LISTEN tcp 0 0 172.17.0.4:33875 91.189.88.153:80 TIME_WAIT tcp 0 0 172.17.0.4:56956 172.17.0.2:4567 ESTABLISHED tcp 0 0 172.17.0.4:57475 172.17.0.3:4567 ESTABLISHED tcp6 0 0 :::3306 :::* LISTEN |
Summary
I’ve shown you the following:
* Create a Dockerfile and my.cnf file
* Build a docker container using the created Dockerfile
* Made a few changes on some wsrep_ options on my.cnf
* Bootstrap the first node
* Start the other nodes of the cluster
* Check cluster status and confirmed all nodes are in the cluster
There are other ways to setup Docker with Percona XtraDB Cluster 5.6 such as using vagrant + docker which further automates the whole process or by using shell scripts, this article however shows you the basics of how to accomplish the same task.
On my next post, I will show how to setup Percona ClusterControl on another Docker container and adding this three-node PXC 5.6 cluster to it.
Read related posts here:
* Using MySQL Sandbox with Percona Server
* Testing Percona XtraDB Cluster with Vagrant
* Percona XtraDB Cluster: Setting up a simple cluster
Jericho,
Nice! Though I wonder if it is possible to automate it to be more user friendly. As a user getting test cluster what I would do is to get back the cluster nodes IP (without messing with setting them in config) or might be even IP of HaProxy which is staying in front of the cluster.
Peter,
I think Pipework [Software-defined networking for Linux Containers] would be the solution here, although I’ve read that there are other “hackish” ways to accomplish the job. Currently, Docker is in heavy development and I would assume that eventually it would be possible to explicitly set each node’s IP much like how vagrant does it with “private network”.
You might want to use ADD in your dockerfile to add the percona.list and my.cnf files.
Daniel,
Thanks for pointing that out.
Yes indeed, we can include the ADD option in the dockerfile like below:
ADD ./percona.list /etc/apt/sources.list.d/percona.list
ADD ./my.cnf /etc/my.cnf
Assuming the file contents are correct, it would make the process of provisioning the node clusters much simpler.
Isn’t it a very bad style to rely on the IP-address of the container?!
Most likley the IP is going to change after i.e. a restart. Propagating the usage of docker should not use any IP at all.
Also configuration is done *not* accessing the container :/
@erkules,
Its about showing the basics of setting up Percona XtraDB Cluster 5.6 on Docker, regarding whether this is the safest way to do container networking is beyond the scope of this blog IMHO. Readers may refer to Docker’s documentation for further reference here: https://docs.docker.com/articles/networking/#communication-between-containers (which is an advanced topic) and https://docs.docker.com/userguide/dockerlinks/.
In hindsight, readers should be aware that this is only good for testing purposes and not fit (maybe not yet) for production use. I’ve also encountered some limitations with docker and on those rare occasions I tend to use Virtualbox. As I’ve also said before, docker is still heavily developed and there could still be changes on how the setup can be done.
First I just wanted to say you a big thank you for this tutorial, I’m a beginner with docker and it has been a great start point for me.
However, I’d pinpoint that it is mentioned a bit late in the article that the TTY must be detached with a ctrl-p ctrl-q.
Especially I’ve struggled with this command before realizing it:
for n in {1..3}; do docker run –name dockerpxc$n -i -t -d ubuntu_1204/percona:galera56 bash; done
Hello Jericho,
Unable to execute below command , not getting keys
RUN apt-key adv –keyserver keys.gnupg.net –recv-keys 1C4CBDCDCD2EFD2A
create this docker image using coreos Virtual machine , any one please let me know how this command we have executed ?