Twitter's Calvin Sun

Twitter’s Calvin Sun

Twitter’s Calvin Sun (@Calvinsun2012) is looking forward to the fast-approaching Percona Live MySQL Conference and Expo this April 1-4 in Santa Clara, Calif. He’ll be speaking, yes, but he’s also looking forward to learning from his peers – particularly those at Facebook. Both companies, he explained, are in the rather unique positions of unprecedented rapid growth and ever-expanding performance demands.

Calvin is a senior engineering manager at Twitter, where he manages MySQL development. Prior to that, he managed the InnoDB team at Oracle. Calvin also worked for MySQL Inc. from 2006 to 2008, managing MySQL storage engines development. He has over 15+ years of database development experience.

He said MySQL is the data storage technology behind most Twitter data: the social graph, timelines, users, direct messages, as well as tweets themselves. With more than 230 million monthly active users and 500 million tweets per day, Twitter pushes MySQL and InnoDB a lot further than most companies. I had an opportunity to chat with Calvin via Skype the other day. Here’s that interview.

Tom: Aside from your talk, “MySQL at Twitter,” which we’ll get to soon, what are you looking forward to the most at this year’s Percona Live MySQL Conference and Expo?

Calvin: This year I’m particularly eager to attend the Facebook sessions because Twitter faces some of the same challenges as they do. Learning from each other within the MySQL community is very valuable and I look forward to this event every April.  For me, and my team, we always get the most excited about the opportunities to learn from the experiences of other vendors and other MySQL users.

We are also very pleased to see Oracle join this conference for the second year in a row. Their attendance is very beneficial to the MySQL community because we want to better understand what’s going on in MySQL development at Oracle. And hopefully we’ll also have some visibility and perhaps even some influence into their roadmap in terms of upcoming releases.

Multi-datacenter support and fast failover are also areas we are interested in. And of course since I’m from the InnoDB team within MySQL, I’d like to see what’s new there, too.

And finally, I’m looking forward to participating in a panel Q&A with our DBA team. The session is called “MySQL at Twitter” and in it we’ll field questions on how MySQL is used within Twitter, along with the challenges we have now and have faced in the past. It will be led by my colleague Will Gunty, a senior MySQL DBA.

Tom: Let’s take a look at some of those Facebook sessions you said you’re interested in – they have several spread out over each day of the conference:

Wow, that is a fantastic collection of sessions just from Facebook alone!

Calvin: Facebook is probably the largest MySQL installation in the world and they are in a lot of ways facing some of the same challenges we are facing. Facebook has a bigger MySQL team than we do and are also a little bit ahead of us, especially on MySQL 5.6 and InnoDB compression, so we can learn new things from those sessions – in addition to having direct conversations with members of the Facebook team outside of those sessions, as we always do. They have created a lot of innovative solutions to address their problems.

Tom: You’ll be leading a session titled “Scaling Twitter with MySQL” along with a couple members of your team.  With Twitter’s explosive growth since its founding in 2006, and as that activity continues to grow with Twitter being used in new ways, how has your team kept pace? And how are you preparing for the future?

Calvin: It is a huge scale. During my session I’ll talk about how MySQL evolved within Twitter over the years. Many small companies just starting out, including Twitter in the early days, use master-slave replication. But as Twitter started to grow, that sort of setup could no longer meet our needs, so over the years we starting developing our own sharding and replication framework. We still use MySQL as our basic data storage, though, and working creatively like this enabled us to scale fast.

For us, usage is in real-time and is event driven.  For example, the World Cup or the Olympics or the Super Bowl – those types of very popular events send peaks up several times higher than average. For that reason we have to always be prepared for whatever sort of usage levels we might encounter.

Tom: Who should make it a point to attend your session?

Calvin: I would say people working at both large companies as well as startups, especially Internet companies that want to use MySQL and who need to learn how to prepare for growth. Also anyone who needs to ensure that their company’s architecture, their infrastructure, is in fact prepared for rapid growth if needed. They’ll learn how Twitter prepared for and addressed specific issues.

Tom: What will they walk away with?

Calvin: They’ll walk away with 1) how to build a scalable infrastructure using MySQL and 2) how to plan for growth.

Tom: What are some of the bigger projects or challenges you and your team face in 2014?

Calvin: One of the bigger things we are in the process of doing is MySQL 5.6 migration. But for Twitter overall, I think that as it grows, ensuring reliability and scalability has always been a priority for us. We are also looking to do a better job at failover – above and beyond failover between our data centers. So we’ll be looking to improve that kind of operational efficiency this year.

Another area is optimization. We’re looking into the structure of our MySQL installation within Twitter to see which areas we can improve; make more efficient. One area is data compression, which we currently don’t use in production. We want to see what we can achieve in reducing data sizes and saving disk spaces.

Tom: Before we started the interview you mentioned that you’ve been attending these MySQL conferences since 2006. Why do you feel it is important to be there personally?

Calvin: Because I strongly support the MySQL community in general and try to do my best to participate and serve the community. For example last year I gave presentations on MySQL/InnoDB in Beijing and Seoul, plus OSCON and MySQL Connect. Those in addition to Percona Live and related events give me a chance to share my own experiences and lessons learned with the community. As I’ve already mentioned it also gives me the chance to learn.

Tom: Thanks very much for speaking with me today, Calvin. I look forward to seeing you in April! And as a small token of my appreciation, here’s a discount code you can share with your friends and colleagues (everyone reading this can take advantage of it, too). Just have them enter “SeeMeSpeak” on the Percona Live 2014 registration page and they’ll automatically save 10 percent.

Calvin: Thanks, Tom. We really appreciate the hard work Percona and the Conference Committee does in terms of organizing these MySQL conferences with all of the excellent sessions. See you in April!

* * *

Update: April 9, 2014
I had the opportunity to interview Calvin last week at Percona Live 2014 about the new WebScaleSQL initiative. WebScaleSQL is a collaboration among engineers from several companies – including Twitter, Google Facebook and LinkedIn – that face similar challenges in running MySQL at scale, and seek greater performance from a database technology tailored for their needs.