It is not so hard actually - we made a short patch for MySQL 5.0 which show innodb buffer pool content

The binaries, RPMS (RedHat/Centos) and full source code also available to download

P.S. There is Jeremy Cole's patch for InnoDB plugin MySQL 5.1, and main idea was taken from there.

Entry posted by Vadim | 6 comments

Add to: | | | |

Today we have published new search functionality for our site which is based on Sphinx. We have developed it as WordPress plugin which will be available as open source software in a few weeks - just want to test it a bit more and write proper documentation before announcing. If you would like to test it however let us know.

As you can see search functionality allows you to specify what would you like to search (Posts, Pages or Comments) as well as if you would like to sort results be relevance of freshness. You can also use query standard Sphinx query language to search phrases or Search MyISAM in post title.

Besides basic search functionality we've implemented list of last searches and Top searches which if both nice for fun and for SEO.

Hope this addition will make MySQL Performance Blog more useful for you.

UPDATE: For those looking to download plugin code it is here:
https://launchpad.net/wp-sphinx-plugin/+download

Entry posted by peter | 44 comments

Add to: | | | |

Entry posted by Vadim | 4 comments

Add to: | | | |

Because of charter flight schedule this was rather long meeting - we had whole week in the Egypt in 5 star all inclusive resort (FYI: It is nowhere near European/American 5 stars in terms of quality or price). We had work sessions for 4.5 days, One day we took full day excursion to Luxor and another half a day we took introductionary diving and snorkeling tour. Some of us having late plane departure got almost another day to relax on the beach.

If you're wondering about budget we tried to keep it at just over 1000$ per person which varied a bit depending on cost of travel, such as travel from Krakow or Novosibirsk was surely expensive than from Kiev and Moscow.

Being visitor on plenty of conferences and now involved in organizing one I can tell it is quite a challenging job, even having under 20 attendees. It took a lot of time to purchase the trip from 3 different locations and when it took a lot of time and effort running around to organize various issues. Happily we had Michael doing most of the running otherwise we barely could be able to have any meaningful work done.

Arrival day and day after were especially erratic. Our Russian group plane departure was moved from 8:00 to 18:00 and then further delayed couple of hours do they arrived late at night so it was hard to start early in the morning. Also it turned out our Ukrainian tour organizer who booked conference room for us did not communicated that to the hotel, so we did not get it until 12:00 when hotel manager woke up.

Another fun stuff was Internet connection. The room which was sold as having wireless Internet did not have any Internet and wireless Internet worked only in the lobby and not in the room and was extremely expensive (10EUR per hour per person, with 1 hour "card" being maximum you could buy).

We had one laptop with extremely good wireless card which allowed to pick up the signal from another hotel and we brought wireless router with us so we could organize wireless Internet for us. We also could negotiate better bulk price for a week from wireless provider.

During weekend when this all was happening Internet was quite decent however during week day business hours we were pleased with over huge packet loss and pings over 1000ms for most of the time, so it was not easy to work besides emails and (sometimes) instant messaging.

In the end it was very good idea to have Alexey Kovyrin to cover us for Urgent cases while we were not available.

So we had some fun time and fun challenges to overcome together and I would call it quite successful meeting.

We had bunch of good internal presentations on topics such as Mercurial version control system, Sphinx, LVM, Web Application Securities which will be published soon.

More pictures are to be uploaded soon but for now you can take a look at gallery by Aurimas, our underwater photographer.

P.S Some people would likely wonder which hotel we went to. It is Iberotel Makadi Beach

Entry posted by peter | 7 comments

Add to: | | | |

First we now do Data Recovery for MySQL. We're mainly focused on Innodb with this one because it has distinct page structure which allows good data recovery possibilities even in extreme cases such as melt down file system or broken down RAID. We can undelete Innodb data - Heikki was very nice and treats deleted rows very gently allowing them to be 100% recovered if they were not overwritten yet, as well as recover dropped Innodb tables. We also can get some data recovered from covering indexes and other areas, continually improving our tool set.

This service was mainly created on demand - we had Alexa 100 companies contacting us with recovery request because neither MySQL nor Oracle could help them to recover the date so we had to get our hands dirty and Alexey had series of sleepless nights digging into Innodb page and row structure. When few more companies followed, so we continued to improve our recovery toolset and as we have both tools and experience now it would be silly not to announce it to the public.

Recovering Innodb is not the only recovery task we had to do over last year and a bit. We had to recover data using query log, corrupted binary log and dig into MyISAM tables.

You would ask who would ever need such recovery services - should not people just relay on backups ? Sure they should and most of the people we help started to take their backups much more seriously after they had to go through recovery process and lost some of their data, and we helped to implement proper backup strategy for some.

Sometimes backups stop working in others they are never setup, yet another nice case is weekly backups with no binary log enabled when managed hosting support would happily point to the backups but they are too old to be really useful.

The reason to lose data also varies. It could be SQL injection, full server being compromised or staff mistakes. Many of these would not be helped by having the slave MySQL server - so do not think about your Slave as your backup. Hard Drive failure is also less frequent issue - I think everyone understand risk of loosing hard drive well and important data is stored on RAID or at least replicated.

Alexey also has a task to write more details about Innodb data recovery process and release code for our innodb recovery toolset, so you can recover your data for free if you're ready to spend time on it.

The other Service is official Sphinx Support and Consulting . We're working together with Sphinx Project Team to bring you first hand support and consulting services for Sphinx. We can support your production Sphinx installation help you with development or organize custom feature development for Sphinx project.

We have been active Sphinx users from very early stages and we helped many customers to roll out sphinx solutions so this was quite a natural step.

Entry posted by peter | 2 comments

Add to: | | | |

It turns out however there is significant legal differences - first one would be MySQL Trademark violation but not the second one.
I learned it because MySQL contacted me about our consulting company new "corporate" site and rename Services appropriately.

As Monty explained me once MySQL has to protect the trade mark in all instances as otherwise they may loose rights to it. So this is quite understandable. They however are also quite selective - google around and there would be plenty of companies out where which will offer you services named "MySQL Support" or "MySQL Consulting"

The most well case known with MySQL Trademark enforcement so far is perhaps MySQL Front which MySQL asked to be renamed but MySQL Front team instead selected to stop the project.

What worries me is situation with a lot of Open Source project which use MySQL in the name - there is MySQL Toolkit, MySQL Master Master Manager, MySQL Slow Log Analyzer and a lot of others.

Unless developers have secured permission from MySQL these products can be at risk and MySQL may request them to be renamed. As I understand what you're Open Source Project and make no money out of it does not help here.

Other Projects are smarter with naming, like InnoTop or MyTop which hint on the product names but do not use them directly.

Entry posted by peter | 9 comments

Add to: | | | |

Lets hope this will run stable and we'll not need to move it back in emergency in less than a week as we had to last time.

Up to this point it all was running pretty well, with only minor issues. We got CentOS 5 on the box as we wanted, however we could not request custom partitioning - I really prefer to keep all important data on LVM volume so it is easy to backup.

Entry posted by peter | 4 comments

Add to: | | | |

The project generates decent interest and we have about 3000 sites Registered over this time, which I consider decent number especially as we did not do much of advertisement and PR keeping it low profile and working out few bugs which we might have.

We use GeoIP DNS based load balancing between "gathering" servers in Europe and US which seems to work very well both providing level of HA if one of the servers goes down and allowing to increase accuracy by reducing round trip. Over time we are planning to get more locations with pair of servers in each so we do not need to use relatively slow DNS based fail over if one of the servers goes down.

We get some 600 tracking events per second on our lighttpd tracking servers which currently works well and there is still some capacity available but we're still planning to get rid of little PHP code we have left at this layer to get it even more efficient. It would be good to handle some 5000 events/sec per server.

MySQL 5.1 with Partitioning works nicely for data storage with no MySQL bugs hitting us with this project so far. We use Innodb tables now because checking and repairing MyISAM is nightmare and PBXT which could be good for this work is just not ready.

MySQL Performance is in fact the most serious issues we have to work with, even now reporting only clicks statistics for some huge sites like Mininova may take quite a while to generate.

The typical solution for trackers is to have summary data built one way or around and we might need to do some of it for certain most common queries. At this point however we're looking how much performance we can get from real time aggregation because we want absolutely unrestricted dynamic filters and dynamic timezones and this makes things hard to aggregate. This is surely fun challenge to deal with.

We also continue to work on adding more Advertisers we support and improving a ways we track the old ones. Recently we've added support for Vibrant Media Intellitxt (hovers only at this point) . We also now support ShoppingAds even though they are not yet out of private beta version.

Looking at the site we've added major improvement of saved reports - now you can create custom reports with all filters you would want to and save them for quick use at later time. For example you can track what are the most popular click directions for US audience compared to general audience or track performance for referrals from given domain name to see if partnership makes any sense for you.

Finally we've added Demo Account so you do not have to register any more to see system in action. We used our MySQL Performance Forums site for the demo, which might be a bit low traffic but still good to see how system works. This is the reason why we added Google Adsense Adds on that site.

If you have any other ideas what would you like to see implemented in ClickAider, let us know.

Entry posted by peter | 21 comments

Add to: | | | |

We had purchased this 1and1 server few months ago, before we were running into troubles and as it was running well and because we have rather quick and automated fail over if it ever crashes we do not get into too much trouble.
There is however yet another traffic trick surprise which we got into.

When we purchased the server we specially looked for traffic limits and 1and1 looked very good from the glance view. Promising Unlimited Traffic with 100Mbit transfer speed and when we got the server it indeed had 100Mbps connection.

Yesterday we've noticed server responding slow, showing packet loss as well as very large ping times compared to other servers in the same network. Checking with 1and1 support of course was useless - "if you shut down the web server you do not have any packet loss means it is your application problem". I specially asked them if they could have enabled any traffic shaping or have any concurrent connection limit on their firewall or something similar - they told me they do not.

Looking closer at the problem I discovered my NIC is now switched to 10Mbit/Half duplex mode which was not enough at the times.
Checking with 1and1 support again I was told I'm switched to 10Mbps because I used over 1TB of traffic over month.
Now I'm given the option to unblock it in the control panel manually but I have to do this each 250GB of traffic manually after it blocks, which is of course unusable for any real use.

Indeed you can find about this limit in fine print but not from the outlined offer.

This is not the first time I run into the problem. The last one was with other provider and was even funnier. I got the server with
3000GB per month of traffic, however as I started to put any traffic on it I found my server is down and I was told their system simply shuts down the server if it uses more than 3Mbit/sec for more than 24 hours. And that was not even in the fine print.
The funnies thing is if you do the math here - if you use 5Mbit/Sec for 30 days 24 hours with 100% efficiency you can use only 970GB worth of traffic, which means their advertised offer is simply unreachable with such restrictions.

Generally in my experience (which is at large extent focused on inexpensive hosting) I found it much better to work with Co-Location than dedicated server hosting. If you get a rack or two, especially from the smaller company (which will make you significant client) you can get fair treatment. On other hand if you're unlucky if will be harder to move servers to another location than simply stop renting one dedicated server and get another one. Typically used 95% Commit + Overcharge payment is also fair pricing which means hosting provider is interested for me to use more traffic, not less.

There are some surprised in Co-Location market as well but as setting up your own location is more involved you tend to check things more carefully. For example 24/7 support may not be such or it could be a guy living 30 miles from data center which can come where within an hour during the night if you need him. The "remote hands" may also wary a lot in price and quality.

The other interesting surprises I had were about power consumption. First you really need to make sure what power consumption is stated. Many data centers even though they give you 20A of power will only allow 80% peak use, which makes it 16A. In other cases you could get the amperage you can use fully. Of course I do not speak about 110V vs 208V difference which is pretty obvious one. Another gotcha with power is power availability and density - quite often there would not be extra power available or power density allowed per rack will be very low. Hurricane Electric in one of their data centers for example allows 15A of power per rack and this is it - want more power buy another rack.

Going forward we plan to host our consulting business web sites MySQLPerformanceBlog, Percona high quality dedicated server, mainly to keep it isolated from everything else and be secure, while for our other projects such as ClickAider as development and test servers and some other projects we have in works we will need to organize our little data center in the next few months.

For ClickAider this however would not be good enough - to minimize latency we need several GEOIP based locations which gather the data and that would be way to expensive to organize multiple data centers. So for the coming months I will be on lookout for dedicated physical or virtual servers in different parts of the world.

Entry posted by peter | 12 comments

Add to: | | | |

Our goals are rather simple though the fact we're globally distributed may put us a bit aside from typical small business needs.

Normal Phone Line We're looking to get normal business phone line which would be routed to me and Vadim withing our normal business hours and voicemail outside of them. The same line could offer extensions to reach other team members plus. Plus we'd like to get direct numbers for me and Vadim at least so people do not have to pass through choice system each time. And Yes... Voicemail should be sent by email so it is easy to read it. We would like to keep using normal phones for this kind of service, while being able to use software or hardware VOIP calls for dial out. Phone conferencing features would be good to have to add people to the line.

Emergency Phone Line Providing Emergency Support to our clients we need to put better systems in place. Gurrently we give away few land lines and mobile numbers to reach us and it works well but does not look professional and neither scales as we get more people. We would like instead to get On Call schedule so calls are automatically routed to this person land and mobile numbers and if he does not replies in time followed escalation procedure to get at least anyone to the phone. The alternative is of course to use external on call center and make them to take the callers number reach someone and make him to call back, but I do not like this. From my personal experience if I have an issue I want to get to someone who can help rather than promise for someone to call back.

Of course both lines should be multi-line ie "never busy" and phone numbers should be ours so we can switch solutions while keeping phone numbers.

We considered two ways to implement it. First one is to use Asterisk or some derivatives as TrixBox. This way is flexible and geeky while I do not think we want to get into all aspects to get it usable as business critical solution.

The other one is to use a hosted service so we do not need to support solution ourselves and can simplify setup. However this means this someone should be good both in providing great service quality - uptime, call quality for international calls, support and flexible enough to accomodate our needs.

Any advice you could give us on this topic ?

Entry posted by peter | 10 comments

Add to: | | | |