Whenever I see benchmark results I try to understand if it is technical benchmark – made by people seeking the truth or it is done by Marketing department to wash your brains. Watch out. Whenever you treat marketing benchmarks as technical ones, you make make wrong decision. Take a look at MySQL 5.0 Benchmarks Whitepaper and guess which type is this ?
You can also compare it to my MySQL Performance 5.0 vs 4.1 presentation to have some fun.
What can we see ? Out of all MySQL 4.1 vs 5.0 benchmarks which were done only benchmarks which show MySQL 5.0 is faster were selected and bunch of other benchmarks which show 5.0 is actually slower than 4.1 were hidden under the table.
In general any benchmarks I see which show something being absolute winner I smell something fishy. In software development there is no free lunch and your decisions to speed something up often will mean something else becomes slower. There are of course exceptions with optimizing very bad code, but it barely applies to MySQL – Monty is not bad developer at all 🙂
So in case of MySQL 5.0 a lot of great features were implemented and new optimizations being added which made certain things much faster, sometimes hundreds of times faster. This however came at cost of larger code size and more complex code which had slowed down some things few percent. This is normal and most software developers would understand it. For non developers reading this article for some reason – think about following comparison – Windows 3.1 used to boot in 10 seconds on my old 486 desktop. Would you expect Windows XP to be able to do the same ?
This was similar with MySQL 4.1 – number of great performance fixes but good number of cases when it become few percent slower (especially due to handling multiple character sets).
Of course I would not say it is normal and nothing can be done in all cases. There are performance bugs, some of them as Innodb Scalability were known long before MySQL 5.0 was stable but were not fixed, others, such as broken group commit could have been handled better.
Same applies to comparing MySQL to other vendors. If you ask me if MySQL is faster than Oracle or PostgreSQL I would say yes, in many cases, but not all cases. There are cases when MySQL sucks and you better to do things differently or use other database. For example it happens with many types of subqueries, cases when hash or sort merge join is needed and in many other advanced optimizer features. I can help in many cases to work around MySQL limitations or design application so it uses MySQL strengths but it is other story.
I am always surprised why Marketing department (not only MySQL Marketing, but any marketing) tries to show only one side of the coin ? Do people really believe it or is it just “everyone does it” and so showing weaknesses together with strengths would be understood as poor product ?
In my opinion honesty and openness is best practice. During my work for MySQL I recommended not to use MySQL for certain applications (ie when queries and schema could not be changed migrating from Oracle) more than once or twice, and interesting enough many of these customers stayed with MySQL for other applications or checked MySQL when new version was released.
In general I also have to admit it is hard to find good technical benchmark these days, competitive benchmarks which are done inside companies are rarely published. Benchmark run by individuals are very often of mediocre quality, performed without proper tuning or with other mistakes. One reason for that is – it is hard to be expert in everything, for example if I would do benchmarked of MySQL vs PostgreSQL and try my best to get best performance out of both of them, I still know MySQL much better and so it would not be 100% correct.
To get best results you need to make Vendors to compete in the same benchmark. It ether can be done by some Industry body – such as TPC or SPEC or by mass media – eWeek benchmark or recent CT benchmark are from this type. Interesting enough even in TPC or SPEC benchmarks vendors try to avoid directly comparable results – for example Oracle and Microsoft would run TPC-C on a bit different configurations, or Sun would not publish SpecJAppServer benchmarks on absolutely same configuration.
Looking at any benchmarks also make sure to check how well they apply to your application. The other big secret about benchmarks is – their results way to often have nothing to do with performance in your real environment.
“Slown?”
Slown = slowed down
Thanks for correction. This is far from only mistaken in this post I guess.
And english is far from my native language 🙂
Peter, Interesting findings indeed. I have to say if all the marketing folks across all industries started giving the flip side of the coin, the world will be a totally different place.
Would love to hear your thoughts on how after upgrading to MySQL 5, an application can achieve the same performance as when it was on MySQL 4 since I’ve been reading that upgrading (without making significant changes) can reduce performance.
Frank
Thanks Frank,
Speaking about MySQL 4 -> 5 upgrade there are variants.
1) If you application was not well optimized for MySQL 4 it may work much faster on MySQL 5 – for example if you used OR on different columns which not works well with index merge.
2) You may get benefit from MySQL optimizations – like index merge may improve performance, some changes making group by more efficient, improved join optimizer and all these little things – it is close to one but more subtle.
Also you could get improvement using Innodb tables due to Compact storage format, especially if your database is close to memory size – in this case saving 10-20% space is very important.
3) The way to benefit from MySQL 5.0 is to use its features but of course it should make sense for your application – for example as you could see in benchmarks DBT2 benchmarks can be made 30% faster with 5.0… but if you use Stored procedures. Also you can use triggers instead of separate updates and these kind of things.
hi i wanted to ask a question abaout benchmarks
can anybody advise me a benchmark software that we can use to
see the performance off oracle and mysql
Ayhan,
please do not place your comments to more than one blog post.
As I mentioned you should really try to benchmark on your workload rather than use existing benchmarks which might not be relevant for your application.
Regarding framework to setup benchmarks – perhaps Quest Benchmark Factory could do