Have you seen recent eWeek benchmarks which test OpenSource stacks and compare them to .NET ? Here is what was compared and here are results.
Results make me ask number of questions – why WAMP would perform 6 times better than LAMP ? Why Python would be faster than PHP on Windows but slower on Linux ? Without having such basic questions answered I can’t accept results as valid. If you run benchmark and something unexpected pops up drill it down – quite frequently it would be configuration problem, bug or something similar.
Well, this is however not my main concern with this benchmark. Here is to start:
No Specifications – Good benchmark starts with specifications. What exactly bechmarked system has to be doing ? Normally you would try to optimize system to do only what needs to be done and strip everything else and configure it appropriately. For example different CMS could be doing different things internally – some could be doing live full text search indexing on update, page access statistics storage etc.
No configuration published At least I could not find them – how exactly all of the stacks were configured, what hardware was used etc. Full disclosure report is very important component of any serious benchmark. It should be detailed enough for third party to be able to repeat same results.
Missleading benchmark title – Results are called “stack comparison” while I would rather call it benchmark of different content management systems. Why XOOPS is choosen for PHP or Plone for Python ? Different projects have different quality and different performance goals.
Getting back to results – I’m most surprised about WAMP vs LAMP results (and similar results for Python) – Typically CMS would be CPU bound and unless there are configuration differences, such as acceleratior (eaccelerator, APC) installed on one system but not on the other, so results should be close. There could be 20% difference because of different compliers and OS efficiency but it should not be that high.
If someone knows what really was done let me know. I would be quite curious to take a look at least at WAMP vs LAMP results
Well, they once posted a nice database shootout, this one seems just a non-comparable bunch of data and fud
Pabloj,
Yes. Database comparison was pretty good, and not because MySQL sored top marks in it :). If you would look at list of my concerns they were all pretty much targeted in that benchmark. It was real application which was same for all database vendors which counts as benchmark specs. It was published, so were hardware configuration and database server settings. If I’m not mistaken IBM even repeated the runs with new DB2 versions and showed they are not that bad later on.
1. “why WAMP would perform 6 times better than LAMP ?”
LAMP needs expert tuning to give good performance.
2. “Why Python would be faster than PHP on Windows but slower on Linux ?”
Python relies heavily on OS thread performance, and on Windows it’s better than on Linux.
Overall, you’re right, this benchmark does suck bigtime. 🙂
Alexey,
Honestly I do not buy it 🙂 Tuning of LAMP stack on Windows and Linux is at 95% same – installing PHP accelerator, using proper modules, configuring apache, MySQL Server configuration etc. Going deeper to change kernel options normally gives just few percent difference – OS vendors try to have reasonable defaults. Of course if some “WAMP in the box” which is reasonably optimized was used on Windows while stock Apache+MySQL were used on Linux it could explain the difference.
Speaking about threads… I would not expect it to be that different. MySQL also uses threads very heavily but performance is not that much different.
Well. There is nothing to argue here. We’re just poking finger in the sky until there is some kind of disclosure done.
I think it is obvious when you look at the last slide. It has a screenshot of the software that they used and it was a windows program evaluating localhost. Do you think the loopback would give better performance than physical ethernet?
When I ping localhost vs the default gateway, I get a ping time of 10 times better. Just that alone would account for some of the 10 fold problems with windows vs Linux for the rapid fire tests.
As for the download time, when did it take you to download ANY web page 120 seconds? 2 minutes on a web page? That sounds like a timeout from misconfiguration, not any Linux server I’ve ever used. Maybe they set things up to grab the wrong script or talk to the wrong database (maybe not a local database, but one on the windows box)?