One more note about Quality I’m speaking about here. I think the Quality of release should be mentioned by “weakest link” – if Partitions or Row Level Replication is unstable in MySQL 5.1 GA it is unstable.

At the same time this unstable release may work for your application or actually a lot of applications which simply do not touch this unstable features.

This brings us to another factor – whenever you should use given release for your application in production depends on whenever it works well or it does not. I see GA status not as seal of quality but rather as a promise there are no serious changes going to be done any more so probability of things getting broken is lower compared to early down the road.

For example my application may well work with given Alpha release but not with next one

The fact you do the split developers=employees and users are everybody else means you seems to be thinking about standard close source model. There is indeed fractions of percent of people who can make sense of the technical data in bugs database but with millions of users it gives thousands of people already. I know there are such people in Google, Facebook, Percona, OpenQuery.

But more than that a lot of bug actually have some meaning (or potential meaning) which is clear to the users something like “Innodb could fail to recover with certain log file sizes” – The details it only happen for prime innodb log file numbers because of unexpected hash collision may not be needed for users.

Other things – you’re speaking about various tests you’ve recently developed – I assume these are kept internal and not released as open source ?

Crashing is only one case of the bug. It is the most critical but there also a lot of things from wrong results bad plans and annoyances.

I also do not thing single application, or even single company is enough. If your application uses pretty static set of features sooner or later you have them worked out or worked around and version will run stable for you.

There are also some tricky bugs like race conditions or memory corruptions which can happen in very special conditions which rarely happen during operation – in this case running thousands of boxes makes a difference.

It is also a good question how many of MySQL 5.0 features do you use. I have a feeling you’re mainly using MySQL 4.0 set of features ?

Hey Mark.

I think this does bring up a good point. I think Peter brought it up before.

MySQL really needs to be commended for their open bug database and open source repository. Even if it’s read only. I think we often forget how open they really are even if we want to see improvement.

I’d really like to see Oracle/InnoDB do the same thing. At least with MySQL 5.1 I can look at the bugs to judge the quality of the release before I upgrade.

With InnoDB 5.1 I don’t really have this luxury.

Peter.

I agree that 4.1 wasn’t perfect but for my needs it was nearly perfect. I didn’t use subqueries or charsets (we use UTF-8 encoding via the JDBC driver).

Ken,

“While community and user testing of released software is helpful, it is never a replacement for good design and implementation and robust testing methodology.”

While true, the reverse is also true.

While a robust testing methodology is helpful, it is never a replacement for an open and vibrant open source community and a distributed array of inexpensive unit testers (customers) with access to an open bug database and source repository.

I think we’re a good example. Very little of the work done by MySQL and Oracle/InnoDB over the last few years has been applicable to my company. The work from Google, Percona, and OurDelta has been a breath of fresh air.

“Although the InnoDB Plugin was released in ‘early adopter’ status last April, only two InnoDB Plugin bugs were reported by users in the MySQL bug database.”

This ends up becoming at catch-22… You have to work towards developing an open environmnent for reporting bugs and interacting with teh community.

If you don’t receive extensive bug reports at first don’t give up… The MySQL database in and of itself is good evidence that people are willing to report bugs.

“Just a note on our internally-detected and fixed bugs and the idea of publishing them. Most of our internally-detected bugs involve complex testing scripts and workloads, and logs containing debugging information. This sort of information is helpful for developers, but is not meaningful to users.”

… so then publishing this information can’t hurt anything.

We share all the internal statistics about Spinn3r with our customer base. Every statistic you can think of is exported, graphed, and recorded. Believe me it’s appreciated

In the past year, we have significantly improved our quality assurance (QA) process, resulting in more thorough and systematic testing, including specifically a focus on recovery testing. Many of the problems we have identified … and fixed … in the InnoDB Plugin probably never would have been found and reported by users. Thankfully, most of the time, users don’t run recovery operations, but when they do, they must work. Many recovery-related bugs are not easily reproducible and it is difficult to capture the information required to do so. The kind of stress tests needed to identify timing problems (like race conditions) are not typical of early-adopter workloads. The performance testing we did before the initial release of the InnoDB Plugin lead to the implementation the adaptive LRU mechanism that adjusts the behavior of the system based on whether the workload is cpu or i/o bound.

While community and user testing of released software is helpful, it is never a replacement for good design and implementation and robust testing methodology. Often, early adopter users don’t report bugs (because they may think the problem has been reported by other users or has already been fixed in the next release). Although the InnoDB Plugin was released in “early adopter” status last April, only two InnoDB Plugin bugs were reported by users in the MySQL bug database. A small number of other bugs were reported by users on the InnoDB forums. Most of the bugs were found through extensive internal testing. And, on the other hand, some of the earliest tests of the InnoDB plugin were extremely positive both in terms of performance and reliability.

Just a note on our internally-detected and fixed bugs and the idea of publishing them. Most of our internally-detected bugs involve complex testing scripts and workloads, and logs containing debugging information. This sort of information is helpful for developers, but is not meaningful to users.

We continue to work closely with the Sun/MySQL engineers to maintain the reliability of InnoDB as distributed by MySQL as well as in the InnoDB Plugin. We hope that together we can ensure that available fixes (some 22 of them at the moment) are incorporated more quickly into the standard MySQL distribution.

I really should clarify something. I should have spoken about “Perceived Quality” rather than just Quality. You can find a lot of bugs which were fixed in 4.0 or 3.23 well after they were released.

Really I respect MySQL a lot of having public searchable bug database with bugs found internally reported in it too. This honesty however can scare people, like oh there are hundreds of not fixed bugs in MySQL

I did not mention MySQL 4.1 because it already had its issues, though feature wise it was indeed best release for Web applications. And the good development would be to have Subselects and prepared statements fixed for next release rather than adding more half baked features.

The manor features in MySQL 4.1 were Subselects, Prepared Statements and Character Sets. All they were not done by Monty and these were changes which MySQL architecture was not well designed for.

SubQueries had number of bugs in particular in non trivial cases and still have serve performance limitations as of MySQL 5.1 (so many years and releases away). Prepared statements had to be pretty much redesigned and rewritten from the way they were done originally to make it work, plus they also had their good share of bugs with repeated execution etc.

Finally character sets required changes across all parts of the server and also are generally quite tricky in the cases when you mix collations and use different character sets in the application. It took a good time before 4.1 release to make majority of functions to work good with charsets and avoid breaking apps which worked normally on 4.0 with 4.1 because of charset related glitches.