When you see bug happening you would see MySQL log flooded with error messages like this:

091119 23:03:34 [ERROR] Error in accept: Resource temporarily unavailable
091119 23:03:34 [ERROR] Error in accept: Resource temporarily unavailable
091119 23:03:34 [ERROR] Error in accept: Resource temporarily unavailable
091119 23:03:34 [ERROR] Error in accept: Resource temporarily unavailable

filling out disk space

Depending on the case you may be able to connect to MySQL through Unix Socket or TCP/IP or neither.
It also looks like there is a correlation between having a lot of tables and such condition.

Previously I was unlucky with seeing this issue in production so we had to restart MySQL quickly currently I have a test MySQL showing some behavior.

Here is what strace tells me:

[percona@test9 msb_5_4_2]$ strace -f -p 19229
Process 19229 attached with 23 threads – interrupt to quit
[pid 19286] rt_sigtimedwait([HUP QUIT ALRM TERM TSTP],
[pid 19285] futex(0×165962ec, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19284] select(0, NULL, NULL, NULL, {0, 765000}
[pid 19283] select(0, NULL, NULL, NULL, {0, 412000}
[pid 19248] futex(0×1781193c, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19247] futex(0×178118bc, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19246] futex(0×1781183c, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19245] futex(0×178117bc, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19244] futex(0×1781173c, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19232] futex(0×1781113c, FUTEX_WAIT_PRIVATE, 165, NULL
[pid 19229] accept(16392,
[pid 19241] futex(0×178115bc, FUTEX_WAIT_PRIVATE, 319, NULL
[pid 19243] futex(0×178116bc, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19242] futex(0×1781163c, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19240] futex(0×1781153c, FUTEX_WAIT_PRIVATE, 3, NULL
[pid 19239] futex(0×178114bc, FUTEX_WAIT_PRIVATE, 7, NULL
[pid 19238] futex(0×1781143c, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19237] futex(0×178113bc, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19236] futex(0×1781133c, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19235] futex(0×178112bc, FUTEX_WAIT_PRIVATE, 7, NULL
[pid 19234] futex(0×1781123c, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19233] futex(0×178111bc, FUTEX_WAIT_PRIVATE, 207, NULL
[pid 19231] futex(0×178110bc, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19229] <... accept resumed> 0×7fffffac70b0, [18423225245113516048]) = -1 EAGAIN (Resource temporarily unavailable)
[pid 19229] accept(16392, 0×7fffffac70b0, [18423225245113516048]) = -1 EAGAIN (Resource temporarily unavailable)
[pid 19229] accept(16392, 0×7fffffac70b0, [18423225245113516048]) = -1 EAGAIN (Resource temporarily unavailable)
[pid 19229] accept(16392, 0×7fffffac70b0, [18423225245113516048]) = -1 EAGAIN (Resource temporarily unavailable)
[pid 19229] fcntl(16392, F_SETFL, O_RDWR) = 0
[pid 19229] fcntl(16392, F_SETFL, O_RDWR) = 0
[pid 19229] select(16394, [1025 1040 1042 1044
....
8 9709 9714 9716 9717 15368 15369], NULL, NULL, NULL*** buffer overflow detected ***: strace terminated
======= Backtrace: =========
/lib64/libc.so.6(__chk_fail+0×2f)[0x35f36e6aff]
/lib64/libc.so.6[0x35f36e5ad3]
strace[0x408b60]

So as you can see accept gets pretty high socket number – probably because of large innodb_open_files I tested with in this case – which all can be used during recovery.

Note the process gets accept on the socket to fail with EAGAIN (which is not well described in the manual) and later getting call to select call with 16384 sockets. This does not seems to be the healthy number to work with SELECT call and It is quite possible something goes wrong because of it.

If you have any ideas why could be going wrong in this case.

Entry posted by peter | 5 comments

Add to: | | | |

It is no surprise why I could not find the bug easily – the bug is not really related to multiple key caches but to online key cache resize. It is just this code most actively used in case you’re using multiple key caches. It is very rare one would resize single key cache in production and it only triggers crash sometimes, while if you’re using multiple key caches there are often some scripts in place which adjust their size or change mappings of the tables.

No let me put my complain hat on. Looking into this crash bug I see it was opened in early 2006, this is 3.5 years ago when I was still with MySQL living in Seattle area…. looks like century ago really.
As usually with such tricky bugs Shane Bester steps in to create a test case to crash it soon. Shane seems to be the best guy in the universe when it comes to making non repeatable bugs repeatable.

It took over a year to complete the bug fix. For all this time bug existed in 4.1, 5.0 and 5.1 trees. When bug was fixed and closed (so you will not find this bug if you search for active bugs) but in reality it was only fixed in MySQL 5.1 while even though bug closed MySQL 5.0 and MySQL 4.1 still did not have it fixed.

It was planned for it to be backported in MySQL 5.0 after MySQL 5.1 is in production for a few months but I trust this was forgotten.

Now this looks like the nastiest bug type to me. This is the bug in relatively rarely used feature – I’d expect less than 1% of people would resize key cache while server is up or use multiple key caches. The fix of it is not easy and may affect 100% of the users if bug is introduced. It is also race condition bug so not all people using the feature will run into the bug – In my practice number of people using multiple key caches with relatively light load never ran into this issue.

It is the hard choice what to do in such case but I think closing the bug and leaving it unfixed at the same time is not a good idea. At least I’d like to see the bug “cloned” for version 5.0 and 4.1 so it is can be found as active bug and it is clear these versions still have it. Or may be have some special bug status to indicate it was fixed only in later MySQL versions but left unfixed in former ones ?

I also would like to have no crash bugs in the server. If there are some features which are known to be buggy and crashing I’d like to see them disabled as run time and enabled with –enable-buggy-features switch (I’m sure marketing can come up with better name for this one)

Entry posted by peter | No comment

Add to: | | | |

When you run DROP TABLE there are several things that need to happen – write lock on a table so it cannot be used by any other thread, the data file(s) removal by the storage engine and of course in the end MySQL has to destroy the definition file (.frm). That's not all that happens, there is one other thing:

The entire code which removes a table is surrounded by LOCK_open mutex. The mutex is used in various places inside MySQL, but primarily when tables are being opened or closed. It means that when LOCK_open is locked, no query can be executed because they are stopped from accessing any table.

Here is when the slow file removal on the ext3 file system starts to be a pain. Deleting a 10GB file can take a few seconds and if that is a MySQL table, the mutex remains locked for all that time stalling all queries:

I tried some alternative approaches to trick MySQL into removing smaller files with DROP TABLE to minimize the effect, such as:

• TRUNCATE TABLE large_table; ALTER TABLE large_table ENGINE=…; DROP TABLE large_table;
• TRUNCATE TABLE large_table; OPTIMIZE TABLE large_table; DROP TABLE large_table;

Unfortunately as it turned out each of the administrative commands like ALTER TABLE or OPTIMIZE TABLE one way or another uses LOCK_open mutex when the old table files are deleted:

The only alternative seems to be changing the file system. To XFS for example, which handles file removals much more efficiently:

EXT3

XFS

A better solution through MySQL internals could be to simulate the table drop by renaming the corresponding data file or files and physically removing them outside of the mutex lock. However it may not be that simple, because the actual removal is performed by the storage engine, so it's not something MySQL code can control.

This is certainly not a common situation, but may become a problem to anyone when it's the least expected (e.g. dropping old unused tables).

Entry posted by Maciej Dobrzanski | 19 comments

Add to: | | | |

Working with production systems we assume reads are reads and if we're just reading we can't break anything. OK may be we can crash the server with some select query which runs into some bug but not cause the data loss.

This case teaches us things can be different - reads can in fact cause certain writes (updates) inside which add risk, such as exposed by this bug.

This is why transparency is important - to understand how safe something is it is not enough to know what is this logically but also what really happens inside and so what can go wrong.

Entry posted by peter | 19 comments

Add to: | | | |

It is not the most critical bug by far but this makes it worse - critical bugs would usually cause upgrades already or were worked around while such stuff as causing things like "sometimes my slave clone script does not work" may hang on for years.

Entry posted by peter | One comment

Add to: | | | |

So you can see merge table behaves quite smart. Even though the maximum value in table a2 was 1 it finds out what other subtable had value 2 and assigns the value 3 to the new row.

Let us see how stable merge table auto_increment values are by truncating the table we just inserted data to:

As you see newly inserted row got value 3, which is maximum value which remained in the table plus 1. So auto_increment values are reusable just as with old ISAM tables. Such behavior does not only corresponds to TRUNCATE - deleting last row will cause auto_increment value to be reused too.

Can you use AUTO_INCREMENT clause in CREATE TABLE to get different auto_increment values ? I guess not:

As you can see Merge table does not even stores auto_increment value (and of course does it without any warnings)

Neither setting auto_increment value for underlying MyISAM tables works:

So for bad or for good you should remember auto_increment for Merge Tables works differently from both MyISAM and Innodb tables and being similar to what ISAM tables used to have. As the side effect of this - if you're using auto_increment columns inserting into Merge Table and in the last table directly will have different results.

Another thing I should remind you should be very careful while running ALTER TABLE for underlying tables. Unlike with TRUNCATE TABLE there is no warning displayed but the MERGE TABLE may continue having old file descriptors open not seeing changes done to original table:

As you can see delete is invisible in the merge table after we did alter table until we have run flush tables.

I've done my tests with MySQL 5.0.62.

Entry posted by peter | No comment

Add to: | | | |

You may thing it is minor issue but in fact with large buffer pool this bug makes database virtually unrecoverable (if 10% of progress in 2hours qualifies as that). It is especially nasty as it is quite hard to predict. Both customers had MySQL crash recovery happening in reasonable time... most of the times until they run into this problem.

So what is the point ? Have modest expectations about when your favorite MySQL bugs are fixed (This is actually Innodb one, so Innobase/Oracle is responsible for fixing it not MySQL/Sun but there are MySQL bugs not fixed for years too). Look for workarounds or ways to fix things yourself.

In particular case workaround was rather easy - reducing Innodb buffer pool size to 4GB instead of 24G and disabling innodb_flush_method=O_DIRECT so OS cache can be used for IOs. This made database to complete crash recovery in 30 minutes.

Entry posted by peter | 4 comments

Add to: | | | |

So ANALYZE TABLE should be very fast and non intrusive operation doing just little update on the data. Right ?

Wrong! There is the bug or rather MySQL Design Feature which causes ANALYZE TABLE to block all accesses to this table while it could be flushed from the table cache.

What does this mean in practice ? If you have some long running query accessing Innodb table and you run ANALYZE TABLE you will be unable to access that table with "Waiting for table" lock until the first query completes.

For applications which run short transactions it may not be the big deal but if you mix long reporting together with real time update queries this can be the real issue.

This is generally the type of gotcha I hate the most. From the glance view there are no reasons Innodb can't do this without locks but in practice some old mysql design artifacts result in this behavior and the bug can't be fixed quickly.

Entry posted by peter | 10 comments

Add to: | | | |

The one I'm speaking about comes from the issue with MySQL optimizer stopping using further index key parts as soon as there is a range clause on the previous key part. So if you have INDEX(A,B) and have a where clause A BETWEEN 5 and 10 AND B=6 only the first part (A) of the index will be used which can be seriously affect performance. Of course in this example you can use index (B,A) but there are many similar cases when it is not possible.

I have described couple of solutions to this problem - using IN list instead of range or UNION which however require rather serious application changes and also can result in huge IN lists and suboptimal execution for large ranges.

Lets take a look at very typical reporting query which queries data for date range for multiple of groups (these can be devices, pages, users .... etc)

As you can see from the EXPLAIN this query is expected to analyze over 300.000 of rows which is relatively fast for this (in memory) table but will become unacceptable as soon as you get to do random disk IO.

Note this is also interesting case of EXPLAIN being wrong - it shows key_len=7 which corresponds to the full key while only first key part is used.

Let us now replace the range with IN list in this query:

So we get same result but approximately 50 times faster. In this report we had just one month worth of data - what if you would have a year ? 5 years ? What if you get say thousands of groups at the same time ? Performing such query MySQL has to build (and do lookups) for all combinations which is 31*10=310 in this case. But if it gets to hundreds of thousands this method starts to break (and newer MySQL versions will stop using this optimization method if there are too many combinations to check).

Instead you could use JOIN to get list of days matching range from some pre-generated table and use the join to retrieve the rows from original table:

As you can see it does not work while I know I used exactly this trick to optimize some nasty queries.
It looks like equality propagation is working here (note the number of rows for second table in join is estimated same in original query) and we get the range clause on "info" table instead nested loops join - exactly what we tried to avoid.

It is easy to block equality propagation by using some trivial function:

So we stopped equality propagation but now have another problem - for some reason MySQL decides to only do "ref" on the date only instead of using range on day and list of groups for each join iteration.
This does not make sense but this is how it is.

I also tried to increase cardinality by having all rows to have different group_id and it still does not work.

The trick however does work if you have just one group_id (and in this case you do not even need to trick around equity propagation to make it work)

For original query form with single group_id query was taking 0.95 sec. The query with BETWEEN range replaced with IN list was instant 0.00 sec same as the query using join with day list table.

So we finally managed to get better performance by joining data to yet another table though why it does not work for multiple group remains question to check with MySQL Optimizer team

UPDATE: I just heard back from Igor Babaev saying it was designed this way (because the first component can run through very many values). The second component is simply not considered for range unless it is equality. You always have something to learn about MySQL Optimizer gotchas

At the same time I figured out how to make MySQL Optimizer to do what we want to do - Just add yet another table to the join so the info table just has bunch of ref lookups:

This query looks very scary but in fact perform much better than original one. In the real queries you can use table with ids just as we had table of days with a where clause instead of precreated table.

Entry posted by peter | 12 comments

Add to: | | | |

Lets help MySQL to ensure GA release will be indeed GA quality and support Monty in his call to ensure this RC candidate works well for you.

If you know about any bugs in MySQL 5.1 or MySQL 5.0 (which are still unfixed in MySQL 5.1) make sure to report them on bugs.mysql.com. If there are some bugs already reported but still not fixed as of MySQL 5.1.26 make sure to post a comment to let MySQL know the bug is affecting you as well. The scream factor is important for bug fix prioritization so if you can make a story of your suffering because of the bug to slashdot you should be quite good.

If you're the customer - make sure you put that note in your bug report. Bug reports from customers should be treated more seriously so make sure you use this advantage. And if you're not a customer I guess one of your friends is so you can ask them to vote for a bug for you.

It also helps to be descriptive to explain why exactly this bug is important. Many MySQL developers are isolated to the customer so they may not quite understand how often bug circumstances happen and how bad are consequences.

It is surely helps to have repeatable bug reports but make sure to report bug even if you do not have one. If significant number of people are reporting similar issue it shows some problem exists, even if they can't exactly pinpoint it. It helps MySQL with bug hunting and what is even more important it helps other users which know they are not alone and can find a workaround assuming fix does not exist.

In general I'm quite happy to see MySQL 5.1 is reaching GA stage and I think it is already better than Quality Disaster MySQL 5.0 initial release have been. Though I still think it is not quite where yet when it comes to Quality. Error rate with 5.1 we've seen among the customers during last few months was higher than with 5.0, also Scalability of MySQL 5.1 is often subpar compared to MySQL 5.0 possibly due to this bug which is not fixed up to now (not in 5.1.26 at least)

Entry posted by peter | 4 comments

Add to: | | | |