Looks like this series of posts of workings with large amount of tables matches your envinronment significantly

-Ryan

Thanks. Good catch. This matches my guess as well. select() is way to the trouble especially these days when we deal with a lot of connections, a lot of open files etc.

I filed a bug on it BTW:
http://bugs.mysql.com/bug.php?id=48929

The select() and accept() calls in the trace are from
handle_connections_sockets() in sql/mysqld.cc. It is supposed to select() on
(at most) 2 file descriptors: the IP server socket and the unix server socket.

Now in this case, one of these somehow ended up with file descriptor number
16393.

From `man 2 select`:

“An fd_set is a fixed size buffer. Executing FD_CLR() or FD_SET() with a
value of fd that is negative or is equal to or larger than FD_SETSIZE will
result in undefined behavior. Moreover, POSIX requires fd to be a valid
file descriptor.”

And FD_SETSIZE seems to be 1024 on Linux!

So basically mysqld is now passing random stack memory to select() for file
descriptors 1024-16393 . This also explains why accept() fails with EAGAIN:
there isn’t any new connection pending, it is just that select() gets lots of
random file descriptors in the undefined memory, and one of them happens to
return ready.

So the code in handle_connections_sockets() is really broken, the only reason
it works at all is probably that the server sockets are usually created early
and hence get low file descriptors. But in this case for some reason at least
one of them got the very high number 16393.

Even if we do not exceed the FD_SETSIZE it can still be inefficient to
search a potentially big fd_set for just two descriptors. The code should be
changed to use poll(). I seem to remember seeing a patch for this, or I might
make on up for MariaDB.