Comments on: How is join_buffer_size allocated?

By: Normalex

Normalex — Mon, 03 Oct 2011 23:13:21 +0000

size=max(thd->variables.join_buff_size, cache->length);

and the main question is how does the cache->length is being determined?
If it just waists the memory what is the point to use join_buff_size then?

There is obviously a lack of documentation about it.

By: Baron Schwartz

Baron Schwartz — Thu, 23 Sep 2010 14:02:53 +0000

Shahryar, you are right. The manual (http://dev.mysql.com/doc/refman/5.0/en/nested-loop-joins.html) has the following quote: “Join buffering can be used when the join is of type ALL or index (in other words, when no possible keys can be used, and a full scan is done, of either the data or index rows, respectively), or range.”

Thanks for correcting me!

By: Shahryar Ghazi

Shahryar Ghazi — Wed, 22 Sep 2010 14:46:34 +0000

I disagree with “A join buffer is allocated to cache rows from each table in a join when the join canâ€™t use an index.” because even if MySQL decides to scan all index rows and/or uses range for SELECTion it prefers to use the join buffer. Also, there is another variable that determines the max size of join buffer and that is max_join_size (no. of rows to be joined not the buffer size).

By: Baron Schwartz

Baron Schwartz — Mon, 12 Jul 2010 20:41:02 +0000

Shlomi, buffers are usually deallocated at the end in “cleaning up” state. I haven’t checked where this one is.

Geoff, I am not sure; I did not look that closely at that code to figure out its purpose.

By: Baron Schwartz

Baron Schwartz — Thu, 08 Jul 2010 16:02:19 +0000

It’s not accurate to say it can ONLY be used for that. I explained in the blog post what the server uses it for.

By: Shlomi Noach

Shlomi Noach — Thu, 08 Jul 2010 12:11:45 +0000

Baron,

Very interesting! A buffer-per-join-per query is news to me. What about deallocation, then? Do all these per-join buffers live for the duration of the connection?

Thanks

By: Geoff

Geoff — Thu, 08 Jul 2010 07:59:39 +0000

Is it correct that the allocated cache size is then the maximum of *one* row of the join data and the size specified in the configuration? As I understand, that would allow it to degrade from a Block-Nested Loop Join to a Nested Loop Join should only one row fit in cache.

By: dalin

dalin — Thu, 08 Jul 2010 00:28:00 +0000

Then what is the purpose of a setting that can only be used to waste memory?

By: peter

peter — Mon, 05 Jul 2010 18:35:05 +0000

Baron,

First about allocation – it is checked but it is NOT reported if it failed which can cause someone to set large buffer and get lower performance because it is not used.
I think MySQL should warn about such failures – print stuff in the error log file or at least have a global variable counter for allocation failures (which should stay 0 for any well configured system)

A note on crashing the server – you would do a lot better by allocating very large number of smaller buffers, for example 128K or so. If Server failed to allocate 1GB it still can likely allocate few MB needed to run most of concurrent queries successfully. If however you can’t allocate even 128K any more nothing else probably will work.

Also do not forget about over commit behavior for a lot of systems – they can let you to allocate a lot of memory but may fail when you can actually use it (access it)