Comments on: Air traffic queries in MyISAM and Tokutek (TokuDB)

By: Sheeri K. Cabral (Pythian)

Sheeri K. Cabral (Pythian) — Tue, 10 Nov 2009 23:17:17 +0000

Vadim — great! (in general if people are asking for something, even if it seems silly, I’d rather just do it and show them how silly it is, if it doesn’t take me too much time.)

And getting the difference between how fast compiled-in InnoDB and InnoDB-plugin is a win too — most people only consider upgrading or changing big things like compiled vs. plugin if their db is slow

By: Vadim

Vadim — Tue, 10 Nov 2009 22:32:32 +0000

Sheeri,

Ok, I am sold I will try to manage InnoDB setup before Portland, if not – then after that.
Would is really interesting here for me how fast indexes from InnoDB-plugin will work here.

By: Sheeri K. Cabral (Pythian)

Sheeri K. Cabral (Pythian) — Tue, 10 Nov 2009 22:27:45 +0000

I agree — InnoDB should do much worse than MyISAM…but I’d like to see how much worse. It would be good to be able to compare TokuDB to InnoDB in the case of an OLAP workload….TokuDB is supposed to be good for OLTP too, and you benchmarked that.

I just don’t see that it would take so much time to do — seems simple, no? (I have some benchmarks coming out on TokuDB very soon, so if you can’t do it that’s OK too.)

By: Vadim

Vadim — Tue, 10 Nov 2009 20:26:14 +0000

Hmm, Sheeri, your comment came to my mailbox only today, but WordPress says it was posted Nov-6,

Let me comment it now.

I think I explained why I took TokuDB. It has interesting index structure which may make it more
suitable than MyISAM and InnoDB, and you actually can see that from queries time.

Load time is not good for TokuDB, but as Bradley comments it, it depends how you do load.

I made another run where I loaded data in year-per-chunk way (in contrast all years-at-once as in original post), it took over 9h for MyISAM, but for TokuDB the same 5.5h

By: Vadim

Vadim — Tue, 10 Nov 2009 20:17:02 +0000

Something wrong with comments again.
I see some more comments to this post in my email, but not here.

There are comments from Sheeri K. Cabral (Pythian) and Alexander Mikhailian.

I will try to manage them to appear here…

By: Greg Smith

Greg Smith — Mon, 09 Nov 2009 22:47:59 +0000

I don’t really agree with Bradley’s characterization that “in production, many databases are performing close to worst-case insertion workloads”. Sure, there are some of those. But a good percentage of the data I see in production comes from sources that are time based, and that data tends to be loaded in well clustered clumps that are closer to best-case rather than worst. This is particularly true if you have enough RAM in the system that a decent chunk of index blocks can stay in there and not have to be updated on disk every time they’re touched, which is how a performance-oriented database should be provisioned. It’s only the case where you have a workload with really random UPDATE/INSERTs to data already in the index that really tend toward the worst of the B-tree behaviors, which was the case in Vadim’s extreme fragmentation example. I suspect that the BTS air traffic data won’t degrade anywhere near worst-case even if loaded in chunks, because that data is organized into blocks by time in its original files. I’d guess most of the work will be building new little index sub-trees and attaching them, rather than the worst-case behavior where you’re touching things all across existing index blocks.

By: Martin Kersten

Martin Kersten — Sat, 07 Nov 2009 08:40:15 +0000

Dear Vadim

Before looking into Greenplum, a run against Postgresql would be insightful.
It would demonstrate later the relative gains over Greenplum.

regards, Martin

By: Vadim

Vadim — Sat, 07 Nov 2009 03:07:26 +0000

Peter,

I am open for bigger dataset, though I have hard time finding it. Air traffic data is the biggest one (well, there is data take from telescopes, but I do not see how that fits into analytic databases) available.
For sure I can use (and will) synthetic generated data, but it is less interesting for me.

By: peter

peter — Sat, 07 Nov 2009 00:35:04 +0000

Vadim,

I would note you loaded the data for all engines all at once so you need to compare apples to apples. I would expect things may change for everyone (at various extent) if data is loaded incrementally.

I also would suggest looking at larger data set some time in the future – in the current ran the compression has 2 effects, it is not only much less data to look at but also in memory fit may cause virtually in-memory workload for well compressed data. On larger data sets when you get say 10% instead of 1% of in memory fit things may be a bit different.

By: Vadim

Vadim — Fri, 06 Nov 2009 22:54:29 +0000

Bradley,

I agree that this is best case scenario for MyISAM , and with fragmented indexes the result may be much worse for load time and for query execution. For queries yesterday I posted similar problem with InnoDB http://www.mysqlperformanceblog.com/2009/11/05/innodb-look-after-fragmentation/

I will re-run load with chunk-per-year to compare how it affects load time.

Though I should say the scenario I show here is not fully invalid. I’ve seen DW solution bases on MyISAM, when data is load in periodically. And say if we load data once per months, and full load time is only couple hours, it is worth to re-load data entirely to get optimal index structure.