I will definitely give MonetDB a try.

Bit-wise compression is a two-edge sword. You save on IO and pay with CPU cycles.
The numbers reported for both MonetDB and Infobright on this blog already show
that compression is just one ingredient in the equation.
MonetDB used more diskspace and still performed very well.

In http://homepages.cwi.nl/~mk/ontimeReport you find more recent numbers.

In particular, Q1 runs currently within 1 sec on the 118M tuples.
It is certainly worth trying out the 1B version using the upcoming Feb2010 release.

I forwarded your question to my contacts in MonetDB and InfoBright.

In our company we are looking for D/W solution which will allow us to do a real-time (10-15 sec) queries on 1B+ row tables. InfoBright, MonetDB, InfiniDB are in my short lists. Having read your article first time I was impressed by performance numbers of columnar databases, but now … I think that there are a lot of room for query optimization inside both MonetDB and Infobright.

For example, this query takes 7.9 sec for MonetDB and 12 sec for Infobright.

-Q1: Count flights per day from 2000 to 2008 years
SELECT DayOfWeek, count(*) AS c FROM ontime WHERE YearD BETWEEN 2000 AND 2008 GROUP BY DayOfWeek ORDER BY c DESC

This is a full column scan query. Group BY column has very low cardinality (7) => do grouping and aggregation in memory.
DayOfWeek requires only 3 bits per row => for 117M row table, it will need ~30Mb to store the whole column.

Loading column from disk (30MB), decompressing it and aggregating 117M numbers should not take more than 1 sec (actually, I think I can do it on my laptop in less than 0.5 sec).

Vadim, can you forward my e-mail address and question to MonetDB and Infobright guys?

This is query:

SELECT T.C FROM T WHERE (up to 4 simple filter conditions ANDed) ORDER BY T.B DESC LIMIT N;

The good selectivity is not guaranteed. It may require almost full table scan in a worst case. Can they do this type of query under 15 sec in a worst case on 1B table which is similar to the table from this post (120 columns – most of them with low cardinality – no strings only numbers)

That’s impressive. I can’t recall so fast response with performance fixes with significant improvements.

Is that version available to download ?

This benchmark was a good exercise to re-prioritize some of our development efforts towards improvements for daily BI use.
Although the original results were quite promising, we knew that low hanging fruit in the MonetDB stack was not harvested.

A peek preview on the performance results for the next feature release can be found in http://www.cwi.nl/~mk/ontimeReport

Happy newyear. Martin

I am not expert in MonetDB, but Martin Kersten is.
If he does not notice your question, I will pass it to him.