Very nice blog entry and thanks for the favorable mention of Infobright. We would be more than happy to get you or one of your team started with Infobright. Since Infobright is column oriented and sports very high compression, you will likely find that some of your guidance will change.

Cheers

1) When you’re doing complex analysis that isn’t easily or efficiently expressible in SQL. “Finding the most interesting rows and various other stuff related to them” for example.

2) When you’re only going to query the data one time as an intermediate step. If you just want to do a once-off aggregation, then reading it from disk, putting it into a table (writing it to disk), reading it from the table (from disk, again), aggregating it (possibly involving temp tables on disk and/or filesorting on disk), and writing it to another table (back to disk) can be less efficient than just reading it from disk into your script, aggregating it in memory and writing it to the final table.

We use MySQL based reporting for our internal ad serving analysis and “discovered” most of your recommendations ourselves. Our solution has several layers of aggregation and clustering but if finally ends in one or multiple MySQL servers used for end user reporting. Works fine and scales good.

Some ideas from our experience:
- we do not use replication. Instead we maintain several parallel virtually identical systems. If one fails, we can either copy from another or switch.
- we aggregate some data directly on runtime servers, then in intermediate MySQL nodes and finally when building reporting-ready aggregates.
- aggregation is going on incrementally, so reporting delay for end users is below 1 hour
- Peter is right, aggregation in big (but not too big) chunks is much faster
- we use InnoDB for dimensions that fit in memory and MyISAM for aggregates. We consider using InnoDB though. However, MyISAM allows to copy files between servers, that is very convenient sometimes.
- we store raw data just in case but we do not use it. When it gets too big, we just delete it. Raw data required for aggregates processing is truncated daily.
- we do not use a lot of de-normalization, just try to avoid snow-flakes. If we do need de-normalization, we do it on dimension level but keep aggregate tables compact. After all, MySQL is not column oriented storage, so big tables should have the smallest row length possible for faster access.

It is interesting, that we tried to use Oracle for end user reporting server, but now switching back to MySQL. Oracle is too difficult to maintain.

1) Aggregate. There is really some conflict here – to get the best aggregation speed you need to aggregate chunk at once. Merging 100000 events at once on aggregation is much faster than processing them one by one but yet you need to be careful how how it affects replication or causes table locks if you happen to use MyISAM.

2) Real time vs delayed. I think for many applications semi-real time is a value and as you mentioned it is possible at the low end. So it is very handy to use adaptive technologies which can aggregate small batches in a normal case but fail back to larger batches (and so more efficient processing) if it can’t keep up well enough.

3) Denormalization. A common advice but it makes many people to take it to extreme. For example if you store top countries for the day you do not have to store strings – ID’s are just fine because the lookup table is small and you do few lookups anyway. The Denormalization needs to be done in a way your queries avoid a lot of random lookups but you need to keep the balance of keeping data compact.

4) MyISAM vs Innodb – if you have read only data you can often get data clustering with MyISAM too by ALTER TABLE … ORDER BY X. It is not same as with Innodb (data is never stored in clustered index) but can get your IO pretty sequential for your prevailing access type. Innodb is indeed good as default but tradeoff between space and performance can be important for some applications. Actually this is where I would watch Maria storage closely. It may be handy even in its “crash safe MyISAM” mode

5) About storing raw data. Really storing in MyISAM tables without indexes (say one per day) or later Archive tables can be close to efficiency to storing to the database (MyISAM surely will be bound by sequential disk writes) – this gives convenience of running SQL queries if you need to without extra step of loading the data. I also prefer to keep things one table per day or something like it so it is very easy to move things around boxes (physical copy, just as files) and if you use full table scan MERGE TABLES are very efficient. What is however often a bad choice to have huge single highly indexed table which keeps all events as logs.

Baron, this is a great post! Extremely useful information written down very well. Kudos, I’ll be returning to this post many times I’m sure.

Keep it up,

Roland.