Are most of the reads trying to read recently written records, or old stuff? If old stuff, you could shard on a modulus of TO_DAYS() output (for example, depending on your application) so on any given day you’d only be writing to one table but the reads could come from many others (if a lot of reads read old data). Remember that writing to a table will invalidate any entries in the query cache that relied on the content of that table, so this approach could improve the effectiveness of your cache (again, depending on your application). Sharding by day might not be appropriate for your application though – I couldn’t possibly say as I don’t know what your app does.

There are many ways you could improve on this but without further knowledge of your application it’s going to be hard to say.

And of course server configuration is important too.

Read that book – I’ve got a copy and it’s excellent.

I am working on a system which has currently around 20gb of data (increasing at the rate of ~200mb/day). We need to save complete documents in the system so basically one column has around 70-80% of the data. Our server is Quad Dual Core 4GB Ram, Server 2003 and using MySQL 5.0. Query cache size is 156M and limit is 8M. All tables except one are INNO DB and its buffer pool size is 1024M.

The end user client module is read-only so lots and lots of repeated queries therefore I think query cache helps a lot but one of the problems I am facing is somewhat similar as discussed above. Sometimes the system halts for few mins. I fear that if I disable the query cache it will slow down my system which already is just OK.

Apart from that I also want to handle the future size of the db which is increasing rapidly. I have got hold of High Performance MySQL but any hints where should I be looking at first?

Any help will he highly appreciated. I can also give more details of my system in case if it will help. Thanks a lot!

BTW, what about for very small writes like financial realtime data? Actually some tests we did, Innodb was not well suited, but I think in the time they did some tests, innodb was not configured accordingly. The problem is that our application need “immediately” to write a small amount of data (usually 40 – 100 byte packet) many times in a millisec. Normally we have 70% writes vs 30% reads. But writes are made in 4 – 8 threads meanwhile reads are done through 100 – 200 threads.

Do you think Innodb or XtraDB (well tunned) could be used in this case?

Thanks!

I’ve found that not only does increasing the value of innodb_buffer_pool_size increase performance, but for me it has also made the difference between the database working or not working:

I created a test table with quite a lot of data in it. With the default value for this setting, once I’d got to a certain size I couldn’t insert any new rows at all without mysqld (v5.0.27) falling over and rolling back the latest insert.