http://repcached.sourceforge.net/

Thoughts on this?

Two-phase fetch – get IDs first (to build the list), then get the rest later

The idea was that if we build the list of IDs, then we can call generic accessor functions to get the data, and that is an abstraction layer in itself. It will check the cache, hit the db if needed, etc. – the consumer is none the wiser.

I also like this approach because it can enable sharding easily by adding one more check at the top of the function to determine the appropriate server/location of that data. Then use memcache to glue it together in the cache layer so next time it doesn’t require hitting any of the shards. Hope that makes sense.

I assumed that the foundation inside of the generic functions may change, but the generic functions themselves can be called the same way in the app; the only changes will need to be in the accessor functions for the new data storage architecture… which could be sharding, additional forms of caching, a swap out to another RDBMS, moving to something like a web service or a fully SOA framework of clusters of servers each owning specific services and piecing them together on each request (much like I believe Amazon’s e-Commerce system works)

In your case you assume it is composing full objects is expensive but it well may be finding them. As you grow your application that would become bottleneck sooner or later.

Again to dot get me wrong – memcache is cool and we use it a lot and help a lot of people to deploy it. I’m just saying there are challenges if your goal is to have things consistent as well as remove as much work from MySQL as possible.

On which table ? In this case you’re assuming memcache stores rows similar to the database. In this case things are trivial – removing/updating row in memcached if it is updatated in the database. However how do you handle this in the case I described ?

To build collections, I select only the IDs, based on filtering, sorting/ordering, LIMIT, etc. I build an array based on that. I go directly to the database for that.

while(mysql results) {
$fetch[] = $id;
}

Then feed $fetch to a cache_get() function, which is basically a generic wrapper for memcache_get().

I can’t speak for how massively this scales, but allows us to go to the database only for ordering and other lower (or higher?) level functions.

I will say that the memcache UDF is a cool idea, and would probably make this additional step no longer required, but that still requires a connection through MySQL to the memcache layer. I like keeping MySQL out of the loop as much as possible, to save it more complex work… the above example should save a lot of data transfer and hopefully keep the pipe on the server open for more requests. I suppose just using memcache UDF for cache invalidation/reset, set, and the first get (to warm that cache key) would help though.

Put a trigger on that table such that select memc_delete(b.somecol) from table a, b WHERE a.id = b.id etc.. just hook the dependency with a trigger to “touch” the rows in the other table.

Cheers,
-Brian

I do not really understand what you mean. You’re updated row in the database. That row was also cached as the part of 1000 dependent result sets. How exactly do you propose update or invalidate these automatically.

More specific example say I have my account ballance or expenses in cloths category cached. Now I go ahead and modify
one of transactions (the amount was wrong).

How do I do this with Memcache ?

Now you can implement smart Query Cache like table version approach but it is not good enough. I would prefer cached amount for electronics expenses to remain in the cache because it was not affected.