… in England are interested in the data in Italy, so sharding your data based on country will not help as you will have many cross domain/shard queries. Mongo does not provide any locality functionality to meet…
Comment: mysql-proxy, urgh performance and scalability ?
February 27, 2012 By 27 Comments
… features I want are the following: – Connection pool. – Failed Over. – Sharding. – Query Router. In addition it is better support stored-procedure…
Post: Shard-Query turbo charges Infobright community edition (ICE)
May 6, 2011 By 22 Comments
… and conquer approach that Hive takes in combination with Hadoop. Shard-Query applies a clever approach to parallelism which allows it… subset of the data, performance can be improved significantly: $ cat shards.ini [default] user=mysql db=ontime password= port=5029 column… in development release form as a checkout from SVN) here: Shard-Query Google code project Full disclosure Justin Swanhart, the author…
Post: Shard-Query EC2 images available
May 11, 2011 By 2 Comments
… cluster of 1 to 20 nodes. cd shard-query #generate a config for 20 shards (adjust to your number of nodes) php… statements. The queries for the benchmark are in ~ec2-user/shard-query/queries.sql. I have also provided a convenient script… output from the ./run_query command, called pivot_results cd shard-query/ $ ./run_query < queries.sql | tee raw |./pivot_results & [1…
Post: Why you don't want to shard.
August 6, 2009 By 33 Comments
… faith that Yasufumi is making this better). What types of Sharding are there? Despite my cautions, if you have established that you need to shard there are quite a few options available to you: Sharding Partitioning by Application Function – This… out some of the more complicated sharding architectures. For example; another solution is to have shards all store fragments of data…
Post: Shard-Query adds parallelism to queries
November 15, 2010 By 16 Comments
… queries which examine a lot of data. What is Shard-Query? Shard-Query was initially conceived as a utility to add parallelism… solid #CCC;border-bottom:1px solid #CCC;} .PeriodNo Shard-Query16 worker24 worker32 workerNo Shard-Query16 worker24 worker32 worker.1 year4.5481.0440… solid #CCC;border-bottom:1px solid #CCC;} .Query NumberNo shard-query16 workersNo shard-query16 workers.14.6610.864.6605691909790.8596088886261.284.36219…
Post: Distributed Set Processing with Shard-Query
May 14, 2011 By 27 Comments
… give a quick preview, then explain exactly how Shard-Query works.Yes, Shard-Query scales very well at 20 nodes Distributed set… incremental materialized view optimizations can be applied. Shard-Query works only on sets Shard-Query takes relational algebra to its logical maximum… is massively parallel . In fact, it is embarassingly parallel. Because Shard-Query works on sets, and features pluggable partition mapping, it…
Post: "Shard early, shard often"
November 16, 2009 By 6 Comments
… point at which you need to shard, and that the (often omitted) operational issues created by sharding can be painful. What I… I disagree with a statement I’ve been hearing recently; “shard early, shard often”. Here’s why: There’s an order…/index/schema optimization. The gains from sharding are usually much lower. If you shard first, and then decide you want to…
Post: Sharding and Time Base Partitioning
March 14, 2008 By 12 Comments
For large number of online applications once you implemented proper sharding you can consider your scaling problems solved – by getting more… most optimal way by itself to do things. The “classical” sharding involves partitioning by user_id,site_id or somethat similar…
Comment: Why you don't want to shard.
August 6, 2009 By 33 Comments
…, there really isnt any other option than to shard, but I dont think sharding needs to be as doom and gloom as… framework design, not the fact that it’s sharded. If you design your sharding infrastructure with simplicity for developers as a primary…. – There are also some other benefits to sharding which you dont list here. Shards can be an amazing tool when dealing with…
