… BigData with a talk titled, “Conquering BigData with MySQL and Shard Query” Daniel Nichter, lead of the Percona Toolkit development team…
Post: How people are using MySQL... from 1 user to 100 million (upcoming conference talk)
June 11, 2013 By Leave a Comment
… later you have to think about write scalability, and implement sharding ultimately. I will give a talk on this topic at…
Post: Shard-Query turbo charges Infobright community edition (ICE)
May 6, 2011 By 23 Comments
… and conquer approach that Hive takes in combination with Hadoop. Shard-Query applies a clever approach to parallelism which allows it… subset of the data, performance can be improved significantly: $ cat shards.ini [default] user=mysql db=ontime password= port=5029 column… in development release form as a checkout from SVN) here: Shard-Query Google code project Full disclosure Justin Swanhart, the author…
Post: Shard-Query EC2 images available
May 11, 2011 By 2 Comments
… cluster of 1 to 20 nodes. cd shard-query #generate a config for 20 shards (adjust to your number of nodes) php… statements. The queries for the benchmark are in ~ec2-user/shard-query/queries.sql. I have also provided a convenient script… output from the ./run_query command, called pivot_results cd shard-query/ $ ./run_query < queries.sql | tee raw |./pivot_results & [1…
Post: Why you don't want to shard.
August 6, 2009 By 34 Comments
… faith that Yasufumi is making this better). What types of Sharding are there? Despite my cautions, if you have established that you need to shard there are quite a few options available to you: Sharding Partitioning by Application Function – This… out some of the more complicated sharding architectures. For example; another solution is to have shards all store fragments of data…
Post: Shard-Query adds parallelism to queries
November 15, 2010 By 18 Comments
… queries which examine a lot of data. What is Shard-Query? Shard-Query was initially conceived as a utility to add parallelism… solid #CCC;border-bottom:1px solid #CCC;} .PeriodNo Shard-Query16 worker24 worker32 workerNo Shard-Query16 worker24 worker32 worker.1 year4.5481.0440… solid #CCC;border-bottom:1px solid #CCC;} .Query NumberNo shard-query16 workersNo shard-query16 workers.14.6610.864.6605691909790.8596088886261.284.36219…
Post: Distributed Set Processing with Shard-Query
May 14, 2011 By 27 Comments
… give a quick preview, then explain exactly how Shard-Query works.Yes, Shard-Query scales very well at 20 nodes Distributed set… incremental materialized view optimizations can be applied. Shard-Query works only on sets Shard-Query takes relational algebra to its logical maximum… is massively parallel . In fact, it is embarassingly parallel. Because Shard-Query works on sets, and features pluggable partition mapping, it…
Post: "Shard early, shard often"
November 16, 2009 By 6 Comments
… point at which you need to shard, and that the (often omitted) operational issues created by sharding can be painful. What I… I disagree with a statement I’ve been hearing recently; “shard early, shard often”. Here’s why: There’s an order…/index/schema optimization. The gains from sharding are usually much lower. If you shard first, and then decide you want to…
Post: Webinar: Building a highly scaleable distributed row, document or column store with MySQL and Shard-Query
February 6, 2013 By Leave a Comment
… scaleable distributed row, document or column store with MySQL and Shard-Query” The first part of this webinar will focus on… distributed databases are needed, and on the techniques employed by Shard-Query to implement a distributed MySQL database. The focus will… (massively parallel processing) database applications which can be deployed with Shard-Query and the performance aspects of each. The following types…
Post: Sharding and Time Base Partitioning
March 14, 2008 By 12 Comments
For large number of online applications once you implemented proper sharding you can consider your scaling problems solved – by getting more… most optimal way by itself to do things. The “classical” sharding involves partitioning by user_id,site_id or somethat similar…
