Demonstrating distributed set processing performance Shard-Query + ICE scales very well up to at least 20 nodes … mkdir -p “results/$2/” ./run_query < queries.sql |egrep “rows|^-” > results/$2/raw.$$.$i.txt done; Query processing is handled by a Gearman queue which limits the maximum number of concurrent storage node queries. This prevents the system from being overloading and provides a…

… latency/overhead does Gearman add? In the case of a query that takes 20 seconds (or minutes or hours), I could… more akin to the retail space where some sort of query just can’t be solved in a single-threaded space… at it? So why not just spawn N queries and block until they all return? Is this reduced form of Shard-Query possible?

… sum check from SQL, which you can then distribute with Shard-Query: CREATE TABLE `the list` ( `id` bigint(20) NOT NULL AUTO… check to pass: mysql> insert into data (val) values (16); Query OK, 1 row affected (0.01 sec) mysql> SELECT val…

…) and Shard-Query can distribute count(distinct) over a cluster. My benchmark tests this extensively if you take a look at the queries.

… in my next blog post, where I show how flexible Shard-Query is in its configuration options. The virtual machines have little… to load on another storage engine, or test your own queries over the data.

Regardless, Shard-Query suits the class of problems that I want to target. …

Justin, As far as I understood, all dimension tables in your test are replicated among all nodes. Otherwise, joining fact table (partitioned) with another partitioned table will require totally different technique which is hardly implemented in Shard-Query. Am I right?