… will give us some interesting information about how Shard-Query will perform on each database. To do the size comparison, I used… COMMENT=’Contains all avaialble data from 1988 to 2010′; mysql> use ontime1; Database changed mysql> show table status like ‘ontime_fact’G *************************** 1…
Post: Upgrading MySQL
January 5, 2010 By 24 Comments
…. MySQL always was very good maintaining on disk binary compatibility between version and so you can run MySQL 5.0 database with MySQL 5… different MySQL version, unless you have very detailed automated QA. So what you can do is get the backup of the database… a bit so if something happens (like MySQL crashes) you have only one/few shard to deal with. P.S Happy New…
Post: MySQL Limitations Part 4: One thread per connection
October 27, 2010 By 14 Comments
… way MySQL handles connections, allocating one thread per connection to the server. MySQL is a single process with multiple threads. Not all databases… it were, it wouldn’t be a great OLTP web database too. MySQL replication is one of the core, fundamental features — and… limitations. And because MySQL is a multi-threaded database for Web usage that tends to be used in sharded environments with tons of…
Post: Is your MySQL Application having Busy IO by Oracle Measures ?
January 8, 2012 By 1 Comment
… post I wanted to mention what James calls “Busy” Oracle database application when it comes to IO consumption: It is applications… for “sharded” MySQL the numbers are not that high – many what would be considered medium scale MySQL installations would get there. With MySQL we… at MySQL we often start “sharding” or splitting data other way because operationally working with large table and data sizes in MySQL can…
Post: Shard-Query turbo charges Infobright community edition (ICE)
May 6, 2011 By 23 Comments
Shard-Query is an open source tool kit which helps improve the performance of queries against a MySQL database by distributing the work… following graph: One significant advantage of Shard-Query over Hive is that it works with existing MySQL data sets and queries. Another… the data, performance can be improved significantly: $ cat shards.ini [default] user=mysql db=ontime password= port=5029 column=date_id mapper…
Post: Distributed Set Processing with Shard-Query
May 14, 2011 By 27 Comments
… proxy. The databases do 99.9% of the work, due to the multiple levels of result set reduction. Finally, Shard-Query can… operations to operate fully in parallel. Distributed set processing is database agnostic. Keep in mind, at each level of partitioning only… of compute resource which speaks SQL, but right now only MySQL storage nodes are supported. Amdahl’s law applies to the…
Post: Using any general purpose computer as a special purpose SIMD computer
May 16, 2011 By 5 Comments
… Here is the serial version as run by the native database interface (MySQL): mysql> select word, md5(word), md5(reverse(word)), count(*) from… the same operation on the native database. That doesn’t matter because Shard-Query is a smart database proxy that can add parallelism… query. If your database system can not provide native parallelism, then performance will be poor. — SQL TO SEND TO SHARDS: Array ( [0…
Post: High-Performance Click Analysis with MySQL
December 22, 2008 By 13 Comments
… of things that seem “less advanced” than using a database. Sharding and Partitioning Sharding is inevitable if your write workload exceeds the capacity… save that work by either using MySQL 5.1′s row-based replication, or in MySQL 5.0 and earlier, doing the… and other types of event-analysis databases. In some cases I’ve been able to avoid sharding for a long time and…
Post: Webinar: Building a highly scaleable distributed row, document or column store with MySQL and Shard-Query
February 6, 2013 By Leave a Comment
… focus on why distributed databases are needed, and on the techniques employed by Shard-Query to implement a distributed MySQL database. The focus will then proceed to the types of distributed (massively parallel processing) database applications…
Post: MySQL-Memcached or NOSQL Tokyo Tyrant - part 1
October 15, 2009 By 16 Comments
All to often people force themselves into using a database like MySQL with no thought into whether if its the best solution…. All to often people force themselves into using a database like MySQL with no thought into whether if its the best solution… slaves, the complex sharding systems, the discussions on cluster, more memcached. We need to cover up the databases inefficiencies to ensure that…
