… and is abut not SQL and non relational data processing and storage system – BigTable with MapReduce, Amazon Dynamo and SimpleDB, Hadoop Another angle of customization of data store and processing was using non SQL and…
Post: MySQL: Data Storage or Data Processing
November 18, 2007 By 9 Comments
…not used to process the data, at least not on the large scale – instead it is used for data storage and light duty data retrieval. Even… scale data processing other techniques are used. Google uses some bigtable based technologies for search and Google Analytics (which is quite typical log processing…
Post: Using LVM for MySQL Backup and Replication Setup
August 21, 2006 By 53 Comments
… local disk based storage engines It works with MyISAM and Innodb and BDB, It also should work with Solid, PrimeXT and Falcon storage engines. Fast… for Logical Volume Main/Data . You should specify enough of undo space to hold modifications during backup process – I’ve specified 16GB…
Post: How fast can MySQL Process Data
April 9, 2008 By 3 Comments
… Kickfire Appliance and of course talking to them directly I learned a lot in their product is about beating data processing limitations of current systems. This raises valid question how fast can MySQL process (filter) data using it current… is not that bad considering abstraction of storage engine which requires “row by row” processing which means function calls for each row…
Post: Data mart or data warehouse?
July 15, 2010 By 13 Comments
… aggregation process. Comparing the performance of OLAP with and without aggregation over multiple MySQL storage engines at various data scales. What is a data warehouse…(s) and then cleaned up and inserted into the data warehouse with ETL tools. This process is usually called “conforming” the source data into the…
Post: The relationship between Innodb Log checkpointing and dirty Buffer pool pages
February 17, 2012 By 5 Comments
… essence, is what fuzzy checkpointing is. The checkpoint process is really a logical operation. It occasionally…). It also can quickly starve data reads and writes to the transaction log if …’ (good default) and ‘keep_average’ designed for SSD type storage. Running Percona Server and using this …
Post: Distributed Set Processing with Shard-Query
May 14, 2011 By 27 Comments
… there. You can add new nodes in data center B and distribute the queries over both data centers. The response time should only… model and share a common SQL dialect, then they can all participate in the distributed query processing. There is almost no data shipping… time, and transparently to the source query. Adding a new storage node SQL dialect is almost trivial. I’ll document that process shortly…
Post: Using MyISAM in production
June 17, 2006 By 35 Comments
…happen and go for long unnoticed with MyISAM storage engine. This hidden corruption may later cause crashes wrong query results and further data …tables used in batch processing and other means Exported data If you export data from OLTP system for example to perform data analyses – MyISAM will…
Post: Shard-Query EC2 images available
May 11, 2011 By 2 Comments
… blog post about the usage and performance of the splitter. It is multi-threaded(actually multi-process) and is able to hash split up to 50GB/hour of input data on my i970 test…:1 compared to raw input data), which is quite nice. Each shard contains only 128MB of data! Storage engine makes a big difference…
Post: Recovery beyond data restore
August 2, 2008 By 11 Comments
… data restore process you better look at the whole process which is required to bring system to the working state, including data consistency requirements and… mode for large data sets. Such global recovery can also put very high stress on your network and backup storage and take a lot…
