… interest and we have about 3000 sites Registered over this time, which I consider decent number especially as we did not… and allowing to increase accuracy by reducing round trip. Over time we are planning to get more locations with pair of… get from real time aggregation because we want absolutely unrestricted dynamic filters and dynamic timezones and this makes things hard to aggregate. This is…
Post: ClickAider - Track Adsense Clicks and much more
July 9, 2007 By 17 Comments
… in 1999 you could hardly do much of real time aggregation and we had to store aggregated data for all stats. Compared to SpyLOG… and performance gotchas so we decided to give it some time to settle. We use ClickAider on bunch of our own…
Post: High-Performance Click Analysis with MySQL
December 22, 2008 By 13 Comments
… characteristic of a lot of these systems (real or planned) is the desire for “real-time” analysis. Our customers often want their…, let’s look at “real-time.” None of the big three (Google, Yahoo, MSN) provides real-time reporting last time I was involved with… through the file and parse it 10k lines at a time, aggregating as it goes. When each chunk is done, make…
Comment: High-Performance Click Analysis with MySQL
December 23, 2008 By 13 Comments
… from me 1) Aggregate. There is really some conflict here – to get the best aggregation speed you need to aggregate chunk at once. Merging 100000 events at once on aggregation is much faster than… you happen to use MyISAM. 2) Real time vs delayed. I think for many applications semi-real time is a value and as you…
Post: How to convert MySQL's SHOW PROFILES into a real profile
February 20, 2012 By 12 Comments
… similar phases aggregated together, doesn’t sort them by worst-first, and doesn’t show the relative amount of time consumed. I…’s something still missing: it doesn’t show lost time (the amount of time not accounted by the profiling). That is, the… query’s real response time. If there were, I could add in a UNION to inject another row for “lost time” and show…
Comment: MySQL Replication vs DRBD Battles
April 29, 2008 By 12 Comments
… real-time data warehouse applications that log tons of data and frequently summarize and aggregate will run into problems. Do you run aggregation scripts only on one server and propagate them? Do you run aggregates on…
Comment: Analyzing air traffic performance with InfoBright and MonetDB
February 2, 2010 By 50 Comments
… D/W solution which will allow us to do a real-time (10-15 sec) queries on 1B+ row tables. InfoBright, MonetDB… BY column has very low cardinality (7) => do grouping and aggregation in memory. DayOfWeek requires only 3 bits per row => for… whole column. Loading column from disk (30MB), decompressing it and aggregating 117M numbers should not take more than 1 sec (actually…
Post: Profiling MySQL stored routines
January 19, 2009 By 4 Comments
…[me] @ localhost [] # Thread_id: 3432 Schema: test # Query_time: 0.000695 Lock_time: 0.000284 Rows_sent: 1 Rows_examined: 0 Rows… can still aggregate the slow query log with mk-log-parser and find routines that MySQL spends most of the time on…, this way of manual profiling you can’t really see real time stats, for example if it was slow because of locks…
Post: Implementing efficient counters with MySQL
July 1, 2007 By 16 Comments
… many web sites you would see a counter how many time given object – blog post, forum thread, image, movie etc was… could implement cache with such semantics and it will handle real-time counters very efficiently pushing few updates back to MySQL server… replicated) to log updates in heap/myisam/archive table and aggregate it in the database for example each hour. If your…
Comment: Economics of Performance Optimization
May 5, 2008 By 9 Comments
… RAM. In the schema I use caching tables that are aggregated with several joins and are updated nightly. These are used… searching, etc. and help me having to do joins in real-time. However, the indexes on these tables are big. The indexes…, and is this on a per query basis, or over time? How do I know if faster processors will help? My…
