Comments on: Air traffic queries in InfiniDB: early alpha http://www.mysqlperformanceblog.com/2009/11/02/air-traffic-queries-in-infinidb-early-alpha/ Percona's MySQL & InnoDB performance and scalability blog Sat, 11 Feb 2012 16:45:54 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Weidong Zhou http://www.mysqlperformanceblog.com/2009/11/02/air-traffic-queries-in-infinidb-early-alpha/comment-page-1/#comment-690261 Weidong Zhou Sat, 05 Dec 2009 05:01:55 +0000 http://www.mysqlperformanceblog.com/?p=1593#comment-690261 Jim, The bottleneck is IO, not CPU. The CPUs are waiting for lower level file IO to complete. This is why increasing number of cores not going to help too much. Use the tricks to tune IO will improve the performance. Good luck! Weidong Jim,
The bottleneck is IO, not CPU. The CPUs are waiting for lower level file IO to complete. This is why increasing number of cores not going to help too much. Use the tricks to tune IO will improve the performance. Good luck!

Weidong

]]> By: Jim Tommaney http://www.mysqlperformanceblog.com/2009/11/02/air-traffic-queries-in-infinidb-early-alpha/comment-page-1/#comment-674824 Jim Tommaney Mon, 09 Nov 2009 17:42:02 +0000 http://www.mysqlperformanceblog.com/?p=1593#comment-674824 Just as an aside, we believe we have fixed the memory issue you encountered with load data infile with our latest version on launchpad. Just as an aside, we believe we have fixed the memory issue you encountered with load data infile with our latest version on launchpad.

]]> By: Jim Tommaney http://www.mysqlperformanceblog.com/2009/11/02/air-traffic-queries-in-infinidb-early-alpha/comment-page-1/#comment-673876 Jim Tommaney Fri, 06 Nov 2009 16:17:45 +0000 http://www.mysqlperformanceblog.com/?p=1593#comment-673876 The load process is multi-threaded (for all editions) and would see some benfit from additional cores, but nowhere near linear. My timing to load the 21 years with 8 cores was 6613 seconds vs. above time of 9747 seconds with 2 cores, but there could be a large number of hardware differences besides cores so it is difficult to draw any conclusion. Actual benefits from additional cores depends on a large number of conditions; # of columns, data types, storage, etc. The load process is multi-threaded (for all editions) and would see some benfit from additional cores, but nowhere near linear. My timing to load the 21 years with 8 cores was 6613 seconds vs. above time of 9747 seconds with 2 cores, but there could be a large number of hardware differences besides cores so it is difficult to draw any conclusion. Actual benefits from additional cores depends on a large number of conditions; # of columns, data types, storage, etc.

]]> By: Vadim http://www.mysqlperformanceblog.com/2009/11/02/air-traffic-queries-in-infinidb-early-alpha/comment-page-1/#comment-673853 Vadim Fri, 06 Nov 2009 14:54:51 +0000 http://www.mysqlperformanceblog.com/?p=1593#comment-673853 Jim, Ah, ok, perfect. Now I see, your results for 2 cores are similar with my results, so I assume both cores work here. That makes a lot of sense, actually your engine is first (at least in Open Edition) that can utilize many cores during single query execution. That explains why InfiniDB is fast in many queries. Are many cores used during loading data also ? I am waiting on Q8 optimization and on subqueries, and after that I will give a shoot on 16GB, 8 cores box. Jim,

Ah, ok, perfect. Now I see, your results for 2 cores are similar with my results, so I assume both cores work here.

That makes a lot of sense, actually your engine is first (at least in Open Edition) that can utilize many cores during single query execution. That explains why InfiniDB is fast in many queries.

Are many cores used during loading data also ?

I am waiting on Q8 optimization and on subqueries, and after that I will give a shoot on 16GB, 8 cores box.

]]> By: Jim Tommaney http://www.mysqlperformanceblog.com/2009/11/02/air-traffic-queries-in-infinidb-early-alpha/comment-page-1/#comment-673831 Jim Tommaney Fri, 06 Nov 2009 13:40:21 +0000 http://www.mysqlperformanceblog.com/?p=1593#comment-673831 Sorry, bad link above, this is the right one. http://www.infinidb.org/infinidb-blog/infinidb-parallel-processing-of-airline-on-time-data.html It's just a matter of installing the community edition and running queries. The default thread parameters are sufficient to allow full multi-core processing with up to 16 cores. Beyond, that some attention to a couple of parameters may be needed to maximize system utilization. You should be able to verify with top that cpu utilization approaches 200% with a two cpu system. For example, this is what I see when running Q9 with 6 of 8 cores disable: [root@srvalpha2 ~]# top -d .25 | grep PrimProc 26264 root 18 -1 14.8g 2.8g 7000 S 183 17.9 68:00.43 PrimProc 26264 root 18 -1 14.8g 2.8g 7000 S 243 17.9 68:02.21 PrimProc 26264 root 18 -1 14.8g 2.8g 7000 S 152 17.9 68:03.39 PrimProc 26264 root 18 -1 14.8g 2.8g 7000 S 188 17.9 68:04.15 PrimProc 26264 root 18 -1 14.8g 2.8g 7000 S 196 17.9 68:04.88 PrimProc 26264 root 18 -1 14.8g 2.8g 7000 S 198 17.9 68:06.24 PrimProc 26264 root 18 -1 14.8g 2.8g 7000 S 171 17.9 68:07.44 PrimProc 26264 root 18 -1 14.8g 2.8g 7000 S 197 17.9 68:09.41 PrimProc 26264 root 18 -1 14.8g 2.8g 7000 S 195 17.9 68:10.75 PrimProc 26264 root 18 -1 14.8g 2.8g 7000 S 200 17.9 68:13.26 PrimProc 26264 root 18 -1 14.8g 2.8g 7000 S 200 17.9 68:15.28 PrimProc 26264 root 18 -1 14.8g 2.8g 7000 S 122 17.9 68:15.59 PrimProc Sorry, bad link above, this is the right one.
http://www.infinidb.org/infinidb-blog/infinidb-parallel-processing-of-airline-on-time-data.html

It’s just a matter of installing the community edition and running queries. The default thread parameters are sufficient to allow full multi-core processing with up to 16 cores. Beyond, that some attention to a couple of parameters may be needed to maximize system utilization.

You should be able to verify with top that cpu utilization approaches 200% with a two cpu system. For example, this is what I see when running Q9 with 6 of 8 cores disable:

[root@srvalpha2 ~]# top -d .25 | grep PrimProc
26264 root 18 -1 14.8g 2.8g 7000 S 183 17.9 68:00.43 PrimProc
26264 root 18 -1 14.8g 2.8g 7000 S 243 17.9 68:02.21 PrimProc
26264 root 18 -1 14.8g 2.8g 7000 S 152 17.9 68:03.39 PrimProc
26264 root 18 -1 14.8g 2.8g 7000 S 188 17.9 68:04.15 PrimProc
26264 root 18 -1 14.8g 2.8g 7000 S 196 17.9 68:04.88 PrimProc
26264 root 18 -1 14.8g 2.8g 7000 S 198 17.9 68:06.24 PrimProc
26264 root 18 -1 14.8g 2.8g 7000 S 171 17.9 68:07.44 PrimProc
26264 root 18 -1 14.8g 2.8g 7000 S 197 17.9 68:09.41 PrimProc
26264 root 18 -1 14.8g 2.8g 7000 S 195 17.9 68:10.75 PrimProc
26264 root 18 -1 14.8g 2.8g 7000 S 200 17.9 68:13.26 PrimProc
26264 root 18 -1 14.8g 2.8g 7000 S 200 17.9 68:15.28 PrimProc
26264 root 18 -1 14.8g 2.8g 7000 S 122 17.9 68:15.59 PrimProc

]]> By: Vadim http://www.mysqlperformanceblog.com/2009/11/02/air-traffic-queries-in-infinidb-early-alpha/comment-page-1/#comment-673692 Vadim Fri, 06 Nov 2009 07:18:34 +0000 http://www.mysqlperformanceblog.com/?p=1593#comment-673692 Jim, So how can I reuse two cores on my system ? Jim,

So how can I reuse two cores on my system ?

]]> By: Jim Tommaney http://www.mysqlperformanceblog.com/2009/11/02/air-traffic-queries-in-infinidb-early-alpha/comment-page-1/#comment-673689 Jim Tommaney Fri, 06 Nov 2009 07:12:58 +0000 http://www.mysqlperformanceblog.com/?p=1593#comment-673689 Actually, the multi-threaded behavior is enabled by default in community edition. I agree with regard to Q8, we are doing some profiling and will provide updates when possible. Actually, the multi-threaded behavior is enabled by default in community edition.

I agree with regard to Q8, we are doing some profiling and will provide updates when possible.

]]> By: Vadim http://www.mysqlperformanceblog.com/2009/11/02/air-traffic-queries-in-infinidb-early-alpha/comment-page-1/#comment-673612 Vadim Fri, 06 Nov 2009 03:01:40 +0000 http://www.mysqlperformanceblog.com/?p=1593#comment-673612 Jim, Thank you, so as I understood there is no way to enable multi-thread handling in Community edition, right ? Also queries Q8 looks slow in InfiniDB, is there way to improve ? Jim,

Thank you, so as I understood there is no way to enable multi-thread handling in Community edition, right ?

Also queries Q8 looks slow in InfiniDB, is there way to improve ?

]]> By: Jim Tommaney http://www.mysqlperformanceblog.com/2009/11/02/air-traffic-queries-in-infinidb-early-alpha/comment-page-1/#comment-673547 Jim Tommaney Thu, 05 Nov 2009 22:22:42 +0000 http://www.mysqlperformanceblog.com/?p=1593#comment-673547 Vadim, This is very good analysis and feedback, thank you for taking the time to do this. Of course, InfiniDB is all about a multi-threaded processing model that will benefit from additional cores. So, towards that end, I recreated the data set on two separate InfiniDB instances. A single server installation with a Dell 8-core server @ 2.0GHz, as well as a multi-server implementation. I then used Linux hotplug capabilities to take cores offline to mimic a 2, 4, and 8 core server. In spite of this hack, with 6 out of 8 cores offline the measurements were remarkably similar (a total of 178.6 seconds on the dual Xeon, and a total of 174.56 seconds with 6 cores offline). Because this is absolutely a different server configuration and to avoid confusion on what was run where, the results are posted here: http://www.infinidb.org/myblog-admin/infinidb-parallel-processing-of-airline-on-time-data.html Hopefully, this will give a sense of the multi-threaded capabilities of the system for scan and aggregation. Look to that site for future profiles of our scalable hash-join operation as well. Vadim,

This is very good analysis and feedback, thank you for taking the time to do this.

Of course, InfiniDB is all about a multi-threaded processing model that will benefit from additional cores. So, towards that end, I recreated the data set on two separate InfiniDB instances. A single server installation with a Dell 8-core server @ 2.0GHz, as well as a multi-server implementation. I then used Linux hotplug capabilities to take cores offline to mimic a 2, 4, and 8 core server. In spite of this hack, with 6 out of 8 cores offline the measurements were remarkably similar (a total of 178.6 seconds on the dual Xeon, and a total of 174.56 seconds with 6 cores offline).

Because this is absolutely a different server configuration and to avoid confusion on what was run where, the results are posted here:

http://www.infinidb.org/myblog-admin/infinidb-parallel-processing-of-airline-on-time-data.html

Hopefully, this will give a sense of the multi-threaded capabilities of the system for scan and aggregation. Look to that site for future profiles of our scalable hash-join operation as well.

]]> By: Bob Dempsey http://www.mysqlperformanceblog.com/2009/11/02/air-traffic-queries-in-infinidb-early-alpha/comment-page-1/#comment-673047 Bob Dempsey Wed, 04 Nov 2009 13:19:50 +0000 http://www.mysqlperformanceblog.com/?p=1593#comment-673047 Vadim, There is no way currently to specify optional separators like double-quotes ('"') to InfiniDB's bulk loader. I will open an enhancement request for this. Also, as an FYI, there is no way to store a zero-length string in InfiniDB. All string columns are either NULL or have a length >= 1. Vadim,

There is no way currently to specify optional separators like double-quotes (‘”‘) to InfiniDB’s bulk loader. I will open an enhancement request for this.

Also, as an FYI, there is no way to store a zero-length string in InfiniDB. All string columns are either NULL or have a length >= 1.

]]>