Jim

I don’t want to go deeper on this in a comments thread, but you could take it to our forums. If you have time, you will probably find the help you need in our book. And if you have money but not time, you can hire us

It appears to me that disk is pretty close to saturated:

avg-cpu: %user %nice %system %iowait %steal %idle
0.51 0.00 0.20 54.55 0.00 44.73

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sdb 255.00 221.80 70.60 49.00 2580.80 2159.00 39.63 3.88 32.34 7.66 91.60
sdc 235.20 216.80 65.40 47.80 2360.00 2104.60 39.44 3.82 33.53 7.23 81.80
dm-0 0.00 0.00 628.00 535.20 5024.00 4263.60 7.98 39.04 33.41 0.85 99.20

So what to do when you MySQL queries are IO bound? Any query hints to help it do less work?

Thanks for the advice,

Jim

I have a simple table with lots of data some 60 million rows. Queries take a long, long time though CPU is low – never getting above 4%. What should I start measuring? What can I check to know if its I/O bound or what the bottleneck is?

Table Described:

+—————–+———————-+——+—–+———————+—————-+
| Field | Type | Null | Key | Default | Extra |
+—————–+———————-+——+—–+———————+—————-+
| id | int(11) | NO | PRI | NULL | auto_increment |
| title | varchar(255) | YES | | NULL | |
| description | text | YES | | NULL | |
| full_story | text | YES | | NULL | |
| uri | varchar(255) | YES | MUL | NULL | |
| pubdate | datetime | YES | | 0000-00-00 00:00:00 | |
| pubdate_reverse | int(9) | YES | | NULL | |
| feed_id | int(5) | YES | MUL | NULL | |
+—————–+———————-+——+—–+———————+—————-+

The Query:

Select id, title, uri, pubdate from entries where feed_id = 151 order by pubdate_reverse limit 1000;

1000 rows in set (3 min 34.11 sec)

The explain:

+—-+————-+———+——+——————-+——————-+———+——-+——-+————-+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+—-+————-+———+——+——————-+——————-+———+——-+——-+————-+
| 1 | SIMPLE | entries | ref | index_feed_recent | index_feed_recent | 5 | const | 18032 | Using where |
+—-+————-+———+——+——————-+——————-+———+——-+——-+————-+
1 row in set (0.00 sec)

Hints?

Best,

Jim

Another useful subpoint about the tuning process is this — change one thing at a time. I had a client where two different teams were tuning simualtaneously. One was tuning the database and the other the application. As it turned out, those teams were tossing the bottleneck back and forth between them like a hot potato without even realizing it! Given the extreme panic within the organization, they might never have solved the issue without an outsider like myself stepping in and taking charge of the tuning process.

My favorite tuning challenge was a very tough nut where the key to dramatically improving throughput was inserting a strategically located sleep statement. The over-clogged network and database queue was spewing so many errors that the CPUs were spending most of their time churning through exceptions. The sleep statement forced everything to wait a bit and this dramatically reducing exception processing. From there I was able to tune the system to the point where the sleep was no longer necessary (but neither did it hurt so the customer decided to deploy with it in place just as a safety precaution.)

Well we made a lot of money and saved our clients millions of dollars, but I am very happy to be out of the performance tuning game. It is much harder work than most greenfield development. However I have recently soft lauched a new public site and I wouldn’t mind at all if I was required to performance tune that site. At least I have a firm handle on the system design and that, in the long run, is the key to scalable performance.