My server memory is 1G, but the size of the data file for these ten small tables are 1.7G(ten files and each 170M), and the index files are about 400M (ten files and each 40M). From the mysql manual, mysql only cache index, and the data will use the OS cache. If there is any possible that the memory of my server is two low to cache all the DB data. So when I tried to query from multiple tables, the OS need switch between the data files, and cause the performance get worse.

Today I also tested that reduce the data stored in DB, each table has 40M records, and the size for all the data files are about 200M. And during my testing, there is no difference for the response time no matter I access one tables or access 10 tables alternately. This time, the wa is always 10.
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 70508 146860 24260 653572 0 0 0 0 212 2167 62 38 0 0

Thanks a lot!

It is hard to comment on such issue for me without looking at in details. One thing I should note – the “wa” is very high in your test this means this is IO bound case which is very different story.

In memory meaning tables were small enough to fit in memory completely.

The cpu of hardware is 2.2 GHz, and the memory is 1G.

All the MyISAM tables are in the same structure, the data for them are also similar. Each table has 2M records.
The following is my MyISAM table information:

user_id | int(11)
call_id | int(11)
call_direct | tinyint(4)
my_number | varchar(255)
start_time | datetime
end_time | datetime

The index for the table is: user_id+call_id

We tested to read data from the MyISAM tables with one thread and two threads. table_cache=495, Open_tables=120, Opened_tables=133.
1. Two threads and one table, the average response time is about 143ms
Thread1: access table t1
Thread2: access table t1
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy id wa
1 2 32296 8656 64572 815228 0 0 1788 0 684 692 8 4 0 88
0 2 32296 8648 64572 815508 0 0 1792 0 678 566 2 2 0 96

2. Two thread and two tables, the average response time is about 338ms.
Thread1: access table t1
Thread2: access table t2
r b swpd free buff cache si so bi bo in cs us sy id wa
2 2 30640 8764 64448 820976 0 0 1340 0 539 422 1 3 0 96
2 2 30756 8652 64448 821040 4 116 1244 116 546 406 3 1 0 96

3. One thread and two tables, the average response time is about 137ms.

Thread1: access table t1 and table t2 alternately

4. One thread and one table, the average response time is 77ms.

Thread1: only access table t1

5. We tried use two thread to access MyISAM table and Innodb table, there is no such affect. Thread1 read data from t1, thread2 read data from innodb table test, the response time for thread1 is also about 80ms.

Thread1: access table t1
Thread2: access table test (innodb)

After that, we tried cache the table index by following, but things does not change:
mysql> CACHE INDEX user_call_log_entry_1, user_call_log_entry_2, user_call_log_entry_3,user_call_log_entry_4 IN call_log_cache;
+———————————+———-+———-+
| Table | Op | Msg_type | Msg_text |
+——————————–+———-+———-+
| lps.user_1 | assign_to_keycache | status | OK |
| lps.user_2 | assign_to_keycache | status | OK |
| lps.user_3 | assign_to_keycache | status | OK |
| lps.user_4 | assign_to_keycache | status | OK |
+———————————+———-+———-+
4 rows in set (0.02 sec)

mysql> LOAD INDEX INTO CACHE lps.user_1,lps.user_2,lps.user_3,lps.user_4 IGNORE LEAVES;
+————+————–+———-+———-+
| Table | Op | Msg_type | Msg_text |
+————+————–+———-+———-+
| lps.user_1 | preload_keys | status | OK |
| lps.user_2 | preload_keys | status | OK |
| lps.user_3 | preload_keys | status | OK |
| lps.user_4 | preload_keys | status | OK |
+————+————–+———-+———-+
4 rows in set (2 min 1.85 sec)

The disks – this was RAID array but it does not really matter as it was fully in memory workload.