I will reply you, providing as much of free help as I can (I guess you know we also provide commercial consulting services) but please could you post your message to forums instead:

http://forum.mysqlperformanceblog.com/

Thank you.

We have a really fantastics dating site serving 8000 logged in users. However the service goes down too often to call it stable. We have a montoring tool that states that we often hit the “max_connections” threshold (650) connections.
Cpu is fine (about 100% of 400% available, MPM and dual core reports 4 processors).

We have 6Gb of physical memory available but since we run on 32 bit we cannot turn up the innodb_buffer_pool higher than 1.8G right? 2.4 kernel (cannot change to 2.6 because of internal politics). Tmp area is mounted on RAM disk.

We only have innodb in the app.

The application is designed in a way that if allowed we will have something like 1000+ db connections since every user clicks on a link, refreshes his/her browser every 10 secs or so. We cache alot but still we have db problems.

a. The thing that I really, really would like to know is howto track down where the bottleneck is… I’ve read Peter Z two guides but cannot really convert the gained knowledge to solve our situation.

Lesser questions but probably as well important.
b. The question really is how do you determine the memory consumption per connection in average?
This leads to the next question
c. Can we have a limit of 600 connections when every connection consumes X mb. If the answer to ‘b’ is 2mb then the total amount of memory would be something like 1800 + 2*600 = 3G right? To much to handle for 32 bit?

Some stats at writing time (18:37 friday) System copes fine with these figures.
Queries/sec: 265
Traffic: 1Mb/sec
Connections:259

‘Aborted_clients’, ‘267108′
‘Aborted_connects’, ‘25997′
‘Binlog_cache_disk_use’, ‘145′
‘Binlog_cache_use’, ‘9638599′
‘Bytes_received’, ‘2671219831′
‘Bytes_sent’, ‘2416986544′
‘Com_admin_commands’, ‘76′
‘Com_alter_db’, ‘0′
‘Com_alter_table’, ‘7′
‘Com_analyze’, ‘0′
‘Com_backup_table’, ‘0′
‘Com_begin’, ‘0′
‘Com_change_db’, ‘155′
‘Com_change_master’, ‘0′
‘Com_check’, ‘0′
‘Com_checksum’, ‘0′
‘Com_commit’, ‘9558970′
‘Com_create_db’, ‘0′
‘Com_create_function’, ‘0′
‘Com_create_index’, ‘0′
‘Com_create_table’, ‘31′
‘Com_dealloc_sql’, ‘0′
‘Com_delete’, ‘348788′
‘Com_delete_multi’, ‘1′
‘Com_do’, ‘0′
‘Com_drop_db’, ‘0′
‘Com_drop_function’, ‘0′
‘Com_drop_index’, ‘0′
‘Com_drop_table’, ‘0′
‘Com_drop_user’, ‘0′
‘Com_execute_sql’, ‘0′
‘Com_flush’, ‘108′
‘Com_grant’, ‘0′
‘Com_ha_close’, ‘0′
‘Com_ha_open’, ‘0′
‘Com_ha_read’, ‘0′
‘Com_help’, ‘0′
‘Com_insert’, ‘17889656′
‘Com_insert_select’, ‘11′
‘Com_kill’, ‘47′
‘Com_load’, ‘0′
‘Com_load_master_data’, ‘0′
‘Com_load_master_table’, ‘0′
‘Com_lock_tables’, ‘102′
‘Com_optimize’, ‘0′
‘Com_preload_keys’, ‘0′
‘Com_prepare_sql’, ‘0′
‘Com_purge’, ‘0′
‘Com_purge_before_date’, ‘0′
‘Com_rename_table’, ‘0′
‘Com_repair’, ‘0′
‘Com_replace’, ‘20801587′
‘Com_replace_select’, ‘0′
‘Com_reset’, ‘0′
‘Com_restore_table’, ‘0′
‘Com_revoke’, ‘0′
‘Com_revoke_all’, ‘0′
‘Com_rollback’, ‘4666272′
‘Com_savepoint’, ‘0′
‘Com_select’, ‘378247551′
‘Com_set_option’, ‘285487367′
‘Com_show_binlog_events’, ‘0′
‘Com_show_binlogs’, ‘0′
‘Com_show_charsets’, ‘2′
‘Com_show_collations’, ‘176788′
‘Com_show_column_types’, ‘0′
‘Com_show_create_db’, ‘8′
‘Com_show_create_table’, ‘2661′
‘Com_show_databases’, ‘93′
‘Com_show_errors’, ‘0′
‘Com_show_fields’, ‘4260′
‘Com_show_grants’, ‘0′
‘Com_show_innodb_status’, ‘167091′
‘Com_show_keys’, ‘136′
‘Com_show_logs’, ‘0′
‘Com_show_master_status’, ‘105′
‘Com_show_ndb_status’, ‘0′
‘Com_show_new_master’, ‘0′
‘Com_show_open_tables’, ‘0′
‘Com_show_privileges’, ‘0′
‘Com_show_processlist’, ‘20′
‘Com_show_slave_hosts’, ‘61′
‘Com_show_slave_status’, ‘13834′
‘Com_show_status’, ‘194766′
‘Com_show_storage_engines’, ‘0′
‘Com_show_tables’, ‘2842′
‘Com_show_variables’, ‘176825′
‘Com_show_warnings’, ‘8′
‘Com_slave_start’, ‘0′
‘Com_slave_stop’, ‘0′
‘Com_stmt_close’, ‘304204263′
‘Com_stmt_execute’, ‘304205150′
‘Com_stmt_prepare’, ‘410680831′
‘Com_stmt_reset’, ‘0′
‘Com_stmt_send_long_data’, ‘0′
‘Com_truncate’, ‘0′
‘Com_unlock_tables’, ‘102′
‘Com_update’, ‘15146253′
‘Com_update_multi’, ‘0′
‘Connections’, ‘263413′
‘Created_tmp_disk_tables’, ‘0′
‘Created_tmp_files’, ‘4001′
‘Created_tmp_tables’, ‘669137′
‘Delayed_errors’, ‘0′
‘Delayed_insert_threads’, ‘0′
‘Delayed_writes’, ‘0′
‘Flush_commands’, ‘5′
‘Handler_commit’, ‘19092941′
‘Handler_delete’, ‘0′
‘Handler_discover’, ‘0′
‘Handler_read_first’, ‘1100773′
‘Handler_read_key’, ‘2491307870′
‘Handler_read_next’, ‘1266206297′
‘Handler_read_prev’, ‘0′
‘Handler_read_rnd’, ‘234159075′
‘Handler_read_rnd_next’, ‘2358572961′
‘Handler_rollback’, ‘4947643′
‘Handler_update’, ‘12524′
‘Handler_write’, ‘292923929′
‘Key_blocks_not_flushed’, ‘0′
‘Key_blocks_unused’, ‘28987′
‘Key_blocks_used’, ‘1332′
‘Key_read_requests’, ‘1552507′
‘Key_reads’, ‘1654′
‘Key_write_requests’, ‘114303′
‘Key_writes’, ‘1346′
‘Max_used_connections’, ‘651′
‘Not_flushed_delayed_rows’, ‘0′
‘Open_files’, ‘95′
‘Open_streams’, ‘0′
‘Open_tables’, ‘512′
‘Opened_tables’, ‘183240′
‘Qcache_free_blocks’, ‘136′
‘Qcache_free_memory’, ‘10272736′
‘Qcache_hits’, ‘14774371′
‘Qcache_inserts’, ‘83902401′
‘Qcache_lowmem_prunes’, ‘4314′
‘Qcache_not_cached’, ‘294363493′
‘Qcache_queries_in_cache’, ‘177′
‘Qcache_total_blocks’, ‘498′
‘Questions’, ‘1462755007′
‘Rpl_status’, ‘NULL’
‘Select_full_join’, ‘0′
‘Select_full_range_join’, ‘0′
‘Select_range’, ‘5683551′
‘Select_range_check’, ‘0′
‘Select_scan’, ‘1104831′
‘Slave_open_temp_tables’, ‘0′
‘Slave_retried_transactions’, ‘0′
‘Slave_running’, ‘OFF’
‘Slow_launch_threads’, ‘0′
‘Slow_queries’, ‘27405′
‘Sort_merge_passes’, ‘376′
‘Sort_range’, ‘56479689′
‘Sort_rows’, ‘1344092495′
‘Sort_scan’, ‘935242′
‘Table_locks_immediate’, ‘895966639′
‘Table_locks_waited’, ‘20096′
‘Threads_cached’, ‘0′
‘Threads_connected’, ‘260′
‘Threads_created’, ‘124272′
‘Threads_running’, ‘3′
‘Uptime’, ‘4433556′

# top
18:38:40 up 175 days, 2:56, 1 user, load average: 0.79, 1.00, 0.95
328 processes: 326 sleeping, 2 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 15.0% 0.0% 9.5% 0.0% 0.3% 6.5% 68.4%
cpu00 22.1% 0.0% 11.1% 0.1% 1.1% 6.1% 59.1%
cpu01 14.9% 0.0% 7.5% 0.0% 0.0% 5.9% 71.5%
cpu02 10.5% 0.0% 12.5% 0.0% 0.1% 7.1% 69.5%
cpu03 12.7% 0.0% 6.7% 0.0% 0.0% 6.7% 73.7%
Mem: 5907484k av, 5889348k used, 18136k free, 0k shrd, 34132k buff
4607808k actv, 882264k in_d, 96252k in_c
Swap: 4194224k av, 553604k used, 3640620k free 3352196k cached

#iostat 5 2

avg-cpu: %user %nice %sys %iowait %idle
20.49 0.00 6.29 20.80 52.41

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
cciss/c1d0 39.88 218.38 89.96 3304016450 1360995328
cciss/c1d0p1 39.88 218.38 89.96 3304016194 1360995328
cciss/c1d1 66.15 227.50 255.83 3441930690 3870514728
cciss/c1d1p1 66.15 227.50 255.83 3441930434 3870514728
cciss/c0d0 0.86 3.52 11.26 53198730 170422784
cciss/c0d0p1 0.27 0.48 4.77 7251314 72197656
cciss/c0d0p2 0.01 0.06 0.10 858968 1573160
cciss/c0d0p3 0.00 0.00 0.00 168 0
cciss/c0d0p5 0.12 0.11 1.14 1650506 17248664
cciss/c0d0p6 0.27 1.95 2.77 29569906 41860440
cciss/c0d0p7 0.11 0.38 2.16 5719090 32723824
cciss/c0d0p8 0.08 0.54 0.32 8148218 4819040
cciss/c0d0p9 0.00 0.00 0.00 248 0

# vmstat 5 2
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy id wa
2 2 553604 18116 37400 3349360 0 0 0 0 0 0 1 0 1 1
0 0 553604 18548 37316 3349456 0 0 1210 143 2476 3809 19 1 70 10

mysql conf:

[mysqld]
server-id=5908
port=3309
socket=/tmp/mysql.sock
pid-file=/var/run/mysql/mysql.pid
datadir=/db/disk2/mysql/data
log-bin=/var/log/mysql/log
log-slow-queries=/var/log/mysql/slow-queries.log
max_connections=650
open_files_limit=6144
max_connect_errors=10000
interactive_timeout=3600
wait_timeout=3600
key_buffer_size=32M # Only for MyISAM which isnt used.
sort_buffer_size=2M
query_cache_size=10M # Might be increased
thread_cache_size=40
table_cache=512 # Default 64 was used before.
skip-locking
innodb_data_home_dir=/db/disk2/data
innodb_data_file_path=ibdata1:10240M;ibdata2:10240M;ibdata3:10240M;ibdata4:10240M;ibdata5:10M:autoextend
innodb_log_group_home_dir=/var/log/mysql
innodb_log_arch_dir=/var/log/mysql
innodb_table_locks=0
innodb_buffer_pool_size=1800M # USE ALL MEMORY AVAILABLE
#innodb_log_buffer_size=8M # Lowered from 32M according to MySQL
innodb_additional_mem_pool_size=20M
innodb_file_io_threads=4
innodb_lock_wait_timeout=15 # CHANGED from 50
innodb_log_files_in_group=3
innodb_flush_log_at_trx_commit=0
#innodb_log_file_size=400M # Approx 20% of buffer pool
innodb_thread_concurrency=64 # Default is 8 wich is a way to low.

Puhhhhh!

That was alot of text. Could someone please help us out a bit!

Regards

//Marcus Herou

Here is example - You could have such server running with max_connections=1000 and myisam_sort_buffer_size=256M.

If it happens so all connections will start to repair tables at the same time it will need 1000*256MB = 256GB of memory.

In practice however you would not have more than 2-3 connections doing it at the same time so you happen to be fine.

Is it correct?