Thanks so much for your help.

Our mysql db contains a mix of innodb and myism tables and run on linux with 16gigs or ram.

Here is our my.cnf

# sammple MySQL config file for very large systems.
#
#
# This is for a large system with memory of 1G-2G where the system runs mainly
# MySQL.
#
# You can copy this file to
# /etc/my.cnf to set global options,
# mysql-data-dir/my.cnf to set server-specific options (in this
# installation this directory is /usr/local/mysql/data) or
# ~/.my.cnf to set user-specific options.
#
# In this file, you can use all long options that a program supports.
# If you want to know which options a program supports, run the program
# with the “–help” option.

# The following options will be passed to all MySQL clients
[client]
#password = your_password
port = 3306
socket = /tmp/mysql.sock
#tmpdir =/mysql_tmp/
# Here follows entries for some specific programs

# The MySQL server
[mysqld]
port = 3306
socket = /tmp/mysql.sock
bind-address=10.234.94.71
skip-locking
key_buffer_size = 2000M
max_allowed_packet = 32M

# table_cache=20M
# open-files-limit=20000

table_cache = 3072
open_files_limit = 9216

tmp_table_size=1000M
sort_buffer_size = 100M
read_buffer_size = 100M
read_rnd_buffer_size = 100M
myisam_sort_buffer_size = 100M
max_length_for_sort_data=2048
max_sort_length=2048
long-query-time=5
log-slow-queries=/apps/log/slow-query
interactive_timeout=300
wait_timeout=300
thread_cache = 40
max_connections=500
query_cache_size = 2000M
# Try number of CPU’s*2 for thread_concurrency
thread_concurrency = 8
ft_min_word_len=3
#skip-grant-tables

# Don’t listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the “enable-named-pipe” option) will render mysqld useless!
#
#skip-networking

# Replication Master Server (default)
# binary logging is required for replication
log-bin=db1-bin
log-bin-index=db1-bin.index

binlog-ignore-db=chrome_vin
binlog-ignore-db=dummyData
#binlog-ignore-db=edmunds
#binlog-ignore-db=evox
#binlog-ignore-db=jato
#binlog-ignore-db=kbb
binlog-ignore-db=mysql
binlog-ignore-db=test
#binlog-ignore-db=us_incentives_extract
#binlog-ignore-db=vehicles
#binlog-ignore-db=voiceshot

# required unique id between 1 and 2^32 – 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id = 1

# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
# the syntax is:
#
# CHANGE MASTER TO MASTER_HOST=, MASTER_PORT=,
# MASTER_USER=, MASTER_PASSWORD= ;
#
# where you replace , , by quoted strings and
# by the master’s port number (3306 by default).
#
# Example:
#
# CHANGE MASTER TO MASTER_HOST=’125.564.12.1′, MASTER_PORT=3306,
# MASTER_USER=’joe’, MASTER_PASSWORD=’secret’;
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
# start replication for the first time (even unsuccessfully, for example
# if you mistyped the password in master-password and the slave fails to
# connect), the slave will create a master.info file, and any later
# change in this file to the variables’ values below will be ignored and
# overridden by the content of the master.info file, unless you shutdown
# the slave server, delete master.info and restart the slaver server.
# For that reason, you may want to leave the lines below untouched
# (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 – 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id = 2
#
# The replication master for this slave – required
#master-host =
#
# The username the slave will use for authentication when connecting
# to the master – required
#master-user =
#
# The password the slave will authenticate with when connecting to
# the master – required
#master-password =
#
# The port the master is listening on.
# optional – defaults to 3306
#master-port =
#
# binary logging – not required for slaves, but recommended
#log-bin
# Point the following paths to different dedicated disks
#tmpdir = /tmp/
#log-update = /path-to-dedicated-directory/hostname
tmpdir =/mysql_tmp/:/tmp/
# Uncomment the following if you are using BDB tables
#bdb_cache_size = 384M
#bdb_max_lock = 100000

# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = /usr/local/mysql/data/
#innodb_data_file_path = ibdata1:2000M;ibdata2:10M:autoextend
#innodb_log_group_home_dir = /usr/local/mysql/data/
#innodb_log_arch_dir = /usr/local/mysql/data/
innodb_data_home_dir = /db
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = /db
innodb_log_arch_dir = /db

# You can set .._buffer_pool_size up to 50 – 80 %
# of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 8000M
#innodb_additional_mem_pool_size = 80M
# Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 1000M
#innodb_log_buffer_size = 32M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

#[mysqlhotcopy]
interactive-timeout

It should not be reason why it is slow per say.

I would not be limiting it to single system. If you’re using several boxes you brobably have them on local network with 1GBit connection between them. This allows you do do many thousands of queries per second from single connection. From multiple connections it will be many tens of thousands per second.

Network is fast these days, this is why memcached is getting so popular or MySQL Cluster can exist.

Now you probably do not want to do 100 of the queries instead of one. Please take a close look at my recommendation I recommend using 2-3 queries when MySQL optimizer does not optimize query efficient enough or for some other reason single query requires much more work than separate queries.

This is true, however most developers don’t realize that this is *within the database system only*. ie, if you only go to the database once, then go ahead, run as many queries as you want.

If a system is being developed with a *remote* database in mind, you actually have to find a good balance. If the network back and forth with 100 queries costs more than just doing the complex query in the first place, then it’s faster to do the complex query and only go across the network and back to the database once.

For test dataset rule is pretty simple – system should behave same as it will behave on production. So generate about same amount of data as you would run on the single box in production. Sometimes you want to scale it down a bit if your test system is low end.

Basically you want two things to apply – queries should be executed same way on production and test system. Read EXPLAIN should be the same. Second – cache efficiency should be similar. CPU bound workload can’t be compared to disk bound.