here is data from my tests:
computer 1:

My.ini
[client]
port=3306
[mysql]
default-character-set=latin1
[mysqld]
port=3306
basedir=”C:/Program Files/MySQL/MySQL Server 5.0/”
datadir=”C:/Program Files/MySQL/MySQL Server 5.0/Data/”
default-character-set=latin1
default-storage-engine=INNODB
#default-storage-engine=MyISAM
sql-mode=”STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION”
max_connections=100
query_cache_size=0
table_cache=256
tmp_table_size=93M
thread_cache_size=8
#*** MyISAM Specific options
myisam_max_sort_file_size=100G
myisam_max_extra_sort_file_size=100G
myisam_sort_buffer_size=185M
key_buffer_size=157M
read_buffer_size=64K
read_rnd_buffer_size=256K
sort_buffer_size=256K
#*** INNODB Specific options ***
innodb_additional_mem_pool_size=20M
innodb_flush_log_at_trx_commit=0
innodb_log_buffer_size=4M
innodb_buffer_pool_size=304M
innodb_log_file_size=152M
innodb_thread_concurrency=8

RAM: 1GB
Processor: Intel P4 2.253GHz
Cache size: 512KB
Front side bus: 530 MHz
Harddrive: WDC WD400BB-75DEA0
Max Burn Rate: 100MB/s tested 8MB/s
RPMs: 7200

using multiple inserts per transaction for 1 innodb table (25 inserts or .2 seconds of data to be inserted) I got 1121, 1133, and 1166 inserts per second into that one table.
using autocommit for innodb inserts I got 1125, and 1158 inserts per second into the table
using myisam I got 1186 and 1177 inserts per second into the table
the average data length was 231B. It seemed the writing of the insert is what was taking all the time. Data was coming in much faster than it was able to insert. The data queue was getting quite long. I ran the test for 5 minutes. The inserts per second was pretty constant the little variation arrose in how fast the data was coming in which was no more than 3000 packets of data per second. The processor would go up to about 80% during tests.

please give me feed back. I am trying to get the data to be inserted faster than it is coming in. Please post the my.cnf file that you were able to get 9000 inserts per second and the computer specs for that test.

Thanks.

I may run it on XFS when I have a chance. ReiserFS future is kind of uncertain now and as it is dropped as default filesystem by SuSE and not supported by RedHat I do not see customers eager to use it in production.

Also note just many files and support of many growing files is different things. ReiserFS indeed works well with small files I remember creating 10.000.000 of 100 byte files in the directory and it still was working fine.

It would be nice to see if the difference is seen as clearly with actual data in the files

Absolutely. I specially published the benchmark script so everyone could easily repeat the run with options they like

There are a lot of variables you can play with

Things like dir_index etc but also things like noatime.

Could a new test be performed on the same box using for example noatime on the mount and see how it (if any) changes?

There ARE things filesystem can do. For example you could externally allocate space in extents, related to file size and filesystem size. For example if you have 50MB file allocating in 1MB blocks would still cause no more than 2% space waste. For 10K image of course you do not want.

Another optimization which can be done is called delayed allocation. When you perform writes you can actually allocate space only when you flush data to disk, this way you can accumulate larger fragments.

It is not Innodb single tablespace allocation important here as it is single growing file anyway but how innodb allocates data intermally - which is done in 1M extents after first few pages.

I’m not sure what is default grow increment for innodb_file_per_table tablespace - this one is important as many files grow at the same time.

I agree with your worst case scenario. This is what I tried to point out in my previous comment. Though maybe I didn’t do a good job expressing myself

The point I was trying to make is that in that situation there’s not much the filesystem CAN do.

It could TRY to pre-allocate both files in larger chunks but then you’d have angular velocity kick in on the HDDs.

InnoDB’s grow factor (which by default is 8M I believe) is a good balance.

Of course in pure science you could spend a lot of time on this and test different things… Though I mainly tested what had practical interest for me at that point.

Regarding fragmentation - you can’t really assume 2 tables are enough without knowing about internal implementation of OS and tables. First it is quite possible to design filesystem which would be able to handle small amount of growing files well but not large number of files.

But what is even more important is not all fragmentations are same.

Consider the worst case with 2 files with blocks going as 12121212121212 drive/raid/os or storge engine itself will do read-ahead which will fetch few blocks with single read. 1MB read will have only 512K of data for the given table but it is still much better than getting single row with random read