Comments on: Working with many files and file system fragmentation http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/ Everything about MySQL Performance Tue, 02 Dec 2008 18:08:10 +0000 http://wordpress.org/?v=2.5.1 By: peter http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-255717 peter Sat, 22 Mar 2008 17:26:43 +0000 http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-255717 Daniel, I'm not quite sure from your post - did you create the tablespace out of many files and it was already badly fragmented or it was autoextend during some workload run ? In any case we can learn filesystems may not be as optimal as we would like them to be :) Daniel,

I’m not quite sure from your post - did you create the tablespace out of many files and it was already badly fragmented or it was autoextend during some workload run ?

In any case we can learn filesystems may not be as optimal as we would like them to be :)

]]> By: Daniel Schneller http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-255652 Daniel Schneller Sat, 22 Mar 2008 13:33:39 +0000 http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-255652 Some time ago (2006) I noticed that even on a fully defragged NTFS drive newly created InnoDB data files got scattered around the whole partition. http://jroller.com/dschneller/entry/strange_innodb_fragmentation Some time ago (2006) I noticed that even on a fully defragged NTFS drive newly created InnoDB data files got scattered around the whole partition.
http://jroller.com/dschneller/entry/strange_innodb_fragmentation

]]> By: MySQL File System Fragmentation Benchmarks | MySQL Performance Blog http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-255471 MySQL File System Fragmentation Benchmarks | MySQL Performance Blog Sat, 22 Mar 2008 03:47:03 +0000 http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-255471 [...] days ago I wrote about testing writing to many files and seeing how this affects sequential read performance. I was [...] [...] days ago I wrote about testing writing to many files and seeing how this affects sequential read performance. I was [...]

]]> By: Markus http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-254365 Markus Wed, 19 Mar 2008 06:23:29 +0000 http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-254365 Be aware if you use XFS/ReiserFS on PC-Hardware, you risk loss of data in case of power failure: http://zork.net/~nick/mail/why-reiserfs-is-teh-sukc Be aware if you use XFS/ReiserFS on PC-Hardware, you risk loss of data in case of power failure:

http://zork.net/~nick/mail/why-reiserfs-is-teh-sukc

]]> By: peter http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-254335 peter Wed, 19 Mar 2008 05:17:52 +0000 http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-254335 Brian, Thanks for heads up I've written another blog post on those tools. I'm quite sure that was not caching because number at VMSTAT matches these quite closely. Brian,

Thanks for heads up I’ve written another blog post on those tools.

I’m quite sure that was not caching because number at VMSTAT matches these quite closely.

]]> By: peter http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-254333 peter Wed, 19 Mar 2008 05:16:49 +0000 http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-254333 Thanks tgabi, If you have updates you may be suffering from internal fragmentation in addition to file system fragmentation - such as rows may get more than one piece in MySQL sequential pages in index order happen to be in different locations etc. Good you agree on pre allocation option though I think even much smaller values such as 1MB or 4MB would make things much better. Thanks tgabi,

If you have updates you may be suffering from internal fragmentation in addition to file system fragmentation - such as rows may get more than one piece in MySQL sequential pages in index order happen to be in different locations etc.

Good you agree on pre allocation option though I think even much smaller values such as 1MB or 4MB would make things much better.

]]> By: tgabi http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-254283 tgabi Wed, 19 Mar 2008 02:56:14 +0000 http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-254283 I've noticed this long time ago. I'm fighting this using different measures: a. small tables are optimized periodically b.big tables that are affected by fragmentation can be arranged as 1 file per filesystem c. recently Mysql 5.1 table partitioning allowed to use SSD for most recent data (most used) and disk for historical data (less used). Index files are the most prone to fragmentation - since 1 record inserted/updated can trigger updates on several indices. No filesystem is immune to fragmentation unfortunately, the only way to reduce it is to have some table options that will pre-allocate space in large chunks (like 1GB data, 1 GB index). I’ve noticed this long time ago. I’m fighting this using different measures: a. small tables are optimized periodically b.big tables that are affected by fragmentation can be arranged as 1 file per filesystem c. recently Mysql 5.1 table partitioning allowed to use SSD for most recent data (most used) and disk for historical data (less used). Index files are the most prone to fragmentation - since 1 record inserted/updated can trigger updates on several indices. No filesystem is immune to fragmentation unfortunately, the only way to reduce it is to have some table options that will pre-allocate space in large chunks (like 1GB data, 1 GB index).

]]> By: brian http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-254228 brian Tue, 18 Mar 2008 23:40:14 +0000 http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-254228 Are you sure the speed difference is caused by fragmentation of the filesystem on the disk, and not due to the single large file being in the buffer cache? It might be interesting to use the following to see what's in the buffer cache: http://net.doit.wisc.edu/~plonka/fincore/ http://insights.oetiker.ch/linux/fadvise.html Are you sure the speed difference is caused by fragmentation of the filesystem on the disk, and not due to the single large file being in the buffer cache? It might be interesting to use the following to see what’s in the buffer cache:

http://net.doit.wisc.edu/~plonka/fincore/

http://insights.oetiker.ch/linux/fadvise.html

]]> By: peter http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-254204 peter Tue, 18 Mar 2008 23:09:22 +0000 http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-254204 Oh yes. SSD reduce seek penalty dramatically so fragmentation should be much less issue for them. This is multiple Terabyte data storage project so it is not for SSD yet. Oh yes. SSD reduce seek penalty dramatically so fragmentation should be much less issue for them.

This is multiple Terabyte data storage project so it is not for SSD yet.

]]> By: Bill http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-254178 Bill Tue, 18 Mar 2008 22:08:39 +0000 http://www.mysqlperformanceblog.com/2008/03/18/working-with-many-files-and-file-system-fragmentation/#comment-254178 It would also be interesting to repeat the test with the files on those new solid state drives, to see how much of an improvement you could get. It would also be interesting to repeat the test with the files on those new solid state drives, to see how much of an improvement you could get.

]]>