finally a clear statement on storing images as blobs.
I have been looking for this single-lined answer for quite a long time now…

Tobias

I agree with you in general, but there are applications where storing tons of files in a filesystem is bad.

I ran into problems having many thousands (millions?) of files, where I ran out of inodes in the filesystem I was using. Besides disk size, I did not want to have to worry about keeping track of inodes nor extra options to mkfs to increase inode counts whenever deploying a new server or increasing disk or logical volume size.
If you split things into multiple subdirectories then you also have to ensure your application(s) maintain those directories — create new, delete old when empty, etc.
Also doing an ‘ls’ on a huge directory will tie things up on your server.

The more efficient thing to do is:
A) create 1 table that has ONLY an id sequence column set as primary index, and 2nd column for the blob. This table could conceivably be in a different database if you so choose.

B) All other metadata about this blob must be held in a separate table, and the id column used to associate rows (1-to-1 relationship).

C) Do your hefty queries (with joins, complex where clauses, etc) ONLY against the metadata table, and when you actually need the data from the blob query just that single row from your blob table.

What you achieve:
1) A single large file in the filesystem for that 1 InnoDB table that holds the blob. Queries against this table will use the primary index, and will only pull from the actual table what you need.
If disk space usage is important, use row compression and pay the CPU price on your DB server, or have your application compress/decompress upon INSERT/SELECT to pay that price on your app. server.

2) A well-designed meta-data table will contain everything you need to know about that blob, and you could design things to even contain a field that is, say, the first 50 bytes of your blob — if you need that for indexing or searching.

If you think you need multiple blob columns, then as Peter suggested, consider combining them into 1 blob and on the application side splitting up the blobs. That might not so easy either, so maybe use multiple “blob-only” tables that have just a few blob columns each — less than 10 he said was problematic.

re: “If all columns do not fit to the page completely Innodb will automatically chose some of them to be on the page and some stored externally. This is not clearly documented neither can be hinted or seen.”

Last year as a result of docs Bug#29042 (http://bugs.mysql.com/bug.php?id=29042) we did identify that the manual indeed implied some of the misconceptions to which you allude. We tried to make some improvements in the discussion. See:

http://dev.mysql.com/doc/refman/5.1/en/innodb-file-space.html:

“The maximum row length, except for variable-length columns (VARBINARY, VARCHAR, BLOB and TEXT), is slightly less than half of a database page. That is, the maximum row length is about 8000 bytes. LONGBLOB and LONGTEXT columns must be less than 4GB, and the total row length, including BLOB and TEXT columns, must be less than 4GB.

If a row is less than half a page long, all of it is stored locally within the page. If it exceeds half a page, variable-length columns are chosen for external off-page storage until the row fits within half a page. For a column chosen for off-page storage, InnoDB stores the first 768 bytes locally in the row, and the rest externally into overflow pages. Each such column has its own list of overflow pages. The 768-byte prefix is accompanied by a 20-byte value that stores the true length of the column and points into the overflow list where the rest of the value is stored.”

That doesn’t address newer storage formats provided by InnoDB Plugin, of source. Integrating Plugin information into the manual is a longer-term project still in progress.

Very interesting article, thanks for writing it.

You’re right. Storing images in MySQL is in most cases not a good idea being it MyISAM or Innodb storage engine. I’m not sure how this post made you feel it is being recommended.

Right MySQL Allocates memory for a full blobs furthermore there may be several copies of blobs allocation and conversion. I would not store 1GB blobs in MySQL. I think some 64M is the practical top size limit while if you’re using more than 500K blobs I would really think whenever you’re making a right choice.

Many people use MySQL for BLOB/TEXT fields to keep them accessible together with other data as well as to get MySQL backup and recovery, replication, ACID guarantees etc.

If this is not needed using file storage can be better choice (especially when blobs are really files such as images, pdf documents etc). You can also use various noSQL storage solutions. BlobStreaming with PBXT is another interesting solution.

As I understand it is only TAIL which gets compressed. So if you have 10K blob (does not fit into the page) which can be compressed to 4K (will fit in the page) it will still use external page – 768 out of 10K will be stored on the page and when remaining 9K will be compressed to say 3K and stored on the 16K external page. Whenever 3K or 9K is used on 16K page does not really make a lot of difference.

When it makes the difference is even larger blobs which can use less external pages. When compressed. Check this out:

mysql> create table comptest(b mediumblob);
Query OK, 0 rows affected (0.05 sec)
mysql> insert into comptest values (repeat(‘a’,1000000));
Query OK, 1 row affected (0.02 sec)

mysql> show table status like “comptest” \G
*************************** 1. row ***************************
Name: comptest
Engine: InnoDB
Version: 10
Row_format: Compact
Rows: 1
Avg_row_length: 1589248
Data_length: 1589248
Max_data_length: 0
Index_length: 0
Data_free: 0
Auto_increment: NULL
Create_time: 2010-02-10 09:20:46
Update_time: NULL
Check_time: NULL
Collation: latin1_swedish_ci
Checksum: NULL
Create_options:
Comment:
1 row in set (0.00 sec)

mysql> alter table comptest row_format=compressed;
Query OK, 1 row affected (0.62 sec)
Records: 1 Duplicates: 0 Warnings: 0

mysql> show table status like “comptest” \G
*************************** 1. row ***************************
Name: comptest
Engine: InnoDB
Version: 10
Row_format: Compressed
Rows: 1
Avg_row_length: 32768
Data_length: 32768
Max_data_length: 0
Index_length: 0
Data_free: 0
Auto_increment: NULL
Create_time: 2010-02-10 09:21:08
Update_time: NULL
Check_time: NULL
Collation: latin1_swedish_ci
Checksum: NULL
Create_options: row_format=COMPRESSED
Comment:
1 row in set (0.00 sec)

In this this is highly compressible long blob gets some 50 times smaller

Also note this less than 1MB blob takes about 1.5M this is because space allocation for BLOB happens as well as for other innodb segments – allocation happens in 16K pages for a first few pages and when space is allocated in 1MB blobs. Which means “waste” can be a lot more than 16K.

This is actually a strange decision for blobs as Blobs never grow and could be allocated to exact size. If Blob changes it is stored in a new space.

The people at pbxt have created a solution for their storage engine using a streaming blob protocol, but this requires a change to mysql client as far as i know: http://www.blobstreaming.org/