Comments on: When should you store serialized objects in the database?

By: Diego

Diego — Sat, 06 Feb 2010 20:20:07 +0000

Great post and discussion. IMO, unless someone is really unconfortable with trying new dbs (which are somewhat not as proven as mysql), they shouldn’t mysql that way. It’s harder to administer than any nosql db, and it’s just not the best tool for storing key/value data.

By: Leo Petr

Leo Petr — Thu, 04 Feb 2010 00:16:12 +0000

Hi, I’m dropping by via the High Scalability blog.

This is a neat technique.

Something similar but much simpler can be done in IBM DB2. DB2 has a built-in, indexable XML column type. There’s native support for XPath and XQuery, so you can store a clob with arbitrary fields serialized as XML and then do SQL queries with XPath to extract arbitrary fields, run aggregation functions on them, etc. Effectively, this lets you do exactly the same thing except without the opaqueness and with potentially higher performance depending on what you want.

This is included in the free edition (ibm.com/db2/express/)

Disclaimer: I work on the DB2 team, the opinions are my own, etc.

By: Baron Schwartz

Baron Schwartz — Wed, 27 Jan 2010 01:50:17 +0000

This is completely OT and not meant to be trolling, but one of the things I’ve always thought Postgres could improve is make their data files less architecture-dependent.

By: Rob Wultsch

Rob Wultsch — Wed, 27 Jan 2010 00:01:18 +0000

Mark:
“All table rows are structured in the same way. There is a fixed-size header (occupying 23 bytes on most machines), followed by an optional null bitmap, an optional object ID field, and the user data. The header is detailed in Table 53-4. The actual user data (columns of the row) begins at the offset indicated by t_hoff, which must always be a multiple of the MAXALIGN distance for the platform. The null bitmap is only present if the HEAP_HASNULL bit is set in t_infomask. If it is present it begins just after the fixed header and occupies enough bytes to have one bit per data column (that is, t_natts bits altogether). In this list of bits, a 1 bit indicates not-null, a 0 bit is a null. When the bitmap is not present, all columns are assumed not-null. The object ID is only present if the HEAP_HASOID bit is set in t_infomask. If present, it appears just before the t_hoff boundary. Any padding needed to make t_hoff a MAXALIGN multiple will appear between the null bitmap and the object ID. (This in turn ensures that the object ID is suitably aligned.) ”
http://www.postgresql.org/docs/8.4/interactive/storage-page-layout.html

By: Brian Cavanagh

Brian Cavanagh — Tue, 26 Jan 2010 15:53:14 +0000

Yeah PostgreSQLmight be a good choice if you can get around the lousy replication support, as it will let you store arrays and another data types natively in the fields, so you don’t have to worry about an inaccessible object model.

By: Mark Callaghan

Mark Callaghan — Mon, 25 Jan 2010 02:52:07 +0000

Josh – what is the cost of a null column in Postgres? Do possibly null columns require a bit in the row header?

By: Josh Berkus

Josh Berkus — Mon, 25 Jan 2010 02:44:59 +0000

I just thought I’d mention where PostgreSQL is for this: It’s less costly to have null columns in PostgreSQL than in InnoDB, so having lots of null columns is not a reason to use E-Blob. Postgres is also very good at storing blobs, and compresses them for offline storage automatically. Also, with GIST, GIN, expression indexes and HStore (and the upcoming HStore-to-JSON) there are indexing options for blobs. So thats two reasons TO use them.

Reasons to use E-Blob are:

a) the data stored varies wildly per entity, over time, or between customer installations;

b) the data in the blob is almost always retrieved, read, and updated all-at-once due to the application design;

c) the data is not going to be used for aggregation, ad-hoc querying, or response-time-sensitive filtering (since cheap indexes are impossible);

d) the data does not need to be constrained or used to enforce a constraint

Reasons not to use it are:

1) the above-mentioned update cost: update an entire 1K blob to change one value;

2) complete inability to enforce meaningful constraints on the data, thus allowing garbage to creep into the database;

3) high cost of blanket updates to the data which might be required by application design changes.

Generally, I only consider e-blob for non-essential data which is going to vary by installation, or for specially structured data which is infrequently updated and thus works well with special index types which work with blobs.

By: Sean

Sean — Sun, 24 Jan 2010 11:46:32 +0000

Yup, this is a great technique for storing user preferences, e.g. the layout of a page or enabling/disabling certain feature. We just store a serialized associate array in the database for several different things on our site, and when the user logs in, we grab it, unserialize it, and store it in a session. Works great. It woulud be a huge pain in the ass if everytime we wanted to add a new option to the set, we had to change the table structure.

As long as your datda doens’t need to be indexed or searchable, this can save a ton of time, short term and long term.

By: Baron Schwartz

Baron Schwartz — Fri, 22 Jan 2010 21:52:04 +0000

Steven, I agree with you about where to do compression. Your idea about the client automatically handling it is great. Why not mention it on the Drizzle mailing list?

By: Morgan Tocker

Morgan Tocker — Fri, 22 Jan 2010 20:26:51 +0000

Robert – MongoDB uses a binary representation of JSON. They call it BSON:
http://www.mongodb.org/display/DOCS/BSON

If you’re talking about fast formats for MySQL, I liked Jeremy’s suggestion (Google protobuf).