Comments on: Funniest bug ever

By: Shlomi Noach

Shlomi Noach — Wed, 11 Feb 2009 05:40:50 +0000

@Peter, @Baron,

Thanks for the information. It does sound a lot more reasonable in light of your explanation.

By: Baron Schwartz

Baron Schwartz — Tue, 10 Feb 2009 15:39:38 +0000

No, I don’t, but anecdotally I can say that it matters if you have a lot of files. For example, suppose you have 100k tables, which is pretty common in certain types of apps. That’s at least 300k files if you’re using indexed MyISAM tables (also common for the same scenarios). Now suppose that you’re accessing them all randomly; you can do the math at how many times you’ll be doing an atime/diratime write. In many cases you won’t access a file more than once a second, so each access suffers the hit.

My anecdotal evidence is that I haven’t seen significant performance changes from adding noatime,nodiratime to the mount options on “normal” servers with a few hundred tables. You can change the mount options at runtime so it’s pretty easy to see.

By: peter

peter — Tue, 10 Feb 2009 15:37:17 +0000

Shlomi,

Linus mentions “mail spool” which tend to have a lot of tiny files…. and this is where overhead is significant. Unless you’re dealing with tens of thousands of tables in MySQL you’re in different situation. This also means any benchmarks you would like to do should be workload specific – in your particular case it is possible you will see significant gain.

By: Shlomi Noach

Shlomi Noach — Tue, 10 Feb 2009 08:07:40 +0000

Hi Baron,

Though I haven’t benchmarked myself the difference between ‘atime’ and noatime, please see what Linus Torvalds has to say about it:
http://lkml.org/lkml/2007/8/4/98

He claims more then 10% savings, though he wasn’t testing MySQL.
Do you have any benchmarks comparing with/out “noatime”?

Regards

By: o.u.

o.u. — Mon, 09 Feb 2009 22:25:00 +0000

Thank you Barron – ok, not as crazy as it sounded then, though still a shock.

By: Log Buffer

Log Buffer — Fri, 06 Feb 2009 20:37:45 +0000

"Peter Zaitsev shares the funniest bug ever." -- Log Buffer #134

By: johan

johan — Tue, 03 Feb 2009 08:33:47 +0000

Ha, it was a funny bug indeed. Unfortunately, I found it on a customer site
-j

By: Baron Schwartz

Baron Schwartz — Mon, 02 Feb 2009 22:53:44 +0000

It’s not quite that bad. It’s only once per second. (There’s only a write if the atime has actually changed, which is only true once a second.)

By: o.u.

o.u. — Mon, 02 Feb 2009 03:19:03 +0000

Wow, Shlomi Noach @ 4, re: the atime .. a write on every read, even from cache – I’m kind of shocked.

By: peter

peter — Sun, 01 Feb 2009 06:58:37 +0000

Baron, Matthew,

Right. If you would have a good backup (with point in time recovery) we would not have any recovery tools. In practice however backups sometimes are found to be broken and you have to recover the data. Our experience shows no one is immune – number of companies you’ve think should have a backup have contacted us for help (with Innodb)

With Innodb it is easy – thanks to the page format it is possible to locate data even if filesystem was totally ruined (like RAID meltdown).