MySQL: Beware of sync_binlog on EXT3

Well, I just glazed again over my my.cnf for our web-cluster because I just moved a database from one cluster to another and getting quite different performance from it. So, as I expected, there is a slight difference between both configuration files:

And apparently, according to the MySQL Performance Blog that’s really, really bad (as well, we’re currently running without write caching, as the battery module of the storage is dead).

Linux: Getting information about an EXT3 filesystem

You know, I’m not getting any younger. It’s getting harder remembering every damn command … so here is how you get information out of your EXT3 filesystem:

MD (Multiple Devices) weirdness

Well, I don’t think my problem has anything to do with the DawiControl card anymore. I did a little experiment today. I created a 1TiB EXT3 file system on a single drive (one of the new 1TiB drives obviously) and started syncing data over to it (roughly 800MiB).

Now, then I unmounted the drive(s), ran fsck -C -f /dev/sd${deviceletter}1 and it went through without any trouble. Then I removed the partition and created a 1GiB partition on each drive, which I then used to build a new device mapper RAID5 array (with EXT3 on top …).

And guess what happened after I copied the data over, unmounted the file system and ran fsck ? Sure, same thing as yesterday. Now, this means either it’s a mdadm bug, while creating the array or really MD’s fault (which I can rule out, since the same happens on 2.6.25 as well as on 2.6.28) … *shrug*

SIL 3114 barfing

Well, after I had so much trouble with the USB converter (which isn’t really suited for Linux), I went ahead and bought a DawiControl DC-154 (which is using a SIL3114) controller to migrate my stuff.

After fucking up the new RAID array with the 1TB disks on the old controller (luckily I had the old hard disks still lying around, which still contained the RAID array), I plugged the 1TB disks onto the new controller and started building the array. So after 760 minutes (that’s nearly 13 hours) of synchronizing the newly created array, I was finally able to create the file system — that should be without trouble, right ?

Well, yeah … it was … So I started putting the data on the newly created array (using rsync). Only problem: something seems to be corrupting data (as in EXT3 is barfing up a lot of file system errors).

(fsck.ext3 is returning much, much more ..)

After putting the blame on EXT3, I tried out reiserfs (yeah, yeah I know .. baaaad idea). Well, at first it didn’t put out any errors, but running fsck.reiserfs turned up errors that looked a lot like the ones fsck.ext3 returned.

Then, I started looking at the array size (since I was curious), and it said the new array on four 1TB disks is ~760GB. Now according to my improper math, using 4* 1000GB drives the total usable amount of disk space should be something like 2793.96GB, and not ~760GB. *shrug*

I’m out of idea’s right now, and I’m gonna wait till January till I do anything else.

USB weirdness

Well, I was at work for a brief moment, where I grabbed me one of our SATA->USB bridges, since I need to migrate some (~750GB) data of the old raid-array and onto a new one. The troublesome about that is simply, that the current RAID controller only supports four attached devices, that’s why I do have to use something like this … Sure I could have bought a new RAID controller, but why spend 45+ EUR on something, that you can solve differently ?

Well, after figuring that I need to change my kernel config yet again (didn’t have USB support till Tue Dec 23 ~16:45:00 CET 2008) I attached the adapter to two adjacent USB ports. And shortly after copying 4-10MB, the transfer would result in a read-only EXT3 file system with something like this in the syslog:

Well, now what ? I googled a for a bit, apparently this happens when EHCI tries to write to the device and gets a timeout, cause the device is rather slow — or whatever (or the device drops down to USB 1.1). So, after disabling EHCI, the transfer has been running for about three hours now, and roughly only 1/12 of the data transferred to the external disk. Only trouble with that is, that even USB 1.1 is kinda slow to transfer 750GiB ❗

Followup: Well, due to USB 1.1 being slow as a snail, I went surfing for alternatives using Windows (since I know that the bridge does full USB 2.0 with Windows without any troubles). And guess what I found ?
There’s an EXT2/3 device driver for Windows XP, yay! So I’m copying with full 100Mbit speed right now *shrug*