EXT3 – BAFM

MySQL: Beware of sync_binlog on EXT3

February 23, 2009June 21, 2013 Christian Leave a comment

Well, I just glazed again over my my.cnf for our web-cluster because I just moved a database from one cluster to another and getting quite different performance from it. So, as I expected, there is a slight difference between both configuration files:

@@ -55,8 +58,6 @@
 innodb_log_group_home_dir       = /var/lib/mysql/db
 innodb_log_file_size            = 512M
 innodb_thread_concurrency       = 8
-sync_binlog                     = 1

@@ -55,8 +58,6 @@

innodb_log_group_home_dir = /var/lib/mysql/db

innodb_log_file_size = 512M

innodb_thread_concurrency = 8

-sync_binlog = 1

And apparently, according to the MySQL Performance Blog that’s really, really bad (as well, we’re currently running without write caching, as the battery module of the storage is dead).

Linux: Getting information about an EXT3 filesystem

February 13, 2009June 21, 2013 Christian Leave a comment

You know, I’m not getting any younger. It’s getting harder remembering every damn command … so here is how you get information out of your EXT3 filesystem:

sles10sp2 ~ [0] &gt; tune2fs -l /dev/sda2 | grep &quot;^Filesystem&quot;
Filesystem volume name:   &lt;none&gt;
Filesystem UUID:          8eec8235-4d9e-4b58-acf9-3c68c977d5ea
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal resize_inode filetype 
                          needs_recovery sparse_super large_file
Filesystem state:         clean
Filesystem OS type:       Linux
Filesystem created:       Tue May 27 10:48:56 2008

sles10sp2 ~ [0] > tune2fs -l /dev/sda2 | grep "^Filesystem"

Filesystem volume name: <none>

Filesystem UUID: 8eec8235-4d9e-4b58-acf9-3c68c977d5ea

Filesystem magic number: 0xEF53

Filesystem revision #: 1 (dynamic)

Filesystem features: has_journal resize_inode filetype

needs_recovery sparse_super large_file

Filesystem state: clean

Filesystem OS type: Linux

Filesystem created: Tue May 27 10:48:56 2008

MD (Multiple Devices) weirdness

December 29, 2008August 8, 2014 Christian Leave a comment

Well, I don’t think my problem has anything to do with the DawiControl card anymore. I did a little experiment today. I created a 1TiB EXT3 file system on a single drive (one of the new 1TiB drives obviously) and started syncing data over to it (roughly 800MiB).

Now, then I unmounted the drive(s), ran fsck -C -f /dev/sd${deviceletter}1 and it went through without any trouble. Then I removed the partition and created a 1GiB partition on each drive, which I then used to build a new device mapper RAID5 array (with EXT3 on top …).

And guess what happened after I copied the data over, unmounted the file system and ran fsck ? Sure, same thing as yesterday. Now, this means either it’s a mdadm bug, while creating the array or really MD’s fault (which I can rule out, since the same happens on 2.6.25 as well as on 2.6.28) … *shrug*

SIL 3114 barfing

December 28, 2008August 8, 2014 Christian 2 Comments

Well, after I had so much trouble with the USB converter (which isn’t really suited for Linux), I went ahead and bought a DawiControl DC-154 (which is using a SIL3114) controller to migrate my stuff.

After fucking up the new RAID array with the 1TB disks on the old controller (luckily I had the old hard disks still lying around, which still contained the RAID array), I plugged the 1TB disks onto the new controller and started building the array. So after 760 minutes (that’s nearly 13 hours) of synchronizing the newly created array, I was finally able to create the file system — that should be without trouble, right ?

Well, yeah … it was … So I started putting the data on the newly created array (using rsync). Only problem: something seems to be corrupting data (as in EXT3 is barfing up a lot of file system errors).

Dec 28 08:47:21 epimetheus [67092.652866] EXT3-fs: mounted filesystem with ordered data mode.
Dec 28 09:53:20 epimetheus [71058.253027] EXT3-fs error (device md2): ext3_add_entry: bad entry
 in directory #23371810: directory entry across blocks - offset=260, inode=18964552,
 rec_len=26988, name_len=115
Dec 28 09:53:20 epimetheus [71058.305558] EXT3-fs error (device md2): ext3_add_entry: bad entry
 in directory #23371810: directory entry across blocks - offset=260, inode=18964552,
 rec_len=26988, name_len=115

Dec 28 08:47:21 epimetheus [67092.652866] EXT3-fs: mounted filesystem with ordered data mode.

Dec 28 09:53:20 epimetheus [71058.253027] EXT3-fs error (device md2): ext3_add_entry: bad entry

in directory #23371810: directory entry across blocks - offset=260, inode=18964552,

rec_len=26988, name_len=115

Dec 28 09:53:20 epimetheus [71058.305558] EXT3-fs error (device md2): ext3_add_entry: bad entry

in directory #23371810: directory entry across blocks - offset=260, inode=18964552,

rec_len=26988, name_len=115

(fsck.ext3 is returning much, much more ..)

After putting the blame on EXT3, I tried out reiserfs (yeah, yeah I know .. baaaad idea). Well, at first it didn’t put out any errors, but running fsck.reiserfs turned up errors that looked a lot like the ones fsck.ext3 returned.

Then, I started looking at the array size (since I was curious), and it said the new array on four 1TB disks is ~760GB. Now according to my improper math, using 4* 1000GB drives the total usable amount of disk space should be something like 2793.96GB, and not ~760GB. *shrug*

I’m out of idea’s right now, and I’m gonna wait till January till I do anything else.

USB weirdness

December 23, 2008December 23, 2008 Christian 1 Comment

Well, I was at work for a brief moment, where I grabbed me one of our SATA->USB bridges, since I need to migrate some (~750GB) data of the old raid-array and onto a new one. The troublesome about that is simply, that the current RAID controller only supports four attached devices, that’s why I do have to use something like this … Sure I could have bought a new RAID controller, but why spend 45+ EUR on something, that you can solve differently ?

Well, after figuring that I need to change my kernel config yet again (didn’t have USB support till Tue Dec 23 ~16:45:00 CET 2008) I attached the adapter to two adjacent USB ports. And shortly after copying 4-10MB, the transfer would result in a read-only EXT3 file system with something like this in the syslog:

Dec 23 17:23:57 epimetheus usb 1-2: new high speed USB device using ehci_hcd and address 2
Dec 23 17:23:57 epimetheus usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
Dec 23 17:23:57 epimetheus usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Dec 23 17:23:57 epimetheus usb usb3: Product: OHCI Host Controller
Dec 23 17:23:57 epimetheus usb usb3: Manufacturer: Linux 2.6.25-hardened-r11 ohci_hcd
Dec 23 17:23:57 epimetheus usb usb3: SerialNumber: 0000:00:02.1
Dec 23 17:23:57 epimetheus Initializing USB Mass Storage driver...
Dec 23 17:23:57 epimetheus usb 1-2: reset high speed USB device using ehci_hcd and address 2
Dec 23 17:23:57 epimetheus usb 1-2: device descriptor read/64, error -71
Dec 23 17:23:58 epimetheus usb 1-2: reset high speed USB device using ehci_hcd and address 2
Dec 23 17:23:58 epimetheus usb 1-2: device descriptor read/64, error -71
Dec 23 17:23:58 epimetheus usb 1-2: reset high speed USB device using ehci_hcd and address 2
Dec 23 17:23:58 epimetheus usb 1-2: device not accepting address 2, error -71
Dec 23 17:23:59 epimetheus usb 1-2: reset high speed USB device using ehci_hcd and address 2
Dec 23 17:23:59 epimetheus usb 1-2: device not accepting address 2, error -71
Dec 23 17:23:59 epimetheus usb 1-2: USB disconnect, address 2
Dec 23 17:23:59 epimetheus sd 4:0:0:0: Device offlined - not ready after error recovery
Dec 23 17:23:59 epimetheus sd 4:0:0:0: [sde] Result: hostbyte=0x01 driverbyte=0x00
Dec 23 17:23:59 epimetheus end_request: I/O error, dev sde, sector 1229380295
Dec 23 17:23:59 epimetheus end_request: I/O error, dev sde, sector 1229380535
Dec 23 17:23:59 epimetheus Aborting journal on device sde1.
Dec 23 17:23:59 epimetheus ext3_abort called.
Dec 23 17:23:59 epimetheus EXT3-fs error (device sde1): ext3_journal_start_sb: Detected aborted journal
Dec 23 17:23:59 epimetheus Remounting filesystem read-only
Dec 23 17:23:59 epimetheus EXT3-fs error (device sde1) in ext3_ordered_writepage: IO failure
Dec 23 17:23:59 epimetheus __journal_remove_journal_head: freeing b_frozen_data
Dec 23 17:23:59 epimetheus __journal_remove_journal_head: freeing b_frozen_data
Dec 23 17:23:59 epimetheus __journal_remove_journal_head: freeing b_frozen_data
Dec 23 17:23:59 epimetheus __journal_remove_journal_head: freeing b_frozen_data
Dec 23 17:23:59 epimetheus __journal_remove_journal_head: freeing b_committed_data

Dec 23 17:23:57 epimetheus usb 1-2: new high speed USB device using ehci_hcd and address 2

Dec 23 17:23:57 epimetheus usb usb3: New USB device found, idVendor=1d6b, idProduct=0001

Dec 23 17:23:57 epimetheus usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1

Dec 23 17:23:57 epimetheus usb usb3: Product: OHCI Host Controller

Dec 23 17:23:57 epimetheus usb usb3: Manufacturer: Linux 2.6.25-hardened-r11 ohci_hcd

Dec 23 17:23:57 epimetheus usb usb3: SerialNumber: 0000:00:02.1

Dec 23 17:23:57 epimetheus Initializing USB Mass Storage driver...

Dec 23 17:23:57 epimetheus usb 1-2: reset high speed USB device using ehci_hcd and address 2

Dec 23 17:23:57 epimetheus usb 1-2: device descriptor read/64, error -71

Dec 23 17:23:58 epimetheus usb 1-2: reset high speed USB device using ehci_hcd and address 2

Dec 23 17:23:58 epimetheus usb 1-2: device descriptor read/64, error -71

Dec 23 17:23:58 epimetheus usb 1-2: reset high speed USB device using ehci_hcd and address 2

Dec 23 17:23:58 epimetheus usb 1-2: device not accepting address 2, error -71

Dec 23 17:23:59 epimetheus usb 1-2: reset high speed USB device using ehci_hcd and address 2

Dec 23 17:23:59 epimetheus usb 1-2: device not accepting address 2, error -71

Dec 23 17:23:59 epimetheus usb 1-2: USB disconnect, address 2

Dec 23 17:23:59 epimetheus sd 4:0:0:0: Device offlined - not ready after error recovery

Dec 23 17:23:59 epimetheus sd 4:0:0:0: [sde] Result: hostbyte=0x01 driverbyte=0x00

Dec 23 17:23:59 epimetheus end_request: I/O error, dev sde, sector 1229380295

Dec 23 17:23:59 epimetheus end_request: I/O error, dev sde, sector 1229380535

Dec 23 17:23:59 epimetheus Aborting journal on device sde1.

Dec 23 17:23:59 epimetheus ext3_abort called.

Dec 23 17:23:59 epimetheus EXT3-fs error (device sde1): ext3_journal_start_sb: Detected aborted journal

Dec 23 17:23:59 epimetheus Remounting filesystem read-only

Dec 23 17:23:59 epimetheus EXT3-fs error (device sde1) in ext3_ordered_writepage: IO failure

Dec 23 17:23:59 epimetheus __journal_remove_journal_head: freeing b_frozen_data

Dec 23 17:23:59 epimetheus __journal_remove_journal_head: freeing b_committed_data

Well, now what ? I googled a for a bit, apparently this happens when EHCI tries to write to the device and gets a timeout, cause the device is rather slow — or whatever (or the device drops down to USB 1.1). So, after disabling EHCI, the transfer has been running for about three hours now, and roughly only 1/12 of the data transferred to the external disk. Only trouble with that is, that even USB 1.1 is kinda slow to transfer 750GiB ❗

Followup: Well, due to USB 1.1 being slow as a snail, I went surfing for alternatives using Windows (since I know that the bridge does full USB 2.0 with Windows without any troubles). And guess what I found ?
There’s an EXT2/3 device driver for Windows XP, yay! So I’m copying with full 100Mbit speed right now *shrug*