[mythtv-users] Totally OT - but need help wit RAID5 array
James Pifer
jep at obrien-pifer.com
Thu Nov 16 18:02:17 UTC 2006
Sorry for the OT here, but like the Fedora list, this list is one of the
most knowledgeable, especially in terms of storage. I posted to Fedora's
list, got some help, but nothing in the last few hours...
Yesterday I created a RAID5 array of 3 160 gig disks. Everything seemed
ok, copied a bunch of data to it, and all seemed to go fine. I needed
the space where I was previously holding the data so I had to delete
some of it. I thought I was safe since I now had it on a RAID5 array.
Not so, during the night one of the disks started going nuts and was
apparently removed from the array. See log at bottom.
Here's the really bad part. I found that when I created my array I
screwed it up, so my RAID5 was only two disks not three:
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/hdb /dev/hdc dev/hdd
Notice the missing "/" in front of dev/hdd.
So I restarted the system praying that it would all come up. So far I
can't get hdb and hdc back up in the array, although they seem to be
acting normal. Here are some commands and their output:
[root at storage ~]# mdadm --assemble /dev/md0 /dev/hdb /dev/hdc
mdadm: /dev/md0 assembled from 1 drive - not enough to start the array.
[root at storage ~]# cat /proc/mdstat
Personalities :
md0 : inactive hdc[1] hdb[0]
312581632 blocks super non-persistent
unused devices: <none>
[root at storage ~]# mdadm --query --examine /dev/hdb /dev/hdc
/dev/hdb:
Magic : a92b4efc
Version : 00.90.00
UUID : 44cfc75c:25ae3e32:0fbb5311:50edfa8a
Creation Time : Wed Nov 15 09:42:22 2006
Raid Level : raid5
Device Size : 156290816 (149.05 GiB 160.04 GB)
Array Size : 312581632 (298.10 GiB 320.08 GB)
Raid Devices : 3
Total Devices : 3
Preferred Minor : 0
Update Time : Thu Nov 16 02:49:29 2006
State : active
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Checksum : 85c9306 - correct
Events : 0.57595
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 3 64 0 active sync /dev/hdb
0 0 3 64 0 active sync /dev/hdb
1 1 22 0 1 active sync /dev/hdc
2 2 0 0 2 faulty removed
/dev/hdc:
Magic : a92b4efc
Version : 00.90.00
UUID : 44cfc75c:25ae3e32:0fbb5311:50edfa8a
Creation Time : Wed Nov 15 09:42:22 2006
Raid Level : raid5
Device Size : 156290816 (149.05 GiB 160.04 GB)
Array Size : 312581632 (298.10 GiB 320.08 GB)
Raid Devices : 3
Total Devices : 3
Preferred Minor : 0
Update Time : Thu Nov 16 08:26:55 2006
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 1
Spare Devices : 0
Checksum : 85dc2a0 - correct
Events : 0.57602
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 22 0 1 active sync /dev/hdc
0 0 0 0 0 removed
1 1 22 0 1 active sync /dev/hdc
2 2 0 0 2 faulty removed
Do I have any chance of recovering any of the data on these drives?
Any help is appreciated. (and tolerance of my OT is definitely
appreciated)
Thanks,
James
******** Log of failure *********
Nov 16 02:49:36 storage kernel: hdb: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Nov 16 02:49:36 storage kernel: hdb: dma_intr: error=0x40 { UncorrectableError }, LBAsect=12244384, high=0, low=12244384, sector=12244384
Nov 16 02:49:36 storage kernel: ide: failed opcode was: unknown
Nov 16 02:49:36 storage kernel: end_request: I/O error, dev hdb, sector 12244384
Nov 16 02:49:36 storage kernel: raid5:md0: read error not correctable (sector 12244384 on hdb).
Nov 16 02:49:36 storage kernel: raid5: Disk failure on hdb, disabling device. Operation continuing on 1 devices
Nov 16 02:49:36 storage kernel: Buffer I/O error on device md0, logical block 3061092
Nov 16 02:49:36 storage kernel: lost page write due to I/O error on md0
Nov 16 02:49:36 storage kernel: Buffer I/O error on device md0, logical block 3068016
Nov 16 02:49:36 storage kernel: lost page write due to I/O error on md0
Nov 16 02:49:36 storage kernel: Buffer I/O error on device md0, logical block 3068000
Nov 16 02:49:36 storage kernel: lost page write due to I/O error on md0
Nov 16 02:49:36 storage kernel: Buffer I/O error on device md0, logical block 3068017
Nov 16 02:49:36 storage kernel: lost page write due to I/O error on md0
Nov 16 02:49:36 storage kernel: Buffer I/O error on device md0, logical block 3068001
Nov 16 02:49:39 storage kernel: lost page write due to I/O error on md0
Nov 16 02:49:44 storage kernel: Buffer I/O error on device md0, logical block 3068018
Nov 16 02:49:44 storage kernel: lost page write due to I/O error on md0
Nov 16 02:49:44 storage kernel: Buffer I/O error on device md0, logical block 3068002
Nov 16 02:49:44 storage kernel: lost page write due to I/O error on md0
Nov 16 02:49:44 storage kernel: Buffer I/O error on device md0, logical block 3068019
Nov 16 02:49:44 storage kernel: lost page write due to I/O error on md0
Nov 16 02:49:44 storage kernel: Buffer I/O error on device md0, logical block 3068003
Nov 16 02:49:44 storage kernel: lost page write due to I/O error on md0
Nov 16 02:49:44 storage kernel: Buffer I/O error on device md0, logical block 3068020
Nov 16 02:49:44 storage kernel: lost page write due to I/O error on md0
Nov 16 02:49:44 storage kernel: hdb: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Nov 16 02:49:44 storage kernel: hdb: dma_intr: error=0x40 { UncorrectableError }, LBAsect=12252592, high=0, low=12252592, sector=12252592
Nov 16 02:49:44 storage kernel: ide: failed opcode was: unknown
Nov 16 02:49:44 storage kernel: end_request: I/O error, dev hdb, sector 12252592
Nov 16 02:49:44 storage kernel: raid5:md0: read error not correctable (sector 12252592 on hdb).
Nov 16 02:49:44 storage kernel: hdb: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Nov 16 02:49:44 storage kernel: hdb: dma_intr: error=0x40 { UncorrectableError }, LBAsect=12256696, high=0, low=12256696, sector=12256696
Nov 16 02:49:44 storage kernel: ide: failed opcode was: unknown
Nov 16 02:49:44 storage kernel: end_request: I/O error, dev hdb, sector 12256696
Nov 16 02:49:44 storage kernel: raid5:md0: read error not correctable (sector 12256696 on hdb).
Nov 16 02:49:44 storage kernel: printk: 8920 messages suppressed.
Nov 16 02:49:44 storage kernel: Buffer I/O error on device md0, logical block 3064167
Nov 16 02:49:44 storage kernel: lost page write due to I/O error on md0
Nov 16 02:49:44 storage kernel: EXT3-fs error (device md0): read_block_bitmap: Cannot read block bitmap - block_group = 124, block_bitmap = 4063232
Nov 16 02:49:44 storage kernel: Aborting journal on device md0.
Nov 16 02:49:44 storage kernel: ext3_abort called.
Nov 16 02:49:44 storage kernel: EXT3-fs error (device md0): ext3_journal_start_sb: Detected aborted journal
Nov 16 02:49:44 storage kernel: Remounting filesystem read-only
Nov 16 02:49:44 storage kernel: EXT3-fs error (device md0) in ext3_prepare_write: IO failure
Nov 16 02:49:44 storage kernel: RAID5 conf printout:
Nov 16 02:49:44 storage kernel: --- rd:3 wd:1 fd:2
Nov 16 02:49:44 storage kernel: disk 0, o:0, dev:hdb
Nov 16 02:49:44 storage kernel: disk 1, o:1, dev:hdc
Nov 16 02:49:44 storage kernel: RAID5 conf printout:
Nov 16 02:49:44 storage kernel: --- rd:3 wd:1 fd:2
Nov 16 02:49:44 storage kernel: disk 1, o:1, dev:hdc
Nov 16 02:49:44 storage kernel: __journal_remove_journal_head: freeing b_frozen_data
Nov 16 02:50:19 storage kernel: printk: 111 messages suppressed.
Nov 16 02:50:19 storage kernel: Buffer I/O error on device md0, logical block 1
Nov 16 02:50:19 storage kernel: lost page write due to I/O error on md0
Nov 16 02:50:19 storage kernel: Buffer I/O error on device md0, logical block 2981890
Nov 16 02:50:19 storage kernel: lost page write due to I/O error on md0
Nov 16 02:50:19 storage kernel: Buffer I/O error on device md0, logical block 3007502
Nov 16 02:50:19 storage kernel: lost page write due to I/O error on md0
Nov 16 02:50:19 storage kernel: Buffer I/O error on device md0, logical block 3047424
Nov 16 02:50:19 storage kernel: lost page write due to I/O error on md0
Nov 16 02:50:19 storage kernel: Buffer I/O error on device md0, logical block 3065192
Nov 16 02:50:19 storage kernel: lost page write due to I/O error on md0
Nov 16 02:50:19 storage kernel: Buffer I/O error on device md0, logical block 3066217
Nov 16 02:50:19 storage kernel: lost page write due to I/O error on md0
Nov 16 02:50:19 storage kernel: Buffer I/O error on device md0, logical block 3067242
Nov 16 02:50:19 storage kernel: lost page write due to I/O error on md0
Nov 16 03:16:25 storage smartd[1618]: Device: /dev/hdb, 3 Currently unreadable (pending) sectors
Nov 16 03:16:25 storage smartd[1618]: Sending warning via mail to root ...
Nov 16 03:16:25 storage smartd[1618]: Warning via mail to root: successful
Nov 16 03:46:30 storage smartd[1618]: Device: /dev/hdb, not capable of SMART self-check
Nov 16 03:46:30 storage kernel: hdb: irq timeout: status=0xd0 { Busy }
Nov 16 03:46:30 storage kernel: ide: failed opcode was: 0xb0
Nov 16 03:46:35 storage kernel: hda: status timeout: status=0xd0 { Busy }
Nov 16 03:46:35 storage kernel: ide: failed opcode was: unknown
Nov 16 03:46:35 storage kernel: hda: DMA disabled
Nov 16 03:46:35 storage kernel: hdb: DMA disabled
Nov 16 03:46:35 storage kernel: hda: drive not ready for command
Nov 16 03:46:36 storage kernel: ide0: reset: success
Nov 16 03:46:36 storage smartd[1618]: Sending warning via mail to root ...
Nov 16 03:46:36 storage smartd[1618]: Warning via mail to root: successful
Nov 16 03:46:46 storage smartd[1618]: Device: /dev/hdb, 3 Currently unreadable (pending) sectors
Nov 16 04:05:04 storage kernel: EXT3-fs error (device md0): ext3_get_inode_loc: unable to read inode block - inode=13123606, block=26247170
Nov 16 04:05:04 storage kernel: EXT3-fs error (device md0): ext3_get_inode_loc: unable to read inode block - inode=13123603, block=26247170
Nov 16 04:05:04 storage kernel: printk: 10 messages suppressed.
Nov 16 04:05:04 storage kernel: Buffer I/O error on device md0, logical block 1568
Nov 16 04:05:04 storage kernel: Buffer I/O error on device md0, logical block 1569
Nov 16 04:05:04 storage kernel: Buffer I/O error on device md0, logical block 1570
Nov 16 04:05:04 storage kernel: Buffer I/O error on device md0, logical block 1571
Nov 16 04:06:13 storage init: Trying to re-exec init
Nov 16 04:16:25 storage smartd[1618]: Device: /dev/hdb, 3 Currently unreadable (pending) sectors
Nov 16 04:46:31 storage smartd[1618]: Device: /dev/hdb, 3 Currently unreadable (pending) sectors
Nov 16 05:16:21 storage smartd[1618]: Device: /dev/hdb, 81 Currently unreadable (pending) sectors
Nov 16 05:46:22 storage smartd[1618]: Device: /dev/hdb, 230 Currently unreadable (pending) sectors
Nov 16 06:16:21 storage smartd[1618]: Device: /dev/hdb, 396 Currently unreadable (pending) sectors
Nov 16 06:46:20 storage smartd[1618]: Device: /dev/hdb, 708 Currently unreadable (pending) sectors
Nov 16 07:16:24 storage smartd[1618]: Device: /dev/hdb, 850 Currently unreadable (pending) sectors
Nov 16 07:46:36 storage smartd[1618]: Device: /dev/hdb, 1001 Currently unreadable (pending) sectors
More information about the mythtv-users
mailing list