Remapping bad sectors with Linux and dd
I have one faulty non-RAID disk on server which I can't replace right now so I wanted to see if I could mask problems by manually triggering bad sector reallocation.
After figuring out how to force reallocation I found out there's already document how to do same, but in addition it also includes how to find out which file(s) are affected. That document is bit dated and does NOT cover requirement of using 4k blocks on with new disks so decided to explain how I successfully reallocated stubborn bad block on WDC WD20EARS. Smartmontools doc also lacks crucial "conv=fsync" parameter from dd command line.
Smartctl showed following error.
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 30% 28615 2427271528
LBA uses 512 byte sectors while disk itself is 4096 sector like most recent disks are. Let's confirm this is indeed bad block by attempting to read entire 4k sector.
dd if=/dev/sda of=/tmp/badblock bs=4096 count=1 skip=$((2427271528/8))
dd: reading `/dev/sda': Input/output error
0+0 records in
0+0 records out
0 bytes (0 B) copied, 17.4632 s, 0.0 kB/s
Didn't exactly work - just as expected. Log shows:
sd 2:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
sd 2:0:0:0: [sda] CDB: Read(10): 28 00 90 ad 39 68 00 00 08 00
Since data has already been lost we try to overwrite bad data with zeros hopefully triggering reallocation. It's mandatory to write entire 4k sector at once, trying to use bs=512 count=8 will fail and won't trigger reallocation because then disk is internally trying to do read+modify+write rather than plain write.
dd if=/dev/zero of=/dev/sda bs=4096 count=1 seek=$((2427271528/8)) conv=fsync
dd: fsync failed for `/dev/sda': Input/output error
1+0 records in
1+0 records out
Hmm. Didn't work? Let's retry.
dd if=/dev/zero of=/dev/sda bs=4096 count=1 seek=$((2427271528/8)) conv=fsync
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.432892 s, 9.5 kB/s
Whoo, now it's working. Seems that first write triggered reallocation despite returning error and therefore second write to that block succeeds.
Checking smartctl data shows that we have one more reallocated block.
Before:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 199 199 140 Pre-fail Always - 35
After:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 199 199 140 Pre-fail Always - 36
We can now re-run smart tests to determine if there's more bad blocks waiting to be fixed.
How to figure out what file we need to restore based on offset of bad block can be mapped following those instructions on smartmontools site. Or if you're doing the right thing just trigger ZFS scan and it will tell you even if it can't fix it due lack of redundancy.
After figuring out how to force reallocation I found out there's already document how to do same, but in addition it also includes how to find out which file(s) are affected. That document is bit dated and does NOT cover requirement of using 4k blocks on with new disks so decided to explain how I successfully reallocated stubborn bad block on WDC WD20EARS. Smartmontools doc also lacks crucial "conv=fsync" parameter from dd command line.
Smartctl showed following error.
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 30% 28615 2427271528
LBA uses 512 byte sectors while disk itself is 4096 sector like most recent disks are. Let's confirm this is indeed bad block by attempting to read entire 4k sector.
dd if=/dev/sda of=/tmp/badblock bs=4096 count=1 skip=$((2427271528/8))
dd: reading `/dev/sda': Input/output error
0+0 records in
0+0 records out
0 bytes (0 B) copied, 17.4632 s, 0.0 kB/s
Didn't exactly work - just as expected. Log shows:
sd 2:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
sd 2:0:0:0: [sda] CDB: Read(10): 28 00 90 ad 39 68 00 00 08 00
Since data has already been lost we try to overwrite bad data with zeros hopefully triggering reallocation. It's mandatory to write entire 4k sector at once, trying to use bs=512 count=8 will fail and won't trigger reallocation because then disk is internally trying to do read+modify+write rather than plain write.
dd if=/dev/zero of=/dev/sda bs=4096 count=1 seek=$((2427271528/8)) conv=fsync
dd: fsync failed for `/dev/sda': Input/output error
1+0 records in
1+0 records out
Hmm. Didn't work? Let's retry.
dd if=/dev/zero of=/dev/sda bs=4096 count=1 seek=$((2427271528/8)) conv=fsync
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.432892 s, 9.5 kB/s
Whoo, now it's working. Seems that first write triggered reallocation despite returning error and therefore second write to that block succeeds.
Checking smartctl data shows that we have one more reallocated block.
Before:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 199 199 140 Pre-fail Always - 35
After:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 199 199 140 Pre-fail Always - 36
We can now re-run smart tests to determine if there's more bad blocks waiting to be fixed.
How to figure out what file we need to restore based on offset of bad block can be mapped following those instructions on smartmontools site. Or if you're doing the right thing just trigger ZFS scan and it will tell you even if it can't fix it due lack of redundancy.
Thanks! Worked for me. Like you, I had been unsuccessful using the instructions on the smartmontools page. Curious, I opened "$ man dd" and read the entries for "conv=CONVS" & "fsync" and I'll tell you....I still have no idea what they're talking about. But your tutorial worked and my drive is now being re-verified by the RAID card. If Google hadn't found your post I probably wouldn't have been able to figure this out. Good post. Thank you very much.
ReplyDelete