Saturday, December 20, 2014

Remapping bad sectors with Linux and dd

I have one faulty non-RAID disk on server which I can't replace right now so I wanted to see if I could mask problems by manually triggering bad sector reallocation.


After figuring out how to force reallocation I found out there's already document how to do same, but in addition it also includes how to find out which file(s) are affected. That document is bit dated and does NOT cover requirement of using 4k blocks on with new disks so decided to explain how I successfully reallocated stubborn bad block on WDC WD20EARS. Smartmontools doc also lacks crucial "conv=fsync" parameter from dd command line.

Smartctl showed following error.

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       30%     28615         2427271528

LBA uses 512 byte sectors while disk itself is 4096 sector like most recent disks are.  Let's confirm this is indeed bad block by attempting to read entire 4k sector.

dd if=/dev/sda of=/tmp/badblock bs=4096 count=1 skip=$((2427271528/8))
dd: reading `/dev/sda': Input/output error
0+0 records in
0+0 records out
0 bytes (0 B) copied, 17.4632 s, 0.0 kB/s

Didn't exactly work - just as expected. Log shows:

sd 2:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
sd 2:0:0:0: [sda] CDB: Read(10): 28 00 90 ad 39 68 00 00 08 00

Since data has already been lost we try to overwrite bad data with zeros hopefully triggering reallocation. It's mandatory to write entire 4k sector at once, trying to use bs=512 count=8 will fail and won't trigger reallocation because then disk is internally trying to do read+modify+write rather than plain write.

dd if=/dev/zero of=/dev/sda bs=4096 count=1 seek=$((2427271528/8)) conv=fsync
dd: fsync failed for `/dev/sda': Input/output error
1+0 records in
1+0 records out

Hmm. Didn't work? Let's retry.

dd if=/dev/zero of=/dev/sda bs=4096 count=1 seek=$((2427271528/8)) conv=fsync
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.432892 s, 9.5 kB/s

Whoo, now it's working. Seems that first write triggered reallocation despite returning error and therefore second write to that block succeeds.

Checking smartctl data shows that we have one more reallocated block.

Before:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   199   199   140    Pre-fail  Always       -       35

After:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   199   199   140    Pre-fail  Always       -       36

We can now re-run smart tests to determine if there's more bad blocks waiting to be fixed.

How to figure out what file we need to restore based on offset of bad block can be mapped following those  instructions on smartmontools site. Or if you're doing the right thing just trigger ZFS scan and it will tell you even if it can't fix it due lack of redundancy.

1 comment:

  1. Thanks! Worked for me. Like you, I had been unsuccessful using the instructions on the smartmontools page. Curious, I opened "$ man dd" and read the entries for "conv=CONVS" & "fsync" and I'll tell you....I still have no idea what they're talking about. But your tutorial worked and my drive is now being re-verified by the RAID card. If Google hadn't found your post I probably wouldn't have been able to figure this out. Good post. Thank you very much.

    ReplyDelete

Got something to say?!