Homelab – Bytes&Bolts

I recently received a critical alert email from my FreeNAS box with the following error:

Device: /dev/ada3, Self-Test Log error count increased from 0 to 1
Device: /dev/ada3, 1 Currently unreadable (pending) sectors

Rather glad to know the email alerts I setup is working reliably, but looks as though I might have a few bad sectors on one of my drives.
The following commands resolved the error without resulting in any downtime.

The drive in question was /dev/ada3, so first login to a shell on your FreeNAS box as root and run a SMART long self-test (Replace adaX with your corresponding device).

smartctl -t long /dev/adaX

After the test has finished (It might take a few hours) view the results.

smartctl -a /dev/adaX

From the results remember the sector size and the location of the faulty sector (LBA_of_first_error).
In my case my sector size was 512 and LBA_of_first_error was 3082982872.

To correct the SMART error we will zero out the bad sector(s) on the drive, but first we need to permit access to drive.

sysctl kern.geom.debugflags=16

Now zero the sector stated in self-test results out.
Replace of=/dev/adaX, bs=512, seek=3082982872 with values relevant to your drive.

dd if=/dev/zero of=/dev/ada3 bs=512 count=1 seek=3082982872

Re-run the SMART report command to check if the ‘Current_Pending_Sector’ is now showing 0.

smartctl -a /dev/adaX

To check the ZFS file system integrity run a scrub of the pool, replace poolX with the pool name the drive is under (list pools with ‘zpool list’).

zpool scrub poolX

Finally check the output of the scrub to ensure there are no known data errors.

zpool status -v poolX

Now that my FreeNAS install has been running for some time in ‘Production’ with no problems, I decided to do a little housekeeping to help offset potential downtime by creating a mirror for the USB boot drive incase it fails one day.

The option to create a mirrored boot drive is presented during the initial FreeNAS install, however I didn’t have another USB of the same make/model handy at the time. Fast forward to the present, having now found a spare USB that is identical to the one already installed (SanDisk Cruzer Fit USB 2.0 32GB) it was time to finish the job.

REGIUS USB — There’s a spare in there somewhere.

FreeNAS’ excellent documentation quickly goes over the process of retroactively creating the mirrored drive. Which seemed easy enough until I ran into the following error.

MiddlewareError: Failed to attach disk: cannot attach da1p2 to da0p2: device is too small

Cut a long story short after a couple hours of messing around with both USB drives, I discovered the new drive was ever so slightly smaller in capacity than the original even though both are 32GB. According to the documentation the new drive must either the same size or larger than the original, I didn’t realise it was just so picky on exact capacity.

dan@freenas /% gpart show
=>      34  62530557  da0  GPT  (29G)
        34      1024    1  bios-boot  (512k)
      1058         6       - free -  (3.0k)
      1064  62529520    2  freebsd-zfs  (29G)
  62530584         7       - free -  (3.5k)
=>      34  61055997  da1  GPT  (29G)
        34      1024    1  bios-boot  (512k)
      1058         6       - free -  (3.0k)
      1064  61054960    2  freebsd-zfs  (29G)
  61056024         7       - free -  (3.5k)

As you can see in the above snippet da0 has a total of 62530557 blocks compared to da1 which has a total of 61055997 (meaning da0 > da1).

After some more messing around and speaking to some folks on FreeNAS’ IRC I decided to do a reinstall of FreeNAS onto the smaller drive. I backed up my FreeNAS config (System -> General -> Save Config), shutdown the server, disconnected all my WD Red’s so they couldn’t interfere with the installation and began the fresh install.

Once complete, I restored the config (System -> General -> Upload config -> Reboot) and reconnected the WD Red’s to double checked the ZFS volume was still okay (precious precious data). With everything back up and running I installed the other USB stick and tried once again the mirroring process (System -> Boot -> Status -> freenas-boot -> Attach), which I’m happy to report completed successfully after a few minutes.

Hopefully this is one resiliency feature I won’t have to rely on anytime soon, but one can never be too careful and there is no real excuse with USB drives being so cheap nowadays.
FYI 32GB is overkill really for a FreeNAS boot drive, 16GB will suffice just fine.