Tag: FreeNAS

Tech

Fixing FreeNAS error: Currently unreadable (pending) sectors

Post author By Dan
Post date 30/05/2016

I recently received a critical alert email from my FreeNAS box with the following error:

Device: /dev/ada3, Self-Test Log error count increased from 0 to 1
Device: /dev/ada3, 1 Currently unreadable (pending) sectors

Rather glad to know the email alerts I setup is working reliably, but looks as though I might have a few bad sectors on one of my drives.
The following commands resolved the error without resulting in any downtime.

The drive in question was /dev/ada3, so first login to a shell on your FreeNAS box as root and run a SMART long self-test (Replace adaX with your corresponding device).

smartctl -t long /dev/adaX

After the test has finished (It might take a few hours) view the results.

smartctl -a /dev/adaX

From the results remember the sector size and the location of the faulty sector (LBA_of_first_error).
In my case my sector size was 512 and LBA_of_first_error was 3082982872.

To correct the SMART error we will zero out the bad sector(s) on the drive, but first we need to permit access to drive.

sysctl kern.geom.debugflags=16

Now zero the sector stated in self-test results out.
Replace of=/dev/adaX, bs=512, seek=3082982872 with values relevant to your drive.

dd if=/dev/zero of=/dev/ada3 bs=512 count=1 seek=3082982872

Re-run the SMART report command to check if the ‘Current_Pending_Sector’ is now showing 0.

smartctl -a /dev/adaX

To check the ZFS file system integrity run a scrub of the pool, replace poolX with the pool name the drive is under (list pools with ‘zpool list’).

zpool scrub poolX

Finally check the output of the scrub to ensure there are no known data errors.

zpool status -v poolX

Tags FreeNAS, Homelab, Storage, ZFS

Tech

FreeNAS Mirror Boot Device Error: device is too small

Post author By Dan
Post date 10/11/2015

Now that my FreeNAS install has been running for some time in ‘Production’ with no problems, I decided to do a little housekeeping to help offset potential downtime by creating a mirror for the USB boot drive incase it fails one day.

The option to create a mirrored boot drive is presented during the initial FreeNAS install, however I didn’t have another USB of the same make/model handy at the time. Fast forward to the present, having now found a spare USB that is identical to the one already installed (SanDisk Cruzer Fit USB 2.0 32GB) it was time to finish the job.

REGIUS USB — There’s a spare in there somewhere.

FreeNAS’ excellent documentation quickly goes over the process of retroactively creating the mirrored drive. Which seemed easy enough until I ran into the following error.

MiddlewareError: Failed to attach disk: cannot attach da1p2 to da0p2: device is too small

Cut a long story short after a couple hours of messing around with both USB drives, I discovered the new drive was ever so slightly smaller in capacity than the original even though both are 32GB. According to the documentation the new drive must either the same size or larger than the original, I didn’t realise it was just so picky on exact capacity.

dan@freenas /% gpart show
=>      34  62530557  da0  GPT  (29G)
        34      1024    1  bios-boot  (512k)
      1058         6       - free -  (3.0k)
      1064  62529520    2  freebsd-zfs  (29G)
  62530584         7       - free -  (3.5k)
=>      34  61055997  da1  GPT  (29G)
        34      1024    1  bios-boot  (512k)
      1058         6       - free -  (3.0k)
      1064  61054960    2  freebsd-zfs  (29G)
  61056024         7       - free -  (3.5k)

As you can see in the above snippet da0 has a total of 62530557 blocks compared to da1 which has a total of 61055997 (meaning da0 > da1).

After some more messing around and speaking to some folks on FreeNAS’ IRC I decided to do a reinstall of FreeNAS onto the smaller drive. I backed up my FreeNAS config (System -> General -> Save Config), shutdown the server, disconnected all my WD Red’s so they couldn’t interfere with the installation and began the fresh install.

Once complete, I restored the config (System -> General -> Upload config -> Reboot) and reconnected the WD Red’s to double checked the ZFS volume was still okay (precious precious data). With everything back up and running I installed the other USB stick and tried once again the mirroring process (System -> Boot -> Status -> freenas-boot -> Attach), which I’m happy to report completed successfully after a few minutes.

Hopefully this is one resiliency feature I won’t have to rely on anytime soon, but one can never be too careful and there is no real excuse with USB drives being so cheap nowadays.
FYI 32GB is overkill really for a FreeNAS boot drive, 16GB will suffice just fine.

Tags Backup, FreeNAS, Homelab

Tech

How to: Configure FreeNAS 9.3 for Time Machine with disk quotas

Post author By Dan
Post date 07/08/2015
19 Comments on How to: Configure FreeNAS 9.3 for Time Machine with disk quotas

FreeNAS is an amazing software stack and purpose built for hosting dedicated file storage shares, so it makes for an excellent platform to host a Time Machine compatible network share for use with OS X.

I wanted to build a server that would provide a reliable backup location with data redundancy for multiple Macs, with the ability to scale the storage space to meet future needs. Something which the current line of Apple TimeCapsules don’t offer, not to mention they are expensive for what they offer.

This setup will allow any Mac on the local network to backup to a central server using Ethernet/Wireless. I have gone the extra mile and included the ability of being able to remotely backup my MacBook Pro when away in London to my FreeNAS server at home using my OpenVPN server, but that’s for another post.

To clarify this was my first time using FreeNAS, I had no prior experience with the platform before writing this guide, so anyone should be able to recreate my setup with no prior knowledge of FreeNAS. My post is an updated version of an existing article I found, but is also a result of my own trial and errors. It takes time to figure out what works and what doesn’t.

Hardware

For storage I picked up 4 x WD Red 3TB for NAS (Inc WD Express Warranty) for £91.99 each (£183.98 total). I picked the 3TB over the 4TB partly because of cost, but namely for reliability as numerous forums discuss high failure rates for the 4TB models.

I decided to utilise one of my G7 HP MicroServers as a dedicated FreeNAS server. The 2 WD Red drives were installed for storage, as well as a 32GB SanDisk Cruzer for the FreeNAS OS. I also maxed out the 16GB Kingston ECC RAM to help cope with the ZFS filesystems (the minimum recommended is 8GB).

Prerequisites

You need to have a working install of FreeNAS before you can attempt this guide. I won’t detail over the OS installation as it’s fairly simple and has been documented numerous times over online, without forgetting to mention the amazing documentation that comes with FreeNAS Doc.
FreeNAS 9.3 is the current release at time of writing and is what this guide is based on, although future versions should also work fine.

Create ‘time-machine’ Group

The first step is to create a system group for the Time-Machine share in preparation for adding users.
Under the ‘Account’ menu item, expand the ‘Groups’ item, then select ‘Add Group’. Note that in my screenshots I already have a group called ‘Time-Machine’, your system won’t have until you complete this step.

An ‘Add Group’ dialog box should pop up, prompting you to create the new group.

Set the config as follows:

Leave ‘Group ID’ to whatever it is by default
Set ‘Group Name’ to ‘time-machine’

Leave everything else as default and click OK. Our newly created ‘time-machine’ group should be visible under the ‘Groups’ section now.

Create and Configure Time-Machine ZFS Dataset

Now it’s time to create the ZFS dataset which will be used to store the Time Machine backups. You must have a ZFS volume already created for this step, if you haven’t got one then you should go read through the ZFS primer in the FreeNAS docs.

Under the ‘Storage’ tab select the ‘Volumes’ menu item, then select your ZFS volume (Volume1 in my case) and then select ‘Create Dataset’.

A ‘Create ZFS Dataset’ dialog box should pop up, prompting you to create the new ZFS dataset.

Ensure the wizard is in ‘Advanced Mode’ and then set the config as follows:

Set ‘Dataset Name’ to ‘Time-Machine’
Set ‘Quota for this dataset’ to ‘1000 GiB’

In the section option we are specifying a quota for the dataset, effectively settings the size of available disk space for our Time Machine backups. Change the value if 1000 GiB is not suitable for your setup.
Leave everything else as default and click ‘Add Dataset’. Our newly created ‘Time-Machine’ dataset should be visible under the ‘Volumes’ section now.

Now we need to configure the permissions for our ‘Time-Machine’ dataset, so that our ‘time-machine’ group has read/write access.
Select the dataset (Time-Machine) and then select ‘Change Permissions’.

A ‘Change Permissions’ dialog box should pop up, prompting you to edit the ZFS dataset.

Set the config as follows:

Set ‘Owner (group)’ to ‘time-machine’
Set ‘Mode’ checkboxes to the same as mine in the screenshot

Click ‘By setting the group owner to the ‘time-machine’ group, we are granting any users of that group read/write/execute permissions.

Create Time-Machine Users

Now it’s time to create a separate user to represent each computer that will use the FreeNAS server for Time Machine backups.
Under the ‘Account’ menu item, expand the ‘Users’ item, then select ‘Add User’.

Set the config as follows, but change the relevant information related to your setup:

Leave ‘User ID’ to whatever it is by default
Set ‘Username’ to ‘dans-macbook-pro’
Ensure ‘Create a new primary group’ is deselected
Set ‘Primary Group’ to ‘time-machine’
Set ‘Full Name’ to ‘Dan’s MacBook Pro’
Set ‘Password’ to something strong (mix of; uppercase, lowercase, numbers, 16 chars long)

Leave everything else as default and click OK. Our newly created ‘dans-macbook-pro’ should be visible under the ‘Users’ section now.

Create Time-Machine AFP Share

The last step on the FreeNAS server is to create the AFP Share that will broadcast the storage on the local network.
Under the ‘Sharing’ tab select the ‘Apple (AFP)’ menu item, and then select ‘Add Apple (AFP) Share’.

Ensure the wizard is in ‘Advanced Mode’ and then set the config as follows:

Set ‘Name’ to ‘Time Machine’
Set ‘Path’ to your ZFS dataset path
Set ‘Allow List’ to ‘@time-machine’
Ensure the ‘Time Machine’ box is checked
Ensure the ‘Default file permission’ is set to the same as the screenshot
Ensure the ‘Default directory permission’ is set to the same as the screenshot

Add Time Machine Backup to OS X

Finally the last step is to configure Time Machine itself to backup to the newly created share.
In OS X, select ‘Time Machine’ from within ‘System Preferences’, and then click the ‘Select Disk’ button.

All being well your FreeNAS AFP share should be listed. If you select to use the disk for Time Machine you will be prompted to enter the username and password for the FreeNAS user we created previously. That’s the last step, Time Machine should begin backing up shortly after adding the disk. I recommend that the first backup be completed over Ethernet instead of wireless as the initial backup can take considerable time.

I have used this setup for a couple of years now backing up 4/5 Mac’s with no real issues. Any problems I have ran into have most revolved around sudden shutdowns of the FreeNAS server midway through Time Machine backing up due to power cuts/loss. My Storage is setup using ZFS in striped mirrored mode, meaning I get the best of both for speed and disk redundancy.

Please let me know if you found this guide useful, or spot any mistakes above.

Tags FreeNAS, Mac, OS X, TimeMachine

Tech

FreeNAS disk error nightmare..

Post author By Dan
Post date 28/07/2014
No Comments on FreeNAS disk error nightmare..

When I first started tinkering with FreeNAS I installed it on a spare MicroServer just to try it out. The system only had 4GB of RAM installed, so I opted to create a UFS volume over a ZFS volume, as the latter requires at least 8GB of RAM according to the system requirements.

After some success with getting the UFS volume to play nicely with Time Machine on OS X, I took the plunge and ordered a set of Kingston 16GB (2x8GB) DDR3 1333Mhz with ECC support for £119.56 at the time of writing, the link for Amazon UK can be found here.

After the RAM arrived, I installed into the system and destroyed the old UFS volume. Suddenly I was hit with an issue, any subsequent attempts of attempting to create a fresh ZFS volume would result in failure. Several disk wipes, reboots and volume creation attempts later I was left scratching my head. My gut feeling was telling me that FreeNAS was struggling to wipe the disks (Also checking the logs helped). After googling I was able to wipe the disks after entering sysctl kern.geom.debugflags=0x10.. Little did I know this would be the command that would cause my future problems. After the command executed I was able to wipe the disks and create my new ZFS volume.

This ran for about 4 weeks perfectly, until I upgraded the system from 9.2.1.5 to 9.2.1.6. After the upgrade the system rebooted and I was suddenly presented with a amber alert status. Also my ZFS volume had disappeared, and dsmeg was spitting out errors about my mirror being corrupt (see below).

freenas kernel: GEOM: ada1: the secondary GPT table is corrupt or invalid.
freenas kernel: GEOM: ada1: using the primary only — recovery suggested.
freenas kernel: GEOM: ada2: the secondary GPT table is corrupt or invalid.
freenas kernel: GEOM: ada2: using the primary only — recovery suggested.
freenas kernel: GEOM_MIRROR: Device mirror/system launched (2/2).
freenas kernel: GEOM: mirror/system: corrupt or invalid GPT detected.
freenas kernel: GEOM: mirror/system: GPT rejected — may not be recoverable.

After a few hours goggling I determined it wasn’t anything with the disks that had failed, whilst numerous forum posts pointed to that. Instead it was linked to how I overcame the issue of wiping the disks whilst attempting to delete my old UFS volume. Turns out I hadn’t rebooted the FreeNAS system upon creating the ZFS volume, hence the firmware upgrade had nothing to do with my problems. Instead it was a disk mounting issue that was persistent since the very beginning, I just wasn’t aware of it until that moment.

I deleted the ZFS volume using the FreeNAS GUI and attempted to create a new one in its place, which repeatedly kept failing with no success. In desperation I also attempted to create UFS volumes, but that also resulted in similar errors and failure.

This post significantly helped me with figuring out what was going on. After running gmirror forget, clear and remove I was able to fix my disk creation issue.

To ensure I didn’t run into the original issue I followed information from this post to ensure my disks were wiped correctly.

Finally I was successfully able to create a fresh ZFS volume and start afresh. This time making sure the volume was able to be successfully mounted after a reboot.

Tags FreeNAS