Fixing FreeNAS error: Currently unreadable (pending) sectors

I recently received a critical alert email from my FreeNAS box with the following error:

Device: /dev/ada3, Self-Test Log error count increased from 0 to 1
Device: /dev/ada3, 1 Currently unreadable (pending) sectors

Rather glad to know the email alerts I setup is working reliably, but looks as though I might have a few bad sectors on one of my drives.
The following commands resolved the error without resulting in any downtime.

The drive in question was /dev/ada3, so first login to a shell on your FreeNAS box as root and run a SMART long self-test (Replace adaX with your corresponding device).

smartctl -t long /dev/adaX

After the test has finished (It might take a few hours) view the results.

smartctl -a /dev/adaX

From the results remember the sector size and the location of the faulty sector (LBA_of_first_error).
In my case my sector size was 512 and LBA_of_first_error was 3082982872.

To correct the SMART error we will zero out the bad sector(s) on the drive, but first we need to permit access to drive.

sysctl kern.geom.debugflags=16

Now zero the sector stated in self-test results out.
Replace of=/dev/adaX, bs=512, seek=3082982872 with values relevant to your drive.

dd if=/dev/zero of=/dev/ada3 bs=512 count=1 seek=3082982872

Re-run the SMART report command to check if the ‘Current_Pending_Sector’ is now showing 0.

smartctl -a /dev/adaX

To check the ZFS file system integrity run a scrub of the pool, replace poolX with the pool name the drive is under (list pools with ‘zpool list’).

zpool scrub poolX

Finally check the output of the scrub to ensure there are no known data errors.

zpool status -v poolX

Circus Ponies Notebook is no more

Circus Ponies logo

This afternoon I went on over to Circus Ponies to download a fresh copy of Notebook for a new HackPro of mine, and was presented with this lovely notice.
Thanks for all the fish

Well that was a bit out of the blue, and with no prior notice. No emails, no heads up, no nothing.

Circus Ponies announced today that they are closing up shop, having been acquired by Google. They had been the creators of a fab piece of note taking software called Notebook. It really was a simple bit of kit that did was it was meant to do and nothing more, I used it in conjunction with Owncloud to privately host my project notes which automatically synced between my various computers. Not sure why Google would want their hands on it yet, perhaps to improve their own suite of Apps capable of note taking.

Now begins the joyous task of finding something else that will fit the bill. I had been considering for a while about hosting my own internal wiki to accomplish the same role as Notebook, the only niggle with that idea is I often find myself without a reliable Internet connection (trains) and don’t want to be sat stranded not being able to do work simply because I don’t have the relevant project notes handy.

Anyways It’s not as if the app is crippled without online functionality as it doesn’t really have any. So in the meantime I will continue to use until I can find a suitable replacement or Apple releases an update with breaks compatibility.
That really will be the last nail in the coffin for Notebook.

FreeNAS Mirror Boot Device Error: device is too small

SanDisk Cruzer Fit 32GB USB 2.0

Now that my FreeNAS install has been running for some time in ‘Production’ with no problems, I decided to do a little housekeeping to help offset potential downtime by creating a mirror for the USB boot drive incase it fails one day.

The option to create a mirrored boot drive is presented during the initial FreeNAS install, however I didn’t have another USB of the same make/model handy at the time. Fast forward to the present, having now found a spare USB that is identical to the one already installed (SanDisk Cruzer Fit USB 2.0 32GB) it was time to finish the job.

REGIUS USB
There’s a spare in there somewhere.

FreeNAS’ excellent documentation quickly goes over the process of retroactively creating the mirrored drive. Which seemed easy enough until I ran into the following error.

MiddlewareError: Failed to attach disk: cannot attach da1p2 to da0p2: device is too small

Cut a long story short after a couple hours of messing around with both USB drives, I discovered the new drive was ever so slightly smaller in capacity than the original even though both are 32GB. According to the documentation the new drive must either the same size or larger than the original, I didn’t realise it was just so picky on exact capacity.

dan@freenas /% gpart show
=>      34  62530557  da0  GPT  (29G)
        34      1024    1  bios-boot  (512k)
      1058         6       - free -  (3.0k)
      1064  62529520    2  freebsd-zfs  (29G)
  62530584         7       - free -  (3.5k)
=>      34  61055997  da1  GPT  (29G)
        34      1024    1  bios-boot  (512k)
      1058         6       - free -  (3.0k)
      1064  61054960    2  freebsd-zfs  (29G)
  61056024         7       - free -  (3.5k)

As you can see in the above snippet da0 has a total of 62530557 blocks compared to da1 which has a total of 61055997 (meaning da0 > da1).

After some more messing around and speaking to some folks on FreeNAS’ IRC I decided to do a reinstall of FreeNAS onto the smaller drive. I backed up my FreeNAS config (System -> General -> Save Config), shutdown the server, disconnected all my WD Red’s so they couldn’t interfere with the installation and began the fresh install.

Once complete, I restored the config (System -> General -> Upload config -> Reboot) and reconnected the WD Red’s to double checked the ZFS volume was still okay (precious precious data). With everything back up and running I installed the other USB stick and tried once again the mirroring process (System -> Boot -> Status -> freenas-boot -> Attach), which I’m happy to report completed successfully after a few minutes.

FreeNAS Boot drive mirrors

Hopefully this is one resiliency feature I won’t have to rely on anytime soon, but one can never be too careful and there is no real excuse with USB drives being so cheap nowadays.
FYI 32GB is overkill really for a FreeNAS boot drive, 16GB will suffice just fine.

Spacewalk 2.4 installation on CentOS 7

Spacewalk logo

Spacewalk (not the EVA kind) is an open source (GPLv2) Linux systems management solution and is the upstream project for Red Hat Satellite. The main feature I wished to experiment with was the ability to manage multiple systems packages (upgrades) concurrently. The project also has several other main features, including;

  • Inventory your systems (hardware and software information)
  • Install and update software on your systems
  • Collect and distribute your custom software packages into manageable groups
  • Provision (kickstart) your systems
  • Manage and deploy configuration files to your systems
  • Provision and start/stop/configure virtual guests
  • Distribute content across multiple geographical sites in an efficient manner

Prerequisites

Taken directly from the official Spacewalk docs:

  • Open outbound firewall ports 80, 443
  • Inbound open ports 80, 443, 5222 (covered below later on)
  • Storage for database: 250 KiB per client system + 500 KiB per channel + 230 KiB per package in channel (i.e. 1.1GiB for channel with 5000 packages)
  • Storage for packages (default /var/satellite): Red Hat recommend 6GB per channel for their channels, I used 20GB for storage, as 10GB wasn’t enough
  • 2GB RAM minimum, 4GB recommended
  • Fully up-to-date underlying operating system running a vanilla installation (no user customisation performed yet, fresh system install)

Setup Repos

There are 3 main repos to setup.

Spacewalk repo

rpm -Uvh http://yum.spacewalkproject.org/2.4/RHEL/7/x86_64/spacewalk-repo-2.4-3.el7.noarch.rpm

JPackage

cat > /etc/yum.repos.d/jpackage-generic.repo << EOF
[jpackage-generic]
name=JPackage generic
#baseurl=http://mirrors.dotsrc.org/pub/jpackage/5.0/generic/free/
mirrorlist=http://www.jpackage.org/mirrorlist.php?dist=generic&type=free&release=5.0
enabled=1
gpgcheck=1
gpgkey=http://www.jpackage.org/jpackage.asc
EOF

EPEL

Spacewalk requires Java Virtual Machine 1.6.0 or greater which is available in the EPEL repo.

rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Installation

Time to get on with the install 🙂

Database backend

Spacewalk uses database server to store its primary data. It supports either PostgreSQL (version 8.4 and higher) or Oracle RDBMS. Lets install PostgreSQL and let Spacewalk configure the database for us.

yum install spacewalk-setup-postgresql

Install Spacewalk

Install the Spacewalk package with support for PostgreSQL.

yum install spacewalk-postgresql 

Configure the firewall

Enable inbound firewall (firewalld) ports 80 (http) and 443 (https).

firewall-cmd --add-service=http
firewall-cmd --add-service=https

Configure Spacewalk

Last step.

spacewalk-setup --disconnected

Afterwards we should be able to visit: https://hostname.yourdomain.com to create the Spacewalk admin account and finish the installation.

How to: Configure FreeNAS 9.3 for Time Machine with disk quotas

FreeNAS Logo

FreeNAS is an amazing software stack and purpose built for hosting dedicated file storage shares, so it makes for an excellent platform to host a Time Machine compatible network share for use with OS X.

I wanted to build a server that would provide a reliable backup location with data redundancy for multiple Macs, with the ability to scale the storage space to meet future needs. Something which the current line of Apple TimeCapsules don’t offer, not to mention they are expensive for what they offer.

This setup will allow any Mac on the local network to backup to a central server using Ethernet/Wireless. I have gone the extra mile and included the ability of being able to remotely backup my MacBook Pro when away in London to my FreeNAS server at home using my OpenVPN server, but that’s for another post.

To clarify this was my first time using FreeNAS, I had no prior experience with the platform before writing this guide, so anyone should be able to recreate my setup with no prior knowledge of FreeNAS. My post is an updated version of an existing article I found, but is also a result of my own trial and errors. It takes time to figure out what works and what doesn’t.

Hardware

For storage I picked up 4 x WD Red 3TB for NAS (Inc WD Express Warranty) for £91.99 each (£183.98 total). I picked the 3TB over the 4TB partly because of cost, but namely for reliability as numerous forums discuss high failure rates for the 4TB models.

I decided to utilise one of my G7 HP MicroServers as a dedicated FreeNAS server. The 2 WD Red drives were installed for storage, as well as a 32GB SanDisk Cruzer for the FreeNAS OS. I also maxed out the 16GB Kingston ECC RAM to help cope with the ZFS filesystems (the minimum recommended is 8GB).

Prerequisites

FreeNAS 9.3 Homepage
You need to have a working install of FreeNAS before you can attempt this guide. I won’t detail over the OS installation as it’s fairly simple and has been documented numerous times over online, without forgetting to mention the amazing documentation that comes with FreeNAS Doc.
FreeNAS 9.3 is the current release at time of writing and is what this guide is based on, although future versions should also work fine.

Create ‘time-machine’ Group

The first step is to create a system group for the Time-Machine share in preparation for adding users.
Under the ‘Account’ menu item, expand the ‘Groups’ item, then select ‘Add Group’. Note that in my screenshots I already have a group called ‘Time-Machine’, your system won’t have until you complete this step.
Add Group in FreeNAS for TimeMachine

An ‘Add Group’ dialog box should pop up, prompting you to create the new group.
FreeNAS Group Settings
Set the config as follows:

  • Leave ‘Group ID’ to whatever it is by default
  • Set ‘Group Name’ to ‘time-machine’

Leave everything else as default and click OK. Our newly created ‘time-machine’ group should be visible under the ‘Groups’ section now.

Create and Configure Time-Machine ZFS Dataset

Now it’s time to create the ZFS dataset which will be used to store the Time Machine backups. You must have a ZFS volume already created for this step, if you haven’t got one then you should go read through the ZFS primer in the FreeNAS docs.

Under the ‘Storage’ tab select the ‘Volumes’ menu item, then select your ZFS volume (Volume1 in my case) and then select ‘Create Dataset’.
FreeNAS Create ZFS Dataset for TimeMachine

A ‘Create ZFS Dataset’ dialog box should pop up, prompting you to create the new ZFS dataset.
FreeNAS Create ZFS Dataset Dialog
Ensure the wizard is in ‘Advanced Mode’ and then set the config as follows:

  • Set ‘Dataset Name’ to ‘Time-Machine’
  • Set ‘Quota for this dataset’ to ‘1000 GiB’

In the section option we are specifying a quota for the dataset, effectively settings the size of available disk space for our Time Machine backups. Change the value if 1000 GiB is not suitable for your setup.
Leave everything else as default and click ‘Add Dataset’. Our newly created ‘Time-Machine’ dataset should be visible under the ‘Volumes’ section now.

Now we need to configure the permissions for our ‘Time-Machine’ dataset, so that our ‘time-machine’ group has read/write access.
Select the dataset (Time-Machine) and then select ‘Change Permissions’.
FreeNAS Change ZFS Permissions

A ‘Change Permissions’ dialog box should pop up, prompting you to edit the ZFS dataset.
FreeNAS Change ZFS Permissions Dialog
Set the config as follows:

  • Set ‘Owner (group)’ to ‘time-machine’
  • Set ‘Mode’ checkboxes to the same as mine in the screenshot

Click ‘By setting the group owner to the ‘time-machine’ group, we are granting any users of that group read/write/execute permissions.

Create Time-Machine Users

Now it’s time to create a separate user to represent each computer that will use the FreeNAS server for Time Machine backups.
Under the ‘Account’ menu item, expand the ‘Users’ item, then select ‘Add User’.
FreeNAS Create User Dialog
Set the config as follows, but change the relevant information related to your setup:

  • Leave ‘User ID’ to whatever it is by default
  • Set ‘Username’ to ‘dans-macbook-pro’
  • Ensure ‘Create a new primary group’ is deselected
  • Set ‘Primary Group’ to ‘time-machine’
  • Set ‘Full Name’ to ‘Dan’s MacBook Pro’
  • Set ‘Password’ to something strong (mix of; uppercase, lowercase, numbers, 16 chars long)

Leave everything else as default and click OK. Our newly created ‘dans-macbook-pro’ should be visible under the ‘Users’ section now.

Create Time-Machine AFP Share

The last step on the FreeNAS server is to create the AFP Share that will broadcast the storage on the local network.
Under the ‘Sharing’ tab select the ‘Apple (AFP)’ menu item, and then select ‘Add Apple (AFP) Share’.
FreeNAS Create AFP Share Dialog
Ensure the wizard is in ‘Advanced Mode’ and then set the config as follows:

  • Set ‘Name’ to ‘Time Machine’
  • Set ‘Path’ to your ZFS dataset path
  • Set ‘Allow List’ to ‘@time-machine’
  • Ensure the ‘Time Machine’ box is checked
  • Ensure the ‘Default file permission’ is set to the same as the screenshot
  • Ensure the ‘Default directory permission’ is set to the same as the screenshot

Add Time Machine Backup to OS X

Finally the last step is to configure Time Machine itself to backup to the newly created share.
In OS X, select ‘Time Machine’ from within ‘System Preferences’, and then click the ‘Select Disk’ button.
Add FreeNAS to Time Machine in OS X Dialog
All being well your FreeNAS AFP share should be listed. If you select to use the disk for Time Machine you will be prompted to enter the username and password for the FreeNAS user we created previously. That’s the last step, Time Machine should begin backing up shortly after adding the disk. I recommend that the first backup be completed over Ethernet instead of wireless as the initial backup can take considerable time.

I have used this setup for a couple of years now backing up 4/5 Mac’s with no real issues. Any problems I have ran into have most revolved around sudden shutdowns of the FreeNAS server midway through Time Machine backing up due to power cuts/loss. My Storage is setup using ZFS in striped mirrored mode, meaning I get the best of both for speed and disk redundancy.

Please let me know if you found this guide useful, or spot any mistakes above.