Using Buildroot for IoT Projects

Buildroot RPi Yocto Logos

Early on in my effort to build a digital dash for my car I quickly realised I needed to run a stripped down version of Linux if I was to have any chance of getting the dash to boot up quickly on the Raspberry Pi. A dash that kept you waiting for tens of seconds or at worst minutes to turn on wouldn’t exactly be the most practical of solutions (especially if you want to avoid speeding fines hehe).

So I was faced with the situation of taking one of the existing distributions available for the RPi and modding it for my own use, or having a crack at making something myself that would do the job. Initially I did start tinkering around with seeing how fast I could get stripped down versions of Raspbian and Arch booting up, which although was interesting did feel as though I was spending a lot of time and effort digging down in the dirt and undoing a lot of work someone else had already spent a lot of time doing. Therefor instead of trying to tear something down to suit my needs, I decided to focus my attention to incrementally building something useful from the ground up. I figured doing it this was would be a great source of learning as I would no doubt begin to run into obstacles other people have already created solutions for, meaning I would appreciate the open source communities hard work even more so.

After a bit of searching I found a couple of promising looking projects to base my dash on (I really didn’t feel like totally reinventing the wheel). Both Buildroot and the Yocto Project share the common goal of creating a complete, easy to customise, embedded Linux system. Both compile Linux from source using cross-compilation (which should cover most performance requirements), are well documented, actively maintained and already used within the embedded systems industry. In fact I had already been exposed to the Yocto project before without even realising when I attended an Intel Edison workshop at the London FabLab, as the Linux image supplied by Intel for use on the Edison is a Yocto build.

Whilst I already had some experience with Yocto I decided to kickstart my digital dash using using Buildroot. Namely in the effort of simplicity as to get started with Buildroot is just so easy. The documentation available is awesome and provides a steady learning curve. I also found a pretty good short book on using Buildroot with the RPi which was neat primer. After a bit of tinkering I had the environment setup in a Ubuntu VM as well as a working filesystem that booted in seconds on my RPi2. I quickly appreciated the number of tools available out of the box in major Linux distributions as theres really not much to do in a basic buildroot build other than logging in and a few system utils to work with.

It was not before long I had my buildroot environment stored under Git and had automated a few of the basic build commands, such as copying over custom apps after a successful build and flashing the filesystem to SD card. All in all it didn’t take a ton of effort to get a simple enough system working that booted up fast, was capable of connecting to the Internet and launched my prototype app on boot. My prototype app is built using QT with PyQT bindings, both of which buildroot supports although not the latest versions.
I would happily recommend Buildroot to anyone who wants to learn more about Linux targeted for embedded devices and IoT projects, it really is a pleasure to work with and the community behind it is very resourceful and helpful.

Looking towards the future for my digital dash there may come a time where I may have to look to migrating towards other projects like Yocto. Buildroot is great and pretty much does everything I need at this point in time, however it does have what I consider to be quite a big no-go – especially in the IoT community. There is no package management system and no binary packages, meaning updates to the system are not possible via packages. If a future release of the Linux kernel fixes a security vulnerability, a full system update would be required which would involve physically removing the SD card and updating the software manually. Best practices for building IoT devices include having an inbuilt mechanism from the word go, allowing for critical updates to be issued that threaten to compromise the device without having to rely on the end user to manually update their device.

All of this is subject to change I guess, Buildroot could one day change how they do things. For now I’m sticking with them as I’m the only one working on the dash project, but being the responsible security guy I am if I open up the project for others to tinker with I would want to switch to another project that supported some form of inbuilt updating.

Anyway it’s about time I get back to finishing rebuilding the engine for the Mk1 so I can actually test the dash in-car for once instead of on my desk!

Spacewalk 2.4 installation on CentOS 7

Spacewalk logo

Spacewalk (not the EVA kind) is an open source (GPLv2) Linux systems management solution and is the upstream project for Red Hat Satellite. The main feature I wished to experiment with was the ability to manage multiple systems packages (upgrades) concurrently. The project also has several other main features, including;

  • Inventory your systems (hardware and software information)
  • Install and update software on your systems
  • Collect and distribute your custom software packages into manageable groups
  • Provision (kickstart) your systems
  • Manage and deploy configuration files to your systems
  • Provision and start/stop/configure virtual guests
  • Distribute content across multiple geographical sites in an efficient manner

Prerequisites

Taken directly from the official Spacewalk docs:

  • Open outbound firewall ports 80, 443
  • Inbound open ports 80, 443, 5222 (covered below later on)
  • Storage for database: 250 KiB per client system + 500 KiB per channel + 230 KiB per package in channel (i.e. 1.1GiB for channel with 5000 packages)
  • Storage for packages (default /var/satellite): Red Hat recommend 6GB per channel for their channels, I used 20GB for storage, as 10GB wasn’t enough
  • 2GB RAM minimum, 4GB recommended
  • Fully up-to-date underlying operating system running a vanilla installation (no user customisation performed yet, fresh system install)

Setup Repos

There are 3 main repos to setup.

Spacewalk repo

rpm -Uvh http://yum.spacewalkproject.org/2.4/RHEL/7/x86_64/spacewalk-repo-2.4-3.el7.noarch.rpm

JPackage

cat > /etc/yum.repos.d/jpackage-generic.repo << EOF
[jpackage-generic]
name=JPackage generic
#baseurl=http://mirrors.dotsrc.org/pub/jpackage/5.0/generic/free/
mirrorlist=http://www.jpackage.org/mirrorlist.php?dist=generic&type=free&release=5.0
enabled=1
gpgcheck=1
gpgkey=http://www.jpackage.org/jpackage.asc
EOF

EPEL

Spacewalk requires Java Virtual Machine 1.6.0 or greater which is available in the EPEL repo.

rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Installation

Time to get on with the install 🙂

Database backend

Spacewalk uses database server to store its primary data. It supports either PostgreSQL (version 8.4 and higher) or Oracle RDBMS. Lets install PostgreSQL and let Spacewalk configure the database for us.

yum install spacewalk-setup-postgresql

Install Spacewalk

Install the Spacewalk package with support for PostgreSQL.

yum install spacewalk-postgresql 

Configure the firewall

Enable inbound firewall (firewalld) ports 80 (http) and 443 (https).

firewall-cmd --add-service=http
firewall-cmd --add-service=https

Configure Spacewalk

Last step.

spacewalk-setup --disconnected

Afterwards we should be able to visit: https://hostname.yourdomain.com to create the Spacewalk admin account and finish the installation.

Install and configure OpenVPN on CentOS 6

Overview

This article will provide a quick guide to installing and hosting your own OpenVPN server on CentOS 6.

Prerequisites

First order of business is to ensure you have the Extra Packages for Enterprise Linux (EPEL) repository installed. This a Fedora Project special interest group (SIG) that maintains additional packages for RedHat based Enterprise Linux distributions. It will enable the install of the OpenVPN package.

rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm

OpenVPN installation and configuration

Install the OpenVPN package from the newly added EPEL repository. OpenVPN 2.3.7 is the current version available at the time of writing.

yum install openvpn

Some guides will recommend copying the sample OpenVPN configuration, but I prefer to create one from scratch as it creates a cleaner config file that is easy to read and understand. You can if you wish still copy over the sample and edit as necessary to continue following the guide. Skip the command below if you wish to create one from scratch.

cp /usr/share/doc/openvpn-*/sample/sample-config-files/server.conf /etc/openvpn/server.conf

Create/edit the newly copied server config

nano /etc/openvpn/server.conf

Insert the following to the config. You can omit the comments indicated by ‘#’ if you wish.

# Enable TLS and assume server role during TLS handshake.
tls-server

# Use UDP as the main protocol
proto	udp

# Default OpenVPN port is 1194
port	1194

# Configure TAP interface, this allows for full-frame Ethernet packets to be sent. Useful for AFP required for remote OS X TimeMachine backups			
dev	tap

# IP Address allocation to clients for specified network/netmask. 
# The server will take the '.1' address (192.168.100.1). 
server	192.168.100.0 255.255.255.0

# Absolute paths for server cert's and keys (created later on).
ca 		/etc/openvpn/ca.crt
cert		/etc/openvpn/server.crt
key		/etc/openvpn/server.key
dh		/etc/openvpn/dh2048.pem
tls-auth	/etc/openvpn/ta.key 0

# This is the network/subnet of your physical LAN the OpenVPN server will reside.
# Without this clients will be unable to ping other computers located on the same network as the server. 
push "route 192.168.0.0 255.255.255.0"
topology subnet

# DNS servers to be pushed to clients
push "dhcp-option DNS 8.8.8.8"
push "dhcp-option DNS 8.8.8.4"

# Drop privileges after initialisation to help improve security. 
user    nobody
group   nobody
persist-key
persist-tun

# Used by the client to detect server timeout.
# Ping server every 10 seconds, assume timeout after 60. 
keepalive 10 60
ping-timer-rem

# Enable compression
comp-lzo adaptive

# Run the process as a daemon
daemon

# Set logging verbosity, specify absolute paths for log files.  
verb 4
log-append	/var/log/openvpn.log
status		/var/log/openvpn.status

Certificate and Key generation

Now the OpenVPN configuration is complete, we need to generate some certificates and keys using a package Easy-RSA. Time to install more dependencies.

yum install easy-rsa

With the dependancy installed, it’s time to copy some required files into place.

mkdir -m 700 -p /etc/openvpn/easy-rsa/keys
cp -rp /usr/share/easy-rsa/2.0/* /etc/openvpn/easy-rsa

Now we edit the ‘vars’ file which contains all the necessary values for the Easy-RSA scripts to use.

nano /etc/openvpn/easy-rsa/vars

Change the key variables listed below contained in the ‘vars’ file to reflect your information.
You can omit the comments indicated by ‘#’ if you wish.

export KEY_SIZE=2048	# Can be increased to 4096 if desired
export CA_EXPIRE=3650	# 10 years CA expiration 
export KEY_EXPIRE=1095	# 3 year Certificates expiration
export KEY_COUNTRY="GB"
export KEY_PROVINCE="MyCounty"
export KEY_CITY="MyCity"
export KEY_ORG="MyOrg"
export KEY_EMAIL="email@example.com"
export KEY_OU="MyOrgUnit"
export KEY_NAME="MyServer"
export KEY_CN="server.example.com" # FQDN for server

We’ll now load the variables into the session and make sure the keys/ folder is empty using the clean-all script.

cd /etc/openvpn/easy-rsa
source ./vars
./clean-all

Time to build the CA private key and certificate with a password. Press enter when prompted and use a strong password.

./build-ca --pass

Now we build the server certificate. When prompted to enter the ca.key password, enter the password you used during CA creation in the previous step.

./build-key-server server

We generate our Diffie-Hellman key exchange file for the server. This can take a long time to generate depending on your computer.

./build-dh

The last step is to generate the tls-auth file

openvpn --genkey --secret keys/ta.key

It’s time to generate some client certificates.
This step can be repeated as many times as necessary to generate a unique certificate for each client. Replace ‘client’ with a unique name for each client.

./build-key-pass client

Now we copy all of the generated files into the OpenVPN conf directory.

cd /etc/openvpn/easy-rsa/keys
cp ca.crt server.crt server.key dh2048.pem ta.key /etc/openvpn

Routing config

Packet forwarding needs to be enabled on the server, so first we open the config.

nano /etc/sysctl.conf

Then edit ‘ip_forward’ to 1 if it’s not already set.

net.ipv4.ip_forward = 1

Create an iptables rule that will enable the server to forward packets to the rest of the network, received from VPN clients.

iptables -t nat -I POSTROUTING -s 192.168.100.0/24 -o eth0 -j MASQUERADE

Save the firewall rules, enable the service to start automatically on boot and then restart the system.

service iptables save
chkconfig openvpn on
reboot

Client configuration

The server configuration part of this guide is over, now lets move onto the client configuration.

First we need to create the client config. Similar to the server config it’s easier to create a new client config from scratch.
Create a new file on your client called client.conf

nano client.conf

Insert the following client config below. Replace example.com with the hostname/IP address of your OpenVPN server.

client
proto	udp
remote	example.com
port	1194
dev	tap
nobind

ca		ca.crt
cert		client.crt
key		client.key
tls-auth	ta.key 1

ns-cert-type 	 server

comp-lzo adaptive

Next is to copy over the required certificates and keys from the server. Use some form of transfer; USB drive, SCP, SFTP and move the ca.crt, client.crt, client.key and ta.key to the same directory as the client config.

Mac OS X OpenVPN Client

Now we are ready to load the config into a OpenVPN client and test our setup.
For OS X, Tunnelblink is the best OpenVPN client to use.
Opening the client.conf with Tunnelblink should kickstart the config install, which will load the config, keys and certificates into a Tunnelblink profile. Once complete you should be able to successfully connect to your OpenVPN server.

To test connectivity you should be able to ping the OpenVPN server from the client, as well as Google’s DNS server to confirm external connectivity.

ping 192.168.100.1
ping 8.8.8.8

Auto-starting guest VM’s using Virsh

Running this command for each individual VM, will ensure that they are automatically started and suspended when starting up and shutting down your server.

Enable autostart for VM.

sudo virsh autostart guest-vm-name

Disable autostart for VM.

sudo virsh autostart --disable guest-vm-name

Kind of a nice feature that the VMs are automatically suspended and restored when you need to restart, meaning the VMs will resume to their previous state before you restarted.. as if nothing ever happened.

Preserve original host IP via Apache reverse proxy using mod_extract_forwarded in CentOS

Using Apache as a reverse proxy can cause difficulty when reviewing backend web server logs. The default Apache server configuration will incorrectly log the reverse proxy host as being the origin IP address, instead of the actual IP that sent the request. Luckily there is a pretty simple fix to this using the Apache module mod_extract_forwarded. Using this module is much simpler and easier to install than using the alternative method using the mod_rpaf module.

This guide focuses on Apache installations running on CentOS, however the configuration will be applicable to other Redhat distros too.

First ensure that you have the EPEL repository installed and configured. See here

Install the mod_extract_forwarded Apache module

sudo yum install mod_extract_forwarded

Edit the mod_extract_forwarded.conf, uncomment the MEFaccept line and replace the sample IP addresses with your own reverse proxy IP.

sudo vi /etc/httpd/conf.d/mod_extract_forwarded.conf

Restart Apache to take effect and verify the module is loaded.

sudo service httpd restart
yum list installed | grep forward