Getting started with FreeBSD jails

Prep work

So i’ve got my two servers configured, up-to-date, with cowsay installed, so we’re ready to start building some jails.

The first thing I need is a ZFS zpool, for me a single disk will be fine (since we’re going to automate archiving and backups of our jails between the two servers)

Find your disks, everyone has there own way, I like to grep dmesg:

# dmesg | grep -o "ada[0-9]*.*MB "| sort -u
  ada0: 476940MB
  ada1: 476940MB

ada0 is my boot disk, and it’s where my base jail will run, ada1 is my ‘data’ disk where i’m going to store all my jails and VM’s. Seeing as i’m only going to have a small zpool i’m calling mine puddle:

# zpool create puddle /dev/ada1

I’m going to create a ZFS for jails in my zpool:

# zfs create puddle/jails
Ezjail

Now ZFS is done we need to install ezjail, it’s going to make jails.. easy:

# cd /usr/ports/*/ezjail
# make install

Next up we need to edit the ezjail config to leverage ZFS, which is going to make our jails even easier to manage, and opens up options like disk space quotas, compression, deduplication and lots more. Open up the file /usr/local/­­etc/ezjail.conf in vim and navigate to ZFS options. There’s three lines i’m going to change:

ezjail_use_zfs="YES"
ezjail_use_zfs_for_jails="YES"
ezjail_jailzfs="puddle/jails"

This tells ezjail to create each new jail in a seperate ZFS volume in my pool under puddle/jails. We’re going to want to edit our /­etc/rc.conf file now, to enable ZFS and ezjail at system boot.

zfs_enable="YES"
ezjail_enable="YES"
The Base Jail

Now we can get our first jail installed, the first one is going to be what’s called a base jail, it’s going to contain a full FreeBSD userland that all our other jails can use, this means that new jails after this one will only cost us a few mb and will be created in a matter of seconds. Here we go then, install your base jail with the sources and the ports tree:

# ezjail-admin install -sp

Moving forward you can update all your jails in one go (by updating the base) using:

# ezjail-admin update -u

You should also try to keep your base jail ports tree up to date using:

ezjail-admin update -P
The proper Jail

That’s all the base components sorted, let’s create our first proper jail! For my first jail i’m going to build a DNS and DHCP server. First we need to create a alias from one of our network interfaces for the jail to use. I have a gigabit ethernet, a thunderbolt -> gigabit ethernet and a USB -> gigabit ethernet adapter in each of my mac mini’s, for this jail i’m going to use the interface bge1, the IP address 192.168.0.100, the netmask 255.255.255.0 and the detault gateway already set in my rc.conf, you should replace these values with your own:

# ifconfig bge1 alias 192.168.0.100 netmask 255.255.255.0

You should add it to your rc.conf file as well, so it get’s created at boot:

# echo 'ifconfig_bge1_alias0="inet 192.168.0.100 netmask 255.255.255.0"' >> /­etc/rc.conf

Now i’m going to create my jail using the IP address I set earlier and i’m giving it the name Iris.core.net (I’m using Greek gods for FreeBSD systems and jails and Norse gods for others :), core.net is my internal domain):

# ezjail-admin create Iris.core.net 192.168.0.100

I haven’t got a DNS server set up yet so best copy my resolv.conf into the jail:

cp /­etc/resolv.conf /puddle/jails/Iris.core.net/­etc

Start up the ezjail service:

# service ezjail start

Confirm your jail is running with the jls command:

# jls

Done! Your first FreeBSD jail is up and running, with the exeption of applying updates it is as far as we are concerned a full and ‘proper’ FreeBSD server. For now it’s got frea roam of the hosts resources, but as we progress we’ll start locking things down to limit CPU memory and disk space, building jails that we can failover between hosts and all that other good stuff.

Open up a root console to your first jail and enjoy:

# ezjail-admin console Iris.core.net

Control your jail with:

# ezjail-admin stop|start|restart Iris.core.net

In the next post we’ll turn this jail into the networks primary DNS and DHCP server.

Some post installation packages

a bETTER sHELL

We need a better shell, zsh compiled with perl compatible regular expressions will do nicely:

# cd /usr/ports/*/zsh
# make install
Cowsay

We need cosway, it’s a highly critical package for display important information to users on the system:

# cd /usr/ports/*cowsay
# make install

Cowsay

Git

I need git too, but it’s going to pull down lots of dependencies, I could (and will) use ports as we allready have, but this is going to require babysitting the install and selecting the compile options for each dependency. It’s really great to have this level of control, but it’s time consuming, and many people just don’t care. You have 3 choices:

Ports install, take the time to chose your compile options for git and it’s dependencies:

# cd /usr/ports/*/git
# make install

Ports install with default compile options, the benefits of having a compiled application without having to check each compile option:

# cd /usr/ports/*/git
make install -DBATCH

Use the pkg installer, your going to (usually) get a slightly older version of the application, this is what you normally get from a Linux distro:

# pkg install git

After all that we better find a use for git, oh-my-zsh is a great project on Robby Russell’s github, it turns an already powerful shell into something exceptional. We can install it with:

# sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"

 

The first hurdle – Installing FreeBSD on your mac

DOWNLOAD AND PREP

Since I am going to be using FreeBSD on a mac i’m going to need the UEFI version, you can find it on the FreeBSD FTP site here. I assume you’re going to use a USB drive and not waste a perfectly good DVD, so download the memstick image. While that’s dowloading lets have a look at our macs disks, fire up a terminal and:

# sudo su
# diskutil list

Now insert your USB drive and run diskutil again, the /dev/diskNN that just appeared *should* be your USB drive. Let’s unmount the filesystem on it so we can write to the device with dd:

# diskutil unmountDisk /dev/diskxx

A few notes while we wait for the download:

– I couldn’t get a USB3 drive to work on my MacMini, it kept crashed with the error ufs:/dev/ufs/FreeBSD_Install failed with error 19 FreeBSD doesn’t like the drivers, you could try and drop into a shell and load the xhci driver (if it’s not that driver you could try and find it with pciconf -lv).

– If the installer fails part way through and you get stuck in a loop with the error FreeBSD EFI boot block Loader path: /boot/loader.efi You need to drop into a shell and blitz the HD, check your installation media, dd and try again.

Once the download is finished we’re ready to copy the FreeBSD image to our USB drive, replace “~/Downloads/FreeBSD-10.1-RELEASE-amd64-uefi-memstick.img” with the path to your downloaded image and diskNN with your disk:

# dd if="~/Doownloads/FreeBSD-10.1-RELEASE-amd64-uefi-memstick.img" of=/dev/diskNN bs=1m

Now wait.. and wait.. and wait.. No it hasn’t hung, don’t kill the process.. Done!

 INSTALL

Now boot your mac with the option key held down, select your USB drive and follow the FreeBSD installer, it’s pretty self explanatory, but:

– You can’t boot to a root ZFS volume using the UEFI bootloader yet, so make sure you use UFS for your root volume and we can use ZFS awesomeness everywhere else.

– Create a user when the BSD installer asks you if you want to! The reason for this is that by default FreeBSD prevents logging into the system by SSH as root (a good thing) so if you don’t have any local access to this server drop into a shell when asked and add your user to the Wheel group so you can SSH as yourself then become root:

# pw user mod someuser -G wheel

Another thing you should do at this point if you’re using a mac is tweak your .vimrc file. It seems the backspace key on Apple keyboards is treated as a delete key, this results in very strange vim behaviour, so for the root and your user’s .vimrc file add this line:

set backspace=2

At this point you should reboot and congratulate yourself on a lovely fresh FreeBSD installation running on your mac :)

UPDATE FreeBSD

Now get those security updates applied!

# su
# freebsd-update fetch install

Make sure you check for kernel changes and restart if necessary:

# shutdown -r now
Sync the ports directory

Now we need to sync the ports directory up with the FreeBSD SVN repo, there’s a great tool to help you do it called svnup, and the ports system is how we will install it, and everything else, because it’s wonderful (plenty more on that to come).

# cd /usr/ports/net/svnup
# make config-recursive install clean

Now we need to pick our closest SVN mirror by uncommenting the relevant line out of the svnup config file:

/usr/local/­etc/svnup.conf

Now let’s sync our ports directory with the latest one from the FreeBSD SVN repo:

# svnup release

That should do the trick for now.

You should have a working and up-to-date FreeBSD system with the latest ports available to you. In the next post we’ll start covering the basics of FreeBSD maintenance and administration, then get some jails running and start building systems.

 

Where to start..

FreeBSD. That’s where.
What are we going to do with it?
  • Install it on a pair of quad core i7 MacMini Servers.
  • Go through some FreeBSD basics.
  • Set up DNS and DHCP servers inside FreeBSD jails, they need to failover too.
  • Set up load balanced Squid proxy servers inside FreeBSD jails, configured to route HTTP/S traffic securely through a VPN.
  • Set up a PF firewall inside a FreeBSD jail (but using a a dedicated NIC).
  • Build a FAMP server inside a FreeBSD jail to host this blog (FreeBSD, Apache, MySQL, PHP)
  • Build our own FreeBSD mirror in a FreeBSD jail.
  • Virtulize lots of linux servers and a OS X server using Bhyve (the BSD hypervisor).
  • Lots of in-depth technical fun :)
But before we get to that let’s address the elephant in the room, why FreeBSD?

I have always been ideologically attracted to Unix variants over Linux, though both are fantastic, besides, Linux is my bread and butter.

Philisophically; Chaos vs Order.

Linux is a kernel, a distro takes that kernel and cherry picks a version of ls from here, a version of cat from there, gzip, python, perl, and so on, it’s a conglomerate of utilities layered on top of the Linux kernel and mashed together to make it all work. FreeBSD is a complete operating system, development is completely centralised, the kernel all the way up to the utilities are controlled by a single democratic entity, it uses BSD ls, BSD libc, everything is done with the bigger picture in mind, everything is built for BSD, to fit with BSD.

From my point of view there isn’t much in it in terms of performance and security and so on, FreeBSD has a better IP stack but Linux has KVM, at the end of the day FreeBSD does most of what Linux does and vice versa.. however there are some technologies that make FreeBSD stand out for me; ZFS, Dtrace, Jails, the Ports System (compile everything!) and PF. We’ll cover all of those in time.

FreeBSD and Linux share many benefits, I think this picture summarises one of my favourites nicely, complete control and observability of the entire system:

FreeBSD Tools

(Credit to the legendary Brendan Gregg)

So when a Windows admin tells says they don’t know why their system is running slow, or how they can improve their performance, I look this diagram and smile smugly. Every tool you need to know for observing your system is on this diagram.

Anyway – enough philosophy, next post we start getting technical, here is how the home network will look when it’s done.

Core