Building a Squid HTTP(S) proxy to route traffic through a VPN from inside a FreeBSD jail

In this post I’m going to cover how to build a Squid proxy server to route traffic out securely through your VPN using the openVPN client. Best of all we’re going to run the whole thing inside a FreeBSD jail using the VIMAGE virtualised networking stack. I’m using CBSD instead of ezjail for this post – more on that to come. Anyway, let’s get started:

Prep

First we need to compile our kernel with support for VIMAGE – FreeBSD’s virtulized network stack:

# cd /usr/src/sys/amd64/conf/

Now your going to want to edit your kernel config, probably called GENERIC. Add these two devices:

...
device          epair
device          if_bridge

and enable VIMAGE:

...
options         VIMAGE

Now navigate back to the sources dir, build and then install your new kernel:

# cd /usr/src
# make buildkernel KERNCONF=GENERIC
# make installkernel KERNCONF=GENERIC

 

Reboot into your new kernel:

# shutdown -r now

As we’re going to use the virtualised network stack, don’t worry about creating a loopback device or IP alias, all we need is a devfs rule to allow our jail access to tunnel devices, net devices and the Berkley packet filter. Create a new section in your CBSD devfs.rules:

 

[devfsrules_jail_with_bpf=7]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'bpf*' unhide
add path 'net*' unhide
add path 'tun*' unhide
Building the jail with cbsd

Now use the tui to create the jail:

# cbsd jconstruct-tui

Most of the options are pretty self explanatory so I’m not going to run through it all. Make sure you set these two as a minimum:

devfs_ruleset    6
vnet             [X]

That’s it – start the jail up:

# cbsd jstart jailname
Inside the jail

Do your usual prep for /­­etc/resolv.conf and /­etc/hosts, then add the IP address you created earlier to your new virtual interface in /­etc/rc.conf:

ifconfig_eth0="inet 192.168.0.107 netmask 255.255.255.0"
defaultrouter="192.168.0.1"

next up install openvpn:

# cd /usr/ports/*/openvpn
# make install

enable openvpn and set the interface in your /­etc/rc.conf:

# echo 'openvpn_if="tun"' >> /­etc/rc.conf
# echo 'openvpn_enable="YES"' >> /­etc/rc.conf

add your tunnel:

# kldload if_tun
OpenVPN

Now we can start configuring openvpn, grab your VPN providers cert – I use VyprVPN by Golden Frog:

# cd /usr/local/­etc/openvpn
# fetch http://www.goldenfrog.com/downloads/ca.vyprvpn.com.crt

Now for the openvpn config, here’s mine, make sure you call it openvpn.conf:

port 1194
client
dev tunN
proto udp
remote uk1.vyprvpn.com
resolv-retry infinite
nobind
persist-key
persist-tun
persist-remote-ip
ca ca.vyprvpn.com.crt
tls-remote uk1.vyprvpn.com
auth-user-pass vyprvpn.pas
comp-lzo
verb 4
keepalive 10 60
route-noexec
script-security 2
up /usr/local/­etc/openvpn/set_routes.sh

You will need to tweak these to fit – make sure you create a file with your username and password in this dir, I’ve called mine vyprvpn.pas.

Now you’ll note for the last 3 options I have disabled creating the routes, and enabled a script at openvpn startup, I had problems getting openvpn to create the routes for me so I ended up writing a shell script to read /var/log/messages and pull out the route info and set it after openvpn has connected:

option1=$(tail -r /var/log/messages | grep "Peer Connection Initiated with" | head -n 1 | grep -o "[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*")
option2=$(tail -r /var/log/messages | grep "PUSH: Received control message:" | head -n 1 | grep -o "route-gateway [0-9]*\.[0-9]*\.[0-9]*\.[0-9]*" | awk '{print $2}')
/sbin/route add -net $option1 192.168.0.1 255.255.255.255
/sbin/route add -net 0.0.0.0 $option2 128.0.0.0
/sbin/route add -net 128.0.0.0 $option2 128.0.0.0

I’m going to re-write this when I get time, as it was just a quick-fix and the info to set the routes actually exists in some variables set by openvpn.

Now, you should be connected! As I am using VyprVPN I’ll use their page to check my IP address:

# fetch --no-verify-peer -o - https://www.goldenfrog.com/whatismyipaddress | grep -i "public ip" | grep -o "[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*"
Squid

Install it – review all the options carefully when you compile:

# cd /usr/ports/*/squid
# make install

Here is my config, I don’t have any interest in caching anything for now, I just want to route all my http traffic through my VPN, but in a later post we’ll be setting up caching for the FreeBSD and various linux sites. The config is located in /usr/local/­etc/squid/squid.conf:

acl local_net src 192.168.0.0/24
http_access allow local_net
http_access allow localhost
cache_access_log /var/log/squid/access.log
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log
http_port 8081
visible_hostname Hermes.core.net
cache_effective_user root
cache_effective_group wheel
cache_dir null /tmp
cache deny all
Sharing the config

You can have clients on your network automatically pick up your proxy server by using a pac file. You’ll need to host it on a web server. Set it using DHCP option 252:

dhcp-option=252,http://artemis.core.net/proxy.pac

The contents of my proxy.pac are:

function FindProxyForURL(url, host)
{
if (isInNet(host, "192.168.0.0", "255.255.0.0")) {
     return "DIRECT";
  } else {
     if (shExpMatch(url, "http:*"))
        return "PROXY Hermes:8081" ;
     if (shExpMatch(url, "https:*"))
        return "PROXY Hermes:8081" ;
     return "DIRECT";
  }
}

You should have a proxy server set up, being automatically configured on your clients by your DHCP server and routing all of your traffic through your VPN.

If you have any issues grab me on the FreeBSD forums – username tabs.

Setting up a DNS and DHCP server in a FreeBSD jail

The software

BIND for DNS – the most widely used DNS software on the internet, developed at Berkeley.

ICS-DHCP for DHCP – A DHCP server implementation by the Internet Systems Consortium.

The Plan

Configure two highly available BIND DNS servers in jails running on separate physical hosts and two highly available DHCP servers that will dynamically update our DNS servers. I’m going to split this over two posts, in this one we’re going to get our DNS and DHCP servers working inside our jail.

THE JAILS

For each jail you want to use (i’m going to use 2, each with a DNS and DHCP server) you’re going to need to do some prep.

Create a loopback interface:

# service netif cloneup lo1

Add the loopback interface to your rc.conf:

# echo 'cloned_interfaces="${cloned_interfaces} lo1"' >> /­etc/rc.conf

Now create a IP alias to use:

# ifconfig bge1 alias 192.168.0.111 netmask 255.255.255.255 broadcast 192.168.0.255

Add your IP alias to your rc.conf:

# echo 'ifconfig_bge1_alias="inet 192.168.0.111 netmask 255.255.255.255 broadcast 192.168.0.255"' >> /­etc/rc.conf

Next up let’s create our jail using ezjail, specifying the IP alias and loopback device we created earlier:

# ezjail-admin create Ares 'bge1|192.168.0.111,lo1|127.0.2.1'

Don’t start it up just yet! We have more config to do first. Copy your resolv.conf into your jail so it can use your current DNS server to resolve the BIND and DHCPD software downloads:

# cp /­etc/resolv.conf /usr/jails/Ares/­etc/

Now let’s edit our jails hosts file to reflect our different loopback address – keep your localhost entry but add the FQDN name in:

# vi /usr/jails/Ares/­etc/hosts
...
::1 localhost Ares.core.net
127.0.2.1 localhost Ares.core.net

Next up we need to allow access to the BPF (Berkeley packet filter device) this allows the DHCP server to avoid copying uninteresting packets from the kernel to the software running in user mode. Let’s create a devfs rule:

# vi /­etc/devfs.rules
...
[devfsrules_jail_with_bpf=6]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'bpf*' unhide

Now we can add the rule into our jails config file:

# echo 'export jail_Ares_devfs_ruleset="6"' >> /usr/local/­etc/ezjail/Ares

Next up we’re going to need to allow our jail access to raw sockets for the DHCP server to capture broadcast requests. Add the rule to your jail config file:

# echo 'export jail_Ares_parameters="allow.raw_sockets allow.sysvipc' >> /usr/local/­etc/ezjail/Ares

Now bring up the loopback device you created earlier:

# service netif cloneup lo1

Start up your Jail and log in:

# ezjail-admin start Ares
# ezjail-admin console Ares
BIND

Now you’re in your prepped up jail, install BIND (remove -DBATCH if you want to set any specific compile options):

# cd /usr/ports/dns/bind99/
# make install -DBATCH

Now let’s edit the config file at /usr/local/­etc/namedb/named.conf:

Set a ACL right at the top of the config file so we only accept requests from hosts on our subnet:

acl "trusted" {
     192.168.0.0/24;
     localhost;
     localnets;
};

Next section to edit is the forwarders, I just go with Google’s DNS servers, make sure you limit these using the ACL we set up earlier:

forwarders {
       8.8.8.8;
       8.8.4.4;
};
allow-query       { any; };
allow-recursion   { trusted; };
allow-query-cache { trusted; };

Thats it for a very basic usable config – enable BIND in the rc.conf file:

# echo 'named_enable="YES"'>> /­etc/rc.conf

Start BIND and test it on your favourite web server (set the IP to your new DNS server):

# service named start
# dig @192.168.0.111 guytabrar.co.uk

The line you are looking for is:

;; Got answer:
 ICS-DHCP

So you have a working DNS server, now we’re going to get a working DHCP server and then next post we’ll make it play together and get it failing over. Let’s get compiling:

# cd /usr/ports/net/isc-dhcp43-server
# make install -DBATCH

Luckily there’s not much config needed for a simple set up, the file is in /usr/local/­etc/dhcpd.conf here is mine, I think it’s all pretty self-explanatory:

default-lease-time 600;
max-lease-time 7200;
# Enable this only option only if it's your only DHCP server:
authoritative;
option subnet-mask 255.255.255.0;
option broadcast-address 192.168.0.110;
option routers 192.168.0.1;
option domain-name-servers 192.168.0.110, 192.168.0.111;
option domain-name "core.net";
subnet 192.168.0.0 netmask 255.255.255.0 {
range 192.168.0.2 192.168.0.99;
}

 

Now let’s enable DHCPD and some options in our /­etc/rc.conf file:

...
dhcpd_enable="YES"
dhcpd_ifaces="bge1"
dhcpd_withumask="022"
dhcpd_flags="-q"

Now fire it up with:

# service ics-dhcpd start

Now we need to try it out, before you do let’s get tcpdump running:

# tcpdump -envvvi bge1 port 67 or port 68

That should show you all the DHCP traffic on the network, at this point I grabbed my iPhone and renewed the DHCP lease, you can should see the request and the response in your tcpdump. Here’s my iPhone asking for a DHCP lease:

20:57:17.896014 9c:f3:87:37:ec:bd > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x0, ttl 255, id 3969, offset 0, flags [none], proto UDP (17), length 328)
0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 9c:f3:87:37:ec:bd, length 300, xid 0x564e0c0e, Flags [none] (0x0000)
Client-Ethernet-Address 9c:f3:87:37:ec:bd
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Request
Parameter-Request Option 55, length 6:
Subnet-Mask, Default-Gateway, Domain-Name-Server, Domain-Name
Option 119, Option 252
MSZ Option 57, length 2: 1500
Client-ID Option 61, length 7: ether 9c:f3:87:37:ec:bd
Requested-IP Option 50, length 4: 192.168.0.3
Lease-Time Option 51, length 4: 7776000
Hostname Option 12, length 15: "GuyTabrrsiPhone"
END Option 255, length 0
PAD Option 0, length 0, occurs 6

Here’s my DHCP server responding:

20:57:17.896215 3c:07:54:03:57:f0 > 9c:f3:87:37:ec:bd, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
192.168.0.110.67 > 192.168.0.3.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x564e0c0e, Flags [none] (0x0000)
Your-IP 192.168.0.3
Client-Ethernet-Address 9c:f3:87:37:ec:bd
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: ACK
Server-ID Option 54, length 4: 192.168.0.110
Lease-Time Option 51, length 4: 6688
Subnet-Mask Option 1, length 4: 255.255.255.0
Default-Gateway Option 3, length 4: 192.168.0.1
Domain-Name-Server Option 6, length 8: 192.168.0.110,192.168.0.111
Domain-Name Option 15, length 8: "core.net"
END Option 255, length 0
PAD Option 0, length 0, occurs 12

Getting started with FreeBSD jails

Prep work

So i’ve got my two servers configured, up-to-date, with cowsay installed, so we’re ready to start building some jails.

The first thing I need is a ZFS zpool, for me a single disk will be fine (since we’re going to automate archiving and backups of our jails between the two servers)

Find your disks, everyone has there own way, I like to grep dmesg:

# dmesg | grep -o "ada[0-9]*.*MB "| sort -u
  ada0: 476940MB
  ada1: 476940MB

ada0 is my boot disk, and it’s where my base jail will run, ada1 is my ‘data’ disk where i’m going to store all my jails and VM’s. Seeing as i’m only going to have a small zpool i’m calling mine puddle:

# zpool create puddle /dev/ada1

I’m going to create a ZFS for jails in my zpool:

# zfs create puddle/jails
Ezjail

Now ZFS is done we need to install ezjail, it’s going to make jails.. easy:

# cd /usr/ports/*/ezjail
# make install

Next up we need to edit the ezjail config to leverage ZFS, which is going to make our jails even easier to manage, and opens up options like disk space quotas, compression, deduplication and lots more. Open up the file /usr/local/­­etc/ezjail.conf in vim and navigate to ZFS options. There’s three lines i’m going to change:

ezjail_use_zfs="YES"
ezjail_use_zfs_for_jails="YES"
ezjail_jailzfs="puddle/jails"

This tells ezjail to create each new jail in a seperate ZFS volume in my pool under puddle/jails. We’re going to want to edit our /­etc/rc.conf file now, to enable ZFS and ezjail at system boot.

zfs_enable="YES"
ezjail_enable="YES"
The Base Jail

Now we can get our first jail installed, the first one is going to be what’s called a base jail, it’s going to contain a full FreeBSD userland that all our other jails can use, this means that new jails after this one will only cost us a few mb and will be created in a matter of seconds. Here we go then, install your base jail with the sources and the ports tree:

# ezjail-admin install -sp

Moving forward you can update all your jails in one go (by updating the base) using:

# ezjail-admin update -u

You should also try to keep your base jail ports tree up to date using:

ezjail-admin update -P
The proper Jail

That’s all the base components sorted, let’s create our first proper jail! For my first jail i’m going to build a DNS and DHCP server. First we need to create a alias from one of our network interfaces for the jail to use. I have a gigabit ethernet, a thunderbolt -> gigabit ethernet and a USB -> gigabit ethernet adapter in each of my mac mini’s, for this jail i’m going to use the interface bge1, the IP address 192.168.0.100, the netmask 255.255.255.0 and the detault gateway already set in my rc.conf, you should replace these values with your own:

# ifconfig bge1 alias 192.168.0.100 netmask 255.255.255.0

You should add it to your rc.conf file as well, so it get’s created at boot:

# echo 'ifconfig_bge1_alias0="inet 192.168.0.100 netmask 255.255.255.0"' >> /­etc/rc.conf

Now i’m going to create my jail using the IP address I set earlier and i’m giving it the name Iris.core.net (I’m using Greek gods for FreeBSD systems and jails and Norse gods for others :), core.net is my internal domain):

# ezjail-admin create Iris.core.net 192.168.0.100

I haven’t got a DNS server set up yet so best copy my resolv.conf into the jail:

cp /­etc/resolv.conf /puddle/jails/Iris.core.net/­etc

Start up the ezjail service:

# service ezjail start

Confirm your jail is running with the jls command:

# jls

Done! Your first FreeBSD jail is up and running, with the exeption of applying updates it is as far as we are concerned a full and ‘proper’ FreeBSD server. For now it’s got frea roam of the hosts resources, but as we progress we’ll start locking things down to limit CPU memory and disk space, building jails that we can failover between hosts and all that other good stuff.

Open up a root console to your first jail and enjoy:

# ezjail-admin console Iris.core.net

Control your jail with:

# ezjail-admin stop|start|restart Iris.core.net

In the next post we’ll turn this jail into the networks primary DNS and DHCP server.

Some post installation packages

a bETTER sHELL

We need a better shell, zsh compiled with perl compatible regular expressions will do nicely:

# cd /usr/ports/*/zsh
# make install
Cowsay

We need cosway, it’s a highly critical package for display important information to users on the system:

# cd /usr/ports/*cowsay
# make install

Cowsay

Git

I need git too, but it’s going to pull down lots of dependencies, I could (and will) use ports as we allready have, but this is going to require babysitting the install and selecting the compile options for each dependency. It’s really great to have this level of control, but it’s time consuming, and many people just don’t care. You have 3 choices:

Ports install, take the time to chose your compile options for git and it’s dependencies:

# cd /usr/ports/*/git
# make install

Ports install with default compile options, the benefits of having a compiled application without having to check each compile option:

# cd /usr/ports/*/git
make install -DBATCH

Use the pkg installer, your going to (usually) get a slightly older version of the application, this is what you normally get from a Linux distro:

# pkg install git

After all that we better find a use for git, oh-my-zsh is a great project on Robby Russell’s github, it turns an already powerful shell into something exceptional. We can install it with:

# sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"

 

The first hurdle – Installing FreeBSD on your mac

DOWNLOAD AND PREP

Since I am going to be using FreeBSD on a mac i’m going to need the UEFI version, you can find it on the FreeBSD FTP site here. I assume you’re going to use a USB drive and not waste a perfectly good DVD, so download the memstick image. While that’s dowloading lets have a look at our macs disks, fire up a terminal and:

# sudo su
# diskutil list

Now insert your USB drive and run diskutil again, the /dev/diskNN that just appeared *should* be your USB drive. Let’s unmount the filesystem on it so we can write to the device with dd:

# diskutil unmountDisk /dev/diskxx

A few notes while we wait for the download:

– I couldn’t get a USB3 drive to work on my MacMini, it kept crashed with the error ufs:/dev/ufs/FreeBSD_Install failed with error 19 FreeBSD doesn’t like the drivers, you could try and drop into a shell and load the xhci driver (if it’s not that driver you could try and find it with pciconf -lv).

– If the installer fails part way through and you get stuck in a loop with the error FreeBSD EFI boot block Loader path: /boot/loader.efi You need to drop into a shell and blitz the HD, check your installation media, dd and try again.

Once the download is finished we’re ready to copy the FreeBSD image to our USB drive, replace “~/Downloads/FreeBSD-10.1-RELEASE-amd64-uefi-memstick.img” with the path to your downloaded image and diskNN with your disk:

# dd if="~/Doownloads/FreeBSD-10.1-RELEASE-amd64-uefi-memstick.img" of=/dev/diskNN bs=1m

Now wait.. and wait.. and wait.. No it hasn’t hung, don’t kill the process.. Done!

 INSTALL

Now boot your mac with the option key held down, select your USB drive and follow the FreeBSD installer, it’s pretty self explanatory, but:

– You can’t boot to a root ZFS volume using the UEFI bootloader yet, so make sure you use UFS for your root volume and we can use ZFS awesomeness everywhere else.

– Create a user when the BSD installer asks you if you want to! The reason for this is that by default FreeBSD prevents logging into the system by SSH as root (a good thing) so if you don’t have any local access to this server drop into a shell when asked and add your user to the Wheel group so you can SSH as yourself then become root:

# pw user mod someuser -G wheel

Another thing you should do at this point if you’re using a mac is tweak your .vimrc file. It seems the backspace key on Apple keyboards is treated as a delete key, this results in very strange vim behaviour, so for the root and your user’s .vimrc file add this line:

set backspace=2

At this point you should reboot and congratulate yourself on a lovely fresh FreeBSD installation running on your mac :)

UPDATE FreeBSD

Now get those security updates applied!

# su
# freebsd-update fetch install

Make sure you check for kernel changes and restart if necessary:

# shutdown -r now
Sync the ports directory

Now we need to sync the ports directory up with the FreeBSD SVN repo, there’s a great tool to help you do it called svnup, and the ports system is how we will install it, and everything else, because it’s wonderful (plenty more on that to come).

# cd /usr/ports/net/svnup
# make config-recursive install clean

Now we need to pick our closest SVN mirror by uncommenting the relevant line out of the svnup config file:

/usr/local/­etc/svnup.conf

Now let’s sync our ports directory with the latest one from the FreeBSD SVN repo:

# svnup release

That should do the trick for now.

You should have a working and up-to-date FreeBSD system with the latest ports available to you. In the next post we’ll start covering the basics of FreeBSD maintenance and administration, then get some jails running and start building systems.

 

Where to start..

FreeBSD. That’s where.
What are we going to do with it?
  • Install it on a pair of quad core i7 MacMini Servers.
  • Go through some FreeBSD basics.
  • Set up DNS and DHCP servers inside FreeBSD jails, they need to failover too.
  • Set up load balanced Squid proxy servers inside FreeBSD jails, configured to route HTTP/S traffic securely through a VPN.
  • Set up a PF firewall inside a FreeBSD jail (but using a a dedicated NIC).
  • Build a FAMP server inside a FreeBSD jail to host this blog (FreeBSD, Apache, MySQL, PHP)
  • Build our own FreeBSD mirror in a FreeBSD jail.
  • Virtulize lots of linux servers and a OS X server using Bhyve (the BSD hypervisor).
  • Lots of in-depth technical fun :)
But before we get to that let’s address the elephant in the room, why FreeBSD?

I have always been ideologically attracted to Unix variants over Linux, though both are fantastic, besides, Linux is my bread and butter.

Philisophically; Chaos vs Order.

Linux is a kernel, a distro takes that kernel and cherry picks a version of ls from here, a version of cat from there, gzip, python, perl, and so on, it’s a conglomerate of utilities layered on top of the Linux kernel and mashed together to make it all work. FreeBSD is a complete operating system, development is completely centralised, the kernel all the way up to the utilities are controlled by a single democratic entity, it uses BSD ls, BSD libc, everything is done with the bigger picture in mind, everything is built for BSD, to fit with BSD.

From my point of view there isn’t much in it in terms of performance and security and so on, FreeBSD has a better IP stack but Linux has KVM, at the end of the day FreeBSD does most of what Linux does and vice versa.. however there are some technologies that make FreeBSD stand out for me; ZFS, Dtrace, Jails, the Ports System (compile everything!) and PF. We’ll cover all of those in time.

FreeBSD and Linux share many benefits, I think this picture summarises one of my favourites nicely, complete control and observability of the entire system:

FreeBSD Tools

(Credit to the legendary Brendan Gregg)

So when a Windows admin tells says they don’t know why their system is running slow, or how they can improve their performance, I look this diagram and smile smugly. Every tool you need to know for observing your system is on this diagram.

Anyway – enough philosophy, next post we start getting technical, here is how the home network will look when it’s done.

Core