Illumos, Zones and DNS

Why illumos

My offsite Backup-Server running Fedora CoreOS on a Btrfs RAID-5 died recently due to Btrfs Shooting its own foot of, since a single disk-failure with Btrfs RAID-5 obviously means that the whole array is unrecoverable…

So in what was i thought was better than Btrfs RAID-5 i decided to install OmniOS which is a illumos distribution stemming from the OpenSolaris days at SUN Microsystems.

I always thought of Solaris as this elusive highly advanced UNIX since it had so many really advanced features way before any other *NIX ZFS, DTrace, Crossbow and Zones so with this impressive repertoir i decided to give OmniOS a shot.

Getting in the ZONE

Since Zones are such a core feature of any illumos distribution, i simply had to use them. And OmniOS also has quite a number of different zone ‘brands’ which allow a zone to run foreign Software like Solaris 10, emulated aarch64 illumos or even Linux userland within a LX-branded Zone!

My Networking Problem

Now to my problem with zones, they (as far as i was able to find out) prefer/require to setup their IP-Addressing manually in advance. Since i was used to Linux and how the libvirt stack does it’s internal NAT-Networking via the default virbr0 bridge i tried to do something equivalent in OmniOS.

Stubbing my Ethernet

After looking in the Oracle Solaris 10 (1) documentation i found something maybe just right etherstub which looks like it functions akin to a empty bridge interface on Linux, provides a detached L2 Interface to which one can attach VNICs to.

So this is how i set up my virbr0 like interface in OmniOS:

1dladm create-etherstub virbr0
2dladm create-vnic -l virbr0 global0
3dladm create-vnic -l virbr0 zone1
4ipadm create-if global0
5ipadm create-addr -T static -a 192.168.0.1 global0/v4

The above creates a etherstub with a couple VNICs one for the “Global Zone” aka the Host and for the “Guest” Zone, assigns a static IP to the global0 “Interface” but there is still further setup to do…

Since this is the land of the SUN managing Link and IP configuration is done quite different to the way i know of even from the BSDs like FreeBSD and OpenBSD. The dladm command manages the “Ethernet” Layer 2 Interfaces eg. DataLinkADM and ipadm manages the IP addressing be it static or DHCP.

Networking the Masquerade ipfilter Edition

Now we have a etherstub “bridge” a VNIC for the Host with a static IP but we still need to setup the Firewall to allow the in and out traffic from the Zones.

For that we have to tell the ipf that it should allow inbound traffic on global0 with this minimal and most permissive Firewall config for the /etc/ipf/ipf.conf file:

1### no restrictions on loopback interface
2pass in quick on lo0 all
3pass out quick on lo0 all
4### allow traffic on our etherstub interface
5pass in on global0 from any to any
6pass out on global0 from any to any
7### allow any other traffic like default without IPF
8pass in all
9pass out all

To configure NAT with the Solaris ipfilter we add to the /etc/ipf/ipnat.conf file:

1### Masquerade the outbound traffic on bge1 IP
2map bge1 192.168.0.0/24 -> 0/32 portmap tcp/udp auto
3map bge1 192.168.0.0/24 -> 0/32

Where bge1 is network connected “upstream” Interface.

With the Firewall configured we just need to actually start the ipfilter SMF service:

1svcadm enable ipv4-forwarding
2svcadm enable ipv6-forwarding
3svcadm enable network/ipfilter
4ipf -Fa -f /etc/ipf/ipf.conf

The last ipf -Fa -f command flushes and reloads the filter. And yes Solaris had a system service manager before systemd even was a thing with its SMF.

Creating the Zone

With the Networking stuff prepared now its time to get in the Zone:

 1zonecfg -z zone1
 2zonecfg:zone1> create
 3zonecfg:zone1> set zonepath=/zones/zone1
 4zonecfg:zone1> brand=sparse
 5zonecfg:zone1> autoboot=false
 6zonecfg:zone1> ip-type=exclusive
 7zonecfg:zone1> add net
 8zonecfg:zone1:net> set physical=zone1
 9zonecfg:zone1:net> set allowed-address=192.168.0.2
10zonecfg:zone1:net> set defrouter=192.168.0.1
11zonecfg:zone1:net> end
12zonecfg:zone1> verify
13zonecfg:zone1> commit
14zonecfg:zone1> exit

With the Zone defined it is now ready to install and start:

1zoneadm -z zone1 install
2zoneadm -z zone1 boot

Enter the Zone

Now to configure the Zone itself we have to enter the zone:

1zlogin zone1

Running ifconfig confirms that the Zone has the configured IP address:

1lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
2	inet 127.0.0.1 netmask ff000000 
3zone1: flags=40001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,L3PROTECT> mtu 9000 index 2
4	inet 192.168.0.2 netmask ffffff00 broadcast 192.168.0.255
5	ether 2:8:20:7c:9e:37 
6lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
7	inet6 ::1/128

Solaris ifconfig has separate entries v4 and v6 addresses which show the Address.

So now the Zone has a IP and can ping the internet so everything works now, right? haha no:

1% ping 8.8.8.8
28.8.8.8 is alive

Ping works but SunOS/Solaris dates back to a time before DNS was even a thing so its not really in the default configuration of Zones…

For that we to configure the Zone to use DNS via NSS:

1% cp /etc/nsswitch.{dns,conf}
2% echo nameserver 1.1.1.1 > /etc/resolv.conf
3% svcadm enable svc:/network/dns/client:default
4% ping google.de
5google.de is alive

Now the Zone can talk to the Internet.

Notes & Resources

• [1] Solaris 11 was released after moving Solaris back to a closed Software license and has significant differences to the last OpenSolaris release of which illumos was forked of from.
• A extremely helpful cheat sheet if found later: https://github.com/jpdasma/unix-cheat-sheet

• Illumos
• Zones