So i decided to deploy Opnsense onto my network to add an additional layer off security.

Current network situation

#

So our ISP is Ziggo and our network plan is 1 Gigabit down and 100 Megabit up. Ziggo is one off the providers thats a bit lacking behind, as they are still partially using COAXIAL cables. Not fully, but its a HFC (Hybrid Fiber-Coax) network, which runs fiber cables from the datacenters to the distribution box in the street. From that distribution box, they run oldschool coax cables to the homes. This does work fine, but it makes it so up and down speeds are not symetrica l and also upload speed is limited. We did have another plan before this off a few hundred Megabit down and 90 Megabit up, but they gave us a free upgrade to the 1 Gigabit down and 100 Megabit up plan, which is nice from them. They gave us a new modem and tv box too! The modem is the Sagemcom F3896LG and its powered by a broadcom BCM3390 SOC. The previous modem was an Arris touchstone TG2492LG](https://techinfodepot.shoutwiki.com/wiki/Virgin_Media_Super_Hub_3), branded “connectbox” which also is sold by Virgin media as the VM Superhub3. This one is powered by the very problematic Intel Puma chipset See also this article so i am happy that we got rid off it as we had tons of stability issues which is linked to a silicon defect in these chips. We also got some Wifi Accesspoints as the old ones (Deco M4R) had older security and where not fully working at the full speed and had only WiFi5, so we asked if they could send us some newer WiFi6 ones. That was not an issue and we got 2x the Sagemcom F266V2LG accespoints. These hook up wirelessly to the main router. They where so nice to let us keep the old stuff. We did not have to send the Deco’s back and i could repurpose them later. We also got to keep the old modem and settopbox which I tore down and tried to reverse engineer a bit. Our old tv box was a Samsung SMT-G7401, also labeled “Horizon” and makes use off the “Horizon EOS” platform. The new one is marked “Arris VIP5202” and also sold by KPN and i believe it makes use off the KreaTV platform. It has the same GUI so i don’t think there are much differences in software and both of course run Linux.

The fact we need those AP’s to extend the WiFi to the TV in the living room and upstairs to my setup, means we cant put the main router in bridge-mode as that disables all router features and thus also that proprietary mesh link. So we have to place the Opnsense box further up the network chain. I will describe further in this post where i ended up placing it.

What is Opnsense?

#

So Opnsense is a Opensource Firewall/Router operating system thats based on a Unix-Like operating system called FreeBSD.

What is FreeBSD?

#

FreeBSD is as i said, a UNIX-Like operating system that unlike (GNU/)Linux is more closer to the traditional UNIX off the old days and in the past it had large portions off original AT&T Unix code. Also its more off a Unified Operating system as all components are developed in-House by a small team, as opposed to (GNU/)Linux, where you have different developers and organizations in the community develop and maintain different components. And its a mix off a large collection of operating system utilities (largely provided by the GNU(GNU’s Not Unix) project off Richard Stallman) and the Linux kernel by Linus Torvalds, hence its often called “Gnu/Linux”. That means that it also depends on a ton of other developers and organizations to have their work ready when a new release is planned and also you cant easily fix bugs yourself and instead need to notify the maintainer themselves. With FreeBSD its all in-house developed and there are little to no outside people. This also means its a much more stable OS due to the unified, small-team nature.

Commercial uses of FreeBSD

#

FreeBSD is also used widely in commercial products. Sony’s playstation series (3 and upwards) run Orbis-OS which is based on FreeBSD. Not only Sony, but also Apple uses FreeBSD, but inside of its Darwin Operating-System, which powers Apple’s MacOS-X, IOS, IpadOS, HomeOS, WatchOS, etc etc. And is based on the XNU kernel(X is Not Unix), which incorporates a lot of components and code from FreeBSD (althought modified by Apple). The FreeBSD part even makes their OS officially UNIX-Certified. Also Netflix is believed to use FreeBSD on their large streaming CDN(Content delivery Network) as stability is ofcourse critical for them with millions of users. FreeBSD’s network stack is miles more stable than that off (GNU/)Linux. That network stack mature nature and stability is also why Opnsense and PFSense (another Opensource firewall/router solution) use FreeBSD as their core. if you come from (GNU/)Linux or any other UNIX-Like os, FreeBSD will be very similar, but there are some differences and bits that are different across them.

Limitations

#

Because FreeBSD is a smaller project than GNU/Linux, it also has less devs and thus progresses less. This also makes it so it lacks some softwares that other platforms have had for some time and also FreeBSD lacks a major amount of drivers that GNU?Linux does have. This also means that Opnsense has a lot less drivers for network interface cards (NIC’s), so that limits the available network card options that it supports. FreeBSD does have amazing support for Intel cards and Realtek cards also work good enough, but less amazing than intel cards. SO we have to take that into consideration first. Any no-name adapters could have issues so going for Intel or realtek is the best option.

The hardware setup

#

So i needed to find hardware that could run Opnsense. I did have an old HP 280g2 pc which was my first NAS and i bought for 40Euro, that was laying around the place doing nothing but catching dust. So i decided to turn it into a firewall. Its specs are: CPU: I3-6100, RAM: 8GB DDR3-1600 and a 128GB ssd as the bootdrive. Opnsense is not that bloated so it should work out. The NIC (Network interface) is a Realtek RTL8111G-CG which should work with FreeBSD and thus Opnsense. Sadly this is the only NIC so i have to add another one externally. Either via PCIe or USB. Because i dont have a PCIe nic laying around at the moment and am a bit low on money, i decided to go for a temporary solution off hooking up a USB nic. I can always migrate later to a proper PCIe nic. For now i just need things to work out. I had a USB nic that was a generic no name one which ofcourse i could not use. I also had a USB-C hub that had an internal realtek NIC. That one is supported so i used that one. Its a bit off a janky setup as i have to convert usb 3.2 to usb c and then plug in the usb-c hub and then use the NIC off that hub, but it should work out as a temporary kind of emergency solution. Below are a few pictures off the setup.

The Opnsense setup

#

So installing Opnsense was not hard at all. I downloaded The officiall IMG file from the Opensense Website, then on my Linux host, i simply plugged in a 8GB usb i had laying around, used Disks to write the Image file and i was done. Then I plugged i the usb drive, a monitor and keyboard and booted into the installer. You definitely know is based on FreeBSd as it has the same Loader, just the beastie logo (Beastie is the mascott off FreeBSD) replaced with that off Opnsense. I could wait for the timeout to go by or i could press 1 to boot the os. Then it loaded up and you saw some normal FreeBSD stuff going on until it booted into the installer, which was a program that had a TUI (Terminal user interface). This installer had some easy to follow steps and i could configure what NIC my WAN(Whole area network. The Connection that has access to the internet) and what NIC my LAN(Local area network. Basically the “Output” off the router/firewall going to the client devices) was on. Then i could boot into the WebUI and do the rest off the configure steps there. I like that after it has booted up, that it will use the PC speaker/buzzer to generate some nice beeps to show “I am alive and running” and on shutdown or reboot it would output a pattern off beeps to show “i am shutting down”. Below is a picture off the WEBUI login page. I logged in as ‘root’ and with the standard password of “opnsense”.

Setting up everything in the WebUI

#

In that WebUI i could configure stuff like the WAN interface’s IP and the LAN IP and subnet. Instead of 192.168.x.x24, i choose a custom subnet like 10.10.x.x/24, the “24” bit is the CIDR (Classless Inter-Domain Routing) number. After that i changed some rules and set it to double-NAT. Then i changed some options like domain and dns. I could also add DHCP boot options in opnsense. For that i created some tags and then had it that if it got client-arch=0, it would set the tag BIOS and then the DHCP boot option for BIOS would be chosen. Otherwise if client-arch=6 it would choose 32bit-EFI and choose its boot-option. Same method for UEFI64, with with client-arch=7. I previously had them in PiHole, but since Pihole was not anymore the DHCP server for the Homelab (It still is on the normal house’s network as i want to block ads and the only way to set the DNS server to the IP of pihole was to bypass the DHCP server inside the ISP’s router), i ported them over and everything works nicely. I might also deploy pihole on the network for the homelab but i am not sure if i will.

Organizing and converting all devices from the current 192.168.x.x subnet to the new one.

#

Also i wanted to have all devices neatly organized. See below table for all my devices and network infrastructure.

Nas Server

#

Transferring over the nas from the old 192.168.x.x subnet to the new 10.10.10.x subnet was not that difficult. I plugged in a monitor and a keyboard and then could enter the CLI. From there i could select option 1 which was “Change network” or something like that. That opened a new TUI where i could set a new IP address. Then i rebooted everything just in case and it came up and i could reach it on the new 10.10.10.10 IP.

Proxmox server

#

Transfering over the main proxmox server was a bit harder. I also plugged in a monitor and keyboard here, but there was no easy user-friendly way to change the IP using a terminal option or TUI. SO i had to look it up and then i found i could edit the “/etc/network/interfaces file” and change the option below to the new subnet.

Old option

#

iface vmbr0 inet static
        address 192.168.x.x/24
        gateway 192.168.x.1
        bridge-ports nic0
        bridge-stp off
        bridge-fd 0

New option

#

iface vmbr0 inet static
        address 10.10.10.9/24
        gateway 10.10.10.1
        bridge-ports nic0
        bridge-stp off
        bridge-fd 0

After changing those, i rebooted the server and i could reach the webui at the new IP addresss.

Homeassitant server

#

Changig the IP on the Homeasistant server was like on the nas, not that difficult. I could simply enter the command i listed below.

network update enp2s0 --ipv4-gateway 10.10.10.1 --ipv4-address 10.10.10.8

And boom, its done. Easy peasy!

OpenBEKEN

#

For most off the IOT gear (The OpenBEKEN and WLED flashed devices) i could set a static IP in their configuration. For OpenBEKEN it was dead simple, just powercycle the device 5x and then it would enter a special safe-mode. In this mode the device activates an Accespoint and allows you to connect to the device via WiFi. Then you can go to the default IP and change some settings like WiFi or IP-Address. After changing those from the old 192.168.x.x subnet to the 10.10.10.X subnet, you simply reboot the device and it will work. Then also you can enter the WebUI off the device at the new IP and change the mqtt settings so it points to the new IP-Address off the Homeassistant server and then Ha will detect it again.

W-LED

#

For WLED it was a diferent story. Because i had already set them up to connect to 309Electronics_IOT but with the old subnet of 192.168.x.x, i could not easily force them into AP mode. Likely because 309Electronics_IOT was a network they could connect to and everything was the same except for the subnet and IP and it did not cause them to go into AP mode automatically as technically they where still able to connect. So i had tot disconnect them from the ledstrips and then use a USB cable and hook it up to my laptop. Then i could go to the WLED Install website site and then i could get an option to change wifi. Because the Master Deco used to be connected directly to our main ISP router, i temporarily set it up so i could connect it to that network. Then i could reach the ip off the device as it was in that 192.168.x.x subnet and go to settings and then change the settings to the 309Electronics_IOT network and update the IP address so it falls in the 10.10.10.x range. That was all had to be done. So not totally flawless and it did take like 20 minutes to do all off them, but in the end hard work pays off and all where recognised and had the new 10.10.10.x subnet!

The rest off the containers and VMS in proxmox

#

So changing the IP on the containers was also easy. I could simply go to proxmox webui, go to the container, then go to “network” tab and then change the interface’s IP and gateway. For the VMS i changed the IPs in their Operating system. For the Tailscale VPN i had to edit the SystemD service i created. Instead off

--advertise-routes=192.168.x.x/24 

i put

--advertise-routes=10.10.10.0/24

and then i simply restarted tailscale. Then the last thing to do was to change the IP addresses in the Caddyfile in the VPS, so i ssh’ed into the VPS and edited the /etc/caddy/Caddyfile and changed the IP’s from 192.168.178.5:80 to 10.10.10.5:80, aka the IP off the NPM instance. After that restart caddy, restart tailscale and my services are reachable again!

Conclusion/final thoughts.

#

So now everything is back up and running and i now have an additional layer of security and isolation. I will work on some firewall rules to further tighten up security but i am happy that everything is working. I do notice that the USB NIC might be the bottleneck as its hooked up via USB and thus has some stuff inbetween that can cause latency and timeouts/dropouts. I did notice some timeouts today (the day after deploying it, when i wrote this article) and sometimes my music would stop or i could not connect to the Homeassitant server to control my lights and appliances but those timeouts and lags where only temporarily. Still i will put in a better Intel NIC card via PCIe into the opnsense box when i buy one. For now it works good enough and i am happy that it works at all! Surprisingly that usb nic is not that hot and its not cooking either. But that is a temporary solution. In the past i have played arround with some BSD’s, but not to a whole extend. I think i am starting to like FreeBSD even more and i like how stable it is and that its a clean OS/base layer that does not have tons of bells and whistle and its also closer to the true UNIX off the older days. A lot of Gnu/Linux distros have SystemD that breaks some off the UNIX rules and mindset like, “Do one thing and do it well” and “Not too complex/fully featured”.

Anyways thats it for now!