Introduction

#

SO on 25 April night, i noticed that my Homelab services/servers where offline and that my tailsclae VPN failed as i could not watch youtube anymore or do a google search. I went that night to bed not thinking much about it and also i was just tired. The next day woke up and decided to debug the issue.

Troubleshooting steps

#

So first steps i did was try and run the trusty ping tet. Simply run ping {address} in your windows/mac/linux terminal. What ping does is it basically checks if you can connect to/reach that server as it will come back with a response like below if the server is available and your internet is working and will come back with nothing if you cant reach that server.

So i tried pinging the upstream ISP router/modem and that worked. Pinging it returned the response below:

64 bytes from 192.168.178.1: icmp_seq=1 ttl=63 time=6.02 ms
64 bytes from 192.168.178.1: icmp_seq=2 ttl=63 time=7.26 ms

So the WAN side of the opnsense box works fine. Then i pinged 10.10.10.1 (Opnsense’s IP) and that also worked!

64 bytes from 10.10.10.1: icmp_seq=1 ttl=64 time=0.372 ms
64 bytes from 10.10.10.1: icmp_seq=2 ttl=64 time=0.322 ms

Then it tried to ping google.com it returned the response down below

ping: google.com: Name or service unknown

I also tried nslookup to lookup the domain on any nameerver (DNS) but that also failed. So that pretty much points my suspicion that its a DNS related issue. But to verify its not just my laptop and that its a network wide DNS issue i logged into my switch via ssh, which is a zyxel GS1900-8port which i installed OpenWRT on. Then i tried the same ping commands and that worked. Then i tried pinging google.com but that failed, so we know that it must be on the Opnsense side as both the switch and my laptop can ping the raw addresses but cant ping the domain names.

As another verification i directly hooked up my 2nd laptop directly to the lan interface on the Opnsense box and got the same responses for ping (raw addresses worked fine but domain names not).

“Its always DNS” comes into clutch.

#

In the networking and IT sector you have this meme as DNS is a really common issue so usually when some service or website/network fails, people say “Its always DNS” or “Its probably DNS related”. So here i also have a DNS issue as domain names cant be resolved correctly/at all and thus clients can never find the real ip address of the server and thus cant connect.

For anyone wondering what this DNS stuff is, i wrote a explanation below. If you are not interested in that or already know what DNS is, you can skip to the fix by clicking This button.

What is DNS?

#

So DNS can be considered the “Phonebook of the internet” as it litteraly can be compared to a phonebook. In the old days you had no whatsapp or a call App with your mom, dad or friends already in the list. You had to manually search up their phone number in a big book with their names and phone numbers and sometimes adresses. So calling your mom would then require you to first search up her name and then find here phone number like this example for explanation: Mom = 31 12343212 and then you would call that number and get tot talk to mom. The internet works in a similar fashion. When you want to access google.com or wikipedia.org, that domain name thats readable and understandable by all humans, needs to be translated to a real server address. You cant just go and call Mom by typing “mom” in the telephone (atleast not in the older days and also have to get her real phone number) and you cant type in wikipedia.org as the ip addres as IP addresses are numbers.

So when you enter wikipedia.org into the searchbar in your browser, your browser then makes a request that first hits a DNS server. The DNS server then checks in its large databse if it can find the name you are looking for (basically like scrolling through a phonebook and seeing if you find the name of the person you want to call with their phone number listed). If it has found the name (like if you found out that your friend Tom is listed in the phonebook and has his number written down), it also knows what the real IP address of the server that hosts wikipedia.org is. Every server has its own IP address like everyone has their own phone number. IP addresses come in different forms and revisions but i will stick with IPV4 (revision 4. The easier to understand one and standard one). IP addresses often look like this: 123.321.123.321 with a string of digits that has a ‘.’ between it and then another string of digits and it repeats this pattern like show above. This IPV4 is a 32bit integer and we kind of ran out of IPV4 addresses so we switched to IPV6 which has a 128 bits integer but its also a way more confusing/obscure string like this: fe12::3456:6dff:fo78:fe90.

So wikipedia.org has a real IP addres (similar to that uniquie phone number of that person) and using a bit of nslookup (a linux command that makes this request and returns the real IP address of the domain you enter), we can see that wikipedia.org has an IP address of: 185.15.59.224. SO if DNS does not work, you cant connect to/access the server of that website and thus cant access the website. Same way if you dont know the phone number of the president that you cant call the president.

The fix

#

SO i decided to Ask claude and gemini (Again i am fine with using AI for troubleshooting reasons or as a tool) and claude went in a total rabbit hole of dnsbl or somethign to do with the blocklists i had setup, while gemini went straight to the point and told me “Maybe this setting is set to Deny so it wont allow any DNS queries to be made”. And that was the fix! Pretty easy fix. When i navigated to Services > Unbound DNS > Access Lists, i found an Option called “Default Action” which was set to “Deny”. Seting that to “Allow” fixed the issues. DNS fixed just because a setting was wrong. That setting is basically what Unbound needs to do/if it has to accept stuff when it gets a DNS query. That being set to Deny ofcourse maks it so the DNS queries made are simply denied/ignored and thus the device/service making the DNS request does not get na answer and thus cant lookup what ip the server of google.com has.

Kind of felt stupid that it was such an easy fix but i am no expert and thats also the beauty of the Homelab hobby, learning to setup systems and learning from your mistakes and experiences. And its a easy to miss setting. I dont know why things like my homelab and servers stil worked, but after looking in their settings, they seemed to use my upstream pihole DNS server or quad9’s DNS server so they ofcourse where working, but things like my VPN and 309Electronics_IOT network rely on the opnsense box as the DNS server. And some devices where set so they had both the opnsense IP and the Pihole IP as their dns server and sometimes they would switch to either and thus one moment they worked and the other moment they would not.

I am not advertising any Ai model here, but that shows that sometimes using 2 AI models by differnet makers can get you the best answer as sometimes an AI goes haywire and starts halucinating and coming up with bizarre things. That was exactly the case with Claude and shows that even the “better AI models” still often Mock-up things or mess up.

So yeah that was my lovely encounter with DNS related problems and now i can finally properly join the meme cult with the “Its always DNS”. But instead of a worldwide problem its a setting i overseen but i think it stil counts.