Last week we explored creating VLANs for network segmentation, learning how separating devices into distinct networks can improve security and organization. That guide laid the foundation for more advanced network and security strategies, showing how logical isolation can reduce risk for critical devices while keeping guest and IoT traffic contained.

This week, we build on that knowledge and explore advanced networking and security in the homelab through a practical case study of my own home network. Rather than prescribing a specific setup, this article follows my journey: the devices, firmware, firewall platforms, and DNS configurations I explored to secure and manage my network more effectively.

We’ll look at my early experiences with thrift store routers, experimenting with third-party firmware, and ultimately adopting OPNSense as a full-featured firewall platform. Along the way, I’ll share what worked, what didn’t, and the lessons learned through hands-on trial and error.

By the end of this case study, you’ll not only understand technical options for homelab security but also see how personal experimentation can shape your approach. This story-centric approach emphasizes learning from mistakes, discovering new tools, and layering security thoughtfully.

Early Firmware Exploration

My first foray into advanced networking came with a couple of thrift store Linksys WRT65G routers.

“If it weren’t for those thrift store WRT65Gs, I might never have gotten hands-on with Linux-based firmware so early.”

Stock firmware was functional but extremely limited—basic DHCP, NAT, and port forwarding, with little to no customization. Third-party firmware opened a whole new set of possibilities. The WRT devices were unique at the time because they ran Linux under the hood, allowing easy installation of alternative firmware and even custom tweaks.

I experimented with both DD-WRT and Tomato, running one on each WRT65G. Over time, I preferred DD-WRT for stability and consistent performance. This early exploration highlighted the value of enhanced functionality: VPN support, granular firewall rules, QoS management, and scripting capabilities. These features, unavailable on stock firmware, were my first steps into real network control.

Tip: Early hands-on experience with firmware teaches more than theory—experiment, break things, and learn from the results.

Lessons Learned

• Third-party firmware can dramatically extend router capabilities beyond stock firmware.
• Older WRT series devices, like the WRT65G, were excellent learning platforms for Linux-based network experimentation; modern models preserve the look but often lack the same open firmware support.
• Hands-on trial and error is invaluable for understanding network behavior and constraints.
• Key Takeaway: Experimentation is critical—don’t fear breaking things in a lab environment.

Router Firmware Exploration

As I continued experimenting, I evaluated firmware options more systematically. While DD-WRT was reliable for older devices, OpenWRT eventually became my primary choice. It was stable, actively maintained, and supported advanced features like VLANs, scripting, and multiple network interfaces.

I also found value in running multiple firmwares in parallel for experimentation. Older WRT65Gs stayed on DD-WRT while the WRT1200AC moved to OpenWRT for advanced home network tasks. This approach allowed me to test capabilities without risking the primary network.

Tip: Keep some devices as “experimental labs” while maintaining a stable primary network.

Lessons Learned

• Firmware choice affects stability, feature set, and manageability.
• OpenWRT is a strong choice for modern homelab routers; DD-WRT remains useful for legacy hardware.
• Running multiple firmware versions simultaneously supports safe experimentation.
• Key Takeaway: Practical experience with firmware informs better decisions.

Adopting OPNSense

Seeking a more centralized firewall solution, I researched PFSense but eventually settled on OPNSense. While PFSense initially attracted me with features like Dan’s Guardian, I was hesitant about recent monetization changes. OPNSense provided a similar, open platform with strong support for VLANs, multiple NICs, Unbound DNS, and VPNs.

“OPNSense became the cornerstone for securing my home network, consolidating firewall, DNS, and VPN management in one place.”

Early configurations included basic firewall rules, NAT, and VLAN awareness. Unbound DNS integration allowed me to consolidate blacklists and content filtering that I had previously tested with Pi-hole. The migration to OPNSense highlighted the advantages of a centralized management platform over scattered tools.

Firewall OS Comparison

Before fully committing to OPNSense, I looked at other firewall platforms to see what features and trade-offs each offered.

This table focuses on platform capabilities. My journey illustrates how personal experimentation determines which features matter most in practice.

Lessons Learned

• OPNSense offers robust, centralized firewall and DNS management.
• Consolidating security functions simplifies administration and improves maintainability.
• Choosing a firewall platform involves balancing features, community support, and long-term viability.
• Key Takeaway: Consolidate services to prevent conflicts and simplify management.

Securing DNS

Once the firewall was in place, the next priority was encrypting DNS traffic with DNS-over-TLS. Protecting queries from eavesdropping is crucial, but filtering also adds a layer of security against malware, ads, and inappropriate content.

Initially, running Pi-hole alongside the firewall caused conflicts, as different blacklists blocked different sites. Consolidating filtering into Unbound DNS solved these issues, and I settled on Cloudflare Family and CleanBrowsing as primary DNS resolution providers. Critical services that were blocked by filters, such as Microsoft email or Minecraft logins, were routed through Cloudflare’s unfiltered DNS.

Tip: Use DNS-over-TLS with multiple providers—filtered for safety, unfiltered for critical services.

Lessons Learned

• DNS encryption enhances privacy and security.
• Provider selection affects filtering and compatibility with services.
• Consolidation of filtering prevents conflicts and eases management.
• Key Takeaway: Layered security, including DNS filtering, strengthens resilience.

Layered DNS Filtering

Before full consolidation, I experimented with Pi-hole on a Raspberry Pi. While functional, it eventually became unreliable and conflicted with the firewall’s DNS rules. Migrating Pi-hole blacklists into Unbound DNS simplified management and kept my network secure.

“Maintaining manual overrides for domains that slipped through the blacklists became a standard practice.”

Layered filtering provides resilience: multiple sources of intelligence strengthen security without creating administrative chaos.

Lessons Learned

• Centralized DNS filtering reduces conflicts.
• Manual overrides are necessary for services that filtering blocks.
• Combining multiple blacklists strengthens protection while simplifying updates.

Access Control and Network Segmentation

Next, I explored network segmentation. Adding a third NIC allowed me to create a 192.168.30.x network for work devices, isolated from the main home network. VLANs expanded segmentation further, separating guests, IoT, and media devices logically while using a single physical switch.

Tip: Network segmentation is as much about management as security—simpler troubleshooting and reduced lateral movement for threats.

Firewall rules ensured proper isolation, and VLAN tagging allowed flexibility for multiple networks over one switch. These strategies are critical when devices with varying trust levels coexist.

Lessons Learned

• NIC-based separation allows physical isolation.
• VLANs offer logical segmentation without additional hardware.
• Segmentation improves both security and network management.
• Key Takeaway: Network segmentation improves both security and device management.

Summary

Advanced networking in a homelab is a process of discovery. From thrift store Linksys WRT65G routers to OpenWRT and OPNSense, each step brought insights into stability, security, and flexibility. DD-WRT and Tomato taught me the power of third-party firmware, while VLANs and NIC-based segmentation demonstrated the benefits of isolation.

Securing DNS involved encryption, layered filtering, and careful provider selection. Consolidating Pi-hole blacklists into Unbound DNS, combined with filtered and unfiltered DNS-over-TLS providers, ensured both security and service compatibility.

This case study emphasizes learning by doing. Hands-on experimentation, incremental improvements, and consolidation of services made my network more resilient and easier to manage. Documenting the journey shows how lessons compound over time, guiding better decisions in firewall, DNS, and segmentation design.

“If it weren’t for hands-on tinkering, I wouldn’t have discovered the subtleties of DNS, VLANs, and firewall rule interplay.”

Beyond the technical lessons, this journey highlights the value of curiosity and iterative problem-solving. Each experiment—whether a firmware tweak, a VLAN test, or a DNS configuration—provided insights that formal documentation alone cannot offer. By combining structured planning with hands-on trial and error, even complex homelab networks become manageable, secure, and adaptable to evolving needs.