This week we had a bit of an update from the FreeBSD community and Netgate. Mainly, the integrity of the WireGuard VPN code was called into question. As a result, the feature is effectively being pulled from FreeBSD 13 and pfSense in a major blow to the ecosystem.
pfSense and FreeBSD Pull Back on Kernel WireGuard Support
For those who may recall, with pfSense 2.5 we got WireGuard VPN support. That is if we are being forthright here, one of the biggest new features in pfSense 2.5. The WireGuard VPN implementation was designed as a kernel-mode solution and then was contributed to FreeBSD. Most Linux distributions have supported WireGuard for some time, and OPNsense, as an example, has had userland WireGuard support. Still, at some point, this needs to be a kernel-mode implementation.
The Netgate team hired a developer to add the feature to pfSense and then it was contributed to FreeBSD, set for FreeBSD 13. The flare-up over the past week, in an abridged form, is that even after public comment and review, the code that was integrated into FreeBSD was found to be sub-standard when subjected to post-deployment review. It is not a great solution to find out that, for example, Jumbo frame enablement can cause security vulnerabilities in a VPN solution that is designed to provide security.
Of course, this is effectively an important feature for FreeBSD and pfSense. So the idea that it will be pulled indefinitely makes little sense. At the same time, the current guidance is to not use the FreeBSD 13 nor pfSense 2.5 kernel WireGuard at this point. There is work being done to rectify the solution since the kernel-mode implementation is something that a lot of folks want.
Final Words
At STH, we have been using pfSense for years. Beyond the politics of open source, pfSense has worked well. At the same time, this is a very good example of where the problem would not exist if pfSense was based on a Linux solution. WireGuard was integrated into the Linux kernel and is trivial to install on most popular distributions. In Ubuntu/ Debian “apt install wireguard” is all one needs to do to get started.
Beyond the immediate impacts of having a re-implementation of the feature, there is a reason other traditional open-source FreeBSD-based projects are moving to Linux. iXsystems started de-emphasizing FreeBSD for its TrueNAS Scale-Out Project. Linux is a bigger ecosystem than FreeBSD. The challenge now is that with one of the larger FreeBSD projects, pfSense, sponsoring a critical feature that fell flat playing catch-up to Linux, it is going to serve as a warning to others. If one chooses FreeBSD and creates a successful project, they have a higher likelihood of having to do something themselves versus having a critical feature mainlined and well tested. This was the first major feature implementation in pfSense in some time and it fell flat.
For now, we recommend our users switch back to IPsec/ OpenVPN. This will be fixed in the future. If you want to keep using WireGuard, ensure you are not changing the MTU to something larger than a MTU of 1420. This is not a great situation by any means since creates challenges for those that have already deployed. We also cannot turn a blind eye to the fact that this situation is out there. There is a userland wireguard-go solution available for FreeBSD and OPNsense, but the kernel version pfSense was using has been moved out of the mainline now, and the fixed version may come back in future updates.
So what linux firewall / router do you recommend that is equal to or better than pfSense / OPNSense?
I’ve worked with the Netgate guys and the reality is TNSR is going to be the product going forward. I have to wonder when they’ll add a GUI on top of it. Once they make it easy to use for normal people who don’t have cli experience, I fully forsee TNSR superseding Pfsense.
At least for up to gigabit WAN installs, I’d take OpenBSD with pf as a router/firewall over FreeBSD or Linux. My trusty little box has only a dual core Pentium G4560 (HT disabled) and it pushes gigabit 24/7 without breaking a sweat. Load averages are 0.41, 0.34, 0.20 on a busy, moderate-sized home network and home lab.
Oh and OpenBSD has a proper, secure in-kernel implementation of WireGuard – and has for a while now.
In my opinion, the chances of finding serious bugs in an existing program when reimplementing it to run on a different system are reason enough to have an independently created version of WireGuard for FreeBSD. In this case bugs were created rather than found, but I find it naive to think that will always be the case.
“…this is a very good example of where the problem would not exist if pfSense was based on a Linux solution.”
I don’t know why, but this sentence really perturbs me. I feel like this is a pretty casual and undeserved swipe at the BSDs. It also completely misses the point of why pfSense exists in the first place.
At the time this project started, the developers purposely chose FreeBSD to base on because the pf firewall in FreeBSD (and OpenBSD from whence it came) was arguably better than iptables in Linux. It went on to become a very successful project used in a lot of home networks and small offices. I myself was a happy user of pfSense for many years until the hard drive in my firewall appliance died just recently.
Whether you use them or not, the ecosystem of open source operating systems is richer for the presence of the BSDs. OpenBSD has pioneered a lot of security hardening techniques that have been adopted by Linux — and even Windows in some cases. With jails, FreeBSD pioneered the idea of containers, which Solaris went on to perfect with Zones. Frankly, it took Linux a long time to catch up with containers due to the patchwork quilt of technologies required to completely isolate a guest environment. It’s much better now, but it was a security nightmare in the early years. FreeBSD also brought in Dtrace and ZFS from Solaris, the latter port resulting in the birth of FreeNAS. Even NetBSD was used a lot in academic research and used to break bandwidth speed records every so often on Internet2.
Linux absolutely does have a much bigger ecosystem of users and developers, and that brings a lot of advantages to solutions built on top of it. For all of its success, however, it too is far from perfect. I’ve observed over the years a colossal amount of wasted effort and missteps. For example, btrfs was supposed to be Linux’s answer to the GPL-incompatible ZFS. Development started in March 2009 according to wikipedia. Twelve years later, btrfs is *still* not ready for production use and probably never will be. Linux has probably a dozen different tracing technologies of various states of maturity, but from a sysadmin perspective at least, none are as approachable or elegant as Dtrace. Ironically, eBPF seems to be the tracing darling of the community these days: note that B stands for Berkely. I’m also greatly concerned about the security of LTS kernels, which are routinely missing important security patches despite being LTS. (One recent example — https://blog.frizn.fr/linux-kernel/cve-2020-14381 — See Patch Gap at the bottom.)
Getting back to WireGaurd, the real problem in this scenario is that the developer Netgate hired never bothered to engage the WireGuard creators and half-assed his way through his contract. Drama aside, we’re actually very fortunate this happened in an open source setting otherwise a lot of people would be running a vulnerable implementation of WireGuard right now.
I don’t see it as a swipe at FreeBSD. It’s kinda factual. For pfSense these days they’re way late on HW support too. Like if pfSense was on linux we’d get new HW support much earlier. We’d also not have to deal with this kind of issue because WG’s been done for almost a year. Its fair to say if the Linux version was done earlier, and was better than this one, it wouldn’t have created for pfSense like this causing the issue.
I think it’s fair, but for me it’s the utter lack of HW support is why I’d prefer Linux. FreeBSD is fine on old stuff but sometimes it takes forever to get new HW working.
Say anything that ruffles FreeBSD people and you get half an encyclopedia of text.
It’s a bit harsh, but it’s the truth. I love using FreeBSD, but using Linux at work, I’d prefer if pfSense was Linux. I also don’t think this is a “swipe” at FreeBSD. I’m for one worried since FreeNAS is already going Linux. That’ll take a big project away. ZoL is the upstream now, and that’ll have more features. It’ll be a big hit to people even getting introduced to FreeBSD.
To be honest, I rather like that pfSense is based on FreeBSD. It’s a very solid OS and you can very much tell that the base of FreeBSD is one project and not assembled from a thousand pieces like Linux distros are. Man pages and other docs for Linux have improved over the years, but I think FreeBSD’s are still better.
However, I don’t find myself doing too much to pfSense from the CLI. I mostly work through the web UI. It’s basically an appliance.
I did try various Linux-based firewalls before settling on pfSense. I found that I preferred pfSense. I’m a bit up-in-the-air about pfSense in some ways, and I don’t like some of the things that Netgate has done. If I do decide to switch to something else, it will likely be to OPNSense.
I’m taking a wait and see approach. Moving to something else will be more or less painful, and my home network is working well at this point. I’m not looking forward to disrupting things and going through an adjustment period again. I have other things to worry about. :-)
I don’t think that it added to the article to bring Linux into this. It wasn’t necessary.
@Tim ,You mention the word open source a lot. I think you completely forget that Netgate just recently announced they will go closed source (and no the CE edition will be left to wither and slowly die) so had this situation happened a bit later this transparency that you tout as a big advantage would not have existed. To sum up: Announcing to go closed source and then shortly after make this giant snafu is very very vad for Netgate and casts a lot of dark clouds over its future.
And we have not even discussed the arguable close to psychotic behaviour by both the contracted developer (look him up) and the Director of development at Netgate who’s blog post frankly is deeply disturbing.
@Tim ,You mention the word open source a lot. I think you completely forget that Netgate just recently announced they will go closed source (and no the CE edition will be left to wither and slowly die) so had this situation happened a bit later this transparency that you tout as a big advantage would not have existed. To sum up: Announcing to go closed source and then shortly after make this giant snafu is very very vad for Netgate and casts a lot of dark clouds over its future.
And we have not even discussed the arguable close to psychotic behaviour by both the contracted developer (look him up) and the Director of development at Netgate who’s blog post frankly is deeply disturbing on many levels.
@Tim, Completely agree. It has a whole paragraph which claims, describe / presume / read as FreeBSD is crap or dead. Without explaining how FreeBSD, or in fact all BSD’s culture is to move slow rather than the move fast and break things model.
Now I have STH as Anti FreeBSD in my book. At least when I read any review, it will add some more perspective.
FreeBSD is for appliances like FreeNAS and pfSense.
FreeNAS is going Linux
Zfs was a biggie to roll your own BSD storage, but no point now with Proxmox and Ubuntu
pfSense won’t because netgate doesn’t care about free software version anymore
It isn’t debatable that FreeBSD is lost to Linux. It is just how long it holds on. Solaris has great tech, and is still around, but over time it lost out
What the freebsd defenders don’t get is that this is just how the industry and data are going. You can have the best tech but something else gets momentum. Blind to data attacks on sites is not helping bring people to FreeBSD. You’re hurting the distro commenting with blinders on
While I am a fan of the BSDs, I’m an equal opportunity user of both Linux and BSD. My career owes a lot to the availability of free, high quality general purpose OSes, regardless of pedigree. We have much to be grateful for on both sides of the fence.
The main point I wanted to make was that pfSense’s real differentiating feature — the thing that made it uniquely special — was that it used the pf firewall instead of iptables. The pf firewall was the new shiny at the time. That’s what attracted me to pfSense in the first place, and probably many others in the early days. If it had been based on iptables (and therefore Linux) from the start, it’s very likely pfSense (or maybe it would have been named ipSense?) would not have mustered the critical mass of users to become the success it did.
Netgate and Kip/Matt Macy is a whole other kettle of fish which didn’t really have anything to do with my point. I don’t intend to support Netgate in the future.
When I rebuilt my firewall recently, I went with VyOS, which is based on Linux. That decision was influenced by my experience with Ubiquiti’s EdgeRouter line of low cost router appliances. EdgeOS shares a common ancestry with VyOS, both having been forked from Vyatta, so if you’re familiar with EdgeOS’s cli, you should feel at home with VyOS.
VyOS doesn’t have a web gui yet, but they’re beginning to work on that now. That will probably make VyOS a non-started for most people for now. I think the differentiating feature with VyOS is the enterprise-router-like cli so that you don’t have to fiddle with iptables commands directly or config files. I really liked working with EdgeOS, but found the hardware and support a bit disappointing. Running VyOS on generic x86 hardware is the best of both worlds, so I’m happy.
And yes, WireGuard is fully supported. :)
@ksec STH is anti BSD ? because they look realistically at how the market for operating systems and features are moving (see ZOL now being the main development, and routing appliances moving to Linux including pfsense and Freenas/Truenas, arguably 2 of the largest stewards of BSD. I think Your statement is having a faint whiff of nervousness for the future of BSD. Your final statement about BSD going slow, is really irrelevant to this whole discussion with pfsense and Truenas both having high-end products in development based on Linux.
I think STH is one of the most pro-BSD sites out there. I just see this as they’re pointing out that WG was done for Linux and the duplicative effort failed.
For the record, I’m pissed since I implemented WG. It’s a pain to replace VPNs after they’re deployed remotely and the VPN is your way in. So as someone who’s impacted, I’d say it’d be better for quality to be on Linux and not have to do this at all.
How dare one of the most insanely pro-FreeBSD sites suggest that Linux does anything better!
Linux already won. Ya’ll just trying to be different like getting a nose to armpit piercing with a thick chain and tattoos around using FreeBSD.
@tim I’m also in the process of moving to VyOS from pfSense. I had some other projects I wanted to experiment with (mainly trying to spin up an IPv6 only lab for learning) so I was already learning the vagaries of VyOS.
Definitely not the easiest of starts, but I am learning much more about networking which was one of my goals.
What a fascinating bunch of comments to read. They range from the FreeBSD supporters and zealots to the Linux crowd to folks that just want something that works and has Wireguard.
Honestly, I have never seen STH being pro or anti anything. STH tells it like it is, and sometimes the truth hurts or conflicts with our own impression of the world. So those placing a red mark next to STH for appearing to be anti-FreeBSD, or slamming Linux, or claiming one or the other is better are just being p0sers, taking a popular position to look kewl in front of their f@ns.
I have used FreeBSD in the past, but I found it’s hardware support lacking when you want to use hardware that is 2 years or old or less, or you want to use hardware that has ‘quirks’. Perhaps FreeBSD has improved it’s hardware support process over the past few years, but I agree with the poster that FreeBSD releases try to avoid the “breaking things” path of hardware support. So FreeBSD is more conservative relative to a more liberal Linux? That would be an interesting comparison & discussion.
Seriously, I have always considered Wireguard to be a serious uplift to any OS. It did not enter the Linux ecosystem easily. I think it was out-of-tree for easily a year or more as Wireguard developers refined the code to a level acceptable to the Linux key developers.
So perhaps FreeBSD took a more liberal approach with introducing Wireguard, an expedited process or relaxed code checks; I don’t know as I did not follow the acceptance process.
Needless to say I find no shame in any release backing away from a major feature when that major feature introduces serious code breakage. To claim otherwise would be like saying, “My car has a hot new engine, but the wheels keep falling off now. So what, I’ll drive it anyway.”
With VPN code the focus should be on secure code over “rush to production”. So in this case I think the FreeBSD folks got it right; secure code comes first.
I find it entertaining that some in here also think that pfsense is going “closed source.” It’s like their brains turned off on reading the announcement Netgate made.
Whatever the case, the guy who wrote Wireguard wading in last minute to wave his epeen around soured me on some great tech. If he’d handled it like an adult, maybe it wouldn’t be such a mess right now.
Eh. I’m switching away from pfsense anyway. The last straw for me was putting QAT behind their paid version. I use pfsense for my homelab (ie, no budget for licensing) and purchased hardware based on their forward looking statement of “QAT coming soon”.
But when it did come it required a paid subscription. I’d be happy to pay for a homelab license but the only option is $399/year for support. That’s a hard no from me.
I’m switching to Unifi for routing and I’ll use this box as a docker host (C3758 – 8c atom) & VPN endpoint with a Linux OS that lets me use QAT.
@Tim
Worth noting that the VyOS/Vyatta Web UI has been in development for at least 5 years. There is a project called VyControl but I suspect it’ll never be completed to any serious degree as most of the features just don’t translate well.
Personally I’m also considering alternatives to PfSense after this situation. I’m quite familiar with VyOS (been using it since way back in the Vyatta days), but it’s a very different piece of software and is more of a pure firewall/router. It lacks a lot of home-router friendly features like the UI, plugins like PfBlocker, no easy/automatic hairpin NAT solutions etc.
For the time being I’ll stick to 2.4.5 and maybe pivot to OPNSense before possibly jumping ship to (preferably) a Linux based router. IPFire is interesting as it’s been actively developed for a long time and has a large number of features/addons which cross-over with things you’d expect from PfSense and other similar fully-featured routing distros like Untangle.
While it is an unfortunate delay; it’s worth noting that, after discovering the quality issues of the Netgate-developed freebsd kernel Wireguard implementation, Jason Donenfeld, the lead developer, started up an effort to produce an implementation of necessary quality and get it into the next version that time allowed(estimates seem to be 13.1).
There was some honestly rather unseemly acrimony on the Wireguard side in response to “developer reports bugs in your pre-release software; voluntarily begins work on a vastly improved version at no cost to you”, which is something that seems like it would be one heck of a deal; but aside from that fuss things look very promising for a solid freeBSD implementation, just not on the original schedule that Netgate wanted.
Honestly, this makes me feel doubts as to FreeBSD’s suitability in 2021 and onwards.
Does their kernel team not review code as thoroughly as the Linux kernel team does? How could such a pile of turds have made it into a “production” release of their kernel?
In Linux it wouldn’t have even gotten into staging, let alone an actual mainline release, in that state. And NetGates little attempts to malign Jason and the FreeBSD kernel devs wouldn’t have been tolerated by the Linux devs. How do you think NetGate would be able to handle Linus Torvalds frying them to a crisp on LKML?
What matters though is that it seems like the FreeBSD devs at least owned that they let something poor through, whereas NG’s response to feedback was an attempt to malign them and the WireGuard devs instead.
That and their smear attempt on opnSense makes me trust pfSense less and I’m thinking of switching to VyOS in the long run. I considered opnSense, but I don’t really want a userspace WG implementation. VyOS is based on Linux, so I’d get a kernel-mode implementation of WG, and probably the best implementation of WG out there right now. Too bad VyOS doesn’t have pfSense’s fantastic UI.
@Lance Longreen The statement about going slow is direct response to the Author claiming FreeBSD “fell flat playing catch-up to Linux”. Linux has always been on the bleeding edge, BSD has always been for the wait and see before jumping in. When this has been going for for over decade, how is that comment not relevant without further explanation?
And your final statement on having linux on two companies product just show how you dont understand the difference.
@Yaro OPNSense now has wireguard-kmod giving in-kernel WireGuard. It was implemented in the 21.1.4 release of 30th March 2021, and OPNSense are keen to note that it’s Jason Donenfeld’s code, not the Netgate one… ;) Why not just use straight OpenBSD or Linux? Much more straightforward. IPFire is another (Linux based) option, and it’s very lean and fast for routing. It doesn’t support WG though, and the dev has made clear he has no interest in it.
https://fingerlessgloves.me/2021/03/31/hello-wireguard-kmod-on-opnsense/