Recent “cluster server” security breach

For the past 4-5 days, users of FreeBSD have been unable to get any updates from servers worldwide. Specifically, this affected the following services:

  • csup and cvsup (for src, ports, doc, etc.)
  • portsnap (for ports only)
  • freebsd-update (for binary updates/security updates, if not using csup/cvsup)
  • Native CVS (note: intentionally using cached copy from Google)
  • GNATS (official FreeBSD problem reporting/bug tracking system)

All that was being stated publicly by FreeBSD Project members was that there was “an issue”, and then later this turned into “maintenance”:

Some others also began to wonder about how the outage was being handled, since nobody had updated the home page (until the aforementioned), updated the official RSS feed, posted anything on Twitter, or (most important of all) sent anything to the official freebsd-announce mailing list.

At the 4-day mark, I began wondering what kind of “maintenance” was unannounced and not done without some degree of prep-work. And it turns out my system administrator gut feeling was right — today the root cause was actually disclosed:

The omission of detail (particularly of the technical nature) in this report is astounding, resulting in almost more questions than answers. But worse, the report appears to be written in such a way to make it into a scapegoat for pushing portsnap and svn (instead of csup/cvsup), which makes absolutely no sense on multiple levels. Some actual FreeBSD Project members have questioned this as well and I applaud them for doing so.

First and foremost, the report seems to indicate the compromise provided the intruder a way to modify or replace packages on the master package server. If that were the case, the attacker then would have been able to modify relevant checksums/hashes of packages. portsnap just acts as a “ports tree snapshot” utility, it doesn’t download or verify each individual ports’ updates and do checksum validation (at least from the client-to-server perspective) — and even if it did, again, if the attacker had access to modify files used by the portsnap master to generate its update snapshot… You can see where I’m going with this.

Secondly, we do not know if this individual had root-level access or not, just that they could have potentially modified packages. So people citing concerns of csup/cvsup being susceptible to MITM has absolute zero relevancy to what actually transpired in this compromise. Unless the intruder had the ability to stop + start the portsnap mirroring daemon (or whatever it uses — there aren’t details) on the portsnap master server itself, or a way to stick a MITM daemon in the middle by siphoning packets to/from another daemon on the machine (or the 2nd machine mentioned in the report), then MITM has no bearing on this situation thus the concern cited is pure FUD with regards to this compromise. We FreeBSD users have an awful tendency to try and divert attention/focus to something else (perfect example) instead of zeroing in on the problem at hand; “how do I solve X” is never answered with “you do Y”, instead it’s “why are you trying to solve X” or “there’s nothing to solve” or “let me give you a one-liner that practically demeans you personally”.

Thirdly, I still cease to see how users using svn (instead of csup/cvsup or cvs natively) would have rectified this situation directly or indirectly. This is a bit long/heavy, but detail and history are needed to understand my as-usual KISS principle stance:

The problem with svn is that it isn’t in the base system. csup is in the base system, which means right out of the box FreeBSD users can immediately begin getting updates to whatever src branch they follow, and/or ports. Users have repeatedly asked what the replacement for csup/cvsup is, and all they’re told is “portsnap and svn”, with no further information given.

Now, anyone who knows me knows I’m generally against the “base system” concept — that is to say, included software should all be packages or equivalent (i.e. no more “OpenSSL ports” vs. “OpenSSL base”, “gcc ports” vs. “gcc base”, etc.). But as long as the “base system” concept is kept, then it needs to be respected and treated as important. And in fact, the security breach report in question even alludes to its importance. Quote:

“No part of the base FreeBSD system has been put at risk. At no point has the intruder modified any part of the FreeBSD base system software in any way.”

The problem with svn is that it pulls in a crapload of runtime dependencies. The biggest and most notorious is SQLite — which I have no problem with as software, but I’m focused on the dependency count here. However, APR is also required (certainly because Subversion is an Apache project, sigh), as is gettext (NLS/character set support), expat (XML support (why?)), and neon29 (needed for HTTPS support — OPTIONS doesn’t say this but believe me it is):

root@icarus:/usr/ports/devel/subversion # make run-depends-list

All of these things would need to be pulled into FreeBSD for svn to make it into the base system. And I’m certain, especially in the case of gettext, there would be licensing concerns.

But maybe a history lesson is needed here…

Older FreeBSD users may remember the days of cvsup, where that also wasn’t included with the base system (because of its dependency on the Modula-3 language). All of us got used to doing pkg_add -r cvsup-without-gui after our OS installation and considered the matter closed, although it pulled in ezm3 (and that was its only dependency). I was one of the few who was annoyed by this dependency, and the author used to repeatedly state that Modula-3 was absolutely needed for some of the design choices of cvsup. It greatly amused me when years later csup was created, which was cvsup written completely in C — as you can see, Modula-3 isn’t needed at all, so this was just another example of a software author making his belief system everyone else’s problem, rather than being pragmatic.

Here we are again, with history repeating itself, just in a different way. So why don’t I just pkg_add -r subversion and be done with it? Mainly because it pulls in 5 dependencies, which means 5 things I have to deal with/maintain when it comes to software upgrades and so on. I can accept gettext (since it’s used by lots of other software anyway), but the others I really don’t want to deal with.

Okay, so what choices do we have? Well, the svn port can be built with the STATIC option, which results in a statically-linked build. That sounds good to me — except I don’t want to deal with building the software. Why? Because building it (vs. running it) pulls in even more dependencies:

root@icarus:/usr/ports/devel/subversion # make all-depends-list

And that brings me to my final point: so why can’t the FreeBSD package/build cluster build a static version of svn for people to pkg_add -r? A wonderful question, and it’s one I’ve asked many times over. Technically there is no reason this can’t be done. Purely on a technical level, building that static package would be quite easy in the situation where any of the runtime dependencies needed to be updated. The maintainer would then simply increment PORTREVISION in devel/subversion/Makefile to act as an indicator that the package/port should be rebuilt due to underlying dependencies changing (and this is how the ports system maintainers already operate). So why can’t subversion-static and subversion16-static packages be made? Nobody seems to know, only that “you can do it yourself” (and I won’t for the reasons listed above — plus this is what the package building cluster at is for).

This is why I continue to stick with csup, and I imagine many other FreeBSD users do (and will) as well. When the cvsup mirrors go down sometime in 2013 (it’s estimated February), I will most likely moving over to Debian Linux. No, Linux doesn’t magically solve all of this (and I will greatly miss the pf firewalling stack), and believe me Debian has its share of issues, but this kind of clown-car seat-of-the-pants operation needs to end — at least for me. The irony, again for me, is that I moved from Linux to FreeBSD in 1997 solely because of clown-car syndrome.