FreeBSD, csup/cvsup, portsnap, and svn

The subject of FreeBSD and it’s migration to SVN has come up on the mailing lists repeatedly over the past 6 months. The biggest problem seems to be the lack of communication and transparency of this change, combined with the recent security breach incident being used as a stepping stone to justify the situation (despite that incident having absolutely nothing to do with the transition of CVS->SVN). Peter Wemm has at least shed some light on the need for the migration itself; but it still leaves end users out in the cold (so-to-speak).

The most common question I’ve seen posted by end users is “how do I migrate?” and “is csup/cvsup broken?” The answers which have been provided by both the developer and user community have been abysmal at best, mainly because nobody actually bothers to take the time to explain the complexities and nuances involved in the migration. The official FreeBSD documentation is overwhelming and doesn’t take into account some of these nuances (nor should it, IMO). The migration itself is actually easy, but there are many catches along the way which are certain to bite even the most educated of administrators. I’ll also use this opportunity to explain why use of portsnap should be avoided if at all possible.

To try and keep it simple, I’ll outline each of the common setups people have, and how exactly to migrate from each one. I should be very clear with my intentions here:

  1. By “migrate” I do not mean “converting your existing data on the filesystem to svn” — instead, all procedures are based upon the “start fresh, delete old cruft, and go from there” concept. It is not worth fiddling around with something so key/important to FreeBSD; think about it.
  2. I strongly advocate management of both src and ports entirely via SVN. If you follow -RELEASE branches and do not care about src at all, I still recommend you use SVN to handle ports rather than portsnap (more on that later).
  3. The SVN checkout URL I use for src is for what has historically been called RELENG_9 (i.e. 9.x-STABLE, 9.x-PRERELEASE, 9.x-RCx, etc.), not -RELEASE branches. I do not follow -RELEASE branches nor do I care about them; -RELEASE branches only contain fixes for security issues, and every single -RELEASE since the 5.x days has had too many general bugs that warrant immediate fixing.

Below is how I migrated my own systems, starting with a fresh OS installation. The following should be performed as root:

rm -fr /usr/ports
rm -fr /var/db/sup/* /usr/sup
portsnap fetch
portsnap extract
cd /usr/ports/devel/subversion
make install
cd /
rm -fr /var/db/portsnap/*
rm -fr /usr/ports
mkdir /usr/ports
svn checkout svn:// /usr/ports
rm -fr /usr/src
mkdir /usr/src
svn checkout svn:// /usr/src
chmod 700 /root/.subversion
...Manually remove any SUP-related variables from /etc/make.conf...

Now to explain this utter madness:

  • Line 1 deletes the existing copy of /usr/ports from the filesystem
  • Line 2 deletes the CVS repository databases associated with src and ports; these are (were) managed by csup and cvsup themselves
  • Expanding on line 2: lots of people don’t know about these databases. Quite simply what’s in /usr/{src,ports} does not necessarily reflect what CVS thinks is in /usr/{src,ports}, and this is how/why there can be mismatches when going from one branch to another. Surprise! This is what bites a lot of people when switching between tags (i.e. going from RELENG_9_0 to RELENG_9 or vice-versa) or versions in general. So to make it clear: csup uses /var/db/sup for its database directory, while cvsup uses /usr/sup
  • Lines 3-4 use portsnap to fetch a copy of the /usr/ports tree. Why do we do this? Because we need to install subversion from ports
  • You may be asking: why can’t you just use pkg_add -r subversion? The reason is explained here by the port maintainer — read slowly, it’s easy to miss. As of this writing, the subversion package on the official FTP/package servers is old (October 13th) and pulls in an absurd number of dependencies when it shouldn’t. So we can’t use the package, thus we must build it from source ourselves. If/when this gets addressed by the maintainer and/or the package build cluster admins, yes, this procedure will become significantly simpler
  • Lines 5-6 build and install the subversion port itself, along with all of its dependencies. Please do not change any of the subversion options! The defaults as of this writing are as minimal as possible
  • At this point we have a working subversion (svn) binary that we can use
  • Line 7 changes the cwd to /, since we’ll soon be nuking /usr/ports (yes you read that correctly)
  • Line 8 deletes the portsnap database/tarball. This is the one of many reasons I cannot stand portsnap; it quite literally keeps an entire copy (tarball) of the /usr/ports tree laying around, by design. I wonder how many people with small /var partitions have gotten bit by this. portsnap also pulls down INDEX-x files for releases which aren’t relevant to your version of FreeBSD (i.e. it will pull down INDEX-7 on a 9.x system). portsnap’s ports tarball also lacks .svn/ directories (starting to get the picture?). All depressingly moronic
  • Lines 9-10 delete /usr/ports and recreate the directory
  • Line 11 checks out a copy of /usr/ports from the official FreeBSD SVN servers
  • Lines 12-14 are the same as 9-11 but for /usr/src and src
  • Please use svn:// and not http:// or https:// URI schemes — see below for why!
  • Line 15 fixes (what I consider to be) a security concern I have over /root/.subversion being globally readable. Not that I change anything in /root/.suvbersion/{config,servers} but some people might. This is the result of root’s default umask (0022), but I’m surprised svn doesn’t issue a chmod 0700 on the directory itself creates
  • Line 16 should be self-explanatory

It’s like I said — a simple procedure which is greatly complicated by many nuances that aren’t documented anywhere or explained by most.

For upgrading a system which already has a copy of /usr/ports, run what’s shown on line 2, then continue onwards starting at line 5. You can skip line 8 too, assuming portsnap has never been used prior.

Some other related topics or answers to questions I’ve seen asked (or asked myself!):

  • To update your ports tree, you can do any of the following commands — they are all effectively the same:
    cd /usr/ports && make update
    cd /usr/ports && svn update
    cd /usr/ports && svn checkout
    svn update svn:// /usr/ports
    svn checkout svn:// /usr/ports
  • To update your src tree, you can do any of the following commands — they are all effectively the same (note that “make update” can no longer be used):
    cd /usr/src && svn update
    cd /usr/src && svn checkout
    svn update svn:// /usr/src
    svn checkout svn:// /usr/src
  • For the ports INDEX-x file itself, just use the not-so-well-documented “fetchindex” target. However, since the security breach, the generation of this file on the official FreeBSD servers has broken many times (3 or 4 so far). This has been complained about repeatedly by users; in that case, the only choice is to generate it yourself, which takes a very long time and often emits lovely warnings. Anyway, using fetchindex is simple:
  • cd /usr/ports && make fetchindex
  • I do not recommend changing the URI scheme from svn:// to http:// or https://. There have been repeated complaints of issues/problems with the HTTP and HTTP/SSL services hosted at, by both users and committers (so many that I do not care to reference them all). Use native svn:// and be done with it; don’t cause yourself headaches using the other schemes. If you absolutely must switch the URI scheme and can deal with the annoyance, follow this procedure.
  • svn will never be added to the base system due to its large number of dependencies. There may also be licensing concerns with regards to some of those dependencies. I am one of the few people who wants to see the “base system” concept done away with entirely (everything should just be a package/port/whatever and managed that way), but the push-back I’ve received over the years has been from developers insisting that the base system must be kept to cater to commercial vendors that use FreeBSD (such as Juniper, NetApp, Citrix, etc.). The disconnect here is hilarious, and I doubt it will ever be fully addressed.
  • cvs will eventually be removed from the base system. I myself use cvs for my own projects; I imagine this won’t impact most people, but it does impact me. I’ll be forced to install devel/cvs from ports.
  • pkgng — if anyone mentions it to you — has absolutely no bearing/relevancy to the subject matter being discussed in this blog post. If someone tells you to install pkgng or switch to it to solve any of this, they are horribly confused and wrong. And besides, this right here is a good reason to avoid pkgng. Be sure to read the entire thread; also as of this writing there have been no follow-ups to it. Remarkable.
  • About these ads