Systems monitoring nightmares on FreeBSD

For quite some time (years) I’ve been dealing with monitoring of FreeBSD systems. “Monitoring” in this case does not mean service availability, it means data/statistics acquisition of key parts of the system — things like memory and VM usage, CPU load, pf (firewall) statistics, NIC statistics, disk usage, disk I/O, and so on.

You still have to store the acquired data somewhere/somehow. And that’s where RRDTool, unfortunately and unjustifiably, comes into play. If you’re an administrator (or developer) that has to deal with systems monitoring and statistics data acquisition in an open-source world, there’s a very good chance (~90%) that you’ve had to painfully deal with this software.

My use of the word “deal” is a bit vague. “Deal” means actually deal with RRDTool, not “install it and use it blindly”. I’m referring to getting into the intricacies and annoyances of rrdcreate, rrdgraph, and rrddump — especially in the case where existing data is already stored.

You see, the problems with RRDTool are almost limitless. The frustration level is astounding; literally every system administrator I have spoken to about software that relies on RRDTool has spewed forth nothing but raw obscenities intermixed with questions/comments. A past colleague of mine once said “If I ever meet the author of RRD, Tobias Oetiker, I’m going to hurt him”. This is a sentiment I can relate to. The most common comments/questions I see are:

  • What happened to my data?! That value wasn’t what I inserted into the database…
  • Why do these graphs look wrong?
  • Where are these NaN values coming from?
  • What do you mean I’ll lose all my data when moving between i386 and amd64, or i386/amd64 and sparc?
  • What do you mean I can’t easily add another data field to my CF without deleting my existing data?
  • Why is this monitoring software that uses RRDTool chewing up so much disk I/O?
  • Why are all my values, when dumped using “rrdtool dump”, shown in exponential format rather than something sane, and how do I get them to be sane?
  • How do I get the grapher to stop using irrelevant units and show exactly what I want?
  • What the hell? “Ordering CD from Amazon”?!

Then there’s the software dependency nightmare. RRDTool version 1.3 and later relies on an outrageous number of graphing and text/font libraries; on FreeBSD, ports/databases/rrdtool pulls in 51 dependencies, including some X11 bits (ed. as of 2015/03/13, this number has bloated to 68). That’s 51 pieces of software just for what should be a “simple database with graphing and basic layout capabilities”. RRDTool version 1.2 (ports/databases/rrdtool12) only has 9 dependencies, almost all of which are image format libraries and a text rendering library (freetype2) (ed. as of 2015/03/13, this number has bloated to 26; I have not been the port maintainer since 2008/12/19). System administrators often refuse to install RRDTool due to the horrible dependency count. I myself have stuck with RRDTool 1.2 for quite some time because of this. I also used to be the FreeBSD port maintainer for ports/databases/rrdtool12, fancy that.

And God forbid you encounter a bug in it — read the CHANGES file sometime. Some of the bugs are astonishing. And for additional amazement, read the mailing list.

Those of us who despise RRDTool want something that doesn’t mess about with your data, doesn’t result in a mesh of software dependency nightmares, performs well, and “just works” — all while trying to comply with the KISS principle as much as possible.

Except there isn’t anything that meets these simple criteria. Instead, what’s out there is a disgrace, often in other ways. Everything violates the concept of KISS as much as possible — why do I need an aircraft carrier if I’m only travelling a couple miles? FreeBSD administrators tend to apply KISS as much as possible; it’s less commonly-applied on Linux, for whatever reason.

What I’ve wanted for years can be described as follows:

A daemon with a reasonable memory footprint that polls hosts/devices using SNMP (version 2) and writes the acquired data to CSV format text files. No graphing is required; that can be done client-side (or with Timeplot if you prefer). Scalability is a must. Use of 64-bit SNMP counters should be applied wherever possible.

Sounds simple doesn’t it? Indeed — except in the open-source world, you won’t find it. What you will find are aircraft carriers, cobbled together with glue and sticks.

The closest thing I’ve found is a decent piece of software called rrdbot, which meets all of the above requirements (and guess what it uses for its SNMP client code? A copy of FreeBSD’s bsnmp library API) — except it uses RRDTool for storage. I’ve used rrdbot for years with success and reliability, but when it comes to adding a new data source, the travesty that is RRDTool kicks in. rrdbot itself is okay — the underlying code and framework could really use some cleaning up. But could it be modified to support CSV? Yes, and that is something I’ve begun to work on, but it’s extremely complex given how integrated RRD is within the software, all the way down to the configuration file format. It’s not a simple task.

Which brings me to an alternative that many Linux users have told me about over the years: collectd. Because “it supports CSV“.

Last night I spent 6-7 hours looking at it and doing my best to implement it. collectd’s SNMP support is a sad joke, both on a configuration level as well as a technical/implementation level. Let’s talk about it.

Firstly, the documentation for the SNMP plugin starts off with this amazing quote:

SNMP is a widespread standard to provide management data from devices such as switches, routers, rack monitoring systems, uninterruptible power supplies (UPS), etc. While theoretically possible, collecting values from other computers via this protocol is discouraged in favor of collectd’s own protocol

Yeah, why use an existing and admitted widespread standard? Who would ever want that!? Just let me know when collectd can run on our HP ProCurve switches, APC AP7900 RackPDUs, and MRV LX-series serial console units… Regardless of the idiotic justification for pushing a proprietary protocol over a standard one, I continued working with collectd.

Secondly, the configuration file format does not “play well” with SNMP. The official documentation doesn’t touch on design choice nuances, probably because they become apparent to any senior administrator attempting to accomplish said task. What am I talking about? Let’s examine these error lines:

[2011-09-13 17:02:43] snmp plugin: DataSet `counter' requires 1 values, but config talks about 6
[2011-09-13 17:02:43] read-function of plugin `snmp-machine.box.lan' failed. Will suspend it for 20 seconds.

These were from starting collectd with the following SNMP plugin configuration bits:

<Plugin snmp>
  <Data "tcp">
    Type "counter"
    Table false
    Instance ""
    Values "TCP-MIB::tcpActiveOpens.0"   \
           "TCP-MIB::tcpPassiveOpens.0"  \
           "TCP-MIB::tcpAttemptFails.0"  \
           "TCP-MIB::tcpEstabResets.0"   \
           "TCP-MIB::tcpCurrEstab.0"     \
           "TCP-MIB::tcpInErrs.0"
  </Data>
  <Host "machine.box.lan">
    Address "machine.box.lan"
    Version 2
    Community "public"
    Collect "tcp"
  </Host>
</Plugin>

I want to point out in advance that this configuration is not entirely correct nor should people try to use it. Specifically, the tcpCurrEstab MIB returns a GAUGE (i.e. a rate), not a COUNTER. I knew this in advance but was simply messing about with collectd’s SNMP support.

Anyway, the first thing that caught my attention was the badly-formatted error line relating to what plugin failed. The Host is not called snmp-machine.box.lan — the “snmp-” part comes from the collectd itself; it’s indicating “the snmp plugin”. Why this error line cannot be changed to “read-function of plugin snmp, host machine.box.lan failed” is beyond me. Please don’t tell me to fix it myself — I would much rather not look at the error handling code for collectd, despite being quite able to.

The dataset called “counter” comes from collectd’s own types.db file (which is, and I quote, “inspired by RRDtool’s data-source specification” — just shoot me now), which only permits 1 argument handed to the Data directive. For 6 arguments, one would need to make a new entry in types.db that allowed for such. But then if you added a new data argument, you would have to edit types.db to extend things, and any collectd.conf bits that used that type. Lather rinse repeat until you’re blue in the face.

To try and work around this absurdity I attempted to use the Type directive to specify multiple “counter” arguments, e.g.:

<Data "tcp">
  Type "counter"  \
       "counter"  \
       "counter"  \
       "counter"  \
       "counter"  \
       "counter"
  ...
</Data>

But was immediately shot down by syntactical limitations of the Type directive itself:

[2011-09-13 17:05:50] snmp plugin: `Type' needs exactly one string argument.

Wonderful. So what we have here is a glaringly obvious design choice limitation, by someone who wasn’t really thinking about what they were doing. The only “solution” is to give every single SNMP OID its own Data section. That isn’t acceptable for two reasons:

  1. Data sections are defined per-Host-section, which means if you had 10 hosts and approximately 25 OIDs per host you wanted to monitor, that’s 250 blocks, and those are usually 10-15 lines each
  2. Under the hood collectd does not aggregate multiple OIDs into a single SNMP GET statement (protocol-wise) — it issues one at a time. On the SNMP server side, this is extremely inefficient and rude

So really the entire situation could get solved if the configuration syntax ordeals were dealt with appropriately. Look at rrdbot’s method — it makes sense, it just has the evil of RRDTool associated with it:

[poll]
interval:            30
activeOpens.source:  snmp2c://public@machine.box.lan/tcpActiveOpens.0
passiveOpens.source: snmp2c://public@machine.box.lan/tcpPassiveOpens.0
attemptFails.source: snmp2c://public@machine.box.lan/tcpAttemptFails.0
estabResets.source:  snmp2c://public@machine.box.lan/tcpEstabResets.0
currEstab.source:    snmp2c://public@machine.box.lan/tcpCurrEstab.0
inErrs.source:       snmp2c://public@machine.box.lan/tcpInErrs.0

[create]
activeOpens.type:    COUNTER
activeOpens.min:     0
passiveOpens.type:   COUNTER
passiveOpens.min:    0
attemptFails.type:   COUNTER
attemptFails.min:    0
estabResets.type:    COUNTER
estabResets.min:     0
currEstab.type:      GAUGE
currEstab.min:       0
inErrs.type:         COUNTER
inErrs.min:          0
cf:                  AVERAGE

archive:   2/minute * 2 days,
           30/hour * 1 month,
           2/hour * 1 year

There are also multiple design problems with the collectd CSV plugin, not to mention a bad buffer size choice too.

Next, collectd on FreeBSD pulls in a bunch of third-party library dependencies. For example, to get network I/O statistics it relies on a library called libstatgrab even though collectd has getifaddrs(3) support natively. But if you want disk I/O statistics, you need libstatgrab because collectd doesn’t have the native code for obtaining such on the BSDs. And once libstatgrab is installed, it’ll use it for network I/O statistics as well. I can stomach this, but I’m not happy about it.

Linux folks have access to the almost all of this through /proc filesystem — which may not be the best place for such statistics, but implementation-wise a filesystem providing such is very much the true UNIX way, which in my opinion makes this method more UNIX-like than the BSDs.

FreeBSD offers nothing like Linux /proc. Systems data acquisition on FreeBSD involves either libc or syscall functions, which means there’s only one language that you can use: C. Calling command-line utilities is not an option — waste of CPU and memory resources due to excessive fork/exec, spawning of shells, etc. is unreasonable, and any administrator worth a quarter of his salary won’t permit it.

So on FreeBSD the best thing we’ve got is an SNMP daemon called bsnmpd(8). But most of the statistics a person might want aren’t provided, requiring one to install ports/net-mgmt/bsnmp-ucd, a shared library that makes available the more commonly-desired OIDs and MIBs for host statistics. FreeBSD bsnmpd is an alternate to net-snmp’s daemon, but much more bare-bones. The daemon has a minimal memory footprint as well. But bsnmpd can return questionable data for some OIDs.

collectd also makes use of a third-party dlopen(3) wrapper called libltdl that comes from its reliance on libtool. There are patches that address this awful choice, but once again, patches upon patches is the nightmare that FreeBSD users generally do not tolerate. I remember my Linux days vividly — patches atop patches atop patches, many of which conflicted with one another. No thanks.

It seems I’m not the only one who wants rrdtool’s graphing nonsense to die a horrible death. Of course, converting RRD data into JSON is probably a waste of time (my opinion is that the individual in the thread is trying to solve the problem at the wrong layer) — CSV would suffice, but I guess JSON would be “okay”. Anyone up for ASN.1? :-)

Finally, I want to talk a bit about my focus on CSV, because RRDTool advocates often point out how CSV does not scale when it comes to long-term data storage or trending. This is true — CSV doesn’t scale. It’s just a text log of comma-delimited values with timestamps. And given that client-side graphing software tends to lean towards CSV, network bandwidth quickly becomes a concern given that a year’s worth of data could result in a multi-megabyte CSV file. Here’s a write-up of mine on the matter and how to solve it efficiently:

One of the downsides to using CSV is that the file can become extremely large depending on how much data is within the file. This is something RRDTool solves by using RRAs; this is not something CSV can solve without implementing a CSV parser and creating “averaged” CSVs which can be read by the software. This might be a worthwhile enhancement, but is currently not implemented.

For example: if your polling interval is once a day, and you’re polling only 2 OIDs, your data file is going to be quite small: maybe 12KBytes for an entire year’s worth of data. That’s 2*24*30*12, or 17280 pieces of data (2 OIDs times 24 hours times 30 days times 12 months).

However, if your polling interval is once every minute, this gets much worse: 2*60*24*30*12, or 1036800 pieces of data (2 OIDs times 60 minutes per hour times 24 hours times 30 days times 12 months). That’s 60 times more data! 12000 bytes * 60 = 720,000 bytes (~700KBytes).

Even on a fast internet connection, 700KBytes of data might take a while. Plus there’s the parsing time by dygraphs, HTTP protocol overhead, and so on. And just as important is bandwidth usage; this could really hurt if you’re on a 95th-percentile connection.

So how do we solve both of these problems?

Web servers like Apache offer server-side gzip compression; in the Apache world this is called mod_deflate. The browser has to support this too, of course, but all present-day browsers do, as well as some previous-generation browsers. It’s been around for a while.

The HTTP server will compress the data sent to the client in real-time. What’s important to keep in mind is that CSV data is ASCII and very repetitious in nature — it compresses **extremely** well.

For example, with mid-level gzip compression (level 4), a 950KByte CSV file compresses down to around 9-10KBytes. Really! So as long as the web server environment is configured to use gzip compression when serving CSV files to the browser (when viewing graphs), and as long as the browser supports gzip compression, we have very little to worry about.

The trade-off is that the HTTP server may have a slightly higher load. As long as thousands of people aren’t pounding the graphing page all at the same time, this shouldn’t be a problem; most present-day processors are amazingly fast, and zlib is extremely optimised.

So use mod_deflate, make sure your webserver is configured to use it for CSV files (client should send an Accept-Encoding HTTP header with gzip/deflate provided, and the server should respond with a Content-Encoding header with gzip provided). There’s an old write-up at LinuxJournal which can help.

I guess for now I’ll stick with using rrdbot and gritting my teeth over RRDTool, until I can finish writing the necessary CSV code bits for rrdbot. Stef Walter, the author of rrdbot, seems to have gone MIA so I wouldn’t even be able to submit patches upstream and have them implemented in a timely manner.

Are we having fun yet? Yeah, me neither.