Generate Nagios config for check_netapp-api.pl

As so often, I wanted a script, that’ll crawl my filers and regenerate the configuration if there are any new volumes/snapvaults/snapmirrors or if one of them has been removed.

Generate Nagios config for NetApp filers

At some point in the last few weeks, I repeatedly had to recreate my Nagios config for currently six filers. After doing that a few times, I ended up (like sooo often) writing a short Bash script, that’ll do this for me – without any fuss.

The only thing the script needs, is that the filers and the filers are registered in DNS … Here’s an example:

With that done, the script will create the necessary Nagios config for those filers.

Read More

NetApp: Monitoring of SnapVault/SnapMirror/LUN/Snapshot information with Nagios

As I wrote before, we have a bunch of filers (and a ton of volumes w/ luns on them), that I need to monitor. At first, I tried the existing NetApp Nagios-Plugin(s), but they all use SNMP and with that I can either watch all volumes or none. And that didn’t satisfy me.

Don’t get me wrong, the existing plugins are okay and I still use them for stuff (like GLOBALSTATUS or FAN/CPU/POWER) which isn’t present in the API or real hard to get at, however I wanted more. So I ended up looking at the NetApp API, and ended up writing a “short” plugin for Nagios using Perl.

Maybe if I’m ever bored, I’ll rewrite it using C, but for now the Perl plugin has to suffice.

So far the plugin supports the following things:

  • Monitoring FlexVolumes (simply watching the free space)
  • Monitoring LUN space (the allocated space inside a FlexVolume for iSCSI/FC LUNs)
  • Monitoring Snapshot space (the allocated space inside a FlexVolume for Snapshots)
  • Monitoring SnapVault relations (and their age)
  • Monitoring SnapMirror relations (and their age)

The plugin will return performance data for most (if not all) of those classes. It needs a user on the filer you wish to monitor – which sadly needs to have the admin role.

Read More

Monitoring Brocade FC switches with SNMP/Nagios

I looked into the mess a bit more, and as it turns out, the weird crap I was talking about only happens if you have a port with LossofSynchronization, LossofSignal or LinkFailures value with the base of ten (i.e. 10, 101 or 10.000).

Additionally, the OID’s for those three failure elements seem to be dependent on the firmware version, as with 6.3.x they appear as different OIDs. So I may need to introduce another command-line switch, which selects the firmware version and depending on that, the OID.

Even despite those problems I just described, I ended up using the plugin to watch our SAN infrastructure. I even wrote a simple pnp4nagios template, so all the data would show up in a single graph and not a graph per data source.

check_snmp_brocade_fcport Graph: 4 Hours
check_snmp_brocade_fcport Graph: 4 Hours

Monitoring Brocade FC switches with Nagios

The last four days I spent looking for ways on monitoring a Brocade Fibrechannel switch (in my case IBM 2145 B32/F40). The first thing I came up with, is using SNMP. As it was already configured for the previous monitoring with Munin, getting information should be quite easy. After looking through Google for a bit, there is already one script that worked for me.

Only trouble I had with that script, is that it crams every single port into one result. As I wanted something, that a) could watch a single port and b) return performance data, I went ahead an used the script to do a basic rewrite. But after a short while, I grew antsy and started writing a script from scratch, using the OIDs I got from that script and a Cacti template.

So far, I got a good plugin, but it’s still lacking a few things:

  1. Support for warning/critical thresholds for each error category
  2. Sadly the important errors (er_link_fail, er_loss_sync and er_loss_sig) are kept in a separate table structure (swEndDeviceRlsEntry), which I can’t seem to access right now; even though the entries are mandatory and according to the MIB should be at least read-only.
  3. The plugin isn’t doing a proper $session->close(); . After moving the snmp stuff into a subroutine, Perl refuses to do the session closing. Don’t know why right now.

Right now, the plugin supports two modes. The first just checks if the port is operational and in sync and the second checks the port status, but also returns the performance data.

Only do a basic check if the Port is in operational status

Check the port status, but also return performance data

That might look like much, but Nagios is gonna pass everything after “|” to your performance data command.

List of OIDs, which hold the various information:

The last three OIDs, as well as the ones in FCMGMT-MIB (as I mentioned in the TODO), sadly don’t exist (or I’m doing something wrong ? — no clue right now), so I can’t incorporate them into the script at this time.

However, I found something in a separate OID-tree (also the FCMGMT-MIB), which seems to be exactly what I’m looking for.

Only trouble with those OIDs is, that they are OCTET STRING’s, which right now just return crap (either nothing or just a new-line) with my script. Gonna have to work on that.

If you’re interested in the Perl script (for now, lacking some options, performance data, $session->close();), you’ll find it here.

Configuring nagios-plugins-zypper

Since I’m running check_zypper via nrpe (which in turn runs as nobody), I need to set up sudo. In order for the plugin to work, we need to add the following line to /etc/sudoers (by means of visudo):

(Keep in mind this needs to be a single line …)

Praxisbuch Nagios by Tobias Scherbaum

Tobi recently finished writing yet another book, which he also talked about in a blog post. Shortly after, I asked him a rather curious question. What exactly is the plant or animal on the cover of the book ? He was kind enough to send a voucher copy of the book my way.

Praxisbuch Nagios
Praxisbuch Nagios

He actually mentions it in the credits at the beginning of the book. Turns out it is an animal, a sea pen or sea feather (I’m guessing at Pennatula aculeata).

Now as for the content of the book itself, I do have to admit that I haven’t read the whole book. I just picked a few topics (SNMP-Traps with Nagios, notifications) which I did find rather well written. My (soon ex-) trainee, Michel, however already bugged Tobias about some errors in the book itself, or rather some changes which happened after 3.0.3 (that’s the Nagios version the book is based on).

All in all, I guess I can congratulate Tobias on yet another good written book!

Nagios: Service Check Timed Out

Since I got the pleasure of watching some Windows boxen with Nagios, I took the Windows Update plugin from Michal Jankowski and implemented it. It took me some time, to initially set up the nsclient++ correctly so it just works, but up till now the check plugin sometimes reported the usual “Service Check Timed Out”.

Usually I ended up increasing the cscript timeout, or the nsclient++ socket timeout, but it still kept showing up. Since I rely heavily on my surveillance tools, I have the demand, that as few as possible false positives show up. So I ended up chasing down this error today, and after that I have to say it was quite simple.

In my case, it wasn’t cscript (that timeout is set to 300 seconds), neither nsclient++ (socket timeout is set to 300 seconds too), nor the nrpe plugin itself (that has 300 seconds as well).

As it turns out, Nagios got an additional setting controlling these things, called service_check_timeout which defaults to 60 seconds. Sadly the plugin, or rather Windows needs longer than those 60 seconds to figure out whether or not it needs updating, thus Nagios is killing the plugin and returning a CRITICAL message.

After increasing the value of service_check_timeout that’ll be fixed hopefully.

Nagios: SNMP OID’s for IBM’s RSA II adapter

Well, after some poking around I finally found some OID’s for the RSA’s (only through these two links: check_rsa_fan and check_rsa_temp).

For Nagios, I dismissed the fans, since the fan speed is only passed on in percent values. So I only added this:

Oh, and if anyone else is curious like me, here’s the list with the OID’s, courtesy of Gerhard Gschlad and Leonardo Calamai.

For the fans:

And for the temperatures:

I just found a proper list of OID’s for the IBM RSA adapter. That’s rather nice, since I really was looking for the OID’s for the VRM failure OID and other warning/critical events.

Nagios: check_snmp again

Well, today I had to grind my head again, regarding the way check_snmp handles WARNING and CRITICAL events. From my point of view, check_snmp is really just retarded sometimes.

As you know, all the other plugins accept WARNING and CRITICAL-thresholds based on the calculation, if the return integer is above this threshold it reached WARNING/CRITICAL state. But check_snmp doesn’t play that way.

It expects only ranges, which are NOT gonna result in warning or critical events. Which is kinda stupid, since you gotta rethink twice about the thresholds 😛

All in all, another lesson learned 😮