Dealing with SnapVault replication issues

Well, for the past two months I had a case open with NetApp to figure out this SnapVault replication issue we were seeing. The initial transfer of the SnapVault relation would complete with a hick up, manual snapshot transfers also work – just the scheduled, auto-created Snapshots won’t replicate.

At first I (and the NetApp support) thought this was an issue with SnapVault itself, however after being away for the last four weeks I looked at the issue with fresh eyes. After a short peek into the logs, I found what I had found back when I first looked into this.

SnapVault would create the daily snapshot on the SnapVault Primary and start the replication. However something (or someone, wasn’t clear at this point) then created a FlexClone of a volume … And as, back when we first encountered this, I was kinda puzzled.

But then I decided (please don’t ask me what made me look there) to look at the logs of the NetApp Filer on our logserver. As it turns out, back when I enabled syslogging to an external logserver I seem to have enabled debug logging … and it was great to have that! Below you’ll find the log I found – and as you can see there’s at least a clue as to from where that ghost snapshot is coming from.

Now, with knowing from which corner this issue originated it dawned on me, we have had a similar issue before. A quick peek into TSM Manager and I knew I was on the right track. The daily system backup starts around 21:15. Now our TSM backup includes the System State backup (which in turn utilizes VSS – which triggers the NetApp Snapshot!).

After excluding the System State from the Daily Backup the SnapVault stuff worked without a hickup. I ended up removing SnapDrive from the Server in question, since we don’t really need it there. Snapshots created from SnapDrive of the boot lun are gonna be inconsistent anyhow (doesn’t matter if I do ‘em from SnapDrive or the NetApp CLI).

That restored the default VSS handler, which enables TSM to backup the System State again.

Synology: New openvpn init script

My VPN provider isn’t being supported by the Synology VPN client (because they aren’t using the standard port 1194, instead 1195). After tinkering with the ovpn files the Synology VPN client uses to store the connection settings (and failing), I just installed openvpn with ipkg.

However after tinkering around with the init-script provided by the openvpn ipkg from the NSLU2 feed, I got tired and just rewrote the damn thing:

Place the file in /opt/etc/init.d/S20openvpn, and the client daemon will start on boot.

Synology DS213+: Disable Blue Status LED 4

Well, the title pretty much says what I want to do. Even if the DS213+ is on top of a living room cupboard, it’s still way to bright .. as I don’t need really need the LED, here’s a quick hack (the ID and the device is taken from the Synology Wiki) on how to disable it on each startup:


Just place the file in /opt/etc/init.d/S01leds and the script is gonna turn the LED off during a boot/reboot.

Generate Nagios config for

As so often, I wanted a script, that’ll crawl my filers and regenerate the configuration if there are any new volumes/snapvaults/snapmirrors or if one of them has been removed.

Generate Nagios config for NetApp filers

At some point in the last few weeks, I repeatedly had to recreate my Nagios config for currently six filers. After doing that a few times, I ended up (like sooo often) writing a short Bash script, that’ll do this for me – without any fuss.

The only thing the script needs, is that the filers and the filers are registered in DNS … Here’s an example:

With that done, the script will create the necessary Nagios config for those filers.


NetApp: Establishing SnapMirror relationships

After figuring out the SnapVault stuff, I needed to implement a whole bunch of SnapMirror relations. As I am lazy (as in click-lazy), I ended up writing a somewhat short Bash script, that’ll either establish a bunch of SnapMirror relations (for a single host) or just for a single volume.

The script expects, that SSH public key authentification has been set up, and that the source for the SnapMirror exists and is online/not-restricted.


NetApp: Establishing SnapVault relations 1

I’ve been spending a lot of my time the last week on getting SnapVault with out FAS-filers to work. Out came a script, which does this for a given volume (and of course SnapVault Primary and Secondary).

The script expects, that SSH public key authentification has been set up.


NetApp: SnapVault snapshot retention for non-standard snapshot names

Well, the name says it pretty much. Once you rename the snapshot on the SnapVault destination from daily.0 to something else, the whole builtin SnapVault snapshot retention isn’t gonna work anymore.

Back when I started all the code-writing, I wasn’t aware of this. One of my co-worker complained to me about it on Wednesday that there are an assfull of snapshots on the SnapVault destination (one snapshot each day since the end of October, meaning more than 50 snapshots per volume, in a total of 12 or so FlexVolumes, making the total about 500 snapshots).

So I took the time to write this little Bash script (yeah, I know I’m mixing a bunch of languages – I really like the KISS principle), which will get the necessary information from the filer (snapvault snap sched needs to be set) and then deletes the over-aged snapshots.


NetApp: Monitoring of SnapVault/SnapMirror/LUN/Snapshot information with Nagios 4

As I wrote before, we have a bunch of filers (and a ton of volumes w/ luns on them), that I need to monitor. At first, I tried the existing NetApp Nagios-Plugin(s), but they all use SNMP and with that I can either watch all volumes or none. And that didn’t satisfy me.

Don’t get me wrong, the existing plugins are okay and I still use them for stuff (like GLOBALSTATUS or FAN/CPU/POWER) which isn’t present in the API or real hard to get at, however I wanted more. So I ended up looking at the NetApp API, and ended up writing a “short” plugin for Nagios using Perl.

Maybe if I’m ever bored, I’ll rewrite it using C, but for now the Perl plugin has to suffice.

So far the plugin supports the following things:

  • Monitoring FlexVolumes (simply watching the free space)
  • Monitoring LUN space (the allocated space inside a FlexVolume for iSCSI/FC LUNs)
  • Monitoring Snapshot space (the allocated space inside a FlexVolume for Snapshots)
  • Monitoring SnapVault relations (and their age)
  • Monitoring SnapMirror relations (and their age)

The plugin will return performance data for most (if not all) of those classes. It needs a user on the filer you wish to monitor – which sadly needs to have the admin role.


NetApp: Archive SnapManager SQL Snapinfo

The MSSQL admins decided to dump the SMSQL Snapinfo stuff on a separate volume, that SMSQL also snapshots. Same as before, I need a PowerShell script that’ll archive the snapshot and rename it.