April 2009 – BAFM

TSM: Restoring the database/recovery log to a point-in-time

April 24, 2009June 21, 2013 Christian Leave a comment

Well, my co-worker just called on my cell (it’s Friday, 16:00), and asked me which start-up script he needed to change in order to restore the database. My first response was, “ummm, that’s gonna be hard, we’re using heartbeat”.

Okay, so after a bit of asking I got out of him what he wanted to achieve by changing the start-up script. Apparently he did something to crash Tivoli Storage Manager (or rather repeatedly crash it) and wanted to restore the database. He talked to one of the systems partner we do have (and I’m happy we have them most of the time), who in return told him how to do it, but forgot a minute after he hung up the phone.

So, I went digging while he still was telling me how he got Tivoli to kick his own ass … After a bit, I thought “hrrrrrm, shouldn’t this be covered in the Tivoli documentation ?”, and surprisingly it’s actually covered in the documentation.

It’s actually rather simple.

Stop the dsmserv Linux-HA cluster service (tsm-control ha stop tsm1)
Setup the environment (since we’re running multiple instances of Tivoli Storage Manager – export DSMSERV_DIR, export DSMSERV_CONFIG)
Enter the path of the server
Run dsmserv restore db
Wait some time (took about half an hour to restore the 95G database and the 10G recovery log)
Start the dsmserv Linux-HA cluster service (tsm-control ha start tsm1)
Update the server-to-server communication, since the restore db changes the communication verification token

&gt; tsm-control ha stop tsm1
  - tsm1 (dsmserv) -&gt; ha: [ OK ]
&gt; export DSMSERV_DIR=/opt/tivoli/tsm/server/bin
&gt; export DSMSERV_CONFIG=/opt/tivoli/tsm/server/tsm1/dsmserv.opt
&gt; cd /opt/tivoli/tsm/server/tsm1
&gt; /opt/tivoli/tsm/server/bin/dsmserv restore db todate=TODAY totime=08:00:00 source=dbbackup preview=no
.... wait some time ....
&gt; tsm-control ha start tsm1
  - tsm1 (dsmserv) -&gt; ha: [ OK ]

> tsm-control ha stop tsm1

- tsm1 (dsmserv) -> ha: [ OK ]

> export DSMSERV_DIR=/opt/tivoli/tsm/server/bin

> export DSMSERV_CONFIG=/opt/tivoli/tsm/server/tsm1/dsmserv.opt

> cd /opt/tivoli/tsm/server/tsm1

> /opt/tivoli/tsm/server/bin/dsmserv restore db todate=TODAY totime=08:00:00 source=dbbackup preview=no

.... wait some time ....

> tsm-control ha start tsm1

- tsm1 (dsmserv) -> ha: [ OK ]

Nagios: Service Check Timed Out

April 3, 2009June 21, 2013 Christian 6 Comments

Since I got the pleasure of watching some Windows boxen with Nagios, I took the Windows Update plugin from Michal Jankowski and implemented it. It took me some time, to initially set up the nsclient++ correctly so it just works, but up till now the check plugin sometimes reported the usual “Service Check Timed Out”.

Usually I ended up increasing the cscript timeout, or the nsclient++ socket timeout, but it still kept showing up. Since I rely heavily on my surveillance tools, I have the demand, that as few as possible false positives show up. So I ended up chasing down this error today, and after that I have to say it was quite simple.

In my case, it wasn’t cscript (that timeout is set to 300 seconds), neither nsclient++ (socket timeout is set to 300 seconds too), nor the nrpe plugin itself (that has 300 seconds as well).

As it turns out, Nagios got an additional setting controlling these things, called service_check_timeout which defaults to 60 seconds. Sadly the plugin, or rather Windows needs longer than those 60 seconds to figure out whether or not it needs updating, thus Nagios is killing the plugin and returning a CRITICAL message.

After increasing the value of service_check_timeout that’ll be fixed hopefully.

SLES10: zypper.log

April 3, 2009June 21, 2013 Christian Leave a comment

Well, I just stumbled upon something .. My Nagios at work wasn’t working anymore, and I went looking.

nagios3 ~ [0] &gt; tail -f /var/log/nagios/nagios.log
[1238658394] Error: Unable to save status file: No space left on device
[1238658403] Error: Unable to save status file: No space left on device
[1238658413] Error: Unable to save status file: No space left on device
[1238658423] SERVICE ALERT: tsm1;POWER WARN;OK;SOFT;4;-u OK - 0
[1238658423] Error: Unable to save status file: No space left on device
[1238658433] SERVICE ALERT: tsm2;LOAD;WARNING;SOFT;1;WARNING - load average: 6.25, 5.72, 5.36
[1238658433] Error: Unable to save status file: No space left on device
[1238658443] Error: Unable to save status file: No space left on device
[1238658453] Error: Unable to save status file: No space left on device
[1238658463] Error: Unable

nagios3 ~ [0] > tail -f /var/log/nagios/nagios.log

[1238658394] Error: Unable to save status file: No space left on device

[1238658403] Error: Unable to save status file: No space left on device

[1238658413] Error: Unable to save status file: No space left on device

[1238658423] SERVICE ALERT: tsm1;POWER WARN;OK;SOFT;4;-u OK - 0

[1238658423] Error: Unable to save status file: No space left on device

[1238658433] SERVICE ALERT: tsm2;LOAD;WARNING;SOFT;1;WARNING - load average: 6.25, 5.72, 5.36

[1238658433] Error: Unable to save status file: No space left on device

[1238658443] Error: Unable to save status file: No space left on device

[1238658453] Error: Unable to save status file: No space left on device

[1238658463] Error: Unable

After that, zip – nada. Next thing, check whether or not the device is really full … Okay, df ..

nagios3 ~ [130] &gt; df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             3.5G  1.2G  2.1G  37% /
udev                  506M   88K  506M   1% /dev
/dev/sdb1             7.9G  7.7G     0 100% /var

nagios3 ~ [130] > df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 3.5G 1.2G 2.1G 37% /

udev 506M 88K 506M 1% /dev

/dev/sdb1 7.9G 7.7G 0 100% /var

So, it is actually completely filled up. So, now we need to find who’s hogging the space. Since I had a assumption (pnp4nagios), I went straight for /var/lib …

nagios3 lib [0] &gt; du -sh *
16K     CAM
1.1M    YaST2
8.0K    acpi
4.0K    apache2
28K     autoinstall
16K     dhcpcd
4.0K    empty
96K     hardware
4.0K    logrotate.status
8.0K    misc
78M     mysql
2.1M    nagios
4.0K    net-snmp
4.0K    news
24K     nfs
8.0K    nobody
36K     ntp
4.0K    pam_devperm
824K    php5
359M    pnp4nagios
22M     rpm
28K     scpm
4.0K    smpppd
4.0K    sshd
4.0K    support
8.0K    suseRegister
4.0K    uniconf
4.0K    update-messages
4.0K    wwwrun
33M     zmd
14M     zypp

nagios3 lib [0] > du -sh *

16K CAM

1.1M YaST2

8.0K acpi

4.0K apache2

28K autoinstall

16K dhcpcd

4.0K empty

96K hardware

4.0K logrotate.status

8.0K misc

78M mysql

2.1M nagios

4.0K net-snmp

4.0K news

24K nfs

8.0K nobody

36K ntp

4.0K pam_devperm

824K php5

359M pnp4nagios

22M rpm

28K scpm

4.0K smpppd

4.0K sshd

4.0K support

8.0K suseRegister

4.0K uniconf

4.0K update-messages

4.0K wwwrun

33M zmd

14M zypp

That wasn’t it .. so heading to the next place, that’s suspicious most of the time, /var/log.

nagios3 log [0] &gt; du -sh *
5.2G    YaST2
4.0K    acpid
1.4G    apache2
28K     boot.msg
28K     boot.omsg
4.0K    cups
4.0K    dsmerror.log
148K    dsmsched.log
4.0K    faillog
4.0K    krb5
12K     lastlog
4.0K    localmessages
16K     mail
16K     mail.info
198M    messages
0       mysqld.log
14M     nagios
0       ntp
4.0K    pnp4nagios
4.0K    sa
8.0K    scpm
4.0K    vmdesched.log
16K     vmware-imc
4.0K    vmware-tools-guestd
82M     warn
348K    wtmp
115M    zmd-backend.log
24M     zmd-messages.log

nagios3 log [0] > du -sh *

5.2G YaST2

4.0K acpid

1.4G apache2

28K boot.msg

28K boot.omsg

4.0K cups

4.0K dsmerror.log

148K dsmsched.log

4.0K faillog

4.0K krb5

12K lastlog

4.0K localmessages

16K mail

16K mail.info

198M messages

0 mysqld.log

14M nagios

0 ntp

4.0K pnp4nagios

4.0K sa

8.0K scpm

4.0K vmdesched.log

16K vmware-imc

4.0K vmware-tools-guestd

82M warn

348K wtmp

115M zmd-backend.log

24M zmd-messages.log

I was like “WTF ? 5.2G for YaST2 logs ?” when I initially saw that output … As of now, I got a crontab emptying /var/log/YaST2 every 24 hours …

Nagios: SNMP OID’s for IBM’s RSA II adapter

April 1, 2009June 21, 2013 Christian Leave a comment

Well, after some poking around I finally found some OID’s for the RSA’s (only through these two links: check_rsa_fan and check_rsa_temp).

For Nagios, I dismissed the fans, since the fan speed is only passed on in percent values. So I only added this:

define hostgroup{
  hostgroup_name                  rsa-snmp
  alias                           Remote Supervisor Adapter (allowing SNMP connections)
}

define service{
  use                             generic-perfdata

  check_command                   check_rsa_snmpv1_public!.1.3.6.1.4.1.2.3.51.1.2.1.2.1.1!45!60!°C!Temperature CPU0!
  hostgroup_name                  rsa-snmp
  service_description             TEMP CPU0
}

define service{
  use                             generic-perfdata

  check_command                   check_rsa_snmpv1_public!.1.3.6.1.4.1.2.3.51.1.2.1.2.2.1!45!60!°C!Temperature CPU1!
  hostgroup_name                  rsa-snmp
  service_description             TEMP CPU1
}

define service{
  use                             generic-perfdata

  check_command                   check_rsa_snmpv1_public!.1.3.6.1.4.1.2.3.51.1.2.1.5.1.0!29!35!°C!Temperature Ambient!
  hostgroup_name                  rsa-snmp
  service_description             TEMP AMBIENT
}

define hostgroup{

hostgroup_name rsa-snmp

alias Remote Supervisor Adapter (allowing SNMP connections)

}

define service{

use generic-perfdata

check_command check_rsa_snmpv1_public!.1.3.6.1.4.1.2.3.51.1.2.1.2.1.1!45!60!°C!Temperature CPU0!

hostgroup_name rsa-snmp

service_description TEMP CPU0

}

define service{

use generic-perfdata

check_command check_rsa_snmpv1_public!.1.3.6.1.4.1.2.3.51.1.2.1.2.2.1!45!60!°C!Temperature CPU1!

hostgroup_name rsa-snmp

service_description TEMP CPU1

}

define service{

use generic-perfdata

check_command check_rsa_snmpv1_public!.1.3.6.1.4.1.2.3.51.1.2.1.5.1.0!29!35!°C!Temperature Ambient!

hostgroup_name rsa-snmp

service_description TEMP AMBIENT

}

Oh, and if anyone else is curious like me, here’s the list with the OID’s, courtesy of Gerhard Gschlad and Leonardo Calamai.

For the fans:

Fan1: .1.3.6.1.4.1.2.3.51.1.2.3.1.0
Fan2: .1.3.6.1.4.1.2.3.51.1.2.3.2.0
Fan3: .1.3.6.1.4.1.2.3.51.1.2.3.3.0
Fan4: .1.3.6.1.4.1.2.3.51.1.2.3.4.0
Fan5: .1.3.6.1.4.1.2.3.51.1.2.3.5.0
Fan6: .1.3.6.1.4.1.2.3.51.1.2.3.6.0
Fan7: .1.3.6.1.4.1.2.3.51.1.2.3.7.0
Fan8: .1.3.6.1.4.1.2.3.51.1.2.3.8.0
Fan9: .1.3.6.1.4.1.2.3.51.1.2.3.9.0
Fan10: .1.3.6.1.4.1.2.3.51.1.2.3.10.0
Fan11: .1.3.6.1.4.1.2.3.51.1.2.3.11.0
Fan12: .1.3.6.1.4.1.2.3.51.1.2.3.12.0

Fan1: .1.3.6.1.4.1.2.3.51.1.2.3.1.0

Fan2: .1.3.6.1.4.1.2.3.51.1.2.3.2.0

Fan3: .1.3.6.1.4.1.2.3.51.1.2.3.3.0

Fan4: .1.3.6.1.4.1.2.3.51.1.2.3.4.0

Fan5: .1.3.6.1.4.1.2.3.51.1.2.3.5.0

Fan6: .1.3.6.1.4.1.2.3.51.1.2.3.6.0

Fan7: .1.3.6.1.4.1.2.3.51.1.2.3.7.0

Fan8: .1.3.6.1.4.1.2.3.51.1.2.3.8.0

Fan9: .1.3.6.1.4.1.2.3.51.1.2.3.9.0

Fan10: .1.3.6.1.4.1.2.3.51.1.2.3.10.0

Fan11: .1.3.6.1.4.1.2.3.51.1.2.3.11.0

Fan12: .1.3.6.1.4.1.2.3.51.1.2.3.12.0

And for the temperatures:

CPU1: .1.3.6.1.4.1.2.3.51.1.2.1.2.1.1
CPU2: .1.3.6.1.4.1.2.3.51.1.2.1.2.2.1
CPU3: .1.3.6.1.4.1.2.3.51.1.2.1.2.3.1
CPU4: .1.3.6.1.4.1.2.3.51.1.2.1.2.4.1
Ambient: .1.3.6.1.4.1.2.3.51.1.2.1.5.1.0

CPU1: .1.3.6.1.4.1.2.3.51.1.2.1.2.1.1

CPU2: .1.3.6.1.4.1.2.3.51.1.2.1.2.2.1

CPU3: .1.3.6.1.4.1.2.3.51.1.2.1.2.3.1

CPU4: .1.3.6.1.4.1.2.3.51.1.2.1.2.4.1

Ambient: .1.3.6.1.4.1.2.3.51.1.2.1.5.1.0

I just found a proper list of OID’s for the IBM RSA adapter. That’s rather nice, since I really was looking for the OID’s for the VRM failure OID and other warning/critical events.