Christian – Page 22

Monitoring the IBM BladeCenter chassis with Nagios

February 10, 2009June 21, 2013 Christian Leave a comment

Today I ended up working out the details on what we want to monitor regarding our BladeCenter. The most interesting details (for us that is) are these:

Fan speeds for Chassis Cooling/Power Module Cooling Bay(s)
Temperature
Power Domain utilization

It wasn’t *that* hard to implement. Only trouble(s) I ran into, were (1) IBM did a real shitty job with the MIB’s. If you look closely into the mmblade.mib, you’re gonna notice, that not a single OID is specified for the events. (2) As the MIB’s weren’t documented anywhere, I had to look them up via snmpwalk (which I had never used before). So as a reminder (to myself), here’s how it is done:

snmpwalk -v1 -c public -O n 10.0.0.35 .1.3.6.1.4.1.2.3.51.2.2

1	snmpwalk -v1 -c public -O n 10.0.0.35 .1.3.6.1.4.1.2.3.51.2.2

This will get you a list, with a lot of output (5154 lines to be exact). Lucky me, the web interface of the management module/ssh interface is rather verbose, so all you need to do is compare those values with what you are looking for.

So for myself (and anyone interested) read ahead for the list of checks we are currently running on the management module. Read More

VMware vCenter: is not connected

February 4, 2009August 8, 2014 Christian 3 Comments

Well, today I once again had the case where a virtual machine (in my case a Virtual Machine Template) was kinda stuck. You couldn’t remove the template (as in the entries for “Remove from inventory” was grayed out) and you couldn’t re-add the Virtual Machine’s VMX from the datastore browser either.

VI Client - Disconnected templates — VI Client – Disconnected templates

Though, a simple putting the host into maintenance mode and rebooting helped that problem. Maybe there is a simpler solution for this, I just don’t know about it.

Thanks to Sven in #1, I now know that simple solution for my problem!

[root@esxi root]# /etc/init.d/mgmt-vmware restart
Stopping VMware ESX Server Management services:
   VMware ESX Server Host Agent Watchdog                   [  OK  ]
   VMware ESX Server Host Agent                            [  OK  ]
Starting VMware ESX Server Management services:
   VMware ESX Server Host Agent (background)               [  OK  ]
   Availability report startup (background)                [  OK  ]

[root@esxi root]# /etc/init.d/mgmt-vmware restart

Stopping VMware ESX Server Management services:

VMware ESX Server Host Agent Watchdog [ OK ]

VMware ESX Server Host Agent [ OK ]

Starting VMware ESX Server Management services:

VMware ESX Server Host Agent (background) [ OK ]

Availability report startup (background) [ OK ]

Half a minute, and a heart-stopping moment later (all VM’s on that host turn grey after the first update) the VM’s are accessible again. Thanks again to Sven!

Installing SLES10 via network with no DHCP available

January 30, 2009August 8, 2014 Christian Leave a comment

In our current fight against the BladeCenter switches, we’re currently facing the problem that the blades ain’t able to send/receive DHCP-traffic.

So in order to move forward, we had to use static IP addresses. And since SLES10 ain’t straight forward on that, I had to look it up. Now, here’s for me (and everyone else tired of searching) how to do it:

install=http://ftp.barfoo.org/install/SLES10-x64/CD1
  netdevice=eth0
  hostip=10.0.1.240
  netmask=255.255.255.0
  gateway=10.0.1.1
  nameserver=10.0.1.2
  domain=home.barfoo.org
  insmod=bnx2
  splash=verbose

install=http://ftp.barfoo.org/install/SLES10-x64/CD1

netdevice=eth0

hostip=10.0.1.240

netmask=255.255.255.0

gateway=10.0.1.1

nameserver=10.0.1.2

domain=home.barfoo.org

insmod=bnx2

splash=verbose

Setting up the BladeCenter H

January 28, 2009June 21, 2013 Christian 2 Comments

Well, we finally had our maintenance window today, in which we planned the hardware exchange for our current Dell Blade Chassis (don’t ask!). The exchange went fine, but as we started exploring the components (like the IBM BladeCenter SAN switches — which are in fact Cisco MDS 9100) we hit a few road blocks.

First, the default user name/password combo for the Cisco MDS 9100 for the BladeCenter is USERID/PASSW0RD (just as the rest of the password combinations).

Next, we started tinkering around with the Catalyst Switch modules. A hint to myself:

Whenever setting up the switch via the WebGUI, make sure you setup both passwords. The password for the switch itself (when prompted by the WebGUI, enter “admin” as well as the password you just entered.

Now, you should be able to connect to the switch with telnet and be able to access the EXEC mode (and unlike me who struggled ~30 minutes till one of my trainees told me to enter a switch password — out of curiosity).

Now, here the list of commands I needed to setup the switch’s “basics”:

service password-encryption
username admin privilege 15 password 0 &lt;password&gt;
clock timezone CET 1
clock summer-time CET recurring last Sun Mar 2:00 last Sun Oct 3:00
ip domain-name home.barfoo.org
ip name-server 10.0.0.2
ip name-server 10.0.0.1
line con 0
 login local
line vty 0 4
 password &lt;password&gt;
 login local
line vty 5 15
 password &lt;password&gt;
 login
ntp server 160.45.10.8

service password-encryption

username admin privilege 15 password 0 <password>

clock timezone CET 1

clock summer-time CET recurring last Sun Mar 2:00 last Sun Oct 3:00

ip domain-name home.barfoo.org

ip name-server 10.0.0.2

ip name-server 10.0.0.1

line con 0

line vty 0 4

password <password>

line vty 5 15

password <password>

ntp server 160.45.10.8

Updating path information for TSM

January 28, 2009June 21, 2013 Christian Leave a comment

As I did some switching today (between the new lin_tape version by IBM and our own lin_tape version), I ended up writing those lines a dozen times. Here is (just for me, if you don’t care .. skip ahead) on how to generate a list of commands:

for i in $( seq -w 1 32 ); do
  token=&quot;${i/0/}&quot;
  path_prefix=&quot;/dev/lt/IBMtape12245775&quot;
  echo &quot;DELETE PATH TSM1 VTL1_DR$i SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY=VTL1&quot;
  echo &quot;DEFINE PATH TSM1 VTL1_DR$i SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY=VTL1 DEVICE=$path_prefix$((token+11))A1&quot;
done

for i in $( seq -w 1 32 ); do

token="${i/0/}"

path_prefix="/dev/lt/IBMtape12245775"

echo "DELETE PATH TSM1 VTL1_DR$i SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY=VTL1"

echo "DEFINE PATH TSM1 VTL1_DR$i SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY=VTL1 DEVICE=$path_prefix$((token+11))A1"

done

which should get you a list like this:

...
DELETE PATH TSM1 VTL1_DR21 SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY=VTL1
DEFINE PATH TSM1 VTL1_DR21 SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY=VTL1 DEVICE=/dev/lt/IBMtape1224577532A1
...

...

DELETE PATH TSM1 VTL1_DR21 SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY=VTL1

DEFINE PATH TSM1 VTL1_DR21 SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY=VTL1 DEVICE=/dev/lt/IBMtape1224577532A1

...

Sidenote: Amount of Slots per Virtual Tape Library

January 27, 2009June 21, 2013 Christian Leave a comment

Well, I just stumbled about this again (and I don’t know right now whether or not this is documented inside a RedBook or not) today, so I thought maybe I’m gonna write it down.

Slot-Amount Property of a Virtual Tape Library

Please keep in mind, when creating the virtual library to think hard about the amount of slots you might need. It ain’t that bad, you just can’t decrease the amount anymore. So if you think about creating 50 different virtual tape libraries with 500 slots each on your TS7530, think again. The current software level only supports 25.000 slots on a global level.

Working with IBM’s Virtualization Engine Console

January 27, 2009June 21, 2013 Christian Leave a comment

Recently, we got the recommendation from our system partner to use static allocated tape cartridges instead of dynamic allocated ones. Apparently using dynamic allocating cartridges comes with a performance penalty if more than a few nodes are backing up a large amount of data at once.

And yet again, I noticed that the IBM Virtualization Engine Console (aka Falconstor Software) is really error prone.

In order to change the allocation type, we had to shred the old cartridges first (500 x ~100M up till now), chance the allocation type at the virtual tape library level, and then recreate the 500 cartridges with a fixed size (500x 102400MB). Now, as I was kinda optimistic, I decided to create all 500 cartridges at once.

So I ended up creating the 500 cartridges in steps of 45. Which isn’t that big of a deal. But, as we do have two separate logical virtual tape libraries (basically the whole TS7530 is partitioned into two tape libraries), we had to do it for the second one too. But I told myself “Come on, try the maximum amount again!” … And guess what:

That worked ❓ Please, don’t ask me why the hell it’s working for one virtual tape library on the same system (well, different virtualization engine), but ain’t for the other one … 😯

Sunday afternoon playtime

January 25, 2009January 25, 2009 Christian Leave a comment

Well, it’s yet again Sunday afternoon. And I had (again I might add) the urge to play around with all the stuff I have at home.

So at first, I “fixed” the ground wire off my NAS box.

Afterwards, I went back upstairs. Hooked my Philips up to my notebook and figured I *really* need a wireless keyboard. Because typing with the Windows on-screen keyboard is a huge pain in the ass (as well as a pain for the mouse-hand)!

Gentoo packages are gone now!

January 25, 2009August 8, 2014 Christian Leave a comment

Just like I announced last week, my binary packages repository is now history. I’m gonna remove the redirections from lighttpd and DNS in a few.

Automatic updates on SUSE Linux Enterprise 10

January 23, 2009June 21, 2013 Christian Leave a comment

I had the problem, that the automatic update function of YaST doesn’t work like I want it to. I just wanted it to install only those updates, that ain’t interactive, don’t need a service restart and don’t need a reboot.

YaST does only feature an online update that skips “interactive” updates (I’ve never even encountered an interactive update up till now). So I went ahead and wrote a (hackish) script, that achieves what I need.

#!/bin/bash

TMPDIR=&quot;$( mktemp -d /tmp/autoupdate-XXXXXX )&quot;

# Wake up rug
rug --quiet ping -a

# Get the patchlist from rug
rug --terse pch -u | grep &quot;|Needed&quot; | cut -d| -f2 | 
  sed -e &quot;s,sdkp2-,,&quot; -e &quot;s,slesp2-,,&quot; | 
  sort -u &gt; $TMPDIR/patch-list

for package in $( &lt; $TMPDIR/patch-list ); do
  # Get the patch-details
  rug --terse patch-info slesp2-$package &gt; $TMPDIR/patch-details

  # Parse those patch-details for &quot;unwanted&quot; interactions
  is_interactive=&quot;$( cat $TMPDIR/patch-details | 
    grep &quot;^Interactive:&quot; | cut -d &quot; &quot; -f2 )&quot;
  needs_restart=&quot;$( cat $TMPDIR/patch-details | 
    grep &quot;^Restart Required:&quot; | cut -d &quot; &quot; -f3 )&quot;
  needs_reboot=&quot;$( cat $TMPDIR/patch-details | 
    grep &quot;^Reboot Required:&quot; | cut -d &quot; &quot; -f3 )&quot;

  if [ &quot;$is_interactive&quot; == &quot;No&quot; -a &quot;$needs_restart&quot; == &quot;No&quot; 
      -a &quot;$needs_reboot&quot; == &quot;No&quot; ] ; then
    tmp_package=&quot;$( cat $TMPDIR/patch-details | grep ^atom | cut -d  -f2 |
      sed &#039;:a;N;$!ba;s/n/ /g&#039; )&quot;
    for i in $tmp_package; do
      RPM_STATUS=$( rpm -qi $i )
      if [ &quot;$RPM_STATUS&quot; != &quot;package $i is not installed&quot; -a 
        &quot;$( rug --terse lu | cut -d| -f4 | grep &quot;$i$&quot; )&quot; ] ; then
        patch_list=&quot;$patch_list $i&quot;
      fi
    done
  fi
done

rug --quiet install --no-confirm $patch_list

rug --quiet clean-cache

trap &quot;rm -rf &quot;$TMPDIR&quot; &gt;/dev/null 2&gt;&amp;1&quot; ERR EXIT INT TERM HUP
# vim: set tw=80 ts=2 sw=2 et softtabstop=2

#!/bin/bash

TMPDIR="$( mktemp -d /tmp/autoupdate-XXXXXX )"

# Wake up rug

rug --quiet ping -a

# Get the patchlist from rug

sed -e "s,sdkp2-,," -e "s,slesp2-,," |

sort -u > $TMPDIR/patch-list

for package in $( < $TMPDIR/patch-list ); do

# Get the patch-details

rug --terse patch-info slesp2-$package > $TMPDIR/patch-details

# Parse those patch-details for "unwanted" interactions

is_interactive="$( cat $TMPDIR/patch-details |

grep "^Interactive:" | cut -d " " -f2 )"

needs_restart="$( cat $TMPDIR/patch-details |

grep "^Restart Required:" | cut -d " " -f3 )"

needs_reboot="$( cat $TMPDIR/patch-details |

grep "^Reboot Required:" | cut -d " " -f3 )"

if [ "$is_interactive" == "No" -a "$needs_restart" == "No"

-a "$needs_reboot" == "No" ] ; then

tmp_package="$( cat $TMPDIR/patch-details | grep ^atom | cut -d -f2 |

sed ':a;N;$!ba;s/n/ /g' )"

for i in $tmp_package; do

RPM_STATUS=$( rpm -qi $i )

if [ "$RPM_STATUS" != "package $i is not installed" -a

"$( rug --terse lu | cut -d| -f4 | grep "$i$" )" ] ; then

patch_list="$patch_list $i"

done

rug --quiet install --no-confirm $patch_list

rug --quiet clean-cache

trap "rm -rf "$TMPDIR" >/dev/null 2>&1" ERR EXIT INT TERM HUP

# vim: set tw=80 ts=2 sw=2 et softtabstop=2

And just for me, the crontag entry:

30 22 * * *     root  /usr/local/sbin/autoupdate

1	30 22 * * * root /usr/local/sbin/autoupdate