Tag: IBM

Weird TS3500 problem: redux

Christian Leave a comment

Well, after yesterday’s episode with our tape library today continued to be a taxing day. After restarting a few exports that were hanging yesterday due to our library problems, something similar returned. TSM was unable to locate a few (two to be exact) tapes in the library.

Yet the library reported the tapes were still inventoried. *shrug* Here we are again, looking completely baffled. After a short while trying to figure out what to do, we went through the Data Cartridge inventory again. As it turns out, through putting the library in “Pause”-Mode and restarting TSM multiple times, TSM apparently completely forgot that it had these tapes put into drives.

After manually moving the tapes back to their home slot via the management interface of the TS3500 and setting the volume access mode back to read-write, everything is fine now I could finish my pending exports!

Novell KMP: vmware-tools-kmp and ibm-lin_tape-kmp

Christian Leave a comment

Disclaimer: I don’t take any responsibility for faults within the software, I just provide the RPM’s! Feel free to ask me about stuff concerning these RPM’s, but I ain’t accountable if your stuff goes kaboom … Oh, and those RPM’s aren’t recommended or supported by Novell or IBM!

After working with the novell-kmp solution, I think it’s actually rather easy to create a “Kernel Module Package“. In the end, I created two additional KMP’s, one for the tools component of the VMware-Tools shipped with VMware ESX, and another for the lin_tape SCSI driver, used by our IBM TS3400 as well as the IBM TS7530.

Some parts (especially the build system used within the VMware kernel modules) took some figuring out/playing around, but I actually got it working. Now each time I update the VMware-Tools I just need to install the new RPM, tada! No need for a fully fledged build environment on every box.

SUSE Linux Enterprise Server 10:

  • ibm-lin_tape-1.24.0_2.6.16.60_0.37_f594963d-0.1 (i586, x86_64, SRPM)
    • ibm-lin_tape-kmp-bigsmp (i586)
    • ibm-lin_tape-kmp-debug (i586, x86_64)
    • ibm-lin_tape-kmp-default (i586, x86_64)
    • ibm-lin_tape-kmp-kdump (i586, x86_64)
    • ibm-lin_tape-kmp-kdumppae (i586)
    • ibm-lin_tape-kmp-smp (i586, x86_64)
    • ibm-lin_tape-kmp-vmi (i586)
    • ibm-lin_tape-kmp-vmipae (i586)
  • vmware-tools-kmp-3.5.0_153875_2.6.16.60_0.37_f594963d-0.1 (SRPM)
    • vmware-tools-kmp-bigsmp (i586)
    • vmware-tools-kmp-debug (i586, x86_64)
    • vmware-tools-kmp-default (i586, x86_64)
    • vmware-tools-kmp-kdump (i586, x86_64)
    • vmware-tools-kmp-kdumppae (i586)
    • vmware-tools-kmp-smp (i586, x86_64)
    • vmware-tools-kmp-vmi (i586)
    • vmware-tools-kmp-vmipae (i586)
    • vmware-tools-kmp-xen (i586, x86_64)
    • vmware-tools-kmp-xenpae (i586)

SUSE Linux Enterprise Server 11:

  • ibm-lin_tape-1.24.0_2.6.27.21_0.1-0.1 (i586, x86_64, SRPM)
    • ibm-lin_tape-kmp-debug (i586, x86_64)
    • ibm-lin_tape-kmp-default (i586, x86_64)
    • ibm-lin_tape-kmp-pae (i586)
    • ibm-lin_tape-kmp-trace (i586)
    • ibm-lin_tape-kmp-vm (i586, x86_64)
  • vmware-tools-kmp-3.5.0_153875_2.6.27.21_0.1-0.1 (SRPM)
    • vmware-tools-kmp-debug (i586, x86_64)
    • vmware-tools-kmp-default (i586, x86_64)
    • vmware-tools-kmp-pae (i586)
    • vmware-tools-kmp-trace (i586, x86_64)
    • vmware-tools-kmp-vmi (i586)
    • vmware-tools-kmp-xen (i586, x86_64)

Novell KMP: Useable version of ibm-rdac-ds4000

Christian 1 Comment

After some more tinkering, a lot more looking at the macros in /usr/lib/rpm/rpm-suse-kernel-module-subpackage and /usr/lib/rpm/suse_macros, I think I finally have a usable RPM’ified version of IBM’s Multipathing driver ready for use.

There is still one major annoyance left: each time you install a new ibm-rdac-ds4000-kmp RPM, you also need to reinstall the corresponding ibm-rdac-ds4000-initrd package, as the macros in /usr/lib/rpm don’t allow for custom %post or %postun.

As mentioned before, I’m gonna send them to LSI/IBM for review, and maybe, MAYBE they are actually gonna make use of that.

Without further delay, here’s the list of packages. Just a short explanation: you need mppUtil-%version, in order to install the ibm-rdac-ds4000-kmp.

  • mppUtil-09.03.0C05.0030-0.2 (i586, x86_64, SRPM)
  • ibm-rdac-kmp-09.03.0C05.0030_2.6.16.60_0.37_f594963d-0.2 (SRPM)
    • ibm-rdac-kmp-bigsmp (i586)
    • ibm-rdac-kmp-debug (i586, x86_64)
    • ibm-rdac-kmp-default (i586, x86_64)
    • ibm-rdac-kmp-kdump (i586, x86_64)
    • ibm-rdac-kmp-kdumppae (i586)
    • ibm-rdac-kmp-smp (i586, x86_64)
    • ibm-rdac-kmp-vmi (i586)
    • ibm-rdac-kmp-vmipae (i586)
  • ibm-rdac-ds4000-initrd-09.03.0C05.0030_2.6.16.60_0.37_f594963d-0.2
    • ibm-rdac-initrd-bigsmp (i586)
    • ibm-rdac-initrd-debug (i586, x86_64)
    • ibm-rdac-initrd-default (i586, x86_64)
    • ibm-rdac-initrd-kdump (i586, x86_64)
    • ibm-rdac-initrd-kdumppae (i586)
    • ibm-rdac-initrd-smp (i586, x86_64)
    • ibm-rdac-initrd-vmi (i586)
    • ibm-rdac-initrd-vmipae (i586)

This package should be usable with System Storage DS4000 as well as System Storage DS3000 (they use the exact same source code).

I also know, that this solution isn’t really perfect. I’ve been looking at the %triggerin/%triggerun macros, but right now I can’t draw up a scenario (an easy one at that) to successfully use triggers in this situation. Only idea coming up looks like this:

  1. Put the triggers into ibm-rdac-ds4000
  2. When installing the kernel module packages, write the kernelversion/-flavor into a temporary file (impossible, since the macros don’t let you influence %post), and then let the trigger create/update the MPP initrd

If anyone knows a better solution (as in easier, without the writing to a separate file), I’m all ears.

Novell KMP: KMP’ing IBM’s RDAC driver

Christian 1 Comment

Well, after yesterday’s lesson about getting the IBM RDAC to install for a not-yet-running kernel, I decided to take it a step further. Novell does have some documentation about KMP’s, which is actually rather good, especially the guide written by Andreas Grünbacher.

After a short tinkering, I got it actually working. I was kinda surprised, at how easily it actually is. One problem I still have to deal with, is modifying the %post, to generate the mpp-initrd image. For now, the KMP only contains the default %post, which updates the modules.* stuff.

Now, I’m kinda asking myself, why don’t more vendors submit their drivers to Novell in form of KMP’s … Anyway, I’m gonna send mine the LSI/IBM way, maybe they’ll pick it up …

IBM RDAC: Installing the driver for a (not yet) running version

Christian 2 Comments

Well, kernel updates on our Linux servers running IBM’s RDAC driver (developed by LSI) is a real pest .. especially if you have to reboot the box two times in order to install the drivers/initrd correctly.

So I sat down and looked at the Makefile. Turns out, it just needs four tweaks in order to be working with a different kernel version (which you have to pass using environment variables to make).

After that, a simple make KERNEL_OBJ=/lib/modules/2.6.16.60-0.37_f594963d-smp/build OS_VER=2.6.16.60-0.37_f594963d-smp install correctly installs the modules in /lib/modules, rebuilds the correct modules dependencies and builds the correct initrd image.

Monitoring the IBM BladeCenter chassis with Nagios

Christian Leave a comment

Today I ended up working out the details on what we want to monitor regarding our BladeCenter. The most interesting details (for us that is) are these:

  • Fan speeds for Chassis Cooling/Power Module Cooling Bay(s)
  • Temperature
  • Power Domain utilization

It wasn’t *that* hard to implement. Only trouble(s) I ran into, were (1) IBM did a real shitty job with the MIB’s. If you look closely into the mmblade.mib, you’re gonna notice, that not a single OID is specified for the events. (2) As the MIB’s weren’t documented anywhere, I had to look them up via snmpwalk (which I had never used before). So as a reminder (to myself), here’s how it is done:

This will get you a list, with a lot of output (5154 lines to be exact). Lucky me, the web interface of the management module/ssh interface is rather verbose, so all you need to do is compare those values with what you are looking for.

So for myself (and anyone interested) read ahead for the list of checks we are currently running on the management module. Read More

Setting up the BladeCenter H

Christian 2 Comments

Well, we finally had our maintenance window today, in which we planned the hardware exchange for our current Dell Blade Chassis (don’t ask!). The exchange went fine, but as we started exploring the components (like the IBM BladeCenter SAN switches — which are in fact Cisco MDS 9100) we hit a few road blocks.

First, the default user name/password combo for the Cisco MDS 9100 for the BladeCenter is USERID/PASSW0RD (just as the rest of the password combinations).

Next, we started tinkering around with the Catalyst Switch modules. A hint to myself:

Whenever setting up the switch via the WebGUI, make sure you setup both passwords. The password for the switch itself (when prompted by the WebGUI, enter “admin” as well as the password you just entered.

Now, you should be able to connect to the switch with telnet and be able to access the EXEC mode (and unlike me who struggled ~30 minutes till one of my trainees told me to enter a switch password — out of curiosity).

Now, here the list of commands I needed to setup the switch’s “basics”:

Updating path information for TSM

Christian Leave a comment

As I did some switching today (between the new lin_tape version by IBM and our own lin_tape version), I ended up writing those lines a dozen times. Here is (just for me, if you don’t care .. skip ahead) on how to generate a list of commands:

which should get you a list like this:

Sidenote: Amount of Slots per Virtual Tape Library

Christian Leave a comment

Well, I just stumbled about this again (and I don’t know right now whether or not this is documented inside a RedBook or not) today, so I thought maybe I’m gonna write it down.

Slot-Amount Property of a Virtual Tape Library
Slot-Amount Property of a Virtual Tape Library

Please keep in mind, when creating the virtual library to think hard about the amount of slots you might need. It ain’t that bad, you just can’t decrease the amount anymore.  So if you think about creating 50 different virtual tape libraries with 500 slots each on your TS7530, think again. The current software level only supports 25.000 slots on a global level.

Working with IBM’s Virtualization Engine Console

Christian Leave a comment

Recently, we got the recommendation from our system partner to use static allocated tape cartridges instead of dynamic allocated ones. Apparently using dynamic allocating cartridges comes with a performance penalty if more than a few nodes are backing up a large amount of data at once.

And yet again, I noticed that the IBM Virtualization Engine Console (aka Falconstor Software) is really error prone.

In order to change the allocation type, we had to shred the old cartridges first (500 x ~100M up till now), chance the allocation type at the virtual tape library level, and then recreate the 500 cartridges with a fixed size (500x 102400MB). Now, as I was kinda optimistic, I decided to create all 500 cartridges at once.

Failure during the initiation of 500 virtual catridges

So I ended up creating the 500 cartridges in steps of 45. Which isn’t that big of a deal. But, as we do have two separate logical virtual tape libraries (basically the whole TS7530 is partitioned into two tape libraries), we had to do it for the second one too. But I told myself “Come on, try the maximum amount again!” … And guess what:

Successful initiation of 510 virtual catridges

That worked ❓ Please, don’t ask me why the hell it’s working for one virtual tape library on the same system (well, different virtualization engine), but ain’t for the other one … 😯