Christian – Page 19

Weird TS3500 problem: redux

June 3, 2009August 8, 2014 Christian Leave a comment

Well, after yesterday’s episode with our tape library today continued to be a taxing day. After restarting a few exports that were hanging yesterday due to our library problems, something similar returned. TSM was unable to locate a few (two to be exact) tapes in the library.

ANR8300E I/O error on library LIB3584 (OP=00006C03, CC=314, KEY=05, ASC=3B,
ASCQ=0E, SENSE=70.00.05.00.00.00.00.0A.00.00.00.00.3B.0E.00.C0.00-.04.,
Description=The source slot or drive was empty in an attempt to move a
volume). Refer to Appendix C in the 'Messages' manual for recommended action.
ANR8312E Volume 000400 could not be located in library LIB3584.
ANR8358E Audit operation is required for library LIB3584.
ANR8381E LTO volume 000400 could not be mounted in drive DR6 (/dev/rmt0).
ANR1402W Mount request denied for volume 000400 - volume unavailable.
ANR1410W Access mode for volume 000400 now set to "unavailable".

ANR8300E I/O error on library LIB3584 (OP=00006C03, CC=314, KEY=05, ASC=3B,

ASCQ=0E, SENSE=70.00.05.00.00.00.00.0A.00.00.00.00.3B.0E.00.C0.00-.04.,

Description=The source slot or drive was empty in an attempt to move a

volume). Refer to Appendix C in the 'Messages' manual for recommended action.

ANR8312E Volume 000400 could not be located in library LIB3584.

ANR8358E Audit operation is required for library LIB3584.

ANR8381E LTO volume 000400 could not be mounted in drive DR6 (/dev/rmt0).

ANR1402W Mount request denied for volume 000400 - volume unavailable.

ANR1410W Access mode for volume 000400 now set to "unavailable".

Yet the library reported the tapes were still inventoried. *shrug* Here we are again, looking completely baffled. After a short while trying to figure out what to do, we went through the Data Cartridge inventory again. As it turns out, through putting the library in “Pause”-Mode and restarting TSM multiple times, TSM apparently completely forgot that it had these tapes put into drives.

After manually moving the tapes back to their home slot via the management interface of the TS3500 and setting the volume access mode back to read-write, everything is fine now I could finish my pending exports!

Weird TS3500 problem

June 2, 2009June 21, 2013 Christian 1 Comment

Well, today we had a rather weird problem with our TS3500. TSM running on AIX basically went bonko and spit out weird media sense errors, all stating that there is a hardware or media error of unknown nature:

ANR8943E Hardware or media error on library LIB3584 (OP=00006C03, CC=-1,
KEY=04, ASC=44, ASCQ=00,
SENSE=70.00.04.00.00.00.00.46.00.00.00.00.44.00.00.00.00-
     .00.40.82.00.00.00.40.00.00.02.00.48.01.A1.00.00.00.00.0-
      0.06.1B.00.01.09.00.00.00.00.00.00.00.00.00.00.00.00.00.-
      00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00-
     .00.00.00.00.00., Description=An undetermined error has occurred).
Refer to Appendix C in the &#039;Messages&#039; manual for recommended action.
ANR8381E LTO volume HG4480L4 could not be mounted in drive DR9 (/dev/rmt8).

ANR8943E Hardware or media error on library LIB3584 (OP=00006C03, CC=-1,

KEY=04, ASC=44, ASCQ=00,

SENSE=70.00.04.00.00.00.00.46.00.00.00.00.44.00.00.00.00-

.00.40.82.00.00.00.40.00.00.02.00.48.01.A1.00.00.00.00.0-

0.06.1B.00.01.09.00.00.00.00.00.00.00.00.00.00.00.00.00.-

00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00-

.00.00.00.00.00., Description=An undetermined error has occurred).

Refer to Appendix C in the 'Messages' manual for recommended action.

ANR8381E LTO volume HG4480L4 could not be mounted in drive DR9 (/dev/rmt8).

After restarting the TSM server (as in the service, not the whole box) five times, which didn’t resolve squat we decided to take a look at the TS3500 itself. We opened up the Management interface and tried moving a tape into a drive. That didn’t work. Hrmmmmm.

We tried the manual move from the LCD display mounted on the front of the TS3500 base frame, that didn’t work either. So we figured the gripper was stuck and placed a call with our trustworthy support provider.

After a few minutes, they called us back and told us: “Try the following: Place the library in “Pause”-Mode and open it up, maybe a tape fell down …“.

We did exactly that, the gripper moved back to it’s pause position (which is in the base frame), and we started looking inside after opening up the base frame and an expansion frame. Nothing …

So we closed it back up, and let the base frame resume it’s normal duties … guess what: After resuming normal operations, it worked again *shrug*

Reviewing Mobile Admin by Rove

June 1, 2009June 1, 2009 Christian 1 Comment

Well, Laura Kedziora (Marketing Manager) from Rove contacted me about two months ago, whether or not I’d be willing to write up a review about their upcoming release of Mobile Admin supporting Nagios!

At first, I didn’t even know what “Mobile Admin” was .. though I heard some good things about Rove. After playing with it in a VM, I do have to admit that it’s a real neat piece of software.

You can cover the whole (well, pretty much everything) shebang dealing with Nagios with the cell phone. But not just that. I could also manage our VMware Infrastructure (or vCenter) with it, as well as a SCCM environment (if we had any that is). Since I only have a shitty-old Nokia (not the required Nokia 6xxx), I ended up borrowing my brothers BlackBerry to take some photos. Read More

XBMC: Adding the ppa keys to apt

May 15, 2009May 15, 2009 Christian 1 Comment

I recently bought an Acer Aspire Revo and had one of my trainees put XMBC on a SDHC card today. So after a bit of toying earlier, I started looking at the thing (from the command line that is).

One thing, if you enable the PPA (ppa.launchpad.net) sources, apt/aptitude is gonna babble something about an unverified key.

Fetched 119kB in 2s (41.4kB/s)
Reading package lists... Done
W: GPG error: http://ppa.launchpad.net jaunty Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6D975C4791E7EE5E
W: GPG error: http://ppa.launchpad.net jaunty Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 2BBD133164234534
W: GPG error: http://ppa.launchpad.net jaunty Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A956EB81318C7509

Fetched 119kB in 2s (41.4kB/s)

Reading package lists... Done

W: GPG error: http://ppa.launchpad.net jaunty Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6D975C4791E7EE5E

W: GPG error: http://ppa.launchpad.net jaunty Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 2BBD133164234534

W: GPG error: http://ppa.launchpad.net jaunty Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A956EB81318C7509

I ended up looking the error up (since I only have an Ubuntu desktop). There’s a simple solution for this:

for i in 2BBD133164234534 A956EB81318C7509 6D975C4791E7EE5E
do
  gpg --keyserver wwwkeys.eu.pgp.net --recv-keys $i
  gpg --export --armor $i | apt-key add -
done

for i in 2BBD133164234534 A956EB81318C7509 6D975C4791E7EE5E

gpg --keyserver wwwkeys.eu.pgp.net --recv-keys $i

gpg --export --armor $i | apt-key add -

done

Alternatively you can also use this:

for i in 2BBD133164234534 A956EB81318C7509 6D975C4791E7EE5E
  apt-key adv --keyserver=wwwkeys.de.pgp.net --recv-key $i
done

for i in 2BBD133164234534 A956EB81318C7509 6D975C4791E7EE5E

apt-key adv --keyserver=wwwkeys.de.pgp.net --recv-key $i

done

DawiControl SIL3114

May 10, 2009August 8, 2014 Christian Leave a comment

Well, if you remember back to last December, I had some severe troubles with this fake ass RAID controller. At the time I posted my last problem (or rather, my conclusion) I already bought a 3WARE Escalade 9550SXU-8LP on EBay (used, since I didn’t want to spend ~400 EUR). Now after the controller arrived in mid-January, I ended up buying a new main-board, processor, memory, … in order to put the 3WARE card to use; initially I thought the 3WARE card would fit into my A7N8X-X, but apparently that only had a 16bit PCI-slot.

Now, after reinstalling the box, and a short while in the 3WARE controller BIOS, I ended up with a 2,9TiB RAID5 array. That’s it for this lovely DawiControl crap. If anyone needs one, I still have the one I bought at work (lying around in a shelf), holler me.

Novell KMP: vmware-tools-kmp and ibm-lin_tape-kmp

May 10, 2009August 8, 2014 Christian Leave a comment

Disclaimer: I don’t take any responsibility for faults within the software, I just provide the RPM’s! Feel free to ask me about stuff concerning these RPM’s, but I ain’t accountable if your stuff goes kaboom … Oh, and those RPM’s aren’t recommended or supported by Novell or IBM!

After working with the novell-kmp solution, I think it’s actually rather easy to create a “Kernel Module Package“. In the end, I created two additional KMP’s, one for the tools component of the VMware-Tools shipped with VMware ESX, and another for the lin_tape SCSI driver, used by our IBM TS3400 as well as the IBM TS7530.

Some parts (especially the build system used within the VMware kernel modules) took some figuring out/playing around, but I actually got it working. Now each time I update the VMware-Tools I just need to install the new RPM, tada! No need for a fully fledged build environment on every box.

SUSE Linux Enterprise Server 10:

ibm-lin_tape-1.24.0_2.6.16.60_0.37_f594963d-0.1 (i586, x86_64, ~~SRPM~~)
- ibm-lin_tape-kmp-bigsmp (i586)
- ibm-lin_tape-kmp-debug (i586, x86_64)
- ibm-lin_tape-kmp-default (i586, x86_64)
- ibm-lin_tape-kmp-kdump (i586, x86_64)
- ibm-lin_tape-kmp-kdumppae (i586)
- ibm-lin_tape-kmp-smp (i586, x86_64)
- ibm-lin_tape-kmp-vmi (i586)
- ibm-lin_tape-kmp-vmipae (i586)
vmware-tools-kmp-3.5.0_153875_2.6.16.60_0.37_f594963d-0.1 (~~SRPM~~)
- vmware-tools-kmp-bigsmp (i586)
- vmware-tools-kmp-debug (i586, x86_64)
- vmware-tools-kmp-default (i586, x86_64)
- vmware-tools-kmp-kdump (i586, x86_64)
- vmware-tools-kmp-kdumppae (i586)
- vmware-tools-kmp-smp (i586, x86_64)
- vmware-tools-kmp-vmi (i586)
- vmware-tools-kmp-vmipae (i586)
- vmware-tools-kmp-xen (i586, x86_64)
- vmware-tools-kmp-xenpae (i586)

SUSE Linux Enterprise Server 11:

ibm-lin_tape-1.24.0_2.6.27.21_0.1-0.1 (i586, x86_64, ~~SRPM~~)
- ibm-lin_tape-kmp-debug (i586, x86_64)
- ibm-lin_tape-kmp-default (i586, x86_64)
- ibm-lin_tape-kmp-pae (i586)
- ibm-lin_tape-kmp-trace (i586)
- ibm-lin_tape-kmp-vm (i586, x86_64)
vmware-tools-kmp-3.5.0_153875_2.6.27.21_0.1-0.1 (~~SRPM~~)
- vmware-tools-kmp-debug (i586, x86_64)
- vmware-tools-kmp-default (i586, x86_64)
- vmware-tools-kmp-pae (i586)
- vmware-tools-kmp-trace (i586, x86_64)
- vmware-tools-kmp-vmi (i586)
- vmware-tools-kmp-xen (i586, x86_64)

Novell KMP: Useable version of ibm-rdac-ds4000

May 9, 2009August 8, 2014 Christian 1 Comment

After some more tinkering, a lot more looking at the macros in /usr/lib/rpm/rpm-suse-kernel-module-subpackage and /usr/lib/rpm/suse_macros, I think I finally have a usable RPM’ified version of IBM’s Multipathing driver ready for use.

There is still one major annoyance left: each time you install a new ibm-rdac-ds4000-kmp RPM, you also need to reinstall the corresponding ibm-rdac-ds4000-initrd package, as the macros in /usr/lib/rpm don’t allow for custom %post or %postun.

As mentioned before, I’m gonna send them to LSI/IBM for review, and maybe, MAYBE they are actually gonna make use of that.

Without further delay, here’s the list of packages. Just a short explanation: you need mppUtil-%version, in order to install the ibm-rdac-ds4000-kmp.

mppUtil-09.03.0C05.0030-0.2 (i586, x86_64, ~~SRPM~~)
ibm-rdac-kmp-09.03.0C05.0030_2.6.16.60_0.37_f594963d-0.2 (SRPM)
- ibm-rdac-kmp-bigsmp (i586)
- ibm-rdac-kmp-debug (i586, x86_64)
- ibm-rdac-kmp-default (i586, x86_64)
- ibm-rdac-kmp-kdump (i586, x86_64)
- ibm-rdac-kmp-kdumppae (i586)
- ibm-rdac-kmp-smp (i586, x86_64)
- ibm-rdac-kmp-vmi (i586)
- ibm-rdac-kmp-vmipae (i586)
ibm-rdac-ds4000-initrd-09.03.0C05.0030_2.6.16.60_0.37_f594963d-0.2
- ibm-rdac-initrd-bigsmp (i586)
- ibm-rdac-initrd-debug (i586, x86_64)
- ibm-rdac-initrd-default (i586, x86_64)
- ibm-rdac-initrd-kdump (i586, x86_64)
- ibm-rdac-initrd-kdumppae (i586)
- ibm-rdac-initrd-smp (i586, x86_64)
- ibm-rdac-initrd-vmi (i586)
- ibm-rdac-initrd-vmipae (i586)

This package should be usable with System Storage DS4000 as well as System Storage DS3000 (they use the exact same source code).

I also know, that this solution isn’t really perfect. I’ve been looking at the %triggerin/%triggerun macros, but right now I can’t draw up a scenario (an easy one at that) to successfully use triggers in this situation. Only idea coming up looks like this:

Put the triggers into ibm-rdac-ds4000
When installing the kernel module packages, ~~write the kernelversion/-flavor into a temporary file~~ (impossible, since the macros don’t let you influence %post), and then let the trigger create/update the MPP initrd

If anyone knows a better solution (as in easier, without the writing to a separate file), I’m all ears.

Novell KMP: KMP’ing IBM’s RDAC driver

May 8, 2009August 8, 2014 Christian 1 Comment

Well, after yesterday’s lesson about getting the IBM RDAC to install for a not-yet-running kernel, I decided to take it a step further. Novell does have some documentation about KMP’s, which is actually rather good, especially the guide written by Andreas Grünbacher.

After a short tinkering, I got it actually working. I was kinda surprised, at how easily it actually is. One problem I still have to deal with, is modifying the %post, to generate the mpp-initrd image. For now, the KMP only contains the default %post, which updates the modules.* stuff.

Name:            ibm-rdac
License:         GPL
Group:           System/Kernel
Summary:         IBM Multipathing driver for DS4000 disk subsystems.

Version:         09.03.0C05.0030
Release:         0.1
Source0:         rdac-LINUX-%version-source.tar.gz
BuildRequires:   kernel-source kernel-syms
BuildRoot:       %{_tmppath}/%{name}-%{version}-build

Requires(post):    mppUtil-%{version}, module-init-tools
Requires(preun):   module-init-tools
Requires(postun):  module-init-tools

%suse_kernel_module_package um

%description
This is the Engenio Linux RDAC Driver Package for IBM DS4000
Storage Subsystems.

%package KMP
Group:     System/Kernel
Summary:   IBM Multipathing driver for DS4000 disk subsystems.

%description KMP
This is the Engenio Linux RDAC Driver Package for IBM DS4000
Storage Subsystems.

%prep
%setup -n linuxrdac-%{version}
set -- *
mkdir source
mv "$@" source/
mkdir obj

%build
export EXTRA_CFLAGS='-DVERSION="%version"'
for flavor in %flavors_to_build; do
    rm -rf obj/$flavor
    cp -r source obj/$flavor
    make -C /usr/src/linux-obj/%_target_cpu/$flavor modules 
            M=$PWD/obj/$flavor
done

%install
export INSTALL_MOD_PATH=$RPM_BUILD_ROOT
export INSTALL_MOD_DIR=updates
for flavor in %flavors_to_build; do
    make -C /usr/src/linux-obj/%_target_cpu/$flavor modules_install 
            M=$PWD/obj/$flavor
done

%changelog
* Thu May 07 2009 - 09.03.0C05.0030-0.1
  - Initial package.

Name: ibm-rdac

License: GPL

Group: System/Kernel

Summary: IBM Multipathing driver for DS4000 disk subsystems.

Version: 09.03.0C05.0030

Release: 0.1

Source0: rdac-LINUX-%version-source.tar.gz

BuildRequires: kernel-source kernel-syms

BuildRoot: %{_tmppath}/%{name}-%{version}-build

Requires(post): mppUtil-%{version}, module-init-tools

Requires(preun): module-init-tools

Requires(postun): module-init-tools

%suse_kernel_module_package um

%description

This is the Engenio Linux RDAC Driver Package for IBM DS4000

Storage Subsystems.

%package KMP

Group: System/Kernel

Summary: IBM Multipathing driver for DS4000 disk subsystems.

%description KMP

This is the Engenio Linux RDAC Driver Package for IBM DS4000

Storage Subsystems.

%prep

%setup -n linuxrdac-%{version}

set -- *

mkdir source

mv "$@" source/

mkdir obj

%build

export EXTRA_CFLAGS='-DVERSION="%version"'

for flavor in %flavors_to_build; do

rm -rf obj/$flavor

cp -r source obj/$flavor

make -C /usr/src/linux-obj/%_target_cpu/$flavor modules

M=$PWD/obj/$flavor

done

%install

export INSTALL_MOD_PATH=$RPM_BUILD_ROOT

export INSTALL_MOD_DIR=updates

for flavor in %flavors_to_build; do

make -C /usr/src/linux-obj/%_target_cpu/$flavor modules_install

M=$PWD/obj/$flavor

done

%changelog

* Thu May 07 2009 - 09.03.0C05.0030-0.1

- Initial package.

Now, I’m kinda asking myself, why don’t more vendors submit their drivers to Novell in form of KMP’s … Anyway, I’m gonna send mine the LSI/IBM way, maybe they’ll pick it up …

IBM RDAC: Installing the driver for a (not yet) running version

May 7, 2009June 21, 2013 Christian 2 Comments

Well, kernel updates on our Linux servers running IBM’s RDAC driver (developed by LSI) is a real pest .. especially if you have to reboot the box two times in order to install the drivers/initrd correctly.

So I sat down and looked at the Makefile. Turns out, it just needs four tweaks in order to be working with a different kernel version (which you have to pass using environment variables to make).

--- linuxrdac-09.03.0C05.0030.orig/Makefile
+++ linuxrdac-09.03.0C05.0030/Makefile
@@ -25,7 +25,10 @@
 #
 #######################################################################
 
-OS_VER := $(shell uname -r)
+ifeq (&quot;$(OS_VER)&quot;,&quot;&quot;)
+	OS_VER := $(shell uname -r)
+endif
+
 HOST_TYPE := $(shell uname -m)
 IS_SMP := $(shell (uname -v | grep -c &quot;SMP&quot;))
 # Kernel Distribution (either REDHAT or SUSE)
@@ -170,7 +173,7 @@ copyrpmfiles :
 
 moduledep :
 	@echo &quot;Generating module dependencies...&quot;
-	@/sbin/depmod $(uname -r)
+	@/sbin/depmod $(OS_VER)
 	
 setupfiles :
 	@install -o root -g root -m 0500 -D Makefile $(DEST_DIR)/opt/mpp/makefile.saved
@@ -190,7 +193,7 @@ setupfiles :
 	rm devicemapping; fi
 
 setupdriver:
-	@/opt/mpp/setupDriver.$(DIST)
+	@/opt/mpp/setupDriver.$(DIST) $(OS_VER)
 	@if [ -f /etc/SuSE-release ]; then 
 		/sbin/insserv  /etc/init.d/mpp; 
 	else 
@@ -266,7 +269,7 @@ uninstall_doer:
 	@echo &quot;The mpp RAMdisk image mpp-$(OS_VER).img has been removed. You may want to remove it from your boot loader config file.&quot;
 	@if test ! -s /var/mpp/devicemapping ; then rm -rf /var/mpp/; fi
 	@echo &quot;Generating module dependencies...&quot;
-	@/sbin/depmod $(uname -r)
+	@/sbin/depmod $(OS_VER)
 
 
 LINUX_RDAC_DIR :=  $(shell pwd)

--- linuxrdac-09.03.0C05.0030.orig/Makefile

+++ linuxrdac-09.03.0C05.0030/Makefile

@@ -25,7 +25,10 @@

#######################################################################

-OS_VER := $(shell uname -r)

+ifeq ("$(OS_VER)","")

+ OS_VER := $(shell uname -r)

+endif

HOST_TYPE := $(shell uname -m)

IS_SMP := $(shell (uname -v | grep -c "SMP"))

# Kernel Distribution (either REDHAT or SUSE)

@@ -170,7 +173,7 @@ copyrpmfiles :

moduledep :

@echo "Generating module dependencies..."

- @/sbin/depmod $(uname -r)

+ @/sbin/depmod $(OS_VER)

setupfiles :

@install -o root -g root -m 0500 -D Makefile $(DEST_DIR)/opt/mpp/makefile.saved

@@ -190,7 +193,7 @@ setupfiles :

rm devicemapping; fi

setupdriver:

- @/opt/mpp/setupDriver.$(DIST)

+ @/opt/mpp/setupDriver.$(DIST) $(OS_VER)

@if [ -f /etc/SuSE-release ]; then

/sbin/insserv /etc/init.d/mpp;

else

@@ -266,7 +269,7 @@ uninstall_doer:

@echo "The mpp RAMdisk image mpp-$(OS_VER).img has been removed. You may want to remove it from your boot loader config file."

@if test ! -s /var/mpp/devicemapping ; then rm -rf /var/mpp/; fi

@echo "Generating module dependencies..."

- @/sbin/depmod $(uname -r)

+ @/sbin/depmod $(OS_VER)

LINUX_RDAC_DIR := $(shell pwd)

After that, a simple make KERNEL_OBJ=/lib/modules/2.6.16.60-0.37_f594963d-smp/build OS_VER=2.6.16.60-0.37_f594963d-smp install correctly installs the modules in /lib/modules, rebuilds the correct modules dependencies and builds the correct initrd image.

Winter on Spiekeroog

May 4, 2009August 8, 2014 Christian Leave a comment

Well, as I wrote last year, I spent some time with my brother on Spiekeroog (that’s where he’s currently living and working). As I left on December 4th, it started snowing like crazy ..

If you read the old article, you might remember that I mentioned Spiekeroog is 100% free of fossil fueled vehicles. That being said, let’s continue with the story I’m about to tell.

Anyway, my brother was walking me back to the hotel boat and we waited a bit in the harbour since we were a bit early. Now, every item is delivered by ship to the island, since there’s no other way (besides by air, which is kinda expensive ..). Anyway, them dockworkers ain’t stupid. They needed some snow-free space, and they started clearing the snow. But how ?

Well, again, them ain’t stupid … You take a forklift and a euro-pallet and you got what ? Exactly, a snow plow!

Spiekeroogish snow plow — Spiekeroog’ish snow plow

I know, the images are kinda grainy, sadly I only had my cell phone to take the photo (left the digital camera at home). C’est la vie ..