July 4, 2007

miimon, arp_interval and the code

July 4, 2007August 16, 2014 Christian Leave a comment

After today’s adventure with the kernel bonding, I just took a look at the code …

         if (miimon) {
                 printk(KERN_INFO DRV_NAME
                        ": MII link monitoring set to %d msn",
                        miimon);
         } else if (arp_interval) {
                 int i;

                 printk(KERN_INFO DRV_NAME
                        ": ARP monitoring set to %d ms, validate %s, with %d target(s):",
                        arp_interval,
                        arp_validate_tbl[arp_validate_value].modename,
                        arp_ip_count);

                 for (i = 0; i &lt; arp_ip_count; i++)
                         printk (" %s", arp_ip_target[i]);

                 printk("n");

         } else {
                 /* miimon and arp_interval not set, we need one so things
                  * work as expected, see bonding.txt for details
                  */
                 printk(KERN_WARNING DRV_NAME
                        ": Warning: either miimon or arp_interval and "
                        "arp_ip_target module parameters must be specified, "
                        "otherwise bonding will not detect link failures! see "
                        "bonding.txt for details.n");
         }

if (miimon) {

printk(KERN_INFO DRV_NAME

": MII link monitoring set to %d msn",

miimon);

} else if (arp_interval) {

int i;

printk(KERN_INFO DRV_NAME

": ARP monitoring set to %d ms, validate %s, with %d target(s):",

arp_interval,

arp_validate_tbl[arp_validate_value].modename,

arp_ip_count);

for (i = 0; i < arp_ip_count; i++)

printk (" %s", arp_ip_target[i]);

printk("n");

} else {

/* miimon and arp_interval not set, we need one so things

* work as expected, see bonding.txt for details

printk(KERN_WARNING DRV_NAME

": Warning: either miimon or arp_interval and "

"arp_ip_target module parameters must be specified, "

"otherwise bonding will not detect link failures! see "

"bonding.txt for details.n");

}

If I read it right, you only get the KERN_WARNING for “either miimon or arp_interval” only if miimon or arp_interval isn’t set … but at least my config says it is .. *shrug* .. bed time for me 🙄

Bloody cluster solutions

July 4, 2007June 21, 2013 Christian 1 Comment

In preparation to get our website (and all those other websites – like www.fh-neubrandenburg.de or www.hmt-rostock.de) clustered, someone bought the cluster version of the PacketPro 450. These things are nice, especially considering you don’t need to fiddle around with LVS yourself (which is a *real* pain in the ass).

The only problem I have currently with them is that they scan the database and web nodes every 30 seconds, and since we have an active node and a hot-standby both do this and producing this:

Jul  4 18:27:29 dbc-mysql1 sshd[7313]: Did not receive identification string from 172.16.234.11
Jul  4 18:27:30 dbc-mysql1 sshd[7350]: Did not receive identification string from 172.16.234.12
Jul  4 18:27:59 dbc-mysql1 sshd[7363]: Did not receive identification string from 172.16.234.11
Jul  4 18:28:01 dbc-mysql1 sshd[7364]: Did not receive identification string from 172.16.234.12
Jul  4 18:28:31 dbc-mysql1 sshd[7393]: Did not receive identification string from 172.16.234.11
Jul  4 18:28:33 dbc-mysql1 sshd[7394]: Did not receive identification string from 172.16.234.12
Jul  4 18:29:04 dbc-mysql1 sshd[7417]: Did not receive identification string from 172.16.234.11
Jul  4 18:29:05 dbc-mysql1 sshd[7418]: Did not receive identification string from 172.16.234.12
Jul  4 18:29:36 dbc-mysql1 sshd[7419]: Did not receive identification string from 172.16.234.11
Jul  4 18:29:37 dbc-mysql1 sshd[7420]: Did not receive identification string from 172.16.234.12
Jul  4 18:30:06 dbc-mysql1 sshd[7419]: Did not receive identification string from 172.16.234.11
Jul  4 18:30:07 dbc-mysql1 sshd[7420]: Did not receive identification string from 172.16.234.12

Jul 4 18:27:29 dbc-mysql1 sshd[7313]: Did not receive identification string from 172.16.234.11

Jul 4 18:27:30 dbc-mysql1 sshd[7350]: Did not receive identification string from 172.16.234.12

Jul 4 18:27:59 dbc-mysql1 sshd[7363]: Did not receive identification string from 172.16.234.11

Jul 4 18:28:01 dbc-mysql1 sshd[7364]: Did not receive identification string from 172.16.234.12

Jul 4 18:28:31 dbc-mysql1 sshd[7393]: Did not receive identification string from 172.16.234.11

Jul 4 18:28:33 dbc-mysql1 sshd[7394]: Did not receive identification string from 172.16.234.12

Jul 4 18:29:04 dbc-mysql1 sshd[7417]: Did not receive identification string from 172.16.234.11

Jul 4 18:29:05 dbc-mysql1 sshd[7418]: Did not receive identification string from 172.16.234.12

Jul 4 18:29:36 dbc-mysql1 sshd[7419]: Did not receive identification string from 172.16.234.11

Jul 4 18:29:37 dbc-mysql1 sshd[7420]: Did not receive identification string from 172.16.234.12

Jul 4 18:30:06 dbc-mysql1 sshd[7419]: Did not receive identification string from 172.16.234.11

Jul 4 18:30:07 dbc-mysql1 sshd[7420]: Did not receive identification string from 172.16.234.12

That’s only the logs from three minutes … now figure you have it running for like four days and figure what the average log size due to such crap is … But at least it looks solvable, though I gonna have to call them tomorrow and ask for a patch/update to get their ssh-scan to send some banner when performing the service check.

Adapter teaming on SLES10

July 4, 2007June 21, 2013 Christian 2 Comments

Since one of the requirements for my current project is having NIC redundancy, I didn’t get around looking at the available “adapter teaming” (or adapter bonding) solutions available for Linux/SLES.

First I tried to dig into the Broadcom solution (since the Blade I first implemented the stuff uses a Broadcom NetXtreme II card) , but found out pretty soon that the basp configuration tool, which is *only* available on the Broadcom driver CD’s shipped with the Blade itself, pretty much doesn’t work.

Some hours googling later at how to get the frickin’ Broadcom crap working, I stumbled upon a file linked as bonding.txt. Turns out, that the kernel already supports adapter teaming (only that it’s called adapter bonding) by itself. No need for the Broadcom solution anymore.

Setting it up was rather easy (besides my lazy SUSE admin can’t do it via yast; it has to be done on the file layer since “yast lan” is too stupid to even show the thing), it’s simply creating the interface configs via said “yast lan“, copying one of the “ifcfg-eth-id” files to another file called “ifcfg-bond0“, removing some stuff out of it and cleaning out the other interface configs.

Then simply shove in the following into the ifcfg-bond0 in /etc/sysconfig/network:

IPADDR=&quot;141.53.5.141&quot;
NETMASK=&quot;255.255.255.0&quot;
NETWORK=&quot;141.53.5.0&quot;
MTU=&quot;&quot;
REMOTE_IPADDR=&quot;&quot;
STARTMODE=&quot;auto&quot;
BONDING_MASTER=&quot;yes&quot;
BONDING_MODULE_OPTS=&quot;miimon=100 mode=balance-alb&quot;
BONDING_SLAVE0=&quot;bus-pci-0000:02:00.0&quot;
BONDING_SLAVE1=&quot;bus-pci-0000:06:00.0&quot;

IPADDR_int=&quot;172.16.234.41&quot;
NETMASK_int=&quot;255.255.255.0&quot;
NETWORK_int=&quot;172.16.234.0&quot;
LABEL_int=&quot;int&quot;

IPADDR="141.53.5.141"

NETMASK="255.255.255.0"

NETWORK="141.53.5.0"

MTU=""

REMOTE_IPADDR=""

STARTMODE="auto"

BONDING_MASTER="yes"

BONDING_MODULE_OPTS="miimon=100 mode=balance-alb"

BONDING_SLAVE0="bus-pci-0000:02:00.0"

BONDING_SLAVE1="bus-pci-0000:06:00.0"

IPADDR_int="172.16.234.41"

NETMASK_int="255.255.255.0"

NETWORK_int="172.16.234.0"

LABEL_int="int"

That’s it .. We just defined an adapter IP (the 141.53.5.x) and an virtual interface labeled as “int“. We also configured the MII-Monitor to check every 100ms(?) the link of each interface (those defined in BONDING_SLAVEx) if they are either up or down, as well as the adaptive load balancing (“mode=balance-alb“).

Only thing annoying me with that solution is the following entry in /var/log/messages:

Jul  4 18:32:00 dbc-mysql1 kernel: Ethernet Channel Bonding Driver: v3.0.1 (January 9, 2006)
Jul  4 18:32:00 dbc-mysql1 kernel: bonding: Warning: either miimon or arp_interval and arp_ip_target module parameters must be specified, otherwise bonding will not detect link failures! see bonding.txt for details.
Jul  4 18:32:00 dbc-mysql1 kernel: bonding: bond0: Setting MII monitoring interval to 100.
Jul  4 18:32:00 dbc-mysql1 kernel: bonding: bond0: setting mode to balance-alb (6).
Jul  4 18:32:00 dbc-mysql1 kernel: bnx2: eth1: using MSI
Jul  4 18:32:00 dbc-mysql1 kernel: bonding: bond0: enslaving eth1 as an active interface with a down link.
Jul  4 18:32:00 dbc-mysql1 kernel: bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex
Jul  4 18:32:00 dbc-mysql1 kernel: bonding: bond0: link status definitely up for interface eth1.
Jul  4 18:32:00 dbc-mysql1 kernel: bonding: bond0: making interface eth1 the new active one.
Jul  4 18:32:00 dbc-mysql1 kernel: bnx2: eth0: using MSI
Jul  4 18:32:00 dbc-mysql1 kernel: bonding: bond0: enslaving eth0 as an active interface with a down link.
Jul  4 18:32:00 dbc-mysql1 kernel: bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex
Jul  4 18:32:01 dbc-mysql1 kernel: bonding: bond0: link status definitely up for interface eth0.

Jul 4 18:32:00 dbc-mysql1 kernel: Ethernet Channel Bonding Driver: v3.0.1 (January 9, 2006)

Jul 4 18:32:00 dbc-mysql1 kernel: bonding: Warning: either miimon or arp_interval and arp_ip_target module parameters must be specified, otherwise bonding will not detect link failures! see bonding.txt for details.

Jul 4 18:32:00 dbc-mysql1 kernel: bonding: bond0: Setting MII monitoring interval to 100.

Jul 4 18:32:00 dbc-mysql1 kernel: bonding: bond0: setting mode to balance-alb (6).

Jul 4 18:32:00 dbc-mysql1 kernel: bnx2: eth1: using MSI

Jul 4 18:32:00 dbc-mysql1 kernel: bonding: bond0: enslaving eth1 as an active interface with a down link.

Jul 4 18:32:00 dbc-mysql1 kernel: bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex

Jul 4 18:32:00 dbc-mysql1 kernel: bonding: bond0: link status definitely up for interface eth1.

Jul 4 18:32:00 dbc-mysql1 kernel: bonding: bond0: making interface eth1 the new active one.

Jul 4 18:32:00 dbc-mysql1 kernel: bnx2: eth0: using MSI

Jul 4 18:32:00 dbc-mysql1 kernel: bonding: bond0: enslaving eth0 as an active interface with a down link.

Jul 4 18:32:00 dbc-mysql1 kernel: bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex

Jul 4 18:32:01 dbc-mysql1 kernel: bonding: bond0: link status definitely up for interface eth0.

See the warning ? I can’t get it to shut up .. I also tried loading the mii.ko module, but it won’t shut up … damn 🙁

Well, at least the adapter teaming works as desired (still haven’t measured the performance impact with this setup – really need a clever way to do that) and I can plug one of the two cables connected to this box and still have one interface online and a continuous connection. yay ❗

SLES10 on pSeries

July 4, 2007June 21, 2013 Christian Leave a comment

Okay, yet another day passed by blazing fast. I had a good day at work, spent nearly the whole day trying to get my bloody systems hooked up to our SAN (which was interrupted by a non-working SAN-switch, disappearing WWN’s, lunch and my trainees), messing around with our internal network, hacking our Blade Chassis switches to get me what I want and some random paperwork.

But first things first .. We installed SLES10 on a pSeries box the other day (I think on Monday), and now I’m trying to get the WWN of it’s Emulex HBA, out of either sysfs or procfs. But whatcha’ thinking ?

I can’t get the dreaded WWN our of anything. Emulex’s hbacmd (from their HBAnyware utility) tells me there is no HBA and/or I don’t have the lpfc driver loaded (which can’t be, since I see IBM Tape Drives and my DS4300/FAStT900 via the lpfc), which is like … 😡

So if any Emulex/pSeries expert is reading this, *please* (I beg you) tell me how the frack I get the WWN squashed out of it without looking either at the back of the rack or into the BIOS.

And here’s just for the record (my own – so I don’t need to look it up more often) the way on how to reset the attention indicators (basically LED’s) on the front of a pSeries box running Linux, which gets turned on when either resetting the box or killing it in startup:

# Make sure we have powerpc-utils installed ..
pSeries ~ [0] $ rpm -qa | grep powerpc-utils
powerpc-utils-1.0.0-5.4

# Tell us, which LEDs have which address/status
pSeries ~ [0] $ usysattn
U0.1    [on]

# Turn of the given LED
pSeries ~ [0] $ usysattn -l U0.1 -s normal

# Make sure we have powerpc-utils installed ..

pSeries ~ [0] $ rpm -qa | grep powerpc-utils

powerpc-utils-1.0.0-5.4

# Tell us, which LEDs have which address/status

pSeries ~ [0] $ usysattn

U0.1 [on]

# Turn of the given LED

pSeries ~ [0] $ usysattn -l U0.1 -s normal

That’s it, the LED is off.