Category: Work

Posts related to my Work

Adapter teaming on SLES10

Christian 2 Comments

Since one of the requirements for my current project is having NIC redundancy, I didn’t get around looking at the available “adapter teaming” (or adapter bonding) solutions available for Linux/SLES.

First I tried to dig into the Broadcom solution (since the Blade I first implemented the stuff uses a Broadcom NetXtreme II card) , but found out pretty soon that the basp configuration tool, which is *only* available on the Broadcom driver CD’s shipped with the Blade itself, pretty much doesn’t work.

Some hours googling later at how to get the frickin’ Broadcom crap working, I stumbled upon a file linked as bonding.txt. Turns out, that the kernel already supports adapter teaming (only that it’s called adapter bonding) by itself. No need for the Broadcom solution anymore.

Setting it up was rather easy (besides my lazy SUSE admin can’t do it via yast; it has to be done on the file layer since “yast lan” is too stupid to even show the thing), it’s simply creating the interface configs via said “yast lan“, copying one of the “ifcfg-eth-id” files to another file called “ifcfg-bond0“, removing some stuff out of it and cleaning out the other interface configs.

Then simply shove in the following into the ifcfg-bond0 in /etc/sysconfig/network:

That’s it .. We just defined an adapter IP (the 141.53.5.x) and an virtual interface labeled as “int“. We also configured the MII-Monitor to check every 100ms(?) the link of each interface (those defined in BONDING_SLAVEx) if they are either up or down, as well as the adaptive load balancing (“mode=balance-alb“).

Only thing annoying me with that solution is the following entry in /var/log/messages:

See the warning ? I can’t get it to shut up .. I also tried loading the mii.ko module, but it won’t shut up … damn 🙁

Well, at least the adapter teaming works as desired (still haven’t measured the performance impact with this setup – really need a clever way to do that) and I can plug one of the two cables connected to this box and still have one interface online and a continuous connection. yay ❗

SLES10 on pSeries

Christian Leave a comment

Okay, yet another day passed by blazing fast. I had a good day at work, spent nearly the whole day trying to get my bloody systems hooked up to our SAN (which was interrupted by a non-working SAN-switch, disappearing WWN’s, lunch and my trainees), messing around with our internal network, hacking our Blade Chassis switches to get me what I want and some random paperwork.

But first things first .. We installed SLES10 on a pSeries box the other day (I think on Monday), and now I’m trying to get the WWN of it’s Emulex HBA, out of either sysfs or procfs. But whatcha’ thinking ?

I can’t get the dreaded WWN our of anything. Emulex’s hbacmd (from their HBAnyware utility) tells me there is no HBA and/or I don’t have the lpfc driver loaded (which can’t be, since I see IBM Tape Drives and my DS4300/FAStT900 via the lpfc), which is like … 😡

So if any Emulex/pSeries expert is reading this, *please* (I beg you) tell me how the frack I get the WWN squashed out of it without looking either at the back of the rack or into the BIOS.

And here’s just for the record (my own – so I don’t need to look it up more often) the way on how to reset the attention indicators (basically LED’s) on the front of a pSeries box running Linux, which gets turned on when either resetting the box or killing it in startup:

That’s it, the LED is off.

Continuing on SLES10

Christian Leave a comment

OK, it turns out that I was rather stupid when configuring the my.cnf. As it turned out, the effect I was seeing was due to the presence of two log-bin lines, which looked like the following:

And some lines down there was this:

Now the next thing I encountered was while importing our old databases (they are like 1.1GiB each, 25 databases total). The second MySQL Master (and his Slave) will choke as soon as you dump the data too fast into the first Master, as the binlog seems to be too big for MySQL to transfer it via TCP (smth like “Packet too large – try increasing max_packet_size” in the error-log; only problem was that max_packet_size was already at 1GiB which is the absolut maximum for MySQL 5.0 according to the handbook).

A way around this (thanks to a co-worker who pushed me towards this road) is disabling all the MySQL Master/Slave stuff in your my.cnf, start the mysql daemon as a simple, dumb database, import all your databases, stop the mysql daemon; tar up the whole BASEDIR and scp/rssh it to your second master.

Clean out the BASEDIR on the second master, untar your tarball, edit your my.cnf again to include the whole Master/Slave portions on both boxes and you should be up and running 😀

I haven’t run any tests on the MasterMaster replication yet, but I’ll do that as soon as I’m at work again (which is the 27th June, as I’m off for vacation since yesterday, yay!)

Back at SLES10

Christian Leave a comment

Here I am, sitting at my desk on a Thuesday evening thinking about what happened the last few days.

  1. I finally got to play around with our PacketPro 450 Cluster (nifty LoadBalancing appliance)
  2. We reworked the network the way *we* want it (and not that tool of a wannabe sysadmin)
  3. We mostly figured out how to do the LoadBalancing right, we just need to find some bugs in the LoadBalancer software (like the thing is failing over to its slave from time to time, but keeping the IP address for himself) or let the guys at teamix do their work and hopefully get a working release within the next week or so
  4. I figured out how to setup interface bonding with SLES10 (it was quite straight forward, thanks to the excellent in-kernel documentation), and we’re using an active-backup mode for now
  5. I still need to figure out how to do the MySQL Master<->Master replication right .. I’m currently building fresh RPM’s on one of those Dell blades (yes, they ROCK!) which will hopefully be finished till I’m at the office tomorrow.
  6. Pt. 5 also includes figuring out how to pass MySQL a custom location for the binary-log, at least that’s what the handbook says in Chapter “5.11.3. The Binary Log” …

When started with the –log-bin[=base_name] option, mysqld writes a log file containing all SQL commands that update data. If no base_name value is given, the default name is the name of the host machine followed by -bin. If the basename is given, but not as an absolute pathname, the server writes the file in the data directory. It is recommended that you specify a basename; see Section B.1.8.1, “Open Issues in MySQL”, for the reason.

That behavior works for –log-bin-index (like log-bin-index=/mysql/binlogs/$HOSTNAME.idx), but doesn’t for –log-bin. *shrug* I’ll see if that is fixed with something >5.0.18 (that’s what SLES10 currently ships).

I’m also looking for a network topology drawing program (possibly free), as Microsoft Visio (either 2003 or 2007, Standard or Professional) is nice, but still can’t draw shit correctly. So I stumbled upon yEd, which seems to look nice (I haven’t yet looked at it, but will tomorrow) that hopefully gives me the opportunity to draw/visualize my setup at work 😯

Personal progress

Christian Leave a comment

Well, I haven’t blogged in ages (like about a month).

To answer some of the questions I got either via mail or on IRC …

  1. yes, I’m still alive; you ain’t gonna get away that easy 😛
  2. yes, I finally managed to buy my shiny new car
  3. I’m still working on apache, virtualization, mobile, kernel, hardened …

Though I may haven’t been that active recently (I don’t look at CIA or do I count my own commits), I think I managed to close some apache bugs and some of my own. I’m still looking for someone interested in working on the ipw3945 related packages, as I’m kinda sick of that buggish thing.

On the (kinda) bright side of life, work has been a real blast the last few weeks. I started designing a new cluster/fail over system for our web server, also planning on buying some more hardware (as in 2* DS4700, a Cisco MDS9506, and some thin clients) which will require some interesting, additional work.

I finally got that backstabbing BAOH ( kinda like “BOFH – bastard operator from hell”, it’s my personal bastard administator from hell) off my back, and I’m finally able to do some work 😯 !

My life is still a bit messed up, as is my nutrition; but it’s all getting better 😆 (thanks to a kind person, who’s been telling me that I’m all cute and adorable 🙄 ) Thank you !!

Bla-bla-bla, yada-yada-yada

Christian Leave a comment

Interesting title … *shrug* I’m sitting back home, in my bed, while I’m supposed to be at work. I’m now a certified sicko due to a common cold.

Watching the toaster series once again (that’s Battlestar Galactica) and trying to think about some things. Heh, well thinking isn’t the right term; it’s more like meditate over stuff happening in the last few months.

Guess this stupid cold is dictating my weekend 😡 and hopefully I’m gonna be better on Monday. Have fun.

grub via serial console

Christian Leave a comment

As I have quite some trouble every time I need grub via serial console, here’s just my personal reminder on how to do it right:

Additionally the agetty entry for ttyS0 in /etc/inittab needs to be uncommented and changed accordingly to the serial speed

Cisco sucks

Christian Leave a comment

OK, today we had somewhat of an emergency. The core-router for our entire network at work had some kind of hardware defect and repeatedly rebooted every three minutes caused the whole network to go *cabooom*. Usually (you would think), stuff in the same subnet (or VLAN) would still see each other (again, you would think) .. but apparently the VLAN/subnet database is stored on the core router and took *all* subnets with it.

So the core router took our NAS cluster down (as they lost their “PUBLIC” interface) and apparently a minute after the core router went down, our FC storage started sending resets to the FC bus … and there went our ESX cluster ..

I’m still pretty unsure what *exactly* caused the FC storage to send those TGT resets, but it looks like is has something to do with the core router vanishing, as the same thing happened already two times in the past exactly after the core router blew. Still this shouldn’t be happening, as the FC network and the normal network are completely separated (despite the storage having a management port). So stay tuned for some more IBM fun *sigh*

SLES-9 (once again)

Christian 1 Comment

OK, so today was the highlight of the week … We updated apache2 on Tuesday (yeah, that’s still 2.0.49, so if you have some exploits – try them 😛 ) and now out of the sudden we have major performance issues. We looked nearly the whole forenoon for a reason, *why* the frackin’ apache was using 236% of the CPU’s.

In the afternoon, when my co-worker decided to go home (that was ~1500), I decided to revert back to the old patch level. But that isn’t as easy as you think (at least on SLES). The only thing I wanted to do, was something like this:

Looks like SuSE (or Novell who bought SuSE sometime 3 or 4 years ago) doesn’t consider reverting to an older patch level. Which means I would have to remove apache2, apache2-prefork, apache2-mod_php4; fetch the basic RPMS from our FTP server (which sadly forbids directory listing, so I can’t exactly look for the original RPMS) and I tried to blindly to fetch them.

Foooked. Didn’t work .. now I cron’ed the POS to restart every half an hour, so at least we have *some* solution. Will see about reverting the the last patch tomorrow again, hopefully I’ll find the original RPMS.

Waiting

Christian Leave a comment

We are still waiting for the money promised by the state and the country for our HBFG (again, it’s “Hochschulbauförderungsgesetz”), that hopefully is reducing or eliminating our storage/SAN problem we have currently. Right now we have to Cisco MDS9216 (that’s a 16-port 2GBps SAN-switch, two for redundancy), which means we only have 16 SAN-ports. That isn’t much, but still is to less, as we have like 30 machines or so, that *really* need access to the SAN, so we either end up unplugging some of them from the SAN or merge them onto some big machines (like our x366).

The other side of the problem is the storage .. Currently that isn’t redundant, which means we’re fucked if the storage decides to not come up, or one of the controller smokes .. So were looking at two DS4700 with 2 enclosures each filled with 300GB 2GBps FC disks. That will hopefully also solve our constant lack of rackspace.

Apart from that, we took a look at the terminal server market, heard someone from Citrix, looked ourselves at 2X (and I think we are going with the 2X solution – even if they don’t support the authentication passthrough – yet). We might want to consider buying dedicated hardware for the terminal servers, as I implemented them running on the ESX which isn’t a permanent solution, as at least the students will work on those terminal servers 0700-2200, that means continuous load in that time, which isn’t good for the ESX Cluster, as they are pretty loaded already.

We’re also looking in buying a third box for the ESX Cluster, probably one of the same as we have currently (that is x366 – with 2 DC Xeon’s, 16GB RAM, 2×73 GB SAS, 2x dual-port Intel NIC, 2x dual-port FC HBA) to get some extra capacity.

Recently I did some experiments with Gentoo as MySQL cluster (master< ->master replication for our upcoming database servers – that’s what the blade chassis and the two blades are for) and noticed that the Gentoo VM’s were sucking up RAM and didn’t release it, so I had to reset them every morning, in order to free some RAM. I guess I should poke Chris a bit about that, as he told me back at FOSDEM that he was doing some load testing with a similar setup not so far ago.