Category: Life

The Letting Go (why does it hurt so much ?)

Christian 2 Comments

As some of you know, I had some company the last few weeks by two amiable girls from Belgium. We did some hanging out, went out a couple of times and basically had a great time.

One thing I didn’t thought was possible, is them getting close to me (as in “I’d really like them to stay some time”) and as I gave them a ride to the train station yesterday, I really had my problems. Out of the sudden I had a feeling not to let them go, or is it the feeling to let friends (and I’m talking about real friends) slip by, without even knowing them for more than 3 weeks.

Well, it’s too late now; they are back in Eupen by now (hopefully 😥 ) and I’m probably never gonna see them again.

Post FOSDEM 2007 thoughts (do I know you ?)

Christian Leave a comment

Some of us attended this years FOSDEM in Brussels (thanks to Dimitry it was *really*, *really* great).

We (at least the ones attending) got to know each other a bit better (I even got to know some pre Gentoo devs .. yeah, you), and some time after FOSDEM (I think it was ~3 days afterwards), Petteri (betelgeuse) asked me why people all of the sudden start to call him with his first name on IRC.

I think the cause for that is just seeing each other for some time (like 2 days in a row), talking to the other(s) in person makes you feel closer to him/her (you hopefully know what I mean 😯 ). Or maybe that’s just me.

After FOSDEM the virtual bonds changed, I started to call quite some people by their first name as did others.

Cisco sucks

Christian Leave a comment

OK, today we had somewhat of an emergency. The core-router for our entire network at work had some kind of hardware defect and repeatedly rebooted every three minutes caused the whole network to go *cabooom*. Usually (you would think), stuff in the same subnet (or VLAN) would still see each other (again, you would think) .. but apparently the VLAN/subnet database is stored on the core router and took *all* subnets with it.

So the core router took our NAS cluster down (as they lost their “PUBLIC” interface) and apparently a minute after the core router went down, our FC storage started sending resets to the FC bus … and there went our ESX cluster ..

I’m still pretty unsure what *exactly* caused the FC storage to send those TGT resets, but it looks like is has something to do with the core router vanishing, as the same thing happened already two times in the past exactly after the core router blew. Still this shouldn’t be happening, as the FC network and the normal network are completely separated (despite the storage having a management port). So stay tuned for some more IBM fun *sigh*

SLES-9 (once again)

Christian 1 Comment

OK, so today was the highlight of the week … We updated apache2 on Tuesday (yeah, that’s still 2.0.49, so if you have some exploits – try them 😛 ) and now out of the sudden we have major performance issues. We looked nearly the whole forenoon for a reason, *why* the frackin’ apache was using 236% of the CPU’s.

In the afternoon, when my co-worker decided to go home (that was ~1500), I decided to revert back to the old patch level. But that isn’t as easy as you think (at least on SLES). The only thing I wanted to do, was something like this:

Looks like SuSE (or Novell who bought SuSE sometime 3 or 4 years ago) doesn’t consider reverting to an older patch level. Which means I would have to remove apache2, apache2-prefork, apache2-mod_php4; fetch the basic RPMS from our FTP server (which sadly forbids directory listing, so I can’t exactly look for the original RPMS) and I tried to blindly to fetch them.

Foooked. Didn’t work .. now I cron’ed the POS to restart every half an hour, so at least we have *some* solution. Will see about reverting the the last patch tomorrow again, hopefully I’ll find the original RPMS.

Waiting

Christian Leave a comment

We are still waiting for the money promised by the state and the country for our HBFG (again, it’s “Hochschulbauförderungsgesetz”), that hopefully is reducing or eliminating our storage/SAN problem we have currently. Right now we have to Cisco MDS9216 (that’s a 16-port 2GBps SAN-switch, two for redundancy), which means we only have 16 SAN-ports. That isn’t much, but still is to less, as we have like 30 machines or so, that *really* need access to the SAN, so we either end up unplugging some of them from the SAN or merge them onto some big machines (like our x366).

The other side of the problem is the storage .. Currently that isn’t redundant, which means we’re fucked if the storage decides to not come up, or one of the controller smokes .. So were looking at two DS4700 with 2 enclosures each filled with 300GB 2GBps FC disks. That will hopefully also solve our constant lack of rackspace.

Apart from that, we took a look at the terminal server market, heard someone from Citrix, looked ourselves at 2X (and I think we are going with the 2X solution – even if they don’t support the authentication passthrough – yet). We might want to consider buying dedicated hardware for the terminal servers, as I implemented them running on the ESX which isn’t a permanent solution, as at least the students will work on those terminal servers 0700-2200, that means continuous load in that time, which isn’t good for the ESX Cluster, as they are pretty loaded already.

We’re also looking in buying a third box for the ESX Cluster, probably one of the same as we have currently (that is x366 – with 2 DC Xeon’s, 16GB RAM, 2×73 GB SAS, 2x dual-port Intel NIC, 2x dual-port FC HBA) to get some extra capacity.

Recently I did some experiments with Gentoo as MySQL cluster (master< ->master replication for our upcoming database servers – that’s what the blade chassis and the two blades are for) and noticed that the Gentoo VM’s were sucking up RAM and didn’t release it, so I had to reset them every morning, in order to free some RAM. I guess I should poke Chris a bit about that, as he told me back at FOSDEM that he was doing some load testing with a similar setup not so far ago.

Learning french

Christian 1 Comment

The last few days in Brussels really made me think about learing some french again. There were times, I/we was/were completely lost, and I really hate being lost … So I’m going to look into learning french again. Will see if the local academy has something scheduled in their upcoming program. I really hope so, as I think french is something I could use.

Also I’m thinking about taking dance-lessons (yeah, yeah I know – I suck 😛 ), as I’d really like to dance … *mmmmmh*

Shibboleth (WTF is that?)

Christian Leave a comment

OK, I’m sitting now again in train (hrm, I get the feeling I’ve done that already in the last few days – oh wait, I was doing that just on Monday) this time to Berlin.

My boss ordered me to attend a workshop covering the implementation of Shibboleth (for those of you, who can’t associate anything with that term – it’s an implementation for single sign-on, also covering distributed authorization and authentication) somewhere in Berlin Spandau (Evangelisches Johannesstift Berlin).

Yesterday was quite amazing workwise, we lifted the 75kg Blade Chassis into the rack (*yuck* there was a time I was completely against Dell stuff, but recently that has changed), plugged all four C22 plugs into the rack’s PDU’s and into the chassis, patched the management interface (which is *waaay* to slow for a dedicated management daughter board) and for the first time started the chassis. *ugh* That scared me .. that wasn’t noise like a xSeries or any other rack-based server we have around, more like a starting airplane. You can literally stand in behind of the chassis, and get your hairs dried (if you need to). So I looked at the blades together with my co-worker and we figured, that they don’t have any coolers anymore, they are just using the cooling the chassis provides.

Another surprise awaited us, when we thought, we could use the integrated switch to provide network for both integrated network cards (Broadcome NetExtreme II). *sigh* You need two seperate switches to serve two network cards, even if you only have two blades in the chassis (which provides space for 10 blades). *sigh* That really sucks, but its the same with the FC stuff …

So, we are waiting yet again for Dell to make us an offer, and on top of that, the sales representative doesn’t have the slightest idea if the FC passthrough module includes SFP’s or not … *yuck*

I must say, I’m impressed by the Dell hardware, but I’m really disappointed by their sales representative.

Gentoo/hardened and the new toolchain

Christian Leave a comment

OK, as some of you have noticed; I prepared my box for the new toolchain, recompiled the stuff Kevin mentioned in the exact same order wrote down in his README, and it looks like it actually works with all my stuff I have on my box; except sys-libs/grub! *sigh*

Apparently, grub segfaults at boot and/or while running it from the chroot in the exact same spot, the new QA warnings complain about ..

So, I unpacked the libc and grub debug files, to get a clue where it’s failing and fed the program execution into gdb and viola:

I’m not yet sure if it really is the same spot, but I’ll let Kevin have a shot at it …

FOSDEM 2007 #4 (abstract)

Christian Leave a comment

The last three days have been quite amazing, I’m still stunned by all these impressions (basically meeting all those people you usually know though IRC), how nice people can be if they want (yeah dad, I’m talking about you :P)

Although I was pretty sad about leaving so early (I think, if I’m going to attend next year, I’m going by train and starting on Monday morning so I got one night more in Brussels), I’m also quite happy to be home again, as I was pretty phreaked (haha, self-pun intended) by all those people.

The Booth was fine from my POV, but it looks like users already requested some stuff already on the forums, which we hopefully will have next year/the next event (Christel or even Chris – get the event kits sorted out and prepared – pretty please with sugar on top 😛 ).

The talks in the DevRoom where great (at least the ones I attended) – most of them were well prepared (hi Christel 😛 ) and the company I had for these three days (all the senior devs around – especially vapier, wolf31o2, kingtaco, pylon and some of the former ones – hey Mr. “I want SeJo’s HumpGang on the picture” 😀 ) and of course our youngins (that’s Peter, Alexander and Robert).

FOSDEM 2007 #3 (WHUT)

Christian Leave a comment

Today, I got up again at 0630 *ugh* went to bath, woke up Torsten, afterwards the mystery guy and went downstairs to get some breakfast. I ate some cornflakes and some bread along with a glass of chilled orange juice, then went back up to get all our stuff and prepare the room the way we found it.

After we finished that, we went downstairs and met Alex along the way went to the cars and Torsten led us to the university. I went to the booth and noticed that Torsten and me where the only ones that early at the booth. So we sat down for a moment, cleaned up the booth space a bit and finally sat again preparing the notebooks.

Suddenly some of the Debian guys came along and asked for a Gentoo Linux Install-CD and is was like “WHUT”. Apparently they only needed it to fix a borked lilo.

I went over to the DevRoom to see if anyone else was there already, but there was “only” Petteri doing his presentation about Gentoo/JAVA stuff with Rob’s help (he lend Petteri his somewhat broken notebook because Petteri’s hard disk broke on the day before).

So I watched a bit and went then back to the booth. After hacking for half an hour on my broken system (haha, I tried updating glibc via binary packages yesterday again, I know I do suck 😎 ) slowly the other devs showed up one after another. So we manned the booth yet another day. The booth wasn’t that much visited as yesterday (which I kinda was pretty lucky about since yesterday was kinda rough) and I sort of vanished 1100’ish to get to Mike and Chris’s presentation.

I even managed to catch Danny talking about paludis and enjoyed it. Afterwards was Dad’s talk which was really interesting as well as Chris’s talk, though I mostly knew about that, as I’m currently building the hardened 2007.0 stages (which by the way still suck horribly, somehow the gcc ebuild still creates lib32 and lib64 directories on a x86 stage-build and puts the lib_gcc.so.1 into the lib32 directory, I’ll probably have to poke Chris or Andrew again when I’m back home or try to build a 2006.1 stage with grp’s I may get from the old livecd/dvd).

Afterwards we had sort-of a lunch break (hah, no food again for me) and heard Christel talking about “What makes Gentoo a community based distribution?”. Her talk was interesting (admitted she was a bit unprepared, she admitted it herself, don’t sue me :P), though she prepared the slides like during Mike’s and Chris’s talk. 😮 The presentation was still pretty much interesting (hah, Al-Quaida and P0RN were mentioned on the community’s thoughts *ugh* – I doubt it though) from a users POV.

Afterwards I said goodbye to an pretty “good, old” friend (my handsome Rob that is) and a newly won friend (djay-il from userrep working for exanet.com) as they wanted to get to the airport in time, Rob for his flight to Geneva and Alexandre for his flight back to Israel (I don’t know where exactly). Ah, btw (and on behalf of djay-il) – the recent dbus update broke the complete GNOME, hah 😛

In addition to the previous presentation, Marius talked about portage foo, which I have to admit, only listened to with half an ear, as I was bashing Alex and Peter in #gentoo-dev at that time.

Nearly after Marius’s speech we left for the car to get home, since Markus needed to get to work on the next day. But before we could get outside of the door, Christel snapped us and we all did a sorta-like group photo (even SpanKY is on it, after some persuasion – including Surprise Buttsecks). Afterwards we headed for the car, and started our run for home.

It took us about 8 hours straight to get back to Hamburg (and Marius to Bremen), Markus probably needed half an hour more since he gave me a ride to the central station where I sat in McDonald’s for about 3 hours, eating McChicken, Chicken McNuggets, a McFlurry, drank a Coke, an orange juice (they do offer Hohes C now at McDonald’s) and about 5 or so cups of tea 😎 (I know, I’m crazy .. I usually drink about the same amount for breakfast when I’m supposed to get to work).

I’m sitting now in the train back home (first class of course), writing this blog-o-report for all those people, crazy enough to actually read this “report” about my experiences at this years FOSDEM.