Post FOSDEM 2007 thoughts (do I know you ?)

Some of us attended this years FOSDEM in Brussels (thanks to Dimitry it was *really*, *really* great).

We (at least the ones attending) got to know each other a bit better (I even got to know some pre Gentoo devs .. yeah, you), and some time after FOSDEM (I think it was ~3 days afterwards), Petteri (betelgeuse) asked me why people all of the sudden start to call him with his first name on IRC.

I think the cause for that is just seeing each other for some time (like 2 days in a row), talking to the other(s) in person makes you feel closer to him/her (you hopefully know what I mean 😯 ). Or maybe that’s just me.

After FOSDEM the virtual bonds changed, I started to call quite some people by their first name as did others.

Cisco sucks

OK, today we had somewhat of an emergency. The core-router for our entire network at work had some kind of hardware defect and repeatedly rebooted every three minutes caused the whole network to go *cabooom*. Usually (you would think), stuff in the same subnet (or VLAN) would still see each other (again, you would think) .. but apparently the VLAN/subnet database is stored on the core router and took *all* subnets with it.

So the core router took our NAS cluster down (as they lost their “PUBLIC” interface) and apparently a minute after the core router went down, our FC storage started sending resets to the FC bus … and there went our ESX cluster ..

I’m still pretty unsure what *exactly* caused the FC storage to send those TGT resets, but it looks like is has something to do with the core router vanishing, as the same thing happened already two times in the past exactly after the core router blew. Still this shouldn’t be happening, as the FC network and the normal network are completely separated (despite the storage having a management port). So stay tuned for some more IBM fun *sigh*

SLES-9 (once again)

OK, so today was the highlight of the week … We updated apache2 on Tuesday (yeah, that’s still 2.0.49, so if you have some exploits – try them 😛 ) and now out of the sudden we have major performance issues. We looked nearly the whole forenoon for a reason, *why* the frackin’ apache was using 236% of the CPU’s.

In the afternoon, when my co-worker decided to go home (that was ~1500), I decided to revert back to the old patch level. But that isn’t as easy as you think (at least on SLES). The only thing I wanted to do, was something like this:

Looks like SuSE (or Novell who bought SuSE sometime 3 or 4 years ago) doesn’t consider reverting to an older patch level. Which means I would have to remove apache2, apache2-prefork, apache2-mod_php4; fetch the basic RPMS from our FTP server (which sadly forbids directory listing, so I can’t exactly look for the original RPMS) and I tried to blindly to fetch them.

Foooked. Didn’t work .. now I cron’ed the POS to restart every half an hour, so at least we have *some* solution. Will see about reverting the the last patch tomorrow again, hopefully I’ll find the original RPMS.

Waiting

We are still waiting for the money promised by the state and the country for our HBFG (again, it’s “Hochschulbauförderungsgesetz”), that hopefully is reducing or eliminating our storage/SAN problem we have currently. Right now we have to Cisco MDS9216 (that’s a 16-port 2GBps SAN-switch, two for redundancy), which means we only have 16 SAN-ports. That isn’t much, but still is to less, as we have like 30 machines or so, that *really* need access to the SAN, so we either end up unplugging some of them from the SAN or merge them onto some big machines (like our x366).

The other side of the problem is the storage .. Currently that isn’t redundant, which means we’re fucked if the storage decides to not come up, or one of the controller smokes .. So were looking at two DS4700 with 2 enclosures each filled with 300GB 2GBps FC disks. That will hopefully also solve our constant lack of rackspace.

Apart from that, we took a look at the terminal server market, heard someone from Citrix, looked ourselves at 2X (and I think we are going with the 2X solution – even if they don’t support the authentication passthrough – yet). We might want to consider buying dedicated hardware for the terminal servers, as I implemented them running on the ESX which isn’t a permanent solution, as at least the students will work on those terminal servers 0700-2200, that means continuous load in that time, which isn’t good for the ESX Cluster, as they are pretty loaded already.

We’re also looking in buying a third box for the ESX Cluster, probably one of the same as we have currently (that is x366 – with 2 DC Xeon’s, 16GB RAM, 2×73 GB SAS, 2x dual-port Intel NIC, 2x dual-port FC HBA) to get some extra capacity.

Recently I did some experiments with Gentoo as MySQL cluster (master< ->master replication for our upcoming database servers – that’s what the blade chassis and the two blades are for) and noticed that the Gentoo VM’s were sucking up RAM and didn’t release it, so I had to reset them every morning, in order to free some RAM. I guess I should poke Chris a bit about that, as he told me back at FOSDEM that he was doing some load testing with a similar setup not so far ago.

Learning french

The last few days in Brussels really made me think about learing some french again. There were times, I/we was/were completely lost, and I really hate being lost … So I’m going to look into learning french again. Will see if the local academy has something scheduled in their upcoming program. I really hope so, as I think french is something I could use.

Also I’m thinking about taking dance-lessons (yeah, yeah I know – I suck 😛 ), as I’d really like to dance … *mmmmmh*

Shibboleth (WTF is that?)

OK, I’m sitting now again in train (hrm, I get the feeling I’ve done that already in the last few days – oh wait, I was doing that just on Monday) this time to Berlin.

My boss ordered me to attend a workshop covering the implementation of Shibboleth (for those of you, who can’t associate anything with that term – it’s an implementation for single sign-on, also covering distributed authorization and authentication) somewhere in Berlin Spandau (Evangelisches Johannesstift Berlin).

Yesterday was quite amazing workwise, we lifted the 75kg Blade Chassis into the rack (*yuck* there was a time I was completely against Dell stuff, but recently that has changed), plugged all four C22 plugs into the rack’s PDU’s and into the chassis, patched the management interface (which is *waaay* to slow for a dedicated management daughter board) and for the first time started the chassis. *ugh* That scared me .. that wasn’t noise like a xSeries or any other rack-based server we have around, more like a starting airplane. You can literally stand in behind of the chassis, and get your hairs dried (if you need to). So I looked at the blades together with my co-worker and we figured, that they don’t have any coolers anymore, they are just using the cooling the chassis provides.

Another surprise awaited us, when we thought, we could use the integrated switch to provide network for both integrated network cards (Broadcome NetExtreme II). *sigh* You need two seperate switches to serve two network cards, even if you only have two blades in the chassis (which provides space for 10 blades). *sigh* That really sucks, but its the same with the FC stuff …

So, we are waiting yet again for Dell to make us an offer, and on top of that, the sales representative doesn’t have the slightest idea if the FC passthrough module includes SFP’s or not … *yuck*

I must say, I’m impressed by the Dell hardware, but I’m really disappointed by their sales representative.

Gentoo/hardened and the new toolchain

OK, as some of you have noticed; I prepared my box for the new toolchain, recompiled the stuff Kevin mentioned in the exact same order wrote down in his README, and it looks like it actually works with all my stuff I have on my box; except sys-libs/grub! *sigh*

Apparently, grub segfaults at boot and/or while running it from the chroot in the exact same spot, the new QA warnings complain about ..

So, I unpacked the libc and grub debug files, to get a clue where it’s failing and fed the program execution into gdb and viola:

I’m not yet sure if it really is the same spot, but I’ll let Kevin have a shot at it …

FOSDEM 2007 #4 (abstract)

The last three days have been quite amazing, I’m still stunned by all these impressions (basically meeting all those people you usually know though IRC), how nice people can be if they want (yeah dad, I’m talking about you :P)

Although I was pretty sad about leaving so early (I think, if I’m going to attend next year, I’m going by train and starting on Monday morning so I got one night more in Brussels), I’m also quite happy to be home again, as I was pretty phreaked (haha, self-pun intended) by all those people.

The Booth was fine from my POV, but it looks like users already requested some stuff already on the forums, which we hopefully will have next year/the next event (Christel or even Chris – get the event kits sorted out and prepared – pretty please with sugar on top 😛 ).

The talks in the DevRoom where great (at least the ones I attended) – most of them were well prepared (hi Christel 😛 ) and the company I had for these three days (all the senior devs around – especially vapier, wolf31o2, kingtaco, pylon and some of the former ones – hey Mr. “I want SeJo’s HumpGang on the picture” 😀 ) and of course our youngins (that’s Peter, Alexander and Robert).

FOSDEM 2007 #3 (WHUT)

Today, I got up again at 0630 *ugh* went to bath, woke up Torsten, afterwards the mystery guy and went downstairs to get some breakfast. I ate some cornflakes and some bread along with a glass of chilled orange juice, then went back up to get all our stuff and prepare the room the way we found it.

After we finished that, we went downstairs and met Alex along the way went to the cars and Torsten led us to the university. I went to the booth and noticed that Torsten and me where the only ones that early at the booth. So we sat down for a moment, cleaned up the booth space a bit and finally sat again preparing the notebooks.

Suddenly some of the Debian guys came along and asked for a Gentoo Linux Install-CD and is was like “WHUT”. Apparently they only needed it to fix a borked lilo.

I went over to the DevRoom to see if anyone else was there already, but there was “only” Petteri doing his presentation about Gentoo/JAVA stuff with Rob’s help (he lend Petteri his somewhat broken notebook because Petteri’s hard disk broke on the day before).

So I watched a bit and went then back to the booth. After hacking for half an hour on my broken system (haha, I tried updating glibc via binary packages yesterday again, I know I do suck 😎 ) slowly the other devs showed up one after another. So we manned the booth yet another day. The booth wasn’t that much visited as yesterday (which I kinda was pretty lucky about since yesterday was kinda rough) and I sort of vanished 1100’ish to get to Mike and Chris’s presentation.

I even managed to catch Danny talking about paludis and enjoyed it. Afterwards was Dad’s talk which was really interesting as well as Chris’s talk, though I mostly knew about that, as I’m currently building the hardened 2007.0 stages (which by the way still suck horribly, somehow the gcc ebuild still creates lib32 and lib64 directories on a x86 stage-build and puts the lib_gcc.so.1 into the lib32 directory, I’ll probably have to poke Chris or Andrew again when I’m back home or try to build a 2006.1 stage with grp’s I may get from the old livecd/dvd).

Afterwards we had sort-of a lunch break (hah, no food again for me) and heard Christel talking about “What makes Gentoo a community based distribution?”. Her talk was interesting (admitted she was a bit unprepared, she admitted it herself, don’t sue me :P), though she prepared the slides like during Mike’s and Chris’s talk. 😮 The presentation was still pretty much interesting (hah, Al-Quaida and P0RN were mentioned on the community’s thoughts *ugh* – I doubt it though) from a users POV.

Afterwards I said goodbye to an pretty “good, old” friend (my handsome Rob that is) and a newly won friend (djay-il from userrep working for exanet.com) as they wanted to get to the airport in time, Rob for his flight to Geneva and Alexandre for his flight back to Israel (I don’t know where exactly). Ah, btw (and on behalf of djay-il) – the recent dbus update broke the complete GNOME, hah 😛

In addition to the previous presentation, Marius talked about portage foo, which I have to admit, only listened to with half an ear, as I was bashing Alex and Peter in #gentoo-dev at that time.

Nearly after Marius’s speech we left for the car to get home, since Markus needed to get to work on the next day. But before we could get outside of the door, Christel snapped us and we all did a sorta-like group photo (even SpanKY is on it, after some persuasion – including Surprise Buttsecks). Afterwards we headed for the car, and started our run for home.

It took us about 8 hours straight to get back to Hamburg (and Marius to Bremen), Markus probably needed half an hour more since he gave me a ride to the central station where I sat in McDonald’s for about 3 hours, eating McChicken, Chicken McNuggets, a McFlurry, drank a Coke, an orange juice (they do offer Hohes C now at McDonald’s) and about 5 or so cups of tea 😎 (I know, I’m crazy .. I usually drink about the same amount for breakfast when I’m supposed to get to work).

I’m sitting now in the train back home (first class of course), writing this blog-o-report for all those people, crazy enough to actually read this “report” about my experiences at this years FOSDEM.

FOSDEM 2007 #2 (more friends, Bryan and superman)

So, as I earlier mentioned, we finally found the U.L.B. and also found the booth and staffed it for nearly the whole day (I haven’t managed to view any of the speeches, neither one of the Gentoo ones nor any of the others), I talked to some interested people asking about Gentoo itself, but also asking about the EFIKA’s (the Genesi PPC’s), one by Camille and Julien providing video forwarding (from one EFIKA to another via cross-over cable), and the other one by Chris playing Descent 2 and Quake 2 at the end. Peter (I think the PR guy from Genesi Europe, will have to ask Chris / Mike again).

Ah, and I nearly forgot about the best part. I finally met my “new” dad ;-P though he’s a pole humper, he’s really great and I adore him for being himself (hey SpanKY), along with Chris (wolf31o2) who’s really adorable for being himself, and for being all knowledgeable. Though I also finally met SeJo (who’s really charming but also a bitch :-P) and KingTaco (who constantly tried to hump my dad), met most of our youngins, Alex and the amazing Peter (welp) and of course my all-time-favorite handsome Rob.

I talked pretty long’ish to Chris and to SpanKY (who was always trying to make fun out of me, and succeeded) and gained a lot of insight as well as got to know them. I also managed to get in touch with the rest of the developer attending FOSDEM (that’s genone, jokey, kugelfang, spb, dertobi123, wschlich, grobian, genstef, nattfodd, tove, pylon and all the other …).

Later, in the evening we went to some restaurant (by bus and by foot), which was funny though since we left around 1900 when everyone else was leaving too and about 200 other people were trying to get somewhere else of Brussels. It was really nice, christel shouting all over the street, telling some of us to get on the next bus, half a minute later to stop and get back here to the tram. After Petteri looked at the tram schedule, we agreed to go get the bus, because the tram was already late (we were too).

We drove till Brussels Center (the bus-driver was driving like mad, he took every hole in the street he could find and was waaay to fast), I managed to talk a bit with Rob about his work in Gentoo and real-life issues affecting the development and finally walked the remaining way to the Restaurant (we somehow got lost – again) but found our way to the “Rome”, which was really a nice also as the food (we had a three-course meal with either Tortellini or Carpaccio, and again beef and lamb as main course and weird bakery stuff with butterscotch).

Between the courses I talked some more to Bryan, Alexandre and Christel; all of them we’re quite charming all evening and suddenly Dimitry comes at us and is like, “Hey I need all your money” (OK, *now* I’m lying, that was Christel who came up with this sentence instead of Dimitry telling that he needed the money for the dinner collected :S). So Dimitry and Christel went ahead and collected the money from the people staying at the hostel (so Dimitry would be able to pay the manager). That went pretty good and was without trouble (at least the two didn’t say something about having trouble). So we sat again, and talked some more, some more horseporn came up.

And Dimitry came along yet again, asking poor Christel to collect the money for the dinner and that’s what they did. They went along the table and as they were at the end, where SpanKY and the rich dutch translators sit at, as she didn’t knew their names (the ones from the dutch people), she decided to accidentally piss them off (she labeled them “young one 1” and “young one 2”), which they found out later by looking at the sheet.

Soon they finished and Dimitry was able to pay the dinner. So he talked a bit with the restaurant people and soon found out, that the drinks (even the non-alcohol ones) weren’t free. So he raised some more money from everyone. As they finished, we sat again and talked some more (gay horseporn came up again) along with the “The Internet is for …” song (nooo, I’m not gonna say it!!), but instead Christel tried to convince everyone to sing it loudly, which of course everyone refused to do).

Sometime soon they threw us out, because we were way to much people (~50 or so, because some just came along) and we were quite loud too 😛

So we decided to get some beer (I really *so* wanted to get drunk this evening and probably Christel wanted to see me get drunk), so we went looking for another pub where all 50 of us could stay. We walked and walked and walked and walked and lost some people, as they apparently decided to go into the other direction and then sort of vanished (*pooof*). As we were back at the center of the city, someone decided to get into the street where it was smelling like pot. There was also a pub that was quite nice (according to Alexandre), but quite filled. So some of us decided to go back out, but the rest apparently decided to stay, as they were quite comfortable with it (Peter even asked me later what all that pulling was about).

So we went someplace else (at the Grand Place again), and we sat outside. The rest bought beers and handsome Rob insisted on still owing me a beer for something (I’ve no clue what for he would owe me a beer) so he bought me one ! yay ! Hail Rob !!!!1one

We spend some time sitting there and making fun out of jokey (apparently he was on the menu at this bar, pub, whatever) and had some more fun listening to the “The Internet is for …” song again and some of us ordered another beer. Rob went of to make some photo’s of the city hall for his fiance(?).

He came back a few minutes afterwards telling us (what we already knew) that it is pretty cold (he was all tucked into his jacket, which he bought on advice of his fiance – he should tell that story himself). So we walked back to the hostel (it was quite a bit of a walk, but it was nice), Christel was completely out of breath and we met some of the userrep people back in front on the youth hostel and we had some more talking (again). Somehow, one of them came up with the idea that we should buy some latex or spandex superman-like capes for the next Gentoo Event for all developer attending with like pink or purple color and the Gentoo logo on the chest and Bryan was like “I would definitely wear that” and “that would probably scare off all potential users” and he was really encouraged about it.

We did some more talking and I got a bit insight into the userrep stuff (thanks to Christel and Bryan explaining) and finally went to bed around 0200 after trying to bribe the manager to give some more beer. My other room mates (that’s Marius, Markus, Robert, Torsten and some mystery guy) we’re apparently loooong sleeping.