December 31, 2007

As I pointed out back in October, it’s rather easy to create a setup which syncs a built binary package to a remote node (which is serving them to the world – via http,rsync,ftp – pick your poison).

Now, ever since we had slight space problems on miranda (cough my binpkgs cough), I wanted to look into methods on how to get rid of storing them on the buildnode and the webnode. I think now (hehe, it’s only 7pm), I finally managed to get a “proper” bashrc which does a lot of that foo. Take a look at this:

...
syncpkg() {
  # Syncing the binary tbz2 to my webhost
  if is_feature "buildpkg" &amp;&amp; 
  [[ -n $REPO_HOST &amp;&amp; -n $REPO_BASE &amp;&amp; -n $REPO_PATH ]] ; then
    REMOTE_TARGET="$REPO_HOST:$REPO_BASE/$REPO_PATH"
    REMOTE_ECACHE="$REPO_BASE/$REPO_PATH/settings/.ebuild.x"

    einfo "Publishing data to remote repository ($REPO_PATH) on ${REPO_HOST##*@}"

    if ! $( ssh $REPO_HOST "test -d $REPO_BASE/$REPO_PATH/settings" ) ; then
      ssh $REPO_HOST "mkdir -p $REPO_BASE/$REPO_PATH/settings" 
        &gt;&gt; /var/log/syncpkg.log 2&gt;&amp;1
    fi

    if [ -x /usr/bin/q ] ; then
      /usr/bin/qlist -IvCU &gt; /etc/portage/package.list
    fi

    ebegin " [ SYNC: 1/3] /etc/make.conf, /etc/portage/package* to $REPO_PATH/settings"
    rssh /etc/{make.conf,portage/package*,portage/profile,portage/bin,portage/bashrc} 
      $REMOTE_TARGET/settings/ &gt; /var/log/syncpkg.log 2&gt;&amp;1
    eend $?

    ebegin " [ SYNC: 2/3] $PKGDIR to $REPO_PATH"
    rssh $PKGDIR/ $REMOTE_TARGET/ &gt;&gt; /var/log/syncpkg.log 2&gt;&amp;1
    eend $?

    ebegin " [ SYNC: 3/3] Cleaning $PKGDIR"
    rm -rf $PKGDIR/* &gt;&gt; /var/log/syncpkg.log 2&gt;&amp;1
    eend $?

    # We sync our copy of .ebuild.x over to the webnode, as it might not be
    # synced as often / at the same time as the buildnode. Thus the
    # `qpkg --eclean' would remove packages, which *are* in the tree, but
    # the webnode hasn't synced up yet.

    rssh $PORTDIR/.ebuild.x $REMOTE_TARGET/settings/.ebuild.x 
      &gt;&gt; /var/log/syncpkg.log 2&gt;&amp;1

    if $( ssh $REPO_HOST "test -f $REMOTE_ECACHE" ) ; then
      ebegin " [MAINT: 1/3] Removing stale packages in $REPO_PATH"
      ssh $REPO_HOST "CACHE_EBUILD_FILE=$REMOTE_ECACHE qpkg -Eq -P 
        $REPO_BASE/$REPO_PATH/" &gt;&gt; /var/log/syncpkg.log 2&gt;&amp;1
      eend $?
    fi

    # And now remove the .ebuild.x copy again.
    ssh $REPO_HOST "rm -f $REMOTE_ECACHE" &gt;&gt; /var/log/syncpkg.log 2&gt;&amp;1

    # The user ssh'ing, needs to be in the portage group on the remote
    # end. Otherwise, things *will* go wrong (like genpkgindex being
    # unable to write to /var/cache/edb/xpak). Also /var/cache/edb/xpak
    # needs to be owned by portage:portage, as well as group-writeable.

    if $( ssh $REPO_HOST "test -w /var/cache/edb/xpak" ) ; then
      ebegin " [MAINT: 2/3] Regenerating $REPO_PATH/Packages"
      ssh $REPO_HOST "$REPO_BASE/$REPO_PATH/settings/bin/genpkgindex 
        $REPO_BASE/$REPO_PATH/All" &gt;&gt; /var/log/syncpkg.log 2&gt;&amp;1
      eend $?
    fi

    if ! $( ssh $REPO_HOST "test -L $REPO_BASE/$REPO_PATH/Packages" ) ; then
      ebegin " [MAINT: 3/3] Fixing $REPO_PATH/Packages symlink"
      ssh $REPO_HOST "cd $REPO_BASE/$REPO_PATH; rm -f Packages; ln -s 
        All/Packages" &gt;&gt; /var/log/syncpkg.log 2&gt;&amp;1
      eend $?
    fi
  fi
}
...

...

syncpkg() {

# Syncing the binary tbz2 to my webhost

if is_feature "buildpkg" &&

[[ -n $REPO_HOST && -n $REPO_BASE && -n $REPO_PATH ]] ; then

REMOTE_TARGET="$REPO_HOST:$REPO_BASE/$REPO_PATH"

REMOTE_ECACHE="$REPO_BASE/$REPO_PATH/settings/.ebuild.x"

einfo "Publishing data to remote repository ($REPO_PATH) on ${REPO_HOST##*@}"

if ! $( ssh $REPO_HOST "test -d $REPO_BASE/$REPO_PATH/settings" ) ; then

ssh $REPO_HOST "mkdir -p $REPO_BASE/$REPO_PATH/settings"

>> /var/log/syncpkg.log 2>&1

if [ -x /usr/bin/q ] ; then

/usr/bin/qlist -IvCU > /etc/portage/package.list

ebegin " [ SYNC: 1/3] /etc/make.conf, /etc/portage/package* to $REPO_PATH/settings"

rssh /etc/{make.conf,portage/package*,portage/profile,portage/bin,portage/bashrc}

$REMOTE_TARGET/settings/ > /var/log/syncpkg.log 2>&1

eend $?

ebegin " [ SYNC: 2/3] $PKGDIR to $REPO_PATH"

rssh $PKGDIR/ $REMOTE_TARGET/ >> /var/log/syncpkg.log 2>&1

eend $?

ebegin " [ SYNC: 3/3] Cleaning $PKGDIR"

rm -rf $PKGDIR/* >> /var/log/syncpkg.log 2>&1

eend $?

# We sync our copy of .ebuild.x over to the webnode, as it might not be

# synced as often / at the same time as the buildnode. Thus the

# `qpkg --eclean' would remove packages, which *are* in the tree, but

# the webnode hasn't synced up yet.

rssh $PORTDIR/.ebuild.x $REMOTE_TARGET/settings/.ebuild.x

>> /var/log/syncpkg.log 2>&1

if $( ssh $REPO_HOST "test -f $REMOTE_ECACHE" ) ; then

ebegin " [MAINT: 1/3] Removing stale packages in $REPO_PATH"

ssh $REPO_HOST "CACHE_EBUILD_FILE=$REMOTE_ECACHE qpkg -Eq -P

$REPO_BASE/$REPO_PATH/" >> /var/log/syncpkg.log 2>&1

eend $?

# And now remove the .ebuild.x copy again.

ssh $REPO_HOST "rm -f $REMOTE_ECACHE" >> /var/log/syncpkg.log 2>&1

# The user ssh'ing, needs to be in the portage group on the remote

# end. Otherwise, things *will* go wrong (like genpkgindex being

# unable to write to /var/cache/edb/xpak). Also /var/cache/edb/xpak

# needs to be owned by portage:portage, as well as group-writeable.

if $( ssh $REPO_HOST "test -w /var/cache/edb/xpak" ) ; then

ebegin " [MAINT: 2/3] Regenerating $REPO_PATH/Packages"

ssh $REPO_HOST "$REPO_BASE/$REPO_PATH/settings/bin/genpkgindex

$REPO_BASE/$REPO_PATH/All" >> /var/log/syncpkg.log 2>&1

eend $?

if ! $( ssh $REPO_HOST "test -L $REPO_BASE/$REPO_PATH/Packages" ) ; then

ebegin " [MAINT: 3/3] Fixing $REPO_PATH/Packages symlink"

ssh $REPO_HOST "cd $REPO_BASE/$REPO_PATH; rm -f Packages; ln -s

All/Packages" >> /var/log/syncpkg.log 2>&1

eend $?

}

...

As you can see, it does a lot of things, which are all connected with binary package repositories (including cleaning up old packages no longer in the tree – trying not to waste too much space). Sadly, I currently have to use a custom patched qpkg version, as the one implementing the –eclean features isn’t in the tree yet. When I talked to Ned the other day, he complained about it being slow (well, yeah — it has to go through the whole tree) which I don’t really see when you look at what it’s doing.

Also, I had a weird phenomenon today happening: the buildnode built a binary package, sent it to the webnode, which ran qpkg --eclean' afterwards. But after that the binary package was gone. "Why" you ask now ? Well, apparently the webnode isn't synced the same time the buildnode syncs (the webnode is in Germany, the buildnode in the US). So I had to come up with a trick, in order to fool qpkg into not cleaning the freshly built binary packages. See the rssh’ in front of the qpkg call ? Guess what, that’s the lil’ dirty trick …

Anyway, the full bashrc is available. The next thing I’m gonna have to look at (which Markus already did), is building packages via buildbot.

Update: as you see, I updated the bashrc a bit. That’s because after writing this, I started a new (fresh) binpkg repository (empty), and out of the sudden the thing ain’t syncing correctly (as in no Packages file, no portage settings). Turns out, rsync doesn’t create directories which ain’t there. So another extra `ssh‘ execution to create the settings/ directory inside the repo.

Well, it’s yet again New Year’s Eve. Yet again a whole year passed by blazing fast, I didn’t manage to get everything done like I wanted.

That includes the following things:

getting a better job (and probably better paid too!)
getting a better life (well, it’s as it sounds like – my current life is rather unhealthy, and thanks to a friend I got the grip onto myself and started changing a few things – like doing a small workout every day, a bit more movement all over the day and so forth)

Which also means I do have some resolutions for the next year …

Become more active (like do a longer workout each day)
Get a better paid job (even if that’s going to hurt some people)
Fix my remaining health problems (like my foot, the back, …)

Now that sounds like I didn’t get anything done in the last 365 days, but that I sure did.

I finally managed to make my way through the slackers list (Fabian accused me I’d be orphaning half the tree – if at all, it was 1/12),
I did some major changes at work (though I still need to do some things – like fixing the MySQL replication with TYPO3).

I do have the feeling that the next year is gonna get interesting real soon. I do have a project for the implementation of a VDI based class-room scheduled early next year (budget still pending – so it’s a maybe); we still do have to review the available possibilities (which includes Dell – who apparently implemented exactly this for some university/technical university in Brandenburg), as well as some other small purchases.

Gentoo wise I can’t tell yet whether or not I still want to be part of it. The last few months have been rather tough for me, I’ve been haunted by guilt for other things, so I couldn’t care much about Gentoo. I’ve put away some of the burdens I had, in order to focus on the fun aspects of our beloved distribution (there isn’t much left sadly).

I’d like to thank those who had extra patience with me, thank those who took the time to talk to me, those who cheered me up when I needed it. It’s been a tough time, but thanks to a lot of amazing people (Norman, Michel, Christina, Alex, Diego, Ned, Chris, Robin, …) I got through it and I’m still here – alive and kicking 😛 !

Oh, and a happy new year !!

S	M	T	W	T	F	S
						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30	31

Day: December 31, 2007

Advanced bashrc (‘Turning a simple chroot into a binpkg repository’ continued)

Looking back (yet again)