Load Balancing

PacketPro 1.7.0

January 15, 2008August 16, 2014 Christian Leave a comment

After blogging the last time about the PacketPro 450 LoadBalancer appliance, the guys over at teamix seem to have taken that to heart and implemented a rather nifty thing for their new release.

It’s called “Port forwarding“, which is basically what you’d figure from the name. It bounces ports around the load balancer, but saves you from creating a separate virtual server (and adding the physical servers to that one), but also saves you from modifying the syslog-ng configuration on the balanced servers.

TYPO3 and MySQL replication

September 8, 2007June 21, 2013 Christian Leave a comment

Apparently the TYPO3 version we are using, doesn’t play too nice with the MySQL MasterMaster replication.

Sometimes, something like this is going to happen:

070826  0:44:32 [ERROR] Slave: Error &#039;Duplicate entry &#039;75-222419149&#039; for key 1&#039; on query. Default database: &#039;t3nb&#039;. Query: &#039;INSERT INTO cache_pagesection
070826  0:44:32 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with &quot;SLAVE START&quot;. We stopped at log &#039;dbc-mysql1.000192&#039; position 611861372

070826 0:44:32 [ERROR] Slave: Error 'Duplicate entry '75-222419149' for key 1' on query. Default database: 't3nb'. Query: 'INSERT INTO cache_pagesection

070826 0:44:32 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbc-mysql1.000192' position 611861372

Well, as you can see from the last line in the log, the Slave-SQL thread found a duplicate entry and thought it is smart to just turn off the thread instead of disregarding the just made entry. So from now on, both databases drift since there ain’t no replication anymore until someone kick starts the replication again (someone being me).

Anyway, I think I finally traced the fucker down, supposedly one of the problematic cases is located in t3lib/class.t3lib_tstemplate.php on line 362.

$GLOBALS[&#039;TYPO3_DB&#039;]-&gt;exec_DELETEquery(&#039;cache_pagesection&#039;, &#039;page_id=&#039;.intval($GLOBALS[&#039;TSFE&#039;]-&gt;id).&#039; AND mpvar_hash=&#039;.t3lib_div::md5int($GLOBALS[&#039;TSFE&#039;]-&gt;MP));
$GLOBALS[&#039;TYPO3_DB&#039;]-&gt;exec_INSERTquery(&#039;cache_pagesection&#039;, $insertFields);

1 2	$GLOBALS['TYPO3_DB']->exec_DELETEquery('cache_pagesection', 'page_id='.intval($GLOBALS['TSFE']->id).' AND mpvar_hash='.t3lib_div::md5int($GLOBALS['TSFE']->MP)); $GLOBALS['TYPO3_DB']->exec_INSERTquery('cache_pagesection', $insertFields);

Basically what TYPO3 is doing is a DELETE and an INSERT right afterwards. But apparently, it doesn’t check whether the DELETE even succeeded. I hacked it for now, simply adding this:

-                               $GLOBALS[&#039;TYPO3_DB&#039;]-&gt;exec_INSERTquery(&#039;cache_pagesection&#039;, $insertFields);
+                               // Only insert a new cache entry with the same value, if the DELETE succeeded
+                               if ($GLOBALS[&#039;TYPO3_DB&#039;]-&gt;sql_affected_rows() == 1)
+                                       $GLOBALS[&#039;TYPO3_DB&#039;]-&gt;exec_INSERTquery(&#039;cache_pagesection&#039;, $insertFields);
+

- $GLOBALS['TYPO3_DB']->exec_INSERTquery('cache_pagesection', $insertFields);

+ // Only insert a new cache entry with the same value, if the DELETE succeeded

+ if ($GLOBALS['TYPO3_DB']->sql_affected_rows() == 1)

+ $GLOBALS['TYPO3_DB']->exec_INSERTquery('cache_pagesection', $insertFields);

Sadly, this looks more and more like a race-condition between the two boxes (as in the replication / UPDATE being too slow), when users visit a edited site, that hasn’t had it’s cache regenerated yet. Problem is, it ain’t just this single spot, but also the search indexing, image cache and the whole page cache. For now we switched the cluster to active/passive load balancing, till we have a chance to see if a newer TYPO3 fixes those issues.

miimon, arp_interval and the code

July 4, 2007August 16, 2014 Christian Leave a comment

After today’s adventure with the kernel bonding, I just took a look at the code …

         if (miimon) {
                 printk(KERN_INFO DRV_NAME
                        ": MII link monitoring set to %d msn",
                        miimon);
         } else if (arp_interval) {
                 int i;

                 printk(KERN_INFO DRV_NAME
                        ": ARP monitoring set to %d ms, validate %s, with %d target(s):",
                        arp_interval,
                        arp_validate_tbl[arp_validate_value].modename,
                        arp_ip_count);

                 for (i = 0; i &lt; arp_ip_count; i++)
                         printk (" %s", arp_ip_target[i]);

                 printk("n");

         } else {
                 /* miimon and arp_interval not set, we need one so things
                  * work as expected, see bonding.txt for details
                  */
                 printk(KERN_WARNING DRV_NAME
                        ": Warning: either miimon or arp_interval and "
                        "arp_ip_target module parameters must be specified, "
                        "otherwise bonding will not detect link failures! see "
                        "bonding.txt for details.n");
         }

if (miimon) {

printk(KERN_INFO DRV_NAME

": MII link monitoring set to %d msn",

miimon);

} else if (arp_interval) {

int i;

printk(KERN_INFO DRV_NAME

": ARP monitoring set to %d ms, validate %s, with %d target(s):",

arp_interval,

arp_validate_tbl[arp_validate_value].modename,

arp_ip_count);

for (i = 0; i < arp_ip_count; i++)

printk (" %s", arp_ip_target[i]);

printk("n");

} else {

/* miimon and arp_interval not set, we need one so things

* work as expected, see bonding.txt for details

printk(KERN_WARNING DRV_NAME

": Warning: either miimon or arp_interval and "

"arp_ip_target module parameters must be specified, "

"otherwise bonding will not detect link failures! see "

"bonding.txt for details.n");

}

If I read it right, you only get the KERN_WARNING for “either miimon or arp_interval” only if miimon or arp_interval isn’t set … but at least my config says it is .. *shrug* .. bed time for me 🙄

Adapter teaming on SLES10

July 4, 2007June 21, 2013 Christian 2 Comments

Since one of the requirements for my current project is having NIC redundancy, I didn’t get around looking at the available “adapter teaming” (or adapter bonding) solutions available for Linux/SLES.

First I tried to dig into the Broadcom solution (since the Blade I first implemented the stuff uses a Broadcom NetXtreme II card) , but found out pretty soon that the basp configuration tool, which is *only* available on the Broadcom driver CD’s shipped with the Blade itself, pretty much doesn’t work.

Some hours googling later at how to get the frickin’ Broadcom crap working, I stumbled upon a file linked as bonding.txt. Turns out, that the kernel already supports adapter teaming (only that it’s called adapter bonding) by itself. No need for the Broadcom solution anymore.

Setting it up was rather easy (besides my lazy SUSE admin can’t do it via yast; it has to be done on the file layer since “yast lan” is too stupid to even show the thing), it’s simply creating the interface configs via said “yast lan“, copying one of the “ifcfg-eth-id” files to another file called “ifcfg-bond0“, removing some stuff out of it and cleaning out the other interface configs.

Then simply shove in the following into the ifcfg-bond0 in /etc/sysconfig/network:

IPADDR=&quot;141.53.5.141&quot;
NETMASK=&quot;255.255.255.0&quot;
NETWORK=&quot;141.53.5.0&quot;
MTU=&quot;&quot;
REMOTE_IPADDR=&quot;&quot;
STARTMODE=&quot;auto&quot;
BONDING_MASTER=&quot;yes&quot;
BONDING_MODULE_OPTS=&quot;miimon=100 mode=balance-alb&quot;
BONDING_SLAVE0=&quot;bus-pci-0000:02:00.0&quot;
BONDING_SLAVE1=&quot;bus-pci-0000:06:00.0&quot;

IPADDR_int=&quot;172.16.234.41&quot;
NETMASK_int=&quot;255.255.255.0&quot;
NETWORK_int=&quot;172.16.234.0&quot;
LABEL_int=&quot;int&quot;

IPADDR="141.53.5.141"

NETMASK="255.255.255.0"

NETWORK="141.53.5.0"

MTU=""

REMOTE_IPADDR=""

STARTMODE="auto"

BONDING_MASTER="yes"

BONDING_MODULE_OPTS="miimon=100 mode=balance-alb"

BONDING_SLAVE0="bus-pci-0000:02:00.0"

BONDING_SLAVE1="bus-pci-0000:06:00.0"

IPADDR_int="172.16.234.41"

NETMASK_int="255.255.255.0"

NETWORK_int="172.16.234.0"

LABEL_int="int"

That’s it .. We just defined an adapter IP (the 141.53.5.x) and an virtual interface labeled as “int“. We also configured the MII-Monitor to check every 100ms(?) the link of each interface (those defined in BONDING_SLAVEx) if they are either up or down, as well as the adaptive load balancing (“mode=balance-alb“).

Only thing annoying me with that solution is the following entry in /var/log/messages:

Jul  4 18:32:00 dbc-mysql1 kernel: Ethernet Channel Bonding Driver: v3.0.1 (January 9, 2006)
Jul  4 18:32:00 dbc-mysql1 kernel: bonding: Warning: either miimon or arp_interval and arp_ip_target module parameters must be specified, otherwise bonding will not detect link failures! see bonding.txt for details.
Jul  4 18:32:00 dbc-mysql1 kernel: bonding: bond0: Setting MII monitoring interval to 100.
Jul  4 18:32:00 dbc-mysql1 kernel: bonding: bond0: setting mode to balance-alb (6).
Jul  4 18:32:00 dbc-mysql1 kernel: bnx2: eth1: using MSI
Jul  4 18:32:00 dbc-mysql1 kernel: bonding: bond0: enslaving eth1 as an active interface with a down link.
Jul  4 18:32:00 dbc-mysql1 kernel: bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex
Jul  4 18:32:00 dbc-mysql1 kernel: bonding: bond0: link status definitely up for interface eth1.
Jul  4 18:32:00 dbc-mysql1 kernel: bonding: bond0: making interface eth1 the new active one.
Jul  4 18:32:00 dbc-mysql1 kernel: bnx2: eth0: using MSI
Jul  4 18:32:00 dbc-mysql1 kernel: bonding: bond0: enslaving eth0 as an active interface with a down link.
Jul  4 18:32:00 dbc-mysql1 kernel: bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex
Jul  4 18:32:01 dbc-mysql1 kernel: bonding: bond0: link status definitely up for interface eth0.

Jul 4 18:32:00 dbc-mysql1 kernel: Ethernet Channel Bonding Driver: v3.0.1 (January 9, 2006)

Jul 4 18:32:00 dbc-mysql1 kernel: bonding: Warning: either miimon or arp_interval and arp_ip_target module parameters must be specified, otherwise bonding will not detect link failures! see bonding.txt for details.

Jul 4 18:32:00 dbc-mysql1 kernel: bonding: bond0: Setting MII monitoring interval to 100.

Jul 4 18:32:00 dbc-mysql1 kernel: bonding: bond0: setting mode to balance-alb (6).

Jul 4 18:32:00 dbc-mysql1 kernel: bnx2: eth1: using MSI

Jul 4 18:32:00 dbc-mysql1 kernel: bonding: bond0: enslaving eth1 as an active interface with a down link.

Jul 4 18:32:00 dbc-mysql1 kernel: bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex

Jul 4 18:32:00 dbc-mysql1 kernel: bonding: bond0: link status definitely up for interface eth1.

Jul 4 18:32:00 dbc-mysql1 kernel: bonding: bond0: making interface eth1 the new active one.

Jul 4 18:32:00 dbc-mysql1 kernel: bnx2: eth0: using MSI

Jul 4 18:32:00 dbc-mysql1 kernel: bonding: bond0: enslaving eth0 as an active interface with a down link.

Jul 4 18:32:00 dbc-mysql1 kernel: bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex

Jul 4 18:32:01 dbc-mysql1 kernel: bonding: bond0: link status definitely up for interface eth0.

See the warning ? I can’t get it to shut up .. I also tried loading the mii.ko module, but it won’t shut up … damn 🙁

Well, at least the adapter teaming works as desired (still haven’t measured the performance impact with this setup – really need a clever way to do that) and I can plug one of the two cables connected to this box and still have one interface online and a continuous connection. yay ❗