PHP – BAFM

My neverending lighttpd troubles

July 16, 2009August 8, 2014 Christian Leave a comment

Well, after a day or so my lighttpd troubles reappeared. But this time, the lighttpd process would simply put out this:

(mod_fastcgi.c.2913) backend is overloaded; we'll disable it for 2 seconds and send the request to another backend instead: reconnects: 0 load: 131
(mod_fastcgi.c.2668) fcgi-server re-enabled:  0 /var/run/lighttpd/lighttpd-fastcgi-php-17242.socket
(mod_fastcgi.c.2913) backend is overloaded; we'll disable it for 2 seconds and send the request to another backend instead: reconnects: 0 load: 131
(mod_fastcgi.c.2668) fcgi-server re-enabled:  0 /var/run/lighttpd/lighttpd-fastcgi-php-17242.socket

(mod_fastcgi.c.2913) backend is overloaded; we'll disable it for 2 seconds and send the request to another backend instead: reconnects: 0 load: 131

(mod_fastcgi.c.2668) fcgi-server re-enabled: 0 /var/run/lighttpd/lighttpd-fastcgi-php-17242.socket

(mod_fastcgi.c.2913) backend is overloaded; we'll disable it for 2 seconds and send the request to another backend instead: reconnects: 0 load: 131

(mod_fastcgi.c.2668) fcgi-server re-enabled: 0 /var/run/lighttpd/lighttpd-fastcgi-php-17242.socket

And as the message says, PHP (or rather mod_fastcgi?) would simply stop to process requests. In the end, I tuned some of the lighttpd/mod_fastcgi parameters.

  "max-procs"         =&gt; 2
  "idle-timeout"      =&gt; 20,
  "socket"            =&gt; "/tmp/php.socket-" + var.PID,
  "bin-path"          =&gt; "/usr/bin/php-cgi",
  "bin-environment"   =&gt; ( "PHP_FCGI_CHILDREN" =&gt; "6",
                           "PHP_FCGI_MAX_REQUESTS" =&gt; "7000" )

"max-procs" => 2

"idle-timeout" => 20,

"socket" => "/tmp/php.socket-" + var.PID,

"bin-path" => "/usr/bin/php-cgi",

"bin-environment" => ( "PHP_FCGI_CHILDREN" => "6",

"PHP_FCGI_MAX_REQUESTS" => "7000" )

Up till now (I made the change on July 14th), these changes seem to have fixed the issue, guess I’m still hoping (with the saying “Hope dies last” in mind) it’s gonna fix my problems once and for all.

Lighttpd issues

July 11, 2009August 8, 2014 Christian 1 Comment

At first, it seemed that my lighttpd issues were resolved by updating PHP/remerging lighttpd. But apparently not. After putting in a crontab entry, that restarts lighttpd every 15 minutes (which completely sucks), the issue was minimized in it’s impact but not really solved.

*/15 * * * * root    /etc/init.d/lighttpd restart &amp;&gt;/dev/null

1	/15 * * * root /etc/init.d/lighttpd restart &>/dev/null

Thanks to Michél (I guess, again) — who helped me looking at the strace logs, and of course Christian (aka hoffie — one of my old Gentoo buddies), the issue seems finally resolved. It turns out it was neither a PHP nor lighttpd issue. It was a simple matter of (stale) symlinks in /etc/ssl/certs if you can imagine that. Apparently a stale symlink forced PHP into a loop or something, from which it couldn’t recover on it’s own.

So the thank you is probably to the one, who introduced those lines to the ca-certificates ebuild (guess, that would be vapier, the old code monkey):

  if [[ $badcerts -eq 1 ]]; then
    ewarn "You MUST remove the above broken symlinks"
    ewarn "Otherwise any SSL validation that use the directory may fail!"
    ewarn "To batch-remove them, run:"
    ewarn "find -L ${ROOT}etc/ssl/certs/ -type l -exec rm {} +"
  fi

if [[ $badcerts -eq 1 ]]; then

ewarn "You MUST remove the above broken symlinks"

ewarn "Otherwise any SSL validation that use the directory may fail!"

ewarn "To batch-remove them, run:"

ewarn "find -L ${ROOT}etc/ssl/certs/ -type l -exec rm {} +"

After letting the find run through /etc/ssl/certs and restarting lighttpd in the process, everything is back to working order! Finally!

Lighttpd troubles resolved

June 28, 2009August 8, 2014 Christian 1 Comment

Well, after last weeks lighttpd troubles with PHP (or was it without ?), they finally seem resolved. First thing I did, was upgrade to the new php-version (5.2.10). After that, I ran revdep-rebuild, which apparently found issues with lighttpd being linked to a wrong pcre-version. After remerging lighttpd the issues seem to be gone!

Well, guess I was to quick in saying the problem was resolved .. it’s still there, just not happening as fast as it would in the past ….

Weird lighttpd troubles

June 17, 2009June 17, 2009 Christian 1 Comment

Well, since about a week or so I keep having troubles with my vHost and lighttpd. The point being, after some time (up till now it’s been something between days and minutes) lighttpd completely freezes and doesn’t serve no content anymore. I don’t know if this is related to PHP (might be, I did perform an update to dev-lang/php-5.2.9-r2 on Thu May 28 12:18:57 2009), but I have to figure this out since the restart cron-job is getting annoying.

Well, it seems like lighttpd is getting stuck in mod_fastcgi …

2009-06-23 21:34:40: (mod_access.c.135) -- mod_access_uri_handler called
2009-06-23 21:34:40: (mod_fastcgi.c.3675) handling it in mod_fastcgi
2009-06-23 21:34:40: (mod_fastcgi.c.3005) got proc: pid: 11151 socket: unix:/var/run/lighttpd/lighttpd-fastcgi-php-11147.socket-0 load: 85

2009-06-23 21:34:40: (mod_access.c.135) -- mod_access_uri_handler called

2009-06-23 21:34:40: (mod_fastcgi.c.3675) handling it in mod_fastcgi

2009-06-23 21:34:40: (mod_fastcgi.c.3005) got proc: pid: 11151 socket: unix:/var/run/lighttpd/lighttpd-fastcgi-php-11147.socket-0 load: 85

Usually the last line is followed by a line telling that it released the proc, but not always.

Zend Optimizer again

February 19, 2008June 21, 2013 Christian Leave a comment

Well, I happen to be back at my favorite application. Today I stumbled upon a “nice” thing. If you turn on the Zend Optimizer (doesn’t matter whether it is 2.6.2 or 3.3.0), one of the TYPO3 back ends ain’t showing *any* content in the preview pane. Once you turn the Zend Optimizer stuff off, it works without a problem.

And as Zend stated on their “Support Forum“, they don’t really support the Zend Optimizer stuff in the first place. Which is nice, what for do you need the Zend Guard shit in the first place ??

Well, so I do have two options now:

Disable the one plug-in, which really needs the Zend Optimizer (as it also features the Zend De Guard engine – or whatever you want to call it)
or risk some other things breaking due to the Zend Optimizer engine not working (correctly) with php-5.1.2 (which is rather old considering 5.3.0 is in development right now)

But I will see about that tomorrow …

TYPO3 and MySQL replication

September 8, 2007June 21, 2013 Christian Leave a comment

Apparently the TYPO3 version we are using, doesn’t play too nice with the MySQL MasterMaster replication.

Sometimes, something like this is going to happen:

070826  0:44:32 [ERROR] Slave: Error &#039;Duplicate entry &#039;75-222419149&#039; for key 1&#039; on query. Default database: &#039;t3nb&#039;. Query: &#039;INSERT INTO cache_pagesection
070826  0:44:32 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with &quot;SLAVE START&quot;. We stopped at log &#039;dbc-mysql1.000192&#039; position 611861372

070826 0:44:32 [ERROR] Slave: Error 'Duplicate entry '75-222419149' for key 1' on query. Default database: 't3nb'. Query: 'INSERT INTO cache_pagesection

070826 0:44:32 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbc-mysql1.000192' position 611861372

Well, as you can see from the last line in the log, the Slave-SQL thread found a duplicate entry and thought it is smart to just turn off the thread instead of disregarding the just made entry. So from now on, both databases drift since there ain’t no replication anymore until someone kick starts the replication again (someone being me).

Anyway, I think I finally traced the fucker down, supposedly one of the problematic cases is located in t3lib/class.t3lib_tstemplate.php on line 362.

$GLOBALS[&#039;TYPO3_DB&#039;]-&gt;exec_DELETEquery(&#039;cache_pagesection&#039;, &#039;page_id=&#039;.intval($GLOBALS[&#039;TSFE&#039;]-&gt;id).&#039; AND mpvar_hash=&#039;.t3lib_div::md5int($GLOBALS[&#039;TSFE&#039;]-&gt;MP));
$GLOBALS[&#039;TYPO3_DB&#039;]-&gt;exec_INSERTquery(&#039;cache_pagesection&#039;, $insertFields);

1 2	$GLOBALS['TYPO3_DB']->exec_DELETEquery('cache_pagesection', 'page_id='.intval($GLOBALS['TSFE']->id).' AND mpvar_hash='.t3lib_div::md5int($GLOBALS['TSFE']->MP)); $GLOBALS['TYPO3_DB']->exec_INSERTquery('cache_pagesection', $insertFields);

Basically what TYPO3 is doing is a DELETE and an INSERT right afterwards. But apparently, it doesn’t check whether the DELETE even succeeded. I hacked it for now, simply adding this:

-                               $GLOBALS[&#039;TYPO3_DB&#039;]-&gt;exec_INSERTquery(&#039;cache_pagesection&#039;, $insertFields);
+                               // Only insert a new cache entry with the same value, if the DELETE succeeded
+                               if ($GLOBALS[&#039;TYPO3_DB&#039;]-&gt;sql_affected_rows() == 1)
+                                       $GLOBALS[&#039;TYPO3_DB&#039;]-&gt;exec_INSERTquery(&#039;cache_pagesection&#039;, $insertFields);
+

- $GLOBALS['TYPO3_DB']->exec_INSERTquery('cache_pagesection', $insertFields);

+ // Only insert a new cache entry with the same value, if the DELETE succeeded

+ if ($GLOBALS['TYPO3_DB']->sql_affected_rows() == 1)

+ $GLOBALS['TYPO3_DB']->exec_INSERTquery('cache_pagesection', $insertFields);

Sadly, this looks more and more like a race-condition between the two boxes (as in the replication / UPDATE being too slow), when users visit a edited site, that hasn’t had it’s cache regenerated yet. Problem is, it ain’t just this single spot, but also the search indexing, image cache and the whole page cache. For now we switched the cluster to active/passive load balancing, till we have a chance to see if a newer TYPO3 fixes those issues.

SLES, ZendOptimizer and IBM PowerPC(4)+

July 10, 2007June 21, 2013 Christian 2 Comments

What would you figure from the above ? Hopefully the rather obvious, that it’s a *really* shitty combination.

So we figured it would be a nice thing to test our new setup before going into pre-production testing or production, but we don’t have an extra spare box. So we took one of the power4 boxes we have mounted in the rack basically consuming energy all day (that’s about 38kWh a day) and installed SLES10 onto it. Which wasn’t all that bad (at first the box repeatedly started back to AIX, from CD and after convincing the SMS – that’s basically the bios on the power*-boxes also known as System Management Services with a hammer to boot from the first hard disk).

The real bad part started later. First the box committed suicide sometime on the weekend (the last one that is), which is rather not so good.

So we installed the ocfs2-tools (which is obviously needed if you want do writes on a SAN volume mounted on two separate boxes), configured the o2cb thing to start automatically on boot and added the entry to /etc/fstab.

So far so good, but as we slowly activated the apache-vhosts, we finally came to what cost me about three damned hours of my life:

child pid ### exit signal Segmentation fault (11)

1	child pid ### exit signal Segmentation fault (11)

Now guess what … ZendOptimizer just went bye-bye … Damn and what now ? So I looked at the Knowledgebase on zend.com, even found an Article stating it’d do that from time to time …

And attached also the usual crap .. “Please update to the latest version”. Only problem with that is that the latest version is indeed available for x86_64 (meaning amd64 in Gentoo terms), but ain’t for ppc (even if the product page states it should be).

So I went home, knowing what the problem is – since it was already past 4pm – swearing a short “frack that“.

Now that I’m home, ate something (a rather good salad), listening to some Korn/Kid Rock/Offspring and after doing some undertakers work, I asked myself “Why exactly do we need that crappy application anyway ?” (beyond the obvious point, that the ZendOptimizer is like/ is a php-compiler cache).

It turns out, one of my co-workers wrote a TYPO3-plugin interfacing our local research database .. and the catchy thing is, guess what …

He “guarded” it with ZendGuard, thus we need to use the ZendOptimizer thingy; otherwise we couldn’t use it either … 😯