June 19, 2013

sa-learn, dovecot virtual users and virtual user configs

June 19, 2013June 21, 2013 Christian 2 Comments

Well, I wanted independent SpamAssassin Bayes databases per user (different users, different preferences). For that, RoundCube already set up the Junk folder. However, I wanted the ability (for myself, as well for my other users) to individually mark messages as either Spam or Ham.

Now, as I said before I wanted a trivial way to mark messages as Spam or Ham (without using the command line each time).

Now, that was the mailbox setup part. Now we do have to do some command line foo (yeah, it’s still necessary) to actually learn the mails as spam or ham. First we need a script, which scans the Maildir for each domain/user separately, and then creates the bayes database.

#!/bin/bash

# Script, which allows per-user bayes db's for a dovecot virtual user
# setup. sa-learn parses a set amount of folders (.Junk.Spam and .Junk.Ham) for
# Ham/Spam and adds it to the per-user db.

MAIL_DIR=/var/mail
SPAMASS_DIR=/var/lib/spamassassin
SPAM_FOLDER=".Junk.Spam"
HAM_FOLDER=".Junk.Ham"

# get all mail accounts
for domain in $MAIL_DIR/*; do
        for user in $MAIL_DIR/${domain##*/}/*; do
                mailaccount=${user##*/}
                dbpath=$SPAMASS_DIR/${domain##*/}/$mailaccount
                spamfolder=${domain}/${mailaccount}/Maildir/$SPAM_FOLDER
                hamfolder=${domain}/${mailaccount}/Maildir/$HAM_FOLDER

                if [ -d $spamfolder ] ; then
                        [ ! -d $dbpath ] && mkdir -p ${dbpath}
                        echo "Learning Spam from ${spamfolder} for user ${mailaccount}"
                        nice sa-learn --spam --dbpath ${dbpath}/bayes 
                                --no-sync ${spamfolder}
                fi

                if [ -d $hamfolder ] ; then
                        echo "Learning Ham from ${hamfolder} for user ${mailaccount}"
                        nice sa-learn --ham --dbpath ${dbpath}/bayes 
                                --no-sync ${hamfolder}
                fi

                if [ -d $spamfolder -o -d $hamfolder ] ; then
                        nice sa-learn --sync --dbpath $dbpath

                        # Fix dbpath permissions
                        chown -R mail.mail ${dbpath}
                        chmod 700 ${dbpath}
                fi
        done
done

#!/bin/bash

# Script, which allows per-user bayes db's for a dovecot virtual user

# setup. sa-learn parses a set amount of folders (.Junk.Spam and .Junk.Ham) for

# Ham/Spam and adds it to the per-user db.

MAIL_DIR=/var/mail

SPAMASS_DIR=/var/lib/spamassassin

SPAM_FOLDER=".Junk.Spam"

HAM_FOLDER=".Junk.Ham"

# get all mail accounts

for domain in $MAIL_DIR/*; do

for user in $MAIL_DIR/${domain##*/}/*; do

mailaccount=${user##*/}

dbpath=$SPAMASS_DIR/${domain##*/}/$mailaccount

spamfolder=${domain}/${mailaccount}/Maildir/$SPAM_FOLDER

hamfolder=${domain}/${mailaccount}/Maildir/$HAM_FOLDER

if [ -d $spamfolder ] ; then

[ ! -d $dbpath ] && mkdir -p ${dbpath}

echo "Learning Spam from ${spamfolder} for user ${mailaccount}"

nice sa-learn --spam --dbpath ${dbpath}/bayes

--no-sync ${spamfolder}

if [ -d $hamfolder ] ; then

echo "Learning Ham from ${hamfolder} for user ${mailaccount}"

nice sa-learn --ham --dbpath ${dbpath}/bayes

--no-sync ${hamfolder}

if [ -d $spamfolder -o -d $hamfolder ] ; then

nice sa-learn --sync --dbpath $dbpath

# Fix dbpath permissions

chown -R mail.mail ${dbpath}

chmod 700 ${dbpath}

done

This script is based on work from nesono and workaround.org. Anyhow, the script will scan each user folder (you might need to adjust the MAIL_DIR and SPAMASS_DIR variable, depending on where your MAIL_DIR is located.

Next, we need to adjust the SPAMD options to use the virtual-config-dir (that’s the SPAMD name for this).

--- spamassassin.orig   2013-06-19 19:49:30.000000000 +0200
+++ spamassassin        2013-06-19 19:18:07.000000000 +0200
@@ -14,7 +14,7 @@
 # make sure --max-children is not set to anything higher than 5,
 # unless you know what you're doing.

-OPTIONS="--create-prefs --max-children 5 --helper-home-dir"
+OPTIONS="--create-prefs --max-children 5 --helper-home-dir --virtual-config-dir=/var/lib/spamassassin/%d/%l -x -u mail"

 # Pid file
 # Where should spamd write its PID to file? If you use the -u or

--- spamassassin.orig 2013-06-19 19:49:30.000000000 +0200

+++ spamassassin 2013-06-19 19:18:07.000000000 +0200

@@ -14,7 +14,7 @@

# make sure --max-children is not set to anything higher than 5,

# unless you know what you're doing.

-OPTIONS="--create-prefs --max-children 5 --helper-home-dir"

+OPTIONS="--create-prefs --max-children 5 --helper-home-dir --virtual-config-dir=/var/lib/spamassassin/%d/%l -x -u mail"

# Pid file

# Where should spamd write its PID to file? If you use the -u or

As you can see, I basically appended the following to the OPTIONS variable: –virtual-config-dir=/var/lib/spamassassin/%d/%l -x -u mail

Now, here’s a couple of pointers:

–virtual-config-dir=pattern
This option specifies where per-user preferences can be found for virtual users, for the -x switch. The pattern is used as a base pattern for the directory name. Any of the
following escapes can be used:

%u — replaced with the full name of the current user, as sent by spamc.
%l — replaced with the ‘local part’ of the current username. In other words, if the username is an email address, this is the part before the “@” sign.
%d — replaced with the ‘domain’ of the current username. In other words, if the username is an email address, this is the part after the “@” sign.
%% — replaced with a single percent sign (%).

-u username, –username=username
Run as the named user. If this option is not set, the default behaviour is to setuid() to the user running “spamc”, if “spamd” is running as root.

Note: “–username=root” is not a valid option. If specified, “spamd” will exit with a fatal error on startup.

Now, only a small adjustment is still needed. In order for the inbound mails to be scanned with the per-user db’s, you need to adjust postfix’s master.cf file, to run spamc with the per-user db.

--- master.cf.orig      2013-06-19 19:56:57.000000000 +0200
+++ master.cf   2013-06-19 19:57:09.000000000 +0200
@@ -115,7 +115,7 @@

 # dovecot mail delivery
 dovecot   unix  -       n       n       -       -       pipe
-  flags=DRhu user=vmail:mail argv=/usr/lib/dovecot/deliver -d ${recipient}
+  flags=DRhu user=vmail:mail argv=/usr/bin/spamc -u ${recipient} -e /usr/lib/dovecot/deliver -f ${sender} -d ${recipient}

 amavis unix    -       -       -       -       2       smtp
         -o smtp_data_done_timeout=1200

--- master.cf.orig 2013-06-19 19:56:57.000000000 +0200

+++ master.cf 2013-06-19 19:57:09.000000000 +0200

@@ -115,7 +115,7 @@

# dovecot mail delivery

dovecot unix - n n - - pipe

- flags=DRhu user=vmail:mail argv=/usr/lib/dovecot/deliver -d ${recipient}

+ flags=DRhu user=vmail:mail argv=/usr/bin/spamc -u ${recipient} -e /usr/lib/dovecot/deliver -f ${sender} -d ${recipient}

amavis unix - - - - 2 smtp

-o smtp_data_done_timeout=1200

After that’s done (and a restart of postfix, spamassassin and dovecot) you should be the proud owner of a per-user dovecot/postfix/spamassassin implementation.

Postfix, soft_bounce=yes and redelivering mails

June 19, 2013June 19, 2013 Christian Leave a comment

Well, I’m setting up spam/virus filter at the moment. Somewhere I found, that when doing so one should enable soft_bounce=yes in your /etc/postfix/main.cf. Now, once I finished setting up my mailing setup, I wanted to manually force the delivery.

# Check for mails still in the mail-queue
root:(eris.heimdaheim.de/mailing) PWD:~
Wed Jun 19, 17:57:10 [0] > mailq
-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
B404F1B42C     2751 Wed Jun 19 17:58:01  christian.th.heim@gmail.com
                                                                (user unknown)
                                         christian@heimdaheim.de

C250F1B427     2770 Wed Jun 19 17:53:31  christian.th.heim@gmail.com
(lost connection with 127.0.0.1[127.0.0.1] while receiving the initial server greeting)
                                         christian@heimdaheim.de

-- 6 Kbytes in 2 Requests.

# Check for mails still in the mail-queue

root:(eris.heimdaheim.de/mailing) PWD:~

Wed Jun 19, 17:57:10 [0] > mailq

-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------

B404F1B42C 2751 Wed Jun 19 17:58:01 christian.th.heim@gmail.com

(user unknown)

christian@heimdaheim.de

C250F1B427 2770 Wed Jun 19 17:53:31 christian.th.heim@gmail.com

(lost connection with 127.0.0.1[127.0.0.1] while receiving the initial server greeting)

christian@heimdaheim.de

-- 6 Kbytes in 2 Requests.

Now, if you fixed the mail delivery, you just need to enter the following:

root:(eris.heimdaheim.de/mailing) PWD:~
Wed Jun 19, 17:57:10 [0] > postsuper -r ALL
root:(eris.heimdaheim.de/mailing) PWD:~
Wed Jun 19, 17:57:11 [0] > postfix flush

root:(eris.heimdaheim.de/mailing) PWD:~

Wed Jun 19, 17:57:10 [0] > postsuper -r ALL

root:(eris.heimdaheim.de/mailing) PWD:~

Wed Jun 19, 17:57:11 [0] > postfix flush

However if you want to delete the mail from the postfix queue:

root:(eris.heimdaheim.de/mailing) PWD:~
Wed Jun 19, 17:57:11 [0] > postsuper -d ALL deferred

1 2	root:(eris.heimdaheim.de/mailing) PWD:~ Wed Jun 19, 17:57:11 [0] > postsuper -d ALL deferred

Tiny Tiny RSS init-script for multiple instances on Debian Wheezy

June 19, 2013June 19, 2013 Christian 2 Comments

Well, I have a bunch of Tiny Tiny RSS instances running on my webhost, and I wanted a init-script that starts the update-daemons for all instances.

Now, there’s already a bunch of init scripts for Debian around (1, 2) however none of them were to my liking or did what I wanted it to do. So I ended up (yeah, I know *again*) rewriting them.

#!/bin/sh
### BEGIN INIT INFO
# Provides:          ttrss
# Required-Start:    $local_fs $remote_fs networking
# Required-Stop:     $local_fs $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Tiny Tiny RSS update daemon
# Description:       Update the Tiny Tiny RSS subscribed syndication feeds.
### END INIT INFO

set -e

PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="Tiny Tiny RSS update daemon"
NAME=$(command basename "${0}")

DISABLED=1
FORKING=0

# Read configuration variable file if it is present
[ -r "/etc/default/${NAME}" ] && . "/etc/default/${NAME}"

DAEMON_SCRIPT="update.php --daemon"

if [ $FORKING -ne 0 ] ; then
        DAEMON_SCRIPT="update_daemon2.php"
fi

DAEMON=/usr/bin/php

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

[ -d "${TTRSS_PATH}" ] || exit 0

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

if [ "$DISABLED" != "0" -a "$1" != "stop" ]; then
        log_warning_msg "Not starting $DESC - edit /etc/default/tt-rss and change DISABLED to be 0.";
        exit 0;
fi

# Function that starts the daemon/service
do_start()
{
        for dir in ${TTRSS_PATH}/*; do
                INSTANCE_NAME=${dir##*/}
                start-stop-daemon --start --make-pidfile --background --quiet 
                        --chuid "${TTRSS_USERID}" --group "${TTRSS_GROUPID}" 
                        --chdir "${dir}" 
                        --pidfile "${TTRSS_PIDDIR}/ttrss-${INSTANCE_NAME}.pid" 
                        --exec "${DAEMON}" --test >/dev/null 
                        || return 1

                start-stop-daemon --start --make-pidfile --background --quiet 
                        --chuid "${TTRSS_USERID}" --group "${TTRSS_GROUPID}" 
                        --chdir "${dir}" 
                        --pidfile "${TTRSS_PIDDIR}/ttrss-${INSTANCE_NAME}.pid" 
                        --exec "${DAEMON}" -- ${dir}/${DAEMON_SCRIPT} --log 
                        /var/log/ttrss/${INSTANCE_NAME}.log 
                        || return 2
        done
}

# Function that stops the daemon/service
do_stop()
{
        for dir in ${TTRSS_PATH}/*; do
                INSTANCE_NAME=${dir##*/}
                start-stop-daemon --stop --make-pidfile --quiet 
                        --retry=TERM/1/KILL/5 
                        --pidfile "${TTRSS_PIDDIR}/ttrss-${INSTANCE_NAME}.pid" 

                RETVAL="$?"
                [ "$RETVAL" = 2 ] && return 2

                start-stop-daemon --stop --quiet --oknodo --retry=0/1/KILL/5 
                        --exec ${DAEMON}

                [ "$?" = 2 ] && return 2
                rm -f "${TTRSS_PIDDIR}/ttrss-${INSTANCE_NAME}.pid"
                return $RETVAL
        done
}


case "$1" in
        start)
                log_daemon_msg "Starting $DESC" "$NAME"
                do_start
                case "$?" in
                        0|1) log_end_msg 0 ;;
                        2) log_end_msg 1 ;;
                esac
                ;;
        stop)
                log_daemon_msg "Stopping $DESC" "$NAME"
                do_stop
                case "$?" in
                        0|1) log_end_msg 0 ;;
                        2) log_end_msg 1 ;;
                esac
                ;;
        restart|force-reload)
                # If the "reload" option is implemented then remove the
                # 'force-reload' alias
                log_daemon_msg "Restarting $DESC" "$NAME"
                do_stop
                case "$?" in
                        0|1)
                                do_start
                                case "$?" in
                                        0) log_end_msg 0 ;;
                                        1) log_end_msg 1 ;; # Old process is still running
                                        *) log_end_msg 1 ;; # Failed to start
                                esac
                                ;;
                        *)
                                # Failed to stop
                                log_end_msg 1
                                ;;
                esac
                ;;
        *)
                echo "Usage: ${NAME} {start|stop|restart|force-reload}" >&2
                exit 3
                ;;
esac

:

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

#!/bin/sh

### BEGIN INIT INFO

# Provides: ttrss

# Required-Start: $local_fs $remote_fs networking

# Required-Stop: $local_fs $remote_fs

# Default-Start: 2 3 4 5

# Default-Stop: 0 1 6

# Short-Description: Tiny Tiny RSS update daemon

# Description: Update the Tiny Tiny RSS subscribed syndication feeds.

### END INIT INFO

set -e

PATH=/sbin:/usr/sbin:/bin:/usr/bin

DESC="Tiny Tiny RSS update daemon"

NAME=$(command basename "${0}")

DISABLED=1

FORKING=0

# Read configuration variable file if it is present

[ -r "/etc/default/${NAME}" ] && . "/etc/default/${NAME}"

DAEMON_SCRIPT="update.php --daemon"

if [ $FORKING -ne 0 ] ; then

DAEMON_SCRIPT="update_daemon2.php"

DAEMON=/usr/bin/php

# Exit if the package is not installed

[ -x "$DAEMON" ] || exit 0

[ -d "${TTRSS_PATH}" ] || exit 0

# Load the VERBOSE setting and other rcS variables

. /lib/init/vars.sh

# Define LSB log_* functions.

# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.

. /lib/lsb/init-functions

if [ "$DISABLED" != "0" -a "$1" != "stop" ]; then

log_warning_msg "Not starting $DESC - edit /etc/default/tt-rss and change DISABLED to be 0.";

exit 0;

# Function that starts the daemon/service

do_start()

{

for dir in ${TTRSS_PATH}/*; do

INSTANCE_NAME=${dir##*/}

start-stop-daemon --start --make-pidfile --background --quiet

--chuid "${TTRSS_USERID}" --group "${TTRSS_GROUPID}"

--chdir "${dir}"

--pidfile "${TTRSS_PIDDIR}/ttrss-${INSTANCE_NAME}.pid"

--exec "${DAEMON}" --test >/dev/null

|| return 1

start-stop-daemon --start --make-pidfile --background --quiet

--chuid "${TTRSS_USERID}" --group "${TTRSS_GROUPID}"

--chdir "${dir}"

--pidfile "${TTRSS_PIDDIR}/ttrss-${INSTANCE_NAME}.pid"

--exec "${DAEMON}" -- ${dir}/${DAEMON_SCRIPT} --log

/var/log/ttrss/${INSTANCE_NAME}.log

|| return 2

done

}

# Function that stops the daemon/service

do_stop()

{

for dir in ${TTRSS_PATH}/*; do

INSTANCE_NAME=${dir##*/}

start-stop-daemon --stop --make-pidfile --quiet

--retry=TERM/1/KILL/5

--pidfile "${TTRSS_PIDDIR}/ttrss-${INSTANCE_NAME}.pid"

RETVAL="$?"

[ "$RETVAL" = 2 ] && return 2

start-stop-daemon --stop --quiet --oknodo --retry=0/1/KILL/5

--exec ${DAEMON}

[ "$?" = 2 ] && return 2

rm -f "${TTRSS_PIDDIR}/ttrss-${INSTANCE_NAME}.pid"

return $RETVAL

done

}

case "$1" in

start)

log_daemon_msg "Starting $DESC" "$NAME"

do_start

case "$?" in

0|1) log_end_msg 0 ;;

2) log_end_msg 1 ;;

esac

;;

stop)

log_daemon_msg "Stopping $DESC" "$NAME"

do_stop

case "$?" in

0|1) log_end_msg 0 ;;

2) log_end_msg 1 ;;

esac

;;

restart|force-reload)

# If the "reload" option is implemented then remove the

# 'force-reload' alias

log_daemon_msg "Restarting $DESC" "$NAME"

do_stop

case "$?" in

0|1)

do_start

case "$?" in

0) log_end_msg 0 ;;

1) log_end_msg 1 ;; # Old process is still running

*) log_end_msg 1 ;; # Failed to start

esac

;;

# Failed to stop

log_end_msg 1

;;

esac

;;

echo "Usage: ${NAME} {start|stop|restart|force-reload}" >&2

exit 3

;;

esac

# Set DISABLED to 1 to prevent the daemon from starting.
DISABLED=0

TTRSS_USERID="www-data"
TTRSS_GROUPID="www-data"
TTRSS_PIDDIR="/var/run"
TTRSS_PATH="/var/www/rss.heimdaheim.de/www"

# Allow the update_daemon script to use a forking daemon instead of a
# single-threaded instance.
# This option is only available for TinyTinyRSS 1.2.20 and above.
FORKING=0

# Set DISABLED to 1 to prevent the daemon from starting.

DISABLED=0

TTRSS_USERID="www-data"

TTRSS_GROUPID="www-data"

TTRSS_PIDDIR="/var/run"

TTRSS_PATH="/var/www/rss.heimdaheim.de/www"

# Allow the update_daemon script to use a forking daemon instead of a

# single-threaded instance.

# This option is only available for TinyTinyRSS 1.2.20 and above.

FORKING=0