Greylisting

ze-filter implements greylisting method from Evan Harris.

ze-filter implementation is intended to be used on huge servers. So, it tries to reduce the number of entries on database, and uses results from other filtering methods to remove unwanted entries. Other than original databases (pending and validated entries), ze-filter uses two new databases : whitelisted entries and blacklisted entries. Whitelisted entries database contains only couples of IP and domain names.

ze-filter greylisting works in two modes :

• Standalone - In this mode, there is no information exchange between filters on different servers, unless the same filter is serving different servers.
• Client - You can use this mode, if you're running a farm of servers (with some sort of load balancing) or if you want to share information between your domain MXs. In this mode, launch ze-greyd daemon. Each client maintains it's own database. Each time it needs some information, it will lookup at its local database. If it can't find it, it asks ze-greyd server. In the same way, if it updates some information locally found, it informs ze-greyd. This way, each client maintains his own history and the server maintains the history of all clients.

ze-filter greylisting data is stored in Berkeley DB databases.

Ideas behind ze-filter implementation of greylisting were presented at JRES 2005 or TERENA 2005

• Greylisting introduce inacceptable delays - Yes and No ! Messages are delayed only the first time a triplet is presented. After a triplet has been validated, and if it comes often enough, it isn't delayed anymore. When there are problem, the reason is away - see below for some examples.

• Misconfigured mail servers - Some mail servers behave strangely. Hopefully, these are exceptions.
  • To handle this, you have two options - look for ALL possible misconfigured mail servers before enabling greylisting and add them to ze-policy database, as exceptions, using the GreyCheckConnect prefix or you can enable greylisting and handle each new exception each time you find them AND you detect some inacceptable delay. It's up to you to choose the good option and it depends mostly with the kind of mailservers you usually talk with.
  • Some behaviours we've found are :
    1. They don't retry when they receive a 4xx temporary error
    2. They retry after very some long delay (4 hours or more…)
    3. They handle a tempfail error (4xx) as if it was a permanent failure error (5xx)

• Greylisting and farms of outbound servers*

• Default Greylisting Behaviour - You have to decide if you will enable greylisting for all of your users and disable for those who don't want this feature or if you will enable only for some of your users. Generally, you can start with the last option, with some “beta testers” and if people are happy with greylisting, migrate to the first option and remove those who don't want greylisting (this is how we've done at our domain).

• Is greylisting still effective ??? - We've found some zombies retrying after being rejected (four retries over 20 minutes). But they can't retry indefinitely and retrying means loosing bandwidth. For the moment, greylisting still is a good filtering method.

First of all, activate GREY_CHECK configuration option. You may eventually let other options at their default value.

# GREY_CHECK
#     Greylist default activation
#  Syntax : NO | YES
#     VALUES :  NO  YES 
GREY_CHECK                 YES

Second, you must define, at ze-policy database, default behaviour and which triplets will be submitted to greylisting. The three prefix you can use are : GreyCheckConnect, GreyCheckFrom and GreyCheckTo.

Some rules apply :

# Greylisting is applied to every triplet unless defined otherwise
GreyCheckConnect:DEFAULT             YES
# Greylisting is applied only to SMTP clients, senders or recipients 
# defined at ze-policy database
GreyCheckConnect:DEFAULT             NO

Here is a greylisting policy example :

# default is YES
GreyCheckConnect:DEFAULT                YES
# never greylist my friend mail server - don't even check sender or recipient
GreyCheckConnect:111.222.333.200        NO-QUICK
# default for network 1.2.3 is NO, but let's check sender and recipient
GreyCheckConnect:1.2.3                  NO
# greylist is allways applied if sender is in yahoo.com domain
GreyCheckFrom:yahoo.com                 YES-QUICK
# except if sender is joe@yahoo.com
GreyCheckFrom:joe@yahoo.com             NO
# This user don't want greylisting applied to his messages
GreyCheckTo:martins@                    NO
# math department don't want messages sent to them to be greylisted
GreyCheckTo:math.mydomain.com           NO

You can also use an additional prefix : GreyDomainEquiv. Entries with this prefix are used to define equivalent domains. Two domains are equivalent if messages from one of them can come from the other. One example is :

GreyDomainEquiv:arcelor.com		usinor.com

OBS : if the difference between two domains is only their TLD, you don't need to define equivalent domains entries. Example : renault.fr and renault.com are equivalent and don't need any entry to be added to ze-policy database.

GreyCheckTo:postmaster@mydomain.com           NO
GreyCheckTo:abuse@mydomain.com                NO
GreyCheckTo:hostmaster@mydomain.com           NO

See also: Longer delays observed

########################################################################
#    SECTION  :  Greylisting
########################################################################
# GREY_CHECK
#     Greylist default activation
#  Syntax : NO | YES
#     VALUES :  NO  YES 
GREY_CHECK                 YES
#
# GREY_MODE
#     Greylist mode
#  Syntax : STANDALONE | CLIENT
#     VALUES :  STANDALONE  CLIENT 
GREY_MODE                  STANDALONE
#
# GREY_SOCKET
#     Remote Greylist Server Socket when running in CLIENT mode
GREY_SOCKET                local:/var/ze-filter/ze-greyd.sock
#
# GREY_CONNECT_TIMEOUT
#     Timeout to connect go ze-grey server when running in CLIENT mode
GREY_CONNECT_TIMEOUT       10s
#
# GREY_MIN_DELAY_NORMAL
#     Greylist delay for normal messages
GREY_MIN_DELAY_NORMAL      10m
#
# GREY_MIN_DELAY_NULLSENDER
#     Greylist delay for null sender messages
GREY_MIN_DELAY_NULLSENDER   25m
#
# GREY_MAX_DELAY_NORMAL
#     Lifetime for pending entries (normal messages)
GREY_MAX_DELAY_NORMAL      3d
#
# GREY_MAX_DELAY_NULLSENDER
#     Lifetime for pending entries (null sender messages)
GREY_MAX_DELAY_NULLSENDER   24h
#
# GREY_VALIDLIST_LIFETIME
#     Lifetime for inactive whitelisted entries
GREY_VALIDLIST_LIFETIME    2w
#
# GREY_WHITELIST_LIFETIME
#     Lifetime for inactive whitelisted entries
GREY_WHITELIST_LIFETIME    3w
#
# GREY_BLACKLIST_LIFETIME
#     Lifetime for blacklisted entries (not yet implemented)
GREY_BLACKLIST_LIFETIME    1d
#
# GREY_PENDING_NORMAL
#     Max normal pending messages (not yet implemented)
GREY_PENDING_NORMAL        50
#
# GREY_PENDING_NULLSENDER
#     Max null sender pending messages (not yet implemented)
GREY_PENDING_NULLSENDER    6
#
# GREY_IP_COMPONENT
#     How to construct IP part of ntuple
#  Syntax : NONE | FULL | NET
GREY_IP_COMPONENT          NET
#
# GREY_FROM_COMPONENT
#     How to construct FROM part of ntuple
#  Syntax : NONE | FULL | HOST | USER
GREY_FROM_COMPONENT        HOST
#
# GREY_TO_COMPONENT
#     How to construct FROM part of ntuple
#  Syntax : NONE | FULL | HOST | USER
GREY_TO_COMPONENT          FULL
#
# GREY_CLEANUP_INTERVAL
#     Greylist database cleanup interval
GREY_CLEANUP_INTERVAL      10m
#
# GREY_DEWHITE_FLAGS
#     Which criteria utilise to purge greylisting databases ???
#  Syntax : None BadResolve DomainMatch BadRCPT SpamTrap BadMX BadClient Spammer All
GREY_DEWHITE_FLAGS         DomainMatch