7 Jan
2009
7 Jan
'09
5:54 a.m.
Hello and a happy new year,
a couple of days ago one of our perdition servers exceeded the 1024 pop3
connections limit configured (normal usage is about 30-50 parallel
connections). I turned out that all these connections had been eaten up
by some customer trying to pop his email from China and instead of firing
off a connection every 2 minutes things seemed to fail on his end and
groups of 3-6 parallel connects (which failed in turn again) came hurtling
in.
We have been using perdition for over 4 years and have a huge customer
base with over 1.5 million pop3 connects per day and never seen anything
like this, so my guess here is that this is not (purely) a client thing
but that the grate(sic) firewall of China is involved, too.
Anyways, this is a typical example:
---
Jan 7 12:15:10 pp12 perdition[16881]: Connect: 218.1.143.164->203.216.5.113
Jan 7 12:15:10 pp12 perdition[16886]: Connect: 218.1.143.164->203.216.5.113
Jan 7 12:15:10 pp12 perdition[16889]: Connect: 218.1.143.164->203.216.5.113
Jan 7 12:38:19 pp12 perdition[16881]: io_read: read: Connection timed out
Jan 7 12:38:19 pp12 perdition[16881]: __token_fill_buffer: error reading input:
Connection timed out
Jan 7 12:38:19 pp12 perdition[16881]: token_read: token_fill_buffer
Jan 7 12:38:19 pp12 perdition[16881]: read_line: token_read
Jan 7 12:38:19 pp12 perdition[16881]: pop3_in_get_pw: pop3_in_get_pw: read_line
Jan 7 12:38:19 pp12 perdition[16881]: main: protocol->in_get_pw
Jan 7 12:38:19 pp12 perdition[16881]: Fatal Error reading authentication information from
client "218.1.143.164->203.216.5.113 ": Exiting child
---
Aside from being a good starting measure against a DoS attack in general a
feature where one could set a timeout for the maximum time that a process
is allowed to be in "connect" state (I would set this to 1 minute or less)
would be a very welcome addition.
The problem I raised in
http://lists.vergenet.net/pipermail/perdition-users/2008-February/001973.ht…
would benefit (as in workaround) from such a feature, too.
Regards,
Christian
--
Christian Balzer Network/Systems Engineer NOC
chibi(a)gol.com Global OnLine Japan/Fusion Network Services
http://www.gol.com/
https://secure3.gol.com/mod-pl/ols/index.cgi/?intr_id=F-2ECXvzcr6656
12 Jan
12 Jan
4:56 a.m.
On Wed, Jan 07, 2009 at 02:54:33PM +0900, Christian Balzer wrote:
Hello and a happy new year,
a couple of days ago one of our perdition servers exceeded the 1024 pop3
connections limit configured (normal usage is about 30-50 parallel
connections). I turned out that all these connections had been eaten up
by some customer trying to pop his email from China and instead of firing
off a connection every 2 minutes things seemed to fail on his end and
groups of 3-6 parallel connects (which failed in turn again) came hurtling
in.
We have been using perdition for over 4 years and have a huge customer
base with over 1.5 million pop3 connects per day and never seen anything
like this, so my guess here is that this is not (purely) a client thing
but that the grate(sic) firewall of China is involved, too.
Anyways, this is a typical example:
---
Jan 7 12:15:10 pp12 perdition[16881]: Connect: 218.1.143.164->203.216.5.113
Jan 7 12:15:10 pp12 perdition[16886]: Connect: 218.1.143.164->203.216.5.113
Jan 7 12:15:10 pp12 perdition[16889]: Connect: 218.1.143.164->203.216.5.113
Jan 7 12:38:19 pp12 perdition[16881]: io_read: read: Connection timed out
Jan 7 12:38:19 pp12 perdition[16881]: __token_fill_buffer: error reading input:
Connection timed out
Jan 7 12:38:19 pp12 perdition[16881]: token_read: token_fill_buffer
Jan 7 12:38:19 pp12 perdition[16881]: read_line: token_read
Jan 7 12:38:19 pp12 perdition[16881]: pop3_in_get_pw: pop3_in_get_pw: read_line
Jan 7 12:38:19 pp12 perdition[16881]: main: protocol->in_get_pw
Jan 7 12:38:19 pp12 perdition[16881]: Fatal Error reading authentication information
from client "218.1.143.164->203.216.5.113 ": Exiting child
---
Aside from being a good starting measure against a DoS attack in general a
feature where one could set a timeout for the maximum time that a process
is allowed to be in "connect" state (I would set this to 1 minute or less)
would be a very welcome addition.
The problem I raised in
http://lists.vergenet.net/pipermail/perdition-users/2008-February/001973.ht…
would benefit (as in workaround) from such a feature, too.
Hi Christian,
that sounds like a very reasonable suggestion, and I suspect
something that wouldn't be too hard to add - though I haven't
inspected the relevant code paths recently.
--
Simon Horman
VA Linux Systems Japan K.K., Sydney, Australia Satellite Office
H: www.vergenet.net/~horms/ W: www.valinux.co.jp/en
9:48 a.m.
Hi Christian,
a couple of days ago one of our perdition servers
exceeded
the 1024 pop3 connections limit configured (normal usage is
about 30-50 parallel connections). I turned out that all
these connections had been eaten up by some customer trying
to pop his email from China and instead of firing off a
connection every 2 minutes things seemed to fail on his end
and groups of 3-6 parallel connects (which failed in turn
again) came hurtling in.
We had this problem with badly configured/behaving clients a while back (with UW IMAP) and
fixed it with some iptables rules:
-A INPUT -p tcp -m tcp --dport 110 -m state --state NEW -j pop-monitor
-A INPUT -p tcp -m tcp --dport 110 -j ACCEPT
###### pop-monitor ######
# New rate limiting to max of three a minute
-A pop-monitor --match hashlimit --hashlimit-name pop --hashlimit 5/minute
--hashlimit-burst 3 --hashlimit-mode srcip -j ACCEPT
-A pop-monitor -j LOG --log-prefix "POP syn rate too high "
-A pop-monitor -j REJECT
Not perdition specific, but it's generic enough to cover for other software too.
Ian.
14 Jan
14 Jan
12:52 p.m.
Hello,
On Mon, 12 Jan 2009 09:48:03 -0000 Norton, Ian wrote:
We had this problem with badly configured/behaving clients a while back
(with UW IMAP) and fixed it with some iptables rules:
[...]
I love iptables and am using them in many places, nice recipe.
The problem with iptables in general is that it is Linux specific (and not
all OSS software runs on linux boxes) and more importantly for me (all
Debian boxes here) that this assumes that the person or context running
them is allowed to utilize them. Specific example would be anything
running in a Vserver. Of course you can set iptable rules on the host, but
if the program in question has sensible options and is linked against
tcpwrappers usually that will suffice.
In my particular example the rate might not even have been that high to
trigger things, it was the fact that the connections lingered for over 20
minutes in pre-auth aka connect that exhausted the resources.
Also an IP based solution will need exceptions and monitoring for people
with multiple clients behind NAT (not uncommon here) and of course
something like a webmail server.
Regards,
Christian
--
Christian Balzer Network/Systems Engineer NOC
chibi(a)gol.com Global OnLine Japan/Fusion Network Services
http://www.gol.com/
https://secure3.gol.com/mod-pl/ols/index.cgi/?intr_id=F-2ECXvzcr6656
5583
days inactive
5590
days old
3 comments
3 participants
participants (3)
-
Christian Balzer -
Norton, Ian -
Simon Horman