[Unbound-users] Random subdomain flood query

[battossai]

Hi,


Is there anyone in here get a problem like me.
My Unbound server reveice flood that query randomize subdomain that make
traffic jammed and CPU Load is full ?
Any solution that can be shared ? Thanks
Here is sample log of mine :

*Mar 31 17:56:47 ns1 unbound: [7679:1] info: 49.128.xxx.xxx
cdexevevyp.www.136.xxx. A IN*

*Mar 31 17:56:47 ns1 unbound: [7679:0] info: 103.247.xxx.xxx
cnsjwhclifax.www.136.xxx. A IN*

*Mar 31 17:56:47 ns1 unbound: [7679:0] info: 111.68.xxx.xxx
avezsvuvehgnajun.www.136.xxx. A IN*


*Mar 31 17:56:47 ns1 unbound: [7679:1] info: 119.2.xxx.xxx
epsruvodazqz.www.136.xxx. A IN*

I try to install unbound-bloomfilter but got error.
Any information will be nice.
Thank you.


Regards,
Franky Yu

[Stephane Bortzmeyer]

On Tue, Mar 31, 2015 at 06:09:50PM +0700,

Post by battossai
*Mar 31 17:56:47 ns1 unbound: [7679:1] info: 49.128.xxx.xxx
cdexevevyp.www.136.xxx. A IN*

If using Linux, this Netfilter rule is very useful:

iptables -A INPUT --in-interface eth0 -p udp --dport 53 -m string \
--algo bm --hex-string '|03313336 03787878|' \
--jump DROP

(where 03313336 03787878 = 136.xxx)

[battossai]

Hi,


The problem is that is just one domain that i share, the domain also change
for several time.
But i will try it, thank you in advanced.


Regards,
Franky Yu

Post by Stephane Bortzmeyer
On Tue, Mar 31, 2015 at 06:09:50PM +0700,

Post by battossai
*Mar 31 17:56:47 ns1 unbound: [7679:1] info: 49.128.xxx.xxx
cdexevevyp.www.136.xxx. A IN*

iptables -A INPUT --in-interface eth0 -p udp --dport 53 -m string \
--algo bm --hex-string '|03313336 03787878|' \
--jump DROP
(where 03313336 03787878 = 136.xxx)

[Stephane Bortzmeyer]

On Tue, Mar 31, 2015 at 06:47:28PM +0700,

Post by battossai
The problem is that is just one domain that i share, the domain also
change for several time.

How often? If it not too common, you can still use a hand-crafted
rule.

[battossai]

Hi,

It running every second, i receive about 14 mbit/sec traffic.
Kernel log unable to display it all.

Post by Stephane Bortzmeyer
On Tue, Mar 31, 2015 at 06:47:28PM +0700,

Post by battossai
The problem is that is just one domain that i share, the domain also
change for several time.

How often? If it not too common, you can still use a hand-crafted
rule.

[Stephane Bortzmeyer]

On Tue, Mar 31, 2015 at 08:02:46PM +0700,

Post by battossai
It running every second, i receive about 14 mbit/sec traffic.

DNS attacks typically use small packets, and the stress comes from the
number of packets. Better to express it in Mp/s.

[Franky Yustanto]

Hi,

I got about 3500 pps.


Sent from my iPad

Post by Stephane Bortzmeyer
On Tue, Mar 31, 2015 at 08:02:46PM +0700,

Post by battossai
It running every second, i receive about 14 mbit/sec traffic.

DNS attacks typically use small packets, and the stress comes from the
number of packets. Better to express it in Mp/s.

[Stephane Bortzmeyer]

On Tue, Mar 31, 2015 at 09:12:37PM +0700,

Post by Franky Yustanto
I got about 3500 pps.

And the suffix (136.xxx) actually changes every second?

[Franky Yustanto]

Thanks to Daisuke that help me privately that patch bloomfilter.
Now is worked on my unbound, still try to use it.
The traffic getting little bit down now ... Around 4 til 6 mbps.


Sent from my iPad

Yes, these domains change quite often, unfortunately :(
This is an attack called water torture.
I have developed a solution at work for this,
and am working to get the permission to opensource it.
I can only say that itcounts NXDOMAIN queries for delegation points,
and stops answering queries past a certain threshold.
(The bloom filter solution - checking for long lasting recursive queries
- is not bad at all, though)
Cheers,

Post by Stephane Bortzmeyer
On Tue, Mar 31, 2015 at 09:12:37PM +0700,

Post by Franky Yustanto
I got about 3500 pps.

And the suffix (136.xxx) actually changes every second?
_______________________________________________
Unbound-users mailing list
http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users

--
Stephane LAPIE, EPITA SRS, Promo 2005
"Even when they have digital readouts, I can't understand them."
--MegaTokyo

[Stephane Bortzmeyer]

On Tue, Mar 31, 2015 at 11:34:09PM +0900,

Yes, these domains change quite often, unfortunately :(
This is an attack called water torture.

Actually, no. I've seen very often the "water torture" or "random
qnames" attack and it is the first time I see in the wild such a rapid
change of the suffix.

https://blog.secure64.com/?p=377

[battossai]

I think bloomfilter is help enough for this moment or perhaps the attacker
is turning off the activity.
Let see for tommorow.

Post by Stephane Bortzmeyer
On Tue, Mar 31, 2015 at 11:34:09PM +0900,

Yes, these domains change quite often, unfortunately :(
This is an attack called water torture.

Actually, no. I've seen very often the "water torture" or "random
qnames" attack and it is the first time I see in the wild such a rapid
change of the suffix.
https://blog.secure64.com/?p=377

[battossai]

Hi,

Today i receive about 2000 p/s araound 4 to 5 mb/s.
That's better then yesterday.

Post by Stephane Bortzmeyer
On Tue, Mar 31, 2015 at 11:34:09PM +0900,

Yes, these domains change quite often, unfortunately :(
This is an attack called water torture.

Actually, no. I've seen very often the "water torture" or "random
qnames" attack and it is the first time I see in the wild such a rapid
change of the suffix.
https://blog.secure64.com/?p=377

[Thomas]

Hi,

We have the same problem.

Attacks are random and with many source IPs (botnets). Therefore it is
harder to have an automatic system to block source IPs. Our kind of
"workaround" was to increase the request_list size from the default 1024
to a higher number and to enable jostle-timeout to something like 4sec.
Therefore requests do not stay too long in the request_list once the box
is under load. Manual iptables rules are not maintainable, we only
manually block IPs for the biggest hitter. I agree what we are doing is
_not_ a fix to the problem because we just allocated more resources to
deal with the junk, but jostle-timeout definetely helps. I asked about
it almost a year ago on this mailing-list.

Subject: Unbound DDoS / reflexion attack counter-measure ?
Date: 30/05/14 22:20

Post by battossai
Any solution that can be shared ?

By trying to find my previous post, I actually realised that I missed
Daisuke's email.

Subject: "a mitigation against random subdomain attack"
Date: 24/03/15

His solution: https://github.com/hdais/unbound-bloomfilter

I will test it when I have a bit of time.

Thomas

[Stephane Bortzmeyer]

On Wed, Apr 01, 2015 at 07:53:54AM +1000,

Post by Thomas
We have the same problem.
Attacks are random and with many source IPs (botnets).

Stable suffix or not? battossai claimed that the suffix changed every
second.

Post by Thomas
Therefore it is
harder to have an automatic system to block source IPs.

It's not the source IP that you should block (they are probably forged
so you would block innocent people) but the suffix (I sent the
iptables rule for that a few messages ago).

Post by Thomas
Manual iptables rules are not maintainable,

In my experience, they are, if the attacker does not change the
suffix.

[Stephane LAPIE]

Post by Stephane Bortzmeyer

Post by Thomas
Manual iptables rules are not maintainable,

In my experience, they are, if the attacker does not change the
suffix.

Just my 2 cents here :
The pattern I am seeing on my side does not evolve as fast as one per
second,
but the attacker does change domains every few hours or so.

However, the authoritative servers being hammered as a result do not
change that much.
(Most domains I am seeing are chinese domains related to online gambling
and what not.)

And, in my situation, trying to maintain local zones or iptables rules
is a litteral "whack-a-mole" game,
you can't humanely do that manually for an extended period of time.
It's like, these guys have troves of domains to use and abuse...

(Things get further tricky when some of these domains are set with
wildcard records too)

[Daniel Ryslink]

Post by Stephane LAPIE
And, in my situation, trying to maintain local zones or iptables rules
is a litteral "whack-a-mole" game,
you can't humanely do that manually for an extended period of time.
It's like, these guys have troves of domains to use and abuse...

However, you can maintain local zone list in unbound automatically
fairly easily, we have been doing it for over a year with minimal
necessity of manual intervention. If you wish, have a look at the
attached perl script.

The only other option is to persuade the users of the compromised
machines to clean their systems.
--
Best Regards,
Daniel Ryšlink
System Administrator

Dial Telecom a. s.
Křişíkova 36a/237
186 00 Praha 3, Česká Republika
Tel.:[phone]
***@dialtelecom.cz
-----------------------------------------------
www.dialtelecom.cz
Dial Telecom, a.s.
Jednoduše se připojte
-----------------------------------------------

Post by Stephane LAPIE

Post by Stephane Bortzmeyer

Post by Thomas
Manual iptables rules are not maintainable,

In my experience, they are, if the attacker does not change the
suffix.

The pattern I am seeing on my side does not evolve as fast as one per
second,
but the attacker does change domains every few hours or so.
However, the authoritative servers being hammered as a result do not
change that much.
(Most domains I am seeing are chinese domains related to online gambling
and what not.)
And, in my situation, trying to maintain local zones or iptables rules
is a litteral "whack-a-mole" game,
you can't humanely do that manually for an extended period of time.
It's like, these guys have troves of domains to use and abuse...
(Things get further tricky when some of these domains are set with
wildcard records too)
_______________________________________________
Unbound-users mailing list
http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users

[Daisuke HIGASHI]

However, you can maintain local zone list in unbound automatically fairly
easily, we have been doing it for over a year with minimal necessity of
manual intervention. If you wish, have a look at the attached perl script.

unbound-bloomfilter's attack detection mechanisms implement almost
same thing as your script.
I used public suffix list (source code embedded, currently) to
determine depth of blocking domain
which corresponds to your "third_level_domains.conf".


Note that the bloomfilter itself is a way to reduce collateral damage
caused by filtering.
Of course to reduce damage caused by wrong (false positive) filtering and
to accept legitimate queries for the filtered domain

The only other option is to persuade the users of the compromised machines
to clean their systems.

I agree.

Regards,
--
Daisuke HIGASHI

[Kevin Chadwick]

On Thu, 2 Apr 2015 02:34:48 +0900

Post by Daisuke HIGASHI

The only other option is to persuade the users of the compromised machines
to clean their systems.

I agree.

Which isn't going to happen unless idiots all get chrome books assuming
the update situation their is better than android, which isn't going to
happen because Windows still has third party support others don't and
the Win7 kernel onwards actually has more security features than Linux
or Freebsd but not openbsd or grsec linux. Not to mention out of date
firmwares on phones that laughably are now being branded as a cool
feature by ubuntu mobile (one-time update marketing rubbish). Oh and
those security features that the Win7 kernel now has can't work with
JAVA.

What is the attackers motivation, amplification? If that's true? then
amplification is the real issue and SSL everywhere has a similar
perhaps more dangerous issue that can be fixed as it was with TCP
handshakes sending more than one ACK, the current implementations are
wrong and will now take time to phase out and the sooner that starts the
better.

****The client should have to do more work than he causes****.

[A. Schulze]

Post by Daniel Ryslink
The only other option is to persuade the users of the compromised
machines to clean their systems.

good point.

question to make the problem clear (to me):
the unbound resolver you talk about here provide resolving services for
own/trusted/known/paying users but not for anybody in the world. Correct?

Or do you talk about open/public resolver?

Andreas

[Daniel Ryslink]

Hello,

I have just subscribed here, but we have been dealing with this problem
for about a year.

Here is our solution - a watchdog script that does "unbound-control
dump_requestlist" at regular interval to see how many concurrent
recursive queries are being worked upon.

If there is a flood, this will spike over a defined limit (depending on
normal traffic), and the following action is taken:

The flooding queries have typically the same structure -
<random_string>.<some_domain>, co that the server cannot use cache and
wastes resources on doing a recursive query.

When the number of concurrent queries spike, the script counts them by
domain, and those domains that exceed a defined share (usually over a
quarter) are temporarily blacklisted via "ubound-control local_zone
deny" (you can use "reject" too, or serve an authoritative NXDOMAIN
answer if you prefer). This solution takes advantage of the fact that
legitimate queries are most often quickly finished, and only the bogus
ones pile up and clog the server's memory.

This temporary blacklist is cleared once a day automatically. All
blacklisted zones are logged and I review them regularly, there is an
absolute minimum of false positives. The script also supports
whitelisting of zones you never ever want to blacklist.

I can share the script if anyone is interested.
--
Best regards,
Daniel Ryšlink
System Administrator

Dial Telecom a. s.
Křižíkova 36a/237
186 00 Praha 3, Česká Republika
Tel.:[phone]
***@dialtelecom.cz
-----------------------------------------------
www.dialtelecom.cz
Dial Telecom, a.s.
Jednoduše se připojte
-----------------------------------------------

Post by Thomas
Hi,
We have the same problem.
Attacks are random and with many source IPs (botnets). Therefore it is
harder to have an automatic system to block source IPs. Our kind of
"workaround" was to increase the request_list size from the default
1024 to a higher number and to enable jostle-timeout to something like
4sec. Therefore requests do not stay too long in the request_list once
the box is under load. Manual iptables rules are not maintainable, we
only manually block IPs for the biggest hitter. I agree what we are
doing is _not_ a fix to the problem because we just allocated more
resources to deal with the junk, but jostle-timeout definetely helps.
I asked about it almost a year ago on this mailing-list.
Subject: Unbound DDoS / reflexion attack counter-measure ?
Date: 30/05/14 22:20

Post by battossai
Any solution that can be shared ?

By trying to find my previous post, I actually realised that I missed
Daisuke's email.
Subject: "a mitigation against random subdomain attack"
Date: 24/03/15
His solution: https://github.com/hdais/unbound-bloomfilter
I will test it when I have a bit of time.
Thomas
_______________________________________________
Unbound-users mailing list
http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users

[battossai]

Hi Daniel,


I am interesting to try your script.
Thanks.


Regards,
Franky Yu