[Unbound-users] Python API extension patch proposal

[Stephane Lapie]

Hello,

I am currently in the process of dealing with water torture attacks on
our cache DNS servers (<randomstring>.domain.com queries that never
resolve and end up causing enormous upstream traffic, ultimately
crushing the authoritative server for domain.com).

To this end, I toyed around with unbound, and noticed that
unbound-control's lookup function has a very interesting feature :
$ sudo unbound-control lookup randomtest.google.com
The following name servers are used for lookup of randomtest.google.com.
;rrset 14224 4 0 8 0
google.com. 273424 IN NS ns2.google.com.
google.com. 273424 IN NS ns3.google.com.
google.com. 273424 IN NS ns4.google.com.
google.com. 273424 IN NS ns1.google.com.
;rrset 7350 1 0 8 0
ns1.google.com. 266550 IN A 216.239.32.10
;rrset 57726 1 0 8 0
ns4.google.com. 316926 IN A 216.239.38.10
;rrset 56988 1 0 8 0
ns3.google.com. 316188 IN A 216.239.36.10
;rrset 28120 1 0 8 0
ns2.google.com. 287320 IN A 216.239.34.10
Delegation with 4 names, of which 4 can be examined to query further
addresses.
It provides 4 IP addresses.
216.239.34.10 rto 87 msec, ttl 628, ping 71 var 4 rtt 87, tA 0, tAAAA
0, tother 0, EDNS 0 probed.
216.239.36.10 rto 645 msec, ttl 19, ping 101 var 136 rtt 645, tA 0,
tAAAA 0, tother 0, EDNS 0 probed.
216.239.38.10 rto 113 msec, ttl 31, ping 97 var 4 rtt 113, tA 0,
tAAAA 0, tother 0, EDNS 0 probed.
216.239.32.10 rto 99 msec, ttl 328, ping 47 var 13 rtt 99, tA 0,
tAAAA 0, tother 0, EDNS 0 probed.

Namely, for any given hostname, it can find the closest delegation point
(in this case, remove unambiguously and with no danger the random part
of the attack query, since it goes to the deepest component that retains
any meaning DNS-wise).

I thought that since the information was available within the Unbound
process, coding a Python module that would keep track of the count of
queries to a DDoSed delegation point would be a good start to an
algorithm for effectively blocking water torture attacks, but the
required function, dns_cache_find_delegation() was not available readily
from the Python API.

Therefore, I extended the Python API as per the attached file (also
available at
http://www.yomi.darkbsd.org/~darksoul/lookup-api-extension.patch), to
export struct delegpt and a find_delegation() function that would allow
to acquire the delegation point name, records and servers, from the
Python module environment.

Example of use case (extremely simplified, implementation of
allow_query() not disclosed) :
def operate(id, event, qstate, qdata):
delegation = find_delegation(qstate, qstate.qinfo.qname,
len(qstate.qinfo.qname))

if (event == MODULE_EVENT_NEW) or (event == MODULE_EVENT_PASS):
policy_result = allow_query(qstate.qinfo, delegation,
delegation_name)
if (policy_result): # Pass query to next module
qstate.ext_state[id] = MODULE_WAIT_MODULE
else:
qstate.ext_state[id] = MODULE_ERROR
return True

This patch, along with an actual module that will SERVFAIL (as above)
cache-missing queries going over threshold (therefore reducing upstream
traffic to a tenth of what it would be if honoring DDoS-related queries,
AND keeping it within our AS), has been running in our production
environment at ASAHI Net for several months now, and has been approved
for upstream contribution on our side.

I thought this feature would be very useful to have in the Python module
environment, so would it be possible for you to consider integrating
this patch as a standard feature in Unbound?

Thanks for your time,

[Tarko Tikan]

hey,

Post by Stephane Lapie
I am currently in the process of dealing with water torture attacks on
our cache DNS servers (<randomstring>.domain.com queries that never
resolve and end up causing enormous upstream traffic, ultimately
crushing the authoritative server for domain.com).

I wrote https://github.com/tarko/unbound-reqmon while ago to mitigate
this issue. This will block the domain that is being used for the abuse.

PS! It will need constant attention because it will happily block co.uk,
com.tw etc. at this point. The logic must really be improved if these
attacks persist.

[Stephane Lapie]

Post by Tarko Tikan
hey,

Post by Stephane Lapie
I am currently in the process of dealing with water torture attacks on
our cache DNS servers (<randomstring>.domain.com queries that never
resolve and end up causing enormous upstream traffic, ultimately
crushing the authoritative server for domain.com).

I wrote https://github.com/tarko/unbound-reqmon while ago to mitigate
this issue. This will block the domain that is being used for the abuse.
PS! It will need constant attention because it will happily block
co.uk, com.tw etc. at this point. The logic must really be improved if
these attacks persist.

Yes, I did something similar to this (crunching unbound logs -> local zone), however, as mentioned in my first mail, there is the need to avoid TLDs,
and to figure the longest delegation point for sure, to be 100% sure of what you will end up auto-blocking.

We threw that idea out of the window when "co.jp" got locked out in the dead of the night because one of our corporate customers hammered our DNS server with infected clients on his ActiveDirectory network. (more than 1000 queries on "whatever.domain.co.jp" -> cut last two units -> "co.jp" -> OOPS.)

In Japan, these attacks have reached a point where manual handling and whack-a-mole are not realistic anymore, also the delegation point information lies in the infra cache, and execution of unbound-control is extremely expensive performance-wise, so I came to the conclusion that this processing is best done within Unbound itself, as a patch or a python module.

Alternatively, thinking back to your method : it might be a sound idea to patch unbound-control itself, and to add an option that list requests along with their delegation point name, to perform what you are doing in a more safe manner.

Cheers,

[W.C.A. Wijngaards]

Hi Stephane,

Post by Stephane Lapie
Hello,
I am currently in the process of dealing with water torture attacks
on our cache DNS servers (<randomstring>.domain.com queries that
never resolve and end up causing enormous upstream traffic,
ultimately crushing the authoritative server for domain.com).
To this end, I toyed around with unbound, and noticed that
$ sudo unbound-control lookup randomtest.google.com The following
name servers are used for lookup of randomtest.google.com. ;rrset
14224 4 0 8 0 google.com. 273424 IN NS ns2.google.com.
google.com. 273424 IN NS ns3.google.com. google.com.
273424 IN NS ns4.google.com. google.com. 273424 IN
NS ns1.google.com. ;rrset 7350 1 0 8 0 ns1.google.com. 266550
IN A 216.239.32.10 ;rrset 57726 1 0 8 0 ns4.google.com.
316926 IN A 216.239.38.10 ;rrset 56988 1 0 8 0
ns3.google.com. 316188 IN A 216.239.36.10 ;rrset 28120
1 0 8 0 ns2.google.com. 287320 IN A 216.239.34.10
Delegation with 4 names, of which 4 can be examined to query
further addresses. It provides 4 IP addresses. 216.239.34.10
rto 87 msec, ttl 628, ping 71 var 4 rtt 87, tA 0, tAAAA 0, tother
0, EDNS 0 probed. 216.239.36.10 rto 645 msec, ttl 19, ping
101 var 136 rtt 645, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
216.239.38.10 rto 113 msec, ttl 31, ping 97 var 4 rtt 113, tA
0, tAAAA 0, tother 0, EDNS 0 probed. 216.239.32.10 rto 99
msec, ttl 328, ping 47 var 13 rtt 99, tA 0, tAAAA 0, tother 0, EDNS
0 probed.
Namely, for any given hostname, it can find the closest delegation
point (in this case, remove unambiguously and with no danger the
random part of the attack query, since it goes to the deepest
component that retains any meaning DNS-wise).
I thought that since the information was available within the
Unbound process, coding a Python module that would keep track of
the count of queries to a DDoSed delegation point would be a good
start to an algorithm for effectively blocking water torture
attacks, but the required function, dns_cache_find_delegation() was
not available readily from the Python API.
Therefore, I extended the Python API as per the attached file
(also available at
http://www.yomi.darkbsd.org/~darksoul/lookup-api-extension.patch),
to export struct delegpt and a find_delegation() function that
would allow to acquire the delegation point name, records and
servers, from the Python module environment.
Example of use case (extremely simplified, implementation of
allow_query() not disclosed) : def operate(id, event, qstate,
qdata): delegation = find_delegation(qstate, qstate.qinfo.qname,
len(qstate.qinfo.qname))
policy_result = allow_query(qstate.qinfo, delegation,
delegation_name) if (policy_result): # Pass query to next module
qstate.ext_state[id] = MODULE_ERROR return True
This patch, along with an actual module that will SERVFAIL (as
above) cache-missing queries going over threshold (therefore
reducing upstream traffic to a tenth of what it would be if
honoring DDoS-related queries, AND keeping it within our AS), has
been running in our production environment at ASAHI Net for several
months now, and has been approved for upstream contribution on our
side.

Thank you for the patch, I have put it in the source. Can you tell
the allow_query() details that work for you (the threshold and what
you do with the AS specifically)?

Best regards,
Wouter

Post by Stephane Lapie
I thought this feature would be very useful to have in the Python
module environment, so would it be possible for you to consider
integrating this patch as a standard feature in Unbound?
Thanks for your time,

[Stephane LAPIE]

Hi Wouter,

Post by W.C.A. Wijngaards

Post by Stephane Lapie
This patch, along with an actual module that will SERVFAIL (as
above) cache-missing queries going over threshold (therefore
reducing upstream traffic to a tenth of what it would be if
honoring DDoS-related queries, AND keeping it within our AS), has
been running in our production environment at ASAHI Net for several
months now, and has been approved for upstream contribution on our
side.

Thank you for the patch, I have put it in the source. Can you tell
the allow_query() details that work for you (the threshold and what
you do with the AS specifically)?

Many thanks for your understanding.
I'll try to provide as much notes as I am allowed to.

I don't have total clearance yet to publish the full code, or the exact
thresholds, but basically here's what I am doing and where the reasoning
comes from.



* Storing information :
I am storing query details for :
- the client's address
- the delegpt name (simplified form of qdsjhfoishfsdofjdqs.domain.com,
"domain.com", derived via infra cache), which I will refer hereafter as
"domain"
- the "client-domain" pair

Then, for each of these, I create python dictionaries (it's tricky to
handle locking properly though ;)) that have a series of counters for :
- Normal query count (anything short of ANY)
-> If used in conjunction with the AAAA filter, ignore these queries as
they will never return meaningful information
- ANY query count
-> This is usually blocked at firewall level, but I wanted to try a few
things
- NXDOMAIN count (I get the return code from another event)
-> A server capable of answering NXDOMAINs will crank up this counter
extremely fast in case of a violent attack
- RRSET count (if the query was a success, and had meaningful data, I
check the answer's RRSET count)
-> It could probably be used to detect and block weird forms of AMP
attacks, using the TXT records for instance.

Then, I have set thresholds for each of these counters, applicable to
client, domains, and client-domain pairs.
These thresholds are basically hand-set, applying roughly a 80% ratio on
the kind of fullscale DDoS we take :
- A given DDoS participant gives out 4 attack queries / second to that
domain, which is around 1200 cache-missing queries in 5 minutes.
- A given domain in a DDoS therefore takes around a hundred of these at
the very least.
- One can therefore start flagging a client-domain relation around 5
cache-missing queries in 5 minutes. (not outright blocking ! the purpose
is to contain an attack without harming legitimate use)



* Decision process (summary) :
- In either of these two cases :
-- A client breaches the "single client" threshold (this guy is sending
way too much stuff to be legit, no matter what), or the "client-domain"
maximum threshold
-- A domain is above the DDoS threshold (given how many queries it has
received in five minutes, it's being hammered), and the "client-domain"
is within suspicion range
my implementation of allow_query() will return false, and it will be
blocked.
- For mitigation of false positives :
-- The above is basically mainly using the normal, ANY and NXDOMAIN
counters and positive thresholds
-- In order to avoid false positives, I check if the RRSET counter is
superior to zero, and if so, this means this domain, or client, or
client-domain pair actually have dealt with meaningful info and are
probably not implied in a bonafide DDoS, since they provide stuff that
will fill the cache with meaningful info.



* Data purging :
- Also, every minute, I decrement by a fifth of the total threshold.
This ensures I "forget" about an attacker, but not too fast.
- Last decrement step goes down to the suspicion threshold (if set), and
if the client really has been behaving properly and not hammering, then
it's fully decremented back to zero, otherwise SERVFAIL.
- Every five minutes, I weed out dictionaries entries that have been
already set at zero, to ensure I don't leak memory like crazy.

I could actually also implement dictionaries based on authority servers,
but this is highly unperformant and it's not possible to update all the
dictionaries in the time of a query answer. I think the above algorithm
already considerably slows down the whole thing, but it still works just
fine, thank gods.



* Notes :
- I have actually noticed our attackers have caught on to this blocking
method, and they started attacking domains sharing the same authority
servers, thus splitting the load across several domains.
-> Actually, though, this can probably be blocked with outgoing rate
limit on firewall level, but I was thinking it might make sense for
unbound's iterator module to store information on how much a target
server is solicited to mitigate attacks.
- I am also sending out UDP broadcasts to a monitoring server when
blocking a query, to keep tabs on blocked domains. I eventually think of
having a separate thread to get these broadcasts directly from unbound
so as to share the information that a given client is up to no-good, or
that a domain is hammered, or that a given client is forbidden from
accessing a specific domain.
-> It might be nice to have python module environment variables
accessible via a stats dump or something, food for thought.



* Measured effect :
- A client's queries are blocked, only if they are really hammering the
server, or aggravating an already overwhelmed domain.
-> Legitimate traffic or queries for cached entries are not affected,
even if they are participating in a DDoS. This avoids harming "owned"
customers, and earns the extra time needed to actually help them out.
- Queries that should go upstream to the victim authority server are
stopped, and a SERVFAIL is answered to the client.
-> This not only ensures frivolous recursive queries are not performed,
it also ensures no negative caching is done on Unbound's side, and keeps
memory and network usage tight.

Concretely, this means that, instead of doing this :
1) Client -> ISP network -> Cache server : Sending query for
dsjfiodhsfiodjfiosdfdsq.domain.com (confirming 3MB/s ingoing traffic)
2) Cache server -> Internet peers -> Authority server : Sending query
for dsjfiodhsfiodjfiosdfdsq.domain.com (confirming 30MB/s outgoing
traffic if unfiltered)
3) Cache server <- Internet peers <- Authority server : Receiving
NXDOMAIN or timeouting, thus further insisting
4) Client <- ISP network <- Cache server : Reply timeout for client, or
SERVFAIL if all authority servers for the delegation point are confirmed
dead (which can take a lot of time or never happen), clogging the
recursive client list

Eventually, the query flow becomes this, once a domain has been
confirmed as a DDoS target and a client as an attacker :
1) Client -> ISP network -> Cache server : Sending query for
dsjfiodhsfiodjfiosdfdsq.domain.com (confirming 3MB/s ingoing traffic)
2) Client <- ISP network <- Cache server : Replying SERVFAIL (confirming
3MB/s outgoing traffic, which remains inside of our network and does not
impact any of our network peers), and keeping the recursive client list
clean

This is what I meant by "traffic staying within our AS", this avoids
polluting our network peers' pipes with crud, and unnecessary transit costs.
There is no fancy coordination with our BGP routing tables (yet ;)).


* Conclusion :
Basically, being able to lookup the delegpt name is literally the
cornerstone of the whole above algorithm, as it allows to shove into one
counter hundreds of millions of queries and to finally quantify the
damage in a way that allows for fair blocking.

Cheers,