Discussion:
[Unbound-users] DNS poisoning - any ideas how this can happen?
Martin Bachmann
2015-02-09 17:33:52 UTC
Permalink
Hi all,

We've run into a dns poisoning issue in our company network since Friday.
The issue is being discussed here:
https://forum.pfsense.org/index.php?topic=87491.0 - we use Unbound on a
pfSense. A few other users have the same problem:

- All of a sudden, all host names resolve to a malware host.
- It stops automatically after some time
- There's no arp poisoning going on, so it really comes from Unbound on the
pfSense

Example:

While "on":

$ host omx.ch
omx.ch has address 195.22.26.248
omx.ch mail is handled by 10 mx1.csof.net.
omx.ch mail is handled by 10 mx2.csof.net.

Normally:

$host omx.ch
omx.ch has address 62.48.3.132
omx.ch mail is handled by 10 mxhost1.omx.ch

Other wrongly resolved ips lead to sso.mlwr.io (which tries to redirect
back to xsso.<correcthost.com>/<someidentifier>)

Any ideas?

- Martin
W.C.A. Wijngaards
2015-02-10 08:27:05 UTC
Permalink
Hi Martin,
Post by Martin Bachmann
Hi all,
We've run into a dns poisoning issue in our company network since
https://forum.pfsense.org/index.php?topic=87491.0 - we use Unbound
- All of a sudden, all host names resolve to a malware host. - It
stops automatically after some time - There's no arp poisoning
going on, so it really comes from Unbound on the pfSense
So, unbound comes with a set of commands for unbound-control that
allow you to monitor the runtime settings, and these are exactly meant
to be able to audit the settings in the runtime daemon and if they are
still correct. unbound-control list_forwards.

You could try to use packet capture of traffic going to 8.8.8.8 and
the responses. Or you can get unbound to log verbosely (high
verbosity setting), although much slower at level 4 it'll print a
dig-like output for packets received from upstream, so you can see
where the malicious data comes from.

Or just dig @8.8.8.8 from the commandline, that has the same routing
as the pfSense firewall box with unbound on it, and look at the result.

Unbound has DNSSEC capabilities that are meant to protect against
these sorts of things (only for DNSSEC signed domains of course). You
can easily turn it on with unbound-anchor -a /etc/root.key and putting
auto-trust-anchor-file: "/etc/root.key" in unbound.conf.

Best regards,
Wouter
Post by Martin Bachmann
$ host omx.ch <http://omx.ch> omx.ch <http://omx.ch> has address
195.22.26.248 omx.ch <http://omx.ch> mail is handled by 10
mx1.csof.net <http://mx1.csof.net>. omx.ch <http://omx.ch> mail is
handled by 10 mx2.csof.net <http://mx2.csof.net>.
$host omx.ch <http://omx.ch> omx.ch <http://omx.ch> has address
62.48.3.132 omx.ch <http://omx.ch> mail is handled by 10
mxhost1.omx.ch <http://mxhost1.omx.ch>
Other wrongly resolved ips lead to sso.mlwr.io
<http://sso.mlwr.io> (which tries to redirect back to
xsso.<correcthost.com <http://correcthost.com>>/<someidentifier>)
Any ideas?
- Martin
_______________________________________________ Unbound-users
http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
W.C.A. Wijngaards
2015-02-10 09:18:44 UTC
Permalink
Hi Martin,

Looking through that forum thread, I see that unbound's cache contains
bad items and that the NS records for com, net org edu are point to
nsX.csof.net. And I wonder if unbound is getting cache poisoned or if
your 'upstream resolver' or upstream captive-resolver (i.e. the
8.8.8.8 has been hijacked by your ISP and is serviced by other
software) is getting cache poisoned.

The unbound logs should tell you if you enable verbose logging (I
would recommend level 4, or perhaps level 5 so you can see who
requests those bad domain names in your network). Or when unbound is
misbehaving dig @8.8.8.8 from a box with similar routing, and see if
those responses have been cache poisoned.

When I try to resolve api-nyc01.exip.org here I see unbound complain
that it has to remove potential poisonous DNS RRsets from the answers.
It does so and is not poisoned:

reply from <exip.org.> 54.77.72.254#53
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 26841
;; flags: qr aa ; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 4
;; QUESTION SECTION:
;; api-nyc01.exip.org. IN A

;; ANSWER SECTION:
api-nyc01.exip.org. 10 IN A 195.22.26.248

;; AUTHORITY SECTION:
org. 172800 IN NS ns1.csof.net.
org. 172800 IN NS ns2.csof.net.
org. 172800 IN NS ns3.csof.net.
org. 172800 IN NS ns4.csof.net.

;; ADDITIONAL SECTION:
ns1.csof.net. 100 IN A 54.77.72.254
ns2.csof.net. 100 IN A 212.6.183.201
ns3.csof.net. 100 IN A 195.22.26.199
ns4.csof.net. 100 IN A 54.72.8.183

Humourously (if you think this is funny), it could be a
misconfiguration and bad DNS ISP practices interfering here. Some
domain hosters serve .org and .com zones and they have *.com and *.org
wildcards on those servers. Thus they do not reconfigure their DNS
servers when they buy or sell a domain. But that produces these types
of 'poisonous' answers. Badly written DNS software could then get DNS
cache poisoned as a result. Unbound should not get DNS cache
poisoned, there is explicit fixup code for this.

Because setting different upstream IP forwarders changes the outcome,
I think it may be the upstream DNS server that gets cache poisoned
(after a lookup of one of these affected domains), and some ISPs
interpose all DNS traffic with their own DNS servers, that may be
getting poisoned or maybe only traffic to 'popular' DNS sites. I
doubt google DNS itself is getting cache poisoned, but it is a
technical possibility.

Best regards,
Wouter
Post by W.C.A. Wijngaards
Hi Martin,
Post by Martin Bachmann
Hi all,
We've run into a dns poisoning issue in our company network since
https://forum.pfsense.org/index.php?topic=87491.0 - we use
- All of a sudden, all host names resolve to a malware host. -
It stops automatically after some time - There's no arp
poisoning going on, so it really comes from Unbound on the
pfSense
So, unbound comes with a set of commands for unbound-control that
allow you to monitor the runtime settings, and these are exactly
meant to be able to audit the settings in the runtime daemon and if
they are still correct. unbound-control list_forwards.
You could try to use packet capture of traffic going to 8.8.8.8
and the responses. Or you can get unbound to log verbosely (high
verbosity setting), although much slower at level 4 it'll print a
dig-like output for packets received from upstream, so you can see
where the malicious data comes from.
routing as the pfSense firewall box with unbound on it, and look at
the result.
Unbound has DNSSEC capabilities that are meant to protect against
these sorts of things (only for DNSSEC signed domains of course).
You can easily turn it on with unbound-anchor -a /etc/root.key and
putting auto-trust-anchor-file: "/etc/root.key" in unbound.conf.
Best regards, Wouter
Post by Martin Bachmann
$ host omx.ch <http://omx.ch> omx.ch <http://omx.ch> has address
195.22.26.248 omx.ch <http://omx.ch> mail is handled by 10
mx1.csof.net <http://mx1.csof.net>. omx.ch <http://omx.ch> mail
is handled by 10 mx2.csof.net <http://mx2.csof.net>.
$host omx.ch <http://omx.ch> omx.ch <http://omx.ch> has address
62.48.3.132 omx.ch <http://omx.ch> mail is handled by 10
mxhost1.omx.ch <http://mxhost1.omx.ch>
Other wrongly resolved ips lead to sso.mlwr.io
<http://sso.mlwr.io> (which tries to redirect back to
xsso.<correcthost.com
<http://correcthost.com>>/<someidentifier>)
Any ideas?
- Martin
_______________________________________________ Unbound-users
http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
_______________________________________________ Unbound-users
http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
W.C.A. Wijngaards
2015-02-10 14:50:12 UTC
Permalink
Hi,

After off-list conversation (with conf and logs), the solution is
harden-glue: yes in unbound.conf. The default is yes, but in pfSense
it was turned off.

There is a code fix that still protects NS records if config has
harden-glue no.

Best regards,
Wouter
Post by W.C.A. Wijngaards
Hi Martin,
Looking through that forum thread, I see that unbound's cache
contains bad items and that the NS records for com, net org edu are
point to nsX.csof.net. And I wonder if unbound is getting cache
poisoned or if your 'upstream resolver' or upstream
captive-resolver (i.e. the 8.8.8.8 has been hijacked by your ISP
and is serviced by other software) is getting cache poisoned.
The unbound logs should tell you if you enable verbose logging (I
would recommend level 4, or perhaps level 5 so you can see who
requests those bad domain names in your network). Or when unbound
see if those responses have been cache poisoned.
When I try to resolve api-nyc01.exip.org here I see unbound
complain that it has to remove potential poisonous DNS RRsets from
QUERY, rcode: NOERROR, id: 26841 ;; flags: qr aa ; QUERY: 1,
ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 4 ;; QUESTION SECTION: ;;
api-nyc01.exip.org. IN A
;; ANSWER SECTION: api-nyc01.exip.org. 10 IN A 195.22.26.248
;; AUTHORITY SECTION: org. 172800 IN NS ns1.csof.net. org. 172800
IN NS ns2.csof.net. org. 172800 IN NS ns3.csof.net. org. 172800 IN
NS ns4.csof.net.
;; ADDITIONAL SECTION: ns1.csof.net. 100 IN A 54.77.72.254
ns2.csof.net. 100 IN A 212.6.183.201 ns3.csof.net. 100 IN A
195.22.26.199 ns4.csof.net. 100 IN A 54.72.8.183
Humourously (if you think this is funny), it could be a
misconfiguration and bad DNS ISP practices interfering here. Some
domain hosters serve .org and .com zones and they have *.com and
*.org wildcards on those servers. Thus they do not reconfigure
their DNS servers when they buy or sell a domain. But that
produces these types of 'poisonous' answers. Badly written DNS
software could then get DNS cache poisoned as a result. Unbound
should not get DNS cache poisoned, there is explicit fixup code for
this.
Because setting different upstream IP forwarders changes the
outcome, I think it may be the upstream DNS server that gets cache
poisoned (after a lookup of one of these affected domains), and
some ISPs interpose all DNS traffic with their own DNS servers,
that may be getting poisoned or maybe only traffic to 'popular' DNS
sites. I doubt google DNS itself is getting cache poisoned, but it
is a technical possibility.
Best regards, Wouter
Post by W.C.A. Wijngaards
Hi Martin,
Post by Martin Bachmann
Hi all,
We've run into a dns poisoning issue in our company network
https://forum.pfsense.org/index.php?topic=87491.0 - we use
- All of a sudden, all host names resolve to a malware host. -
It stops automatically after some time - There's no arp
poisoning going on, so it really comes from Unbound on the
pfSense
So, unbound comes with a set of commands for unbound-control that
allow you to monitor the runtime settings, and these are
exactly meant to be able to audit the settings in the runtime
daemon and if they are still correct. unbound-control
list_forwards.
You could try to use packet capture of traffic going to 8.8.8.8
and the responses. Or you can get unbound to log verbosely
(high verbosity setting), although much slower at level 4 it'll
print a dig-like output for packets received from upstream, so
you can see where the malicious data comes from.
routing as the pfSense firewall box with unbound on it, and look
at the result.
Unbound has DNSSEC capabilities that are meant to protect against
these sorts of things (only for DNSSEC signed domains of
course). You can easily turn it on with unbound-anchor -a
/etc/root.key and putting auto-trust-anchor-file: "/etc/root.key"
in unbound.conf.
Best regards, Wouter
Post by Martin Bachmann
$ host omx.ch <http://omx.ch> omx.ch <http://omx.ch> has
address 195.22.26.248 omx.ch <http://omx.ch> mail is handled by
10 mx1.csof.net <http://mx1.csof.net>. omx.ch <http://omx.ch>
mail is handled by 10 mx2.csof.net <http://mx2.csof.net>.
$host omx.ch <http://omx.ch> omx.ch <http://omx.ch> has address
62.48.3.132 omx.ch <http://omx.ch> mail is handled by 10
mxhost1.omx.ch <http://mxhost1.omx.ch>
Other wrongly resolved ips lead to sso.mlwr.io
<http://sso.mlwr.io> (which tries to redirect back to
xsso.<correcthost.com
<http://correcthost.com>>/<someidentifier>)
Any ideas?
- Martin
_______________________________________________ Unbound-users
http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
_______________________________________________ Unbound-users
http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
_______________________________________________ Unbound-users
http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
Dave Warren
2015-02-10 21:49:23 UTC
Permalink
Post by W.C.A. Wijngaards
After off-list conversation (with conf and logs), the solution is
harden-glue: yes in unbound.conf. The default is yes, but in pfSense
it was turned off.
Ouch, that seems like a sub-optimal configuration.

Was this in the unbound package available for pfSense 2.1 and earlier,
or the native unbound implementation in pfSense 2.2? Did you log any
bugs on the pfSense side or does this still need to be done?
--
Dave Warren
http://www.hireahit.com/
http://ca.linkedin.com/in/davejwarren
Dave Warren
2015-02-11 23:52:09 UTC
Permalink
Post by Dave Warren
Post by W.C.A. Wijngaards
After off-list conversation (with conf and logs), the solution is
harden-glue: yes in unbound.conf. The default is yes, but in pfSense
it was turned off.
Ouch, that seems like a sub-optimal configuration.
Was this in the unbound package available for pfSense 2.1 and earlier,
or the native unbound implementation in pfSense 2.2? Did you log any
bugs on the pfSense side or does this still need to be done?
I had a chance to play with pfSense 2.2 a bit, and this is a UI option,
so there's no concern here. Sorry for the additional noise, when I first
read the thread I was assuming that you meant the unbound default is
set, but pfSense turned it off.

tl;dr: Everything is good.
--
Dave Warren
http://www.hireahit.com/
http://ca.linkedin.com/in/davejwarren
Loading...