[Unbound-users] Unbound vs MS Resolver

[Olaf Kolkman]

Folk,

We recently got a query from somebody in the industry asking how Unbound compares to Windows Resolver performance in terms of qps and CPU (memory is under user control).

All our benchmarking is Unix based and we never really did a comparison ourselves. Is there somebody on this list that has performed such measurements, or knows of a reference?

Thanks!

—Olaf Kolkman
NLnet Labs.

[Maurice Walker]

Could I ask if a comparison of Unbound vs BIND has been documented for
a Unix platform?

Thanks,
Maurice

Post by Olaf Kolkman
Folk,
We recently got a query from somebody in the industry asking how
Unbound compares to Windows Resolver performance in terms of qps
and CPU (memory is under user control).
All our benchmarking is Unix based and we never really did a
comparison ourselves. Is there somebody on this list that has
performed such measurements, or knows of a reference?
Thanks!
?Olaf Kolkman NLnet Labs.
_______________________________________________ Unbound-users
http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users

[Olaf Kolkman]

Post by Maurice Walker
Could I ask if a comparison of Unbound vs BIND has been documented for
a Unix platform?

One example, something done by a 3rd party: http://www.iepg.org/2010-11-ietf79/iepg79-mglt.pdf

In general recursive name servers are hard to compare because of how caching affects performance.

—Olaf

[u***@strotmann.de]

Hello Olaf,

Post by Olaf Kolkman
Folk,
We recently got a query from somebody in the industry asking how
Unbound compares to Windows Resolver performance in terms of qps and
CPU (memory is under user control).
All our benchmarking is Unix based and we never really did a
comparison ourselves. Is there somebody on this list that has
performed such measurements, or knows of a reference?

I have a working setup where I could test Unbound vs. MS DNS (the setup
where I do regression tests on the Unbound packages I compile) and would
be very interested in the results as well. The machines are not too
powerful (dual-core with 8GB RAM) compared to todays standards.

I'm pretty busy the next two weeks, but could do the tests later in June
(if that helps).

Is the query about comparing Unbound on Windows with MS DNS, or
comparing Unbound on Unix/Linux with Win DNS?

Carsten

[Dave Warren]

Post by u***@strotmann.de
Is the query about comparing Unbound on Windows with MS DNS, or
comparing Unbound on Unix/Linux with Win DNS?

I'd be interested in the results comparing the two on a
similar/identical platform. In other words, should I uninstall Microsoft
DNS and install unbound on the same system (where unbound is otherwise a
reasonable fit)

Obviously it's not a suitable replacement for Active Directory driven
DNS. However, even here, there's an interesting performance question: Is
it worth installing unbound and forwarding Microsoft DNS to unbound, or
is it better to let Microsoft DNS perform it's own resolution?

[Carsten Strotmann]

Hello Dave,

Post by Dave Warren

Post by u***@strotmann.de
Is the query about comparing Unbound on Windows with MS DNS, or
comparing Unbound on Unix/Linux with Win DNS?

I'd be interested in the results comparing the two on a
similar/identical platform. In other words, should I uninstall Microsoft
DNS and install unbound on the same system (where unbound is otherwise a
reasonable fit)

That is what I will test next week, I will post the results here (and
the test setup).

Post by Dave Warren
Obviously it's not a suitable replacement for Active Directory driven
DNS.

why not? It is best practice to separate DNS resolver (caching DNS
server like Unbound) and authoritative Server. While WinDNS can be used
in both functions, it makes a good resilient and manageable DNS design
to separate the DNS server functions on dedicated machines.

Unbound will nicely work as an secure DNSSEC validating resolver,
resolving Internet names and also (possible) local Active Directory
names that are stored on WinDNS AD integrated servers.

Post by Dave Warren
However, even here, there's an interesting performance question: Is
it worth installing unbound and forwarding Microsoft DNS to unbound, or
is it better to let Microsoft DNS perform it's own resolution?

Forwarding is (today) probably almost always slower than direct name
resolution (and more complicated and brittle), unless you are connected
to the Internet with a slow link. I recommend to not use forwarding
unless there are very special conditions.

Unbound as a direct resolver might be faster than having WinDNS as a
direct resolver.

[Dave Warren]

Post by Carsten Strotmann

Post by Dave Warren
Obviously it's not a suitable replacement for Active Directory driven
DNS.

why not? It is best practice to separate DNS resolver (caching DNS
server like Unbound) and authoritative Server. While WinDNS can be used
in both functions, it makes a good resilient and manageable DNS design
to separate the DNS server functions on dedicated machines.

In general, I agree that it makes sense to split authoritative and
resolver roles. However, in the case of Windows and Active Directory,
Active Directory is built under the assumption that your DNS servers
accept AD authenticated dynamic updates, both from AD itself and from
clients, so it's best practice to only specify Microsoft DNS servers for
Active Directory domain controllers, member servers and workstations
when possible.

While you can do it via other methods (setting up AD's entries manually
or forwarding the appropriate zones), it takes a lot of head-banging to
get everything working and if you mess it up, the effects are subtle and
intermittent since parts of Windows will fall back on broadcasts and
other unreliable methods, and therefore will sometimes work even with
DNS misconfigured.

Also keep in mind that Microsoft's authoritative DNS is multi-master and
site-aware (so a machine registered in the current site will be
immediately available in DNS to the current site, but might take time to
propagate to other physical sites in the same DNS zone, balancing the
need for quick updates vs keeping the number of updates between sites
reasonable)

My theory is that each site (physical location as well as Active
Directory site/subnet) would have one unbound server that performs
internet resolution, with multiple AD servers that forward to the
unbound server.

Post by Carsten Strotmann
Unbound will nicely work as an secure DNSSEC validating resolver,
resolving Internet names and also (possible) local Active Directory
names that are stored on WinDNS AD integrated servers.

Microsoft DNS's DNSSEC support is limited at best, and it has no
pre-fetch support at all, so I'd like to use unbound for primary DNS
resolution. However, hosting Active Directory on anything but
Microsoft's DNS is outside of best practices for Active Directory.

Post by Carsten Strotmann

Post by Dave Warren
However, even here, there's an interesting performance question: Is
it worth installing unbound and forwarding Microsoft DNS to unbound, or is it better to let Microsoft DNS perform it's own resolution?

Forwarding is (today) probably almost always slower than direct name
resolution (and more complicated and brittle), unless you are connected
to the Internet with a slow link. I recommend to not use forwarding
unless there are very special conditions.
Unbound as a direct resolver might be faster than having WinDNS as a
direct resolver.

It might. If so, I'd like to know how much faster or slower the servers
are on their own, but also how much overhead is involved if Microsoft's
DNS sits in the middle to see if complying with best practices is
appropriate, or if there's a technical justification to go with a more
complicated setup.

I have the impression that Microsoft DNS isn't particularly speedy, but
I have not actually attempted to benchmark it since Windows 2003 vs an
appropriate era BIND. At the time, BIND was faster, but only slightly,
but since this design effectively allowed for a shared cache, the
real-world performance was significantly improved.

My guess is that having several Microsoft DNS servers forward to a
single unbound server which does resolution of all non-local zones will
ultimately be slightly faster than having multiple Microsoft DNS servers
do the work themselves, but even if it's ultimately slightly slower,
gaining the benefits of Unbound's DNSSEC validation probably make it
worthwhile. But if it's a lot slower, I would definitely be open to
other configurations.

[Chris Smith]

In general, I agree that it makes sense to split authoritative and resolver
roles. However, in the case of Windows and Active Directory, Active
Directory is built under the assumption that your DNS servers accept AD
authenticated dynamic updates, both from AD itself and from clients, so it's
best practice to only specify Microsoft DNS servers for Active Directory
domain controllers, member servers and workstations when possible.

First a caveat - all of my clients and experience (after my big iron
days that is) are small businesses. Most are non-AD but a couple of
them do have AD domains running on Microsoft servers (no giant
Forests, one domain). One is an inherited account and the other needed
to run software that required an AD. Not my preference (which is
Linux/BSD servers), but they work.

So file this under some guys opinion and probably not applicable to
your environment.

First, unless a box is a server there's no pressing need for it to
have DNS entry. Nice, can be helpful, but not absolutely needed in
most cases.

My theory is that each site (physical location as well as Active Directory
site/subnet) would have one unbound server that performs internet
resolution, with multiple AD servers that forward to the unbound server.

At one site I run the Unbound resolver/cache on an OpenBSD box
configured with stub-zones for the AD domain (forward and reverse).
All clients are configured to query this Unbound box. The AD server
forwards to Unbound anything it is not authoritative for.

The important thing this accomplishes (at least in my paranoid mind)
is that it removes the AD from direct Internet access for DNS purposes
(admittedly I have some trust issues with MS systems and Internet
access). With a side benefit that the AD does not supply DNS answers
directly to the clients - it's DNS workload is very low, as it is a
server with little to no need to resolve outside its authoritative
domain (just the occasional update, new software download, etc.).
Also, and maybe a bit unexpectedly the clients still update their DNS
entries (although as I mentioned earlier I don't find this all that
necessary) on the AD as even though their resolvers point to Unbound
the SRV records (cached by Unbound for all clients) are what allow
them to locate the AD and update.

At any rate the performance is quite good, and (if anecdotal
"evidence" can be offered) improved (I ran no performance tests). But
more importantly I have piece of mind that the AD has very limited
Internet exposure.

I have also run with all clients pointing to the AD with a forward to
the Unbound server (no stub-zone to the AD), which does limit the AD's
Internet footprint, but I like the setup using the stub-zone, reducing
the DNS workload on the AD and even possibly preventing some DNS
nonsense from internal systems.

Chris

[Joe Abley]

Post by Carsten Strotmann

Post by Dave Warren
Obviously it's not a suitable replacement for Active Directory driven
DNS.

why not? It is best practice to separate DNS resolver (caching DNS
server like Unbound) and authoritative Server. While WinDNS can be used
in both functions, it makes a good resilient and manageable DNS design
to separate the DNS server functions on dedicated machines.

In general, I agree that it makes sense to split authoritative and resolver roles. However, in the case of Windows and Active Directory, Active Directory is built under the assumption that your DNS servers accept AD authenticated dynamic updates, both from AD itself and from clients, so it's best practice to only specify Microsoft DNS servers for Active Directory domain controllers, member servers and workstations when possible.

I don't see the logical jump, here.

A DNS UPDATE client can identify the correct domain controller using the SOA MNAME. A recursive resolver can identify the correct domain controller for a zone by following a referral chain. Yes, some environments might have split DNS design decisions that turn out to make this tricky, but really that's more of a reflection of those design decisions than any downstream implementation decision.

There is surely no architectural requirement for the recursive resolver used by any particular stub resolver to run any particular software. "Only specify Microsoft DNS servers" (in the context of resolvers) might make good marketing copy if you're in the business of selling Microsoft DNS servers, but it doesn't sound like it's grounded in logic.


Joe

[Dave Warren]

Post by Joe Abley
I don't see the logical jump, here.
A DNS UPDATE client can identify the correct domain controller using the SOA MNAME. A recursive resolver can identify the correct domain controller for a zone by following a referral chain. Yes, some environments might have split DNS design decisions that turn out to make this tricky, but really that's more of a reflection of those design decisions than any downstream implementation decision.

It's not just that the DNS is split (which Microsoft doesn't even
support split zones within Active Directory enabled zones in a
traditional "split" format), but rather, that updates are done in a
multi-master scenario while sites may have replication intervals in the
period of minutes, hours, or days, and updates are best processed by the
local AD DNS servers (they are not forwarded upstream using the SOA
record or anything else)

In a small environment none of this matters, but in large, multi-site,
physically decentralized environments, you might really want local
clients doing updates to a local AD server so that their DNS records
appear immediately locally and that doesn't happen if you use the
traditional "Update the SOA MNAME and wait for the changes to wander
down to other servers"

Microsoft's DNS server is using a true multi-master, there's nothing
particularly special about the server listed as MNAME, literally any AD
DNS server can process updates locally and will ensure that changes are
replicated out to appropriate partners.

I'm NOT saying it's the only option -- Just that it's Microsoft's best
practice to use Microsoft DNS servers to service Microsoft Active
Directory joined servers and clients, and in my experience, staying
within Microsoft's best practices is usually wise unless you can
articulate a reason to make a difference choice. If you have a practical
reason to do something different, do it! I do a lot of non-standard
stuff in Windows all the time.

(This isn't even a sales point, Microsoft DNS server is a free component
that requires no additional licensing beyond the Active Directory
environment itself.)

But at least in this case, I'm more interested in getting the benefits
of unbound (awesome resolver performance, DNSSEC validation,
pre-fetching, etc) without adding headache (using non-AD DNS for an
Active Directory environment), so using Windows DNS internally and
unbound for external resolution seems like an ideal configuration unless
there are downsides (such as performance)

[Olaf Kolkman]

Post by u***@strotmann.de
Hello Olaf,

Post by Olaf Kolkman
Folk,
We recently got a query from somebody in the industry asking how
Unbound compares to Windows Resolver performance in terms of qps and
CPU (memory is under user control).
All our benchmarking is Unix based and we never really did a
comparison ourselves. Is there somebody on this list that has
performed such measurements, or knows of a reference?

I have a working setup where I could test Unbound vs. MS DNS (the setup
where I do regression tests on the Unbound packages I compile) and would
be very interested in the results as well. The machines are not too
powerful (dual-core with 8GB RAM) compared to todays standards.

Perfect!!!

Post by u***@strotmann.de
I'm pretty busy the next two weeks, but could do the tests later in June
(if that helps).

Hey, I am the one asking a favor ;-)

Post by u***@strotmann.de
Is the query about comparing Unbound on Windows with MS DNS, or
comparing Unbound on Unix/Linux with Win DNS?

The former.

—Olaf

[Carsten Strotmann]

Hi Olaf,
Hi Unbound-Users,

I've did some benchmarking yesterday. I will continue later this month
(with different hardware, see below), but here are the first
results. I'm posting it here because you might have some ideas on how to
improve the benchmark-setup.

DISCLAIMER 1: the hardware used is not "Enterprise" or "Telco" level, I'm
aware of that. It is what I had available.

DISCLAIMER 2: this is a benchmark in a synthetic lab environment. It is
not how DNS resolvers work in the real Internet. This results can give
indications, but do not replace your own measurements in your own
environment using your own hardware.

Lab setup:

* one "load generating" machine running Unbound's own "perf" tool
(Thinkpad E330 4 Core, 16GB RAM, running Fedora Linux 20) , sending
1.000.000 queries to the DNS resolver (out of a zone of 10.000.000
IPv4-A-records). I've tested using more than one "load generating"
machine, but with the limited hardware of the resolver, more
load-machines did not change the measured results

* one "authoritative" DNS server, hosting a root zone containing
10.000.000 A-Records. BIND 9.9.5, Solaris 11, 8 GB. No delegation,
everything is inside one large root zone. TTL is 86400.

* one DNS resolver for testing Unbound vs. WinDNS. AMD Athlon 64 X2 Dual
Core 2.31 Ghz 1 Socket 2 Cores 256 L1 Cache 1 MB L2 Cache, 2GB RAM 533
Mhz DIMM, Ethernet Intel PRO/100S Desktop NIC

All machines were connected via a HP ProCurve 1410 16G Switch

I did 3 tests: Unbound on FreeBSD 10-RELEASE, Unbound on Windows 2012
ServerCore and Windows DNS Server on Windows 2012 ServerCore

Memory was measured by "top" on FreeBSD and by Windows 2012 taskmgr.

Each setup was tested in 2 runs. Run 1 with cold (empty cache) and Run 2
with filled cache (no DNS resolution towards the authoritative).

1 FreeBSD 10 unbound 1.4.20
===========================

1.000.000 Records

Run Memory average qps CPU
Run 1 (cold cache) 9000 90 %
Run 2 (warm cache) 736M 43000 79 %

2 Windows 2012 Core unbound 1.4.22
==================================

1.000.000 Records

Run Memory average qps CPU
Run 1 (cold cache) 4113 53%
Run 2 (warm cache) 398M 30900 50% (63% System)

3 Windows 2012 Core dns.exe 23D99296 (6.2 build 9200)
=====================================================

1.000.000 Records

Run Memory average qps CPU
Run 1 (cold cache) 3160 15-60%
Run 2 (warm cache) 747M 37443 78% (98% System)

Observations:

* According to Sysinternals "process explorer"
(http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx),
Unbound on Windows is a single threaded application. It does not make
use of multiple cores.

* According to the same "process explorer", dns.exe is
multi-threaded.

* dns.exe qps throughput and CPU usage is fluctuating while resolving to the
authoritative servers. QPS was between 1000-6000 with an average on
3160 during this time. Unbound is more steady while doing resolving.

* dns.exe uses more memory during resolution (up to 1.2 GB seen), but
frees memory after getting into stable state (all queries cached)

* dns.exe is faster when only answering from cache (probably due to
multi-threading), Unbound is faster resolving from the authoritative
servers.

* memory allocation on Windows seems to be more resourceful compared to
FreeBSD.

I will re-run these tests on a 4 Core Dell R210 Server later this month
and report the numbers here.

Please let me know if you see flaws in my measurement setup (there are
probably some) and how to fix them.

Best regards

Carsten

Post by Olaf Kolkman
Folk,
We recently got a query from somebody in the industry asking how Unbound compares to Windows Resolver performance in terms of qps and CPU (memory is under user control).
All our benchmarking is Unix based and we never really did a comparison ourselves. Is there somebody on this list that has performed such measurements, or knows of a reference?

[Roland van Rijswijk - Deij]

Hi Olaf, Carsten, (unbound users),

I only just saw this thread and would like to add to Carsten's results;
we have a student working on measuring energy efficiency of DNS
resolvers, who has done some interesting measurements. While there is
probably not a one-to-one correspondence between performance and energy
efficiency we did see some rather strange effects when running Unbound
on Windows Server 2012 that warrant further investigation.

Let me briefly explain his measurement setup: he used an instrumented
server model that has special probes inserted that measure the energy
consumption of most system components (power supply, HDUs, CPU, memory,
...). On this machine he performed repeated experiments where he plays
back a captured set of queries from a live DNS environment to three
different DNS resolver implementations:

- Microsoft built-in DNS
- BIND
- Unbound

He does this with MS Windows Server 2012 as underlying OS and has
repeated the experiment with Centos 6.5 as underlying OS (obviously, he
could not measure MS DNS on that ;-) ). He has measured both without
DNSSEC validation enabled and with DNSSEC validation enabled.

Briefly summarised he found the following:

- MS Windows 2012 baseline power consumption is significantly (~10%)
lower than Centos 6.5 on the same hardware

- MS DNS uses the least energy to process the query data set, followed
by BIND then Unbound on Windows 2012

- On Linux, with DNSSEC disabled BIND uses the least energy followed by
Unbound, with DNSSEC enabled both use about the same amount of energy

All tests were run on a single CPU core.

There is ony important "BUT" here: the measurements for Unbound on
Windows 2012 were extremely unreliable. Where he saw a steady power
consumption coupled with a steady CPU load for MS DNS and BIND, for
Unbound the picture is really garbled. Lots of small peaks in CPU load
(and conversely power consumption) made the measurements extremely
noisy. Unfortunately, he did not have time to get to the bottom of this,
but it seems to suggest that there is some underlying problem here. We
are hypothesising issues were Unbound clashes with the Windows scheduler
in some way, but haven't gotten any further than hypotheses. I'm unsure
whether Unbound is compiled to use libevent on Windows, if that is the
case then that may be worthwhile investigating as well.

Note that on CentOS we do not see this behaviour.

Bottom line: Carsten's results seem to point to Unbound performing
reasonably well on MS Windows 2012, but our student's observations seem
to suggest that something weird is happening when running Unbound on
Windows Server that may be worth investigating.

I plan to find some time to visit NLnet Labs with these results at some
point in the near future, if anybody from labs is planning on attending
the DNSSEC Workshop at ICANN next week give me a shout off list and we
can have a short discussion there.

Hope this helps!

Cheers,

Roland