[Unbound-users] High number of system context switches

[Jan-Frode Myklebust]

Hi,

I'm considering switching from bind to unbound, and have been testing it
on one of our recursive dns servers. Our servers are KVM virtual machines,
running RHEL6.5, 12GB memory, 8 cpu cores, with unbound-1.4.21-1.el6.x86_64
from EPEL.

We typically have high time periods of 12-14k qps on our DNS-servers, and
it's been working fairly well on bind, but with unbound we seem to get into
trouble when the qps exceed 7k. We then see a clear drop in the request
rate, and clients move over to the secondary dns server.

The only problem we've noticed on the unbound server is that context
switches/s is very high. pidstat for unbound doesn't report high
cswch/s, but the system does..

Here's "sar -w" from yesterday evening running unbound:

kl. 19.00 +0200 proc/s cswch/s
kl. 19.10 +0200 3,70 116480,30
kl. 19.20 +0200 3,47 123118,48
kl. 19.30 +0200 3,67 128948,60
kl. 19.40 +0200 3,45 125471,32
kl. 19.50 +0200 3,69 132641,76
kl. 20.00 +0200 3,48 140126,75

while the day before on bind:

kl. 19.00 +0200 1,90 64801,51
kl. 19.10 +0200 2,15 64550,78
kl. 19.20 +0200 1,94 64389,23
kl. 19.30 +0200 2,19 64369,56
kl. 19.40 +0200 1,92 64211,15
kl. 19.50 +0200 2,09 64087,84
kl. 20.00 +0200 1,91 63691,33


Any ideas for what we should try to improve this?

Full unbound.conf stripped for comments:
-------------------------------------------------------------------
server:
verbosity: 1
statistics-interval: 60
statistics-cumulative: yes
extended-statistics: yes
num-threads: 8
interface: 0.0.0.0
interface: ::0
interface-automatic: yes
outgoing-range: 4096
outgoing-port-permit: 32768-65535
outgoing-port-avoid: 0-32767
max-udp-size: 3072
msg-cache-size: 4G
num-queries-per-thread: 4096
rrset-cache-size: 8G
cache-min-ttl: 2
do-ip4: yes
do-ip6: yes
do-udp: yes
do-tcp: yes
access-control: 0.0.0.0/0 allow
access-control: ::0/0 allow
chroot: ""
username: "unbound"
directory: "/etc/unbound"
log-time-ascii: yes
pidfile: "/var/run/unbound/unbound.pid"
harden-glue: yes
harden-dnssec-stripped: yes
harden-below-nxdomain: yes
harden-referral-path: yes
use-caps-for-id: no
unwanted-reply-threshold: 10000000
prefetch: yes
prefetch-key: yes
rrset-roundrobin: yes
minimal-responses: no
dlv-anchor-file: "/etc/unbound/dlv.isc.org.key"
trusted-keys-file: /etc/unbound/keys.d/*.key
auto-trust-anchor-file: "/var/lib/unbound/root.anchor"
val-clean-additional: yes
val-permissive-mode: no
val-log-level: 2
include: /etc/unbound/local.d/*.conf
remote-control:
control-enable: yes
control-interface: 127.0.0.1
control-interface: ::1
server-key-file: "/etc/unbound/unbound_server.key"
server-cert-file: "/etc/unbound/unbound_server.pem"
control-key-file: "/etc/unbound/unbound_control.key"
control-cert-file: "/etc/unbound/unbound_control.pem"
include: /etc/unbound/conf.d/*.conf
-------------------------------------------------------------------




-jf

[Robert Edmonds]

Post by Jan-Frode Myklebust
Any ideas for what we should try to improve this?

Hi!

You might want to try the excellent "perf" tool to record function-level
performance data for the unbound process. On my CentOS 6.5 box it's
located in the "perf" package. See:

https://perf.wiki.kernel.org/

Then you can periodically do something like:

perf record --output="perf.data.$(date +%Y%m%d.%H%M)" --uid=unbound sleep 5

To collect a five second sample of performance data for all processes
running under the "unbound" user.

Then you can use tools like "perf report" and "perf diff" to analyze the
saved samples. Maybe a "perf diff" between samples during low traffic
periods versus high traffic periods will show hot spots.

Also, I notice unbound-1.4.21-1.el6.x86_64 appears to use an old version
of libevent:

linked libs: libevent 1.4.13-stable (it uses epoll), ldns 1.6.16, OpenSSL 1.0.0-fips 29 Mar 2010

IIRC, there are substantial performance improvements in libevent 2.x.

[Paul Wouters]

Post by Robert Edmonds
Also, I notice unbound-1.4.21-1.el6.x86_64 appears to use an old version
linked libs: libevent 1.4.13-stable (it uses epoll), ldns 1.6.16, OpenSSL 1.0.0-fips 29 Mar 2010
IIRC, there are substantial performance improvements in libevent 2.x.

RHEL6 has libevent 1.4.x. RHEL7 will have libevent 2.x

Perhaps run a test with unbound from the RHEL7 beta image and see if
this make a difference?

Paul

[Sotiris Tsimbonis]

Post by Jan-Frode Myklebust
We typically have high time periods of 12-14k qps on our DNS-servers, and
it's been working fairly well on bind, but with unbound we seem to get into
trouble when the qps exceed 7k. We then see a clear drop in the request
rate, and clients move over to the secondary dns server.

Try commenting out the dlv-anchor-file directive.

We had a similar situation about a month ago, where at random times our
unbound servers would start dropping queries.

Running "unbound-control stats_noreset | grep total.requestlist" at the
time showed lots of exceeded requests. We tried increasing various
options (num-threads, num-queries-per-thread etc), but the problem kept
coming back randomly.

At some point, we started recording the output of "unbound-control
dump_requestlist" every minute, and this revealed that when queries were
being dropped, there was a huge delay getting answers from authoritative
NSs of dlv.isc.org.

Maybe RRL on their side? Maybe network trouble in the path between us
that we were not aware of? We never found out, but we turned off dlv and
didn't have a problem since then.

Sot.

[Jan-Frode Myklebust]

Post by Sotiris Tsimbonis
Try commenting out the dlv-anchor-file directive.

Excuse my DNSSEC ignorance, but what's the consequence of commenting out
this directive? Will it still be OK to run a dnssec validating
nameserver, or will too much fail too validate. Or maybe lack of tld
trust anchor means DLV will just be ignored and served as non-validating
dnssec?

Post by Sotiris Tsimbonis
We had a similar situation about a month ago, where at random times our
unbound servers would start dropping queries.

Much appreciated. Thanks! I will try this after easter..



-jf

[Sotiris Tsimbonis]

Post by Jan-Frode Myklebust

Post by Sotiris Tsimbonis
Try commenting out the dlv-anchor-file directive.

Excuse my DNSSEC ignorance, but what's the consequence of commenting out
this directive? Will it still be OK to run a dnssec validating
nameserver, or will too much fail too validate. Or maybe lack of tld
trust anchor means DLV will just be ignored and served as non-validating
dnssec?

You will not validate domains in TLDs that have not been signed yet.

The .au TLD has not been signed yet, but will be signed later this
month. Signed .au domains today can only be validated because of DLV[1].
It fills the trust gap between the signed root and a dnssec signed
domain within a non-signed TLD.

Your dns will still be validating all domains with a "proper" chain of
trust, so later this month we will be able to validate .au domains
without the use of DLV.

http://stats.research.icann.org/dns/tld_report/ shows these numbers today:

532 TLDs in the root zone in total
343 TLDs are signed;
336 TLDs have trust anchors published as DS records in the root zone;
3 TLDs have trust anchors published in the ISC DLV Repository.

Sot.

[1]
http://en.wikipedia.org/wiki/Domain_Name_System_Security_Extensions#DNSSEC_Lookaside_Validation

[Jan-Frode Myklebust]

Post by Sotiris Tsimbonis

Post by Jan-Frode Myklebust
Excuse my DNSSEC ignorance, but what's the consequence of commenting out
this directive? Will it still be OK to run a dnssec validating
nameserver, or will too much fail too validate. Or maybe lack of tld
trust anchor means DLV will just be ignored and served as non-validating
dnssec?

You will not validate domains in TLDs that have not been signed yet.

What was unclear was if the DLV signed domains would SERVFAIL, or if
they would just respond with unauthenticated answer. Seems to be
unauthenticated answer, so I don't see any downside to removing the
DLV anchor.



-jf

[Jan-Frode Myklebust]

Just a follow up to myself, and a big thank you to unbound!

It turned out our virtual DNS-servers were struggeling with a
cpu in 100% softirq, processing incoming packets. This could only run on
one cpu, so we had no way of scaling this out. Newer virtio-net has
up to 8 queues, to scale this over more cpus, but that's not available
on RHEV-H, so we've moved our DNS-servers over to physical hosts.

Still we've been struggeling with the performance of Bind on these
physical servers. Once the primary DNS-server hit ~17-20K qps, we've
notice clients started moving to the secondary server -- but we could
never see anything wrong on the server side.

Yesterday we put unbound (from EPEL6) on the primary dns-server, and
immediately saw a huge effect ! This single server is now answering >30K
qps during prime time, and the secondary server doesn't really have any
peaks anymore -- clients are no longer failing over to secondary.

We're still seeing ~300K cswch/s on this server, which was a bit
worrying initially, but doesn't seem to be any problem yet.. Will
probably be OK for now, until we upgrade to RHEL7 with libevent-2 which
was mentioned might help..



-jf

Post by Jan-Frode Myklebust
Hi,
I'm considering switching from bind to unbound, and have been testing it
on one of our recursive dns servers. Our servers are KVM virtual machines,
running RHEL6.5, 12GB memory, 8 cpu cores, with unbound-1.4.21-1.el6.x86_64
from EPEL.
We typically have high time periods of 12-14k qps on our DNS-servers, and
it's been working fairly well on bind, but with unbound we seem to get into
trouble when the qps exceed 7k. We then see a clear drop in the request
rate, and clients move over to the secondary dns server.
The only problem we've noticed on the unbound server is that context
switches/s is very high. pidstat for unbound doesn't report high
cswch/s, but the system does..
kl. 19.00 +0200 proc/s cswch/s
kl. 19.10 +0200 3,70 116480,30
kl. 19.20 +0200 3,47 123118,48
kl. 19.30 +0200 3,67 128948,60
kl. 19.40 +0200 3,45 125471,32
kl. 19.50 +0200 3,69 132641,76
kl. 20.00 +0200 3,48 140126,75
kl. 19.00 +0200 1,90 64801,51
kl. 19.10 +0200 2,15 64550,78
kl. 19.20 +0200 1,94 64389,23
kl. 19.30 +0200 2,19 64369,56
kl. 19.40 +0200 1,92 64211,15
kl. 19.50 +0200 2,09 64087,84
kl. 20.00 +0200 1,91 63691,33
Any ideas for what we should try to improve this?
-------------------------------------------------------------------
verbosity: 1
statistics-interval: 60
statistics-cumulative: yes
extended-statistics: yes
num-threads: 8
interface: 0.0.0.0
interface: ::0
interface-automatic: yes
outgoing-range: 4096
outgoing-port-permit: 32768-65535
outgoing-port-avoid: 0-32767
max-udp-size: 3072
msg-cache-size: 4G
num-queries-per-thread: 4096
rrset-cache-size: 8G
cache-min-ttl: 2
do-ip4: yes
do-ip6: yes
do-udp: yes
do-tcp: yes
access-control: 0.0.0.0/0 allow
access-control: ::0/0 allow
chroot: ""
username: "unbound"
directory: "/etc/unbound"
log-time-ascii: yes
pidfile: "/var/run/unbound/unbound.pid"
harden-glue: yes
harden-dnssec-stripped: yes
harden-below-nxdomain: yes
harden-referral-path: yes
use-caps-for-id: no
unwanted-reply-threshold: 10000000
prefetch: yes
prefetch-key: yes
rrset-roundrobin: yes
minimal-responses: no
dlv-anchor-file: "/etc/unbound/dlv.isc.org.key"
trusted-keys-file: /etc/unbound/keys.d/*.key
auto-trust-anchor-file: "/var/lib/unbound/root.anchor"
val-clean-additional: yes
val-permissive-mode: no
val-log-level: 2
include: /etc/unbound/local.d/*.conf
control-enable: yes
control-interface: 127.0.0.1
control-interface: ::1
server-key-file: "/etc/unbound/unbound_server.key"
server-cert-file: "/etc/unbound/unbound_server.pem"
control-key-file: "/etc/unbound/unbound_control.key"
control-cert-file: "/etc/unbound/unbound_control.pem"
include: /etc/unbound/conf.d/*.conf
-------------------------------------------------------------------
-jf

[Jan-Frode Myklebust]

I just got a question off-list if I could share my configuration,
by someone that was only able to get 15Kqps measured by query-perf,
on an 8 core, 8GB physical server, with libevent2. I prefer answering
on-list, in hope that others might chime :-)

I don't think there's anything special about our configuration:

HW:
An IBM HS22 blade, with two quad-core Intel E5620 (2.40GHz), with HT
enabled, and 18GB memory.

Software is RHEL6-latest, with unbound from EPEL
(unbound-1.4.22-1.el6.x86_64, libevent-1.4.13-4.el6.x86_64). Only system
tuning is increasing net.nf_conntrack_max to 524288, and
net.netfilter.nf_conntrack_udp_timeout to 10.

Full unbound.conf attached.

My 30Kqps number is from real usage, not benchmark. So we might maybe
be hitting cached entries more often than a synthetic benchmark.



-jf