As I have delved into some of the more modern, cloud focused infrastructure, I have discovered some new applications which are now operating in my personal lab. Most notably, they are CoreDNS, Prometheus, and Grafana. CoreDNS is a DNS server, similar to BIND, but is much easier to setup. Prometheus is a logging and statistics tool which is focused on its query language. Calling it a logging and statistics tool isn’t 100% accurate, but it’s good enough for an introduction. Grafana is a graphing application which takes data from many sources, Prometheus in this case, and creates attractive web-based graphs.
I’ll dedicate another post to general CoreDNS configuration but integration between CoreDNS and Prometheus is easy since CoreDNS provides a Prometheus plugin in its base configuration. Prometheus listens for incoming requests on port 9153 so the following line is added to each zone which should report statistics to Prometheus.
prometheus 0.0.0.0:9153
This line means CoreDNS will listen to all hosts on port 9153 for statistics requests.
Prometheus configuration is a little more complicated, but only a little. Prometheus’ configuration file uses the concept of targets to identify hosts or objects which should be queried. Prometheus just requires a target pointing to CoreDNS.
- job_name: "dns"
static_configs:
- targets: ["192.168.1.20:9153"]
This stanza creates a new job, or query, which should go to the target of 192.168.1.20, port 9153. Restart Prometheus and using the expression tool, type coredns. A list of options will appear providing statistics about CoreDNS. Use any of these to see the values.
If this integration seems easy, that’s because it is. CoreDNS needs to report statistics and Prometheus simply needs to ask for statistics. Grafana is the hardest part and even Grafana isn’t too hard to setup for CoreDNS.
Grafana uses the concept of a ‘data source’. A data source is what Grafana looks at to gather data before building a chart. First, you must configure the built-in Prometheus data source. For basic configurations, the only required is URL. URL should be the HTTP address you access the Prometheus web interface on. By default Prometheus listens on port 9090. http://192.168.1.20:9090 is my setting.
Next, create a new dashboard chart. The important value in the dashboard is the Prometheus query. After some experimentation, I settled on
abs(delta(coredns_dns_request_count_total{zone="."}[15m]))
abs() makes the number its absolute value. Per the Prometheus documentation, delta()
“…calculates the difference between the first and last value of each time series element in a range vector v, returning an instant vector with the given deltas and equivalent labels. The delta is extrapolated to cover the full time range as specified in the range vector selector, so that it is possible to get a non-integer result even if the sample values are all integers.” - prometheus.io
coredns_dns_request_count_total is one of the statistics CoreDNS provides via its Prometheus plugin. To recap, the function takes the absolute value of the difference between the number of DNS requests over two time periods. {zone=“.”} tells Prometheus to only show statistics for DNS requests which should be recursively looked up and aren’t part of another zone I control. Leaving this part out would include all zones. Finally, [15m] sets the start and end times delta() uses. In other words, it takes a 15 minute window and finds the difference between the requests. If the statistic at the start of the 15 minutes was 1000 and the statistic at the end was 1500, the difference would be 500.
Each line in Grafana has a time step field which I set to 15 minutes to match the expression. Format as is set to Time series as well. With all this put together, I get a graph which looks like:
Grafana offers a few improtable CoreDNS graphs. I chose to make my own but also have some imported graphs in another dashboard which I can look at if I want.