dnstap log file not populating for every entry even if repeated.

[davehouser1]

Not sure if this belongs here or somewhere else, so let me know if my question / problem needs to be redirected.

When running dnstap I noticed the binary log only populates when new entries are requested from the named server. Is this expected?

Trouble shooting:

• On bind server make sure dnstap is running, then start the named service.
• Can see the dnstap log gets created, and populated properly on named start up.
• Perform a dig request from a remote client for an entry in the zone db (test.domain.internal). I am performing a watch ls -l on the dnstap file, and see the file increase in size.
• Also monitoring the bind.log file, I see the request come through.
• Perform the same exact dig request (test.domain.internal) from the same remote client to the same named server running dnstap. The bind.log populates with the request, the dnstap log does not increase in size.
• Perform a new request with a different domain (www.internet.com). I see the bind.log populate, then the dnstap log increases in size.
• I use dnstap-read (memory flag) on the file, and can see the CQ entries for the original request (test.domain.internal), the duplicate request (test.domain.internal again), and then the external domain request (www.internet.com).

I was thinking the log would populate for every request no matter how frequent or similar they are. Was there a reason this was disabled? Too much overhead?

Is there a way to enable this feature?

[cmikk]

The dnstap program does not do any deduplication or similar of the messages it receives, but it does use a buffered output when writing to a file. In test setups with low query volumes, this will lead to some queries not being written to the dnstap output file until a subsequent query triggers a buffer flush.

If the duplicate query eventually makes it to the dnstap file, this is probably what is going on. If not, this may take further investigation.

[davehouser1]

@cmikk thanks for the quick response.

We plan to use DNSTAP on bind servers that will received very high volume of forwarding traffic. To confirm, it sounds like dnstap log output is not guaranteed, is that true? Can you share more details on your query volume tests? I would like to mimic with our Ixia test system.

[cmikk]

The current version of the dnstap program in this repository (version 0.3.0) makes every effort to write all data it receives to a destination file or socket. Some earlier versions (prior to 0.2.2) attempted to make this write non-blocking but this interacted poorly with the buffering implementation of a specific dnstap producer, so we backed this out in 0.2.2; all writes to any destination are blocking.

Dnstap producers, on the other hand, are encouraged to drop dnstap data rather than block query processing attempting to export it. In this sense, delivery of dnstap data is not guaranteed, but loss can be reduced through faster (or more minimal) processing.

The most recent volume tests we've done have used resperf to repeat a query at a selected rate and concurrency against the server under test exporting CLIENT_QUERY reports on a unix domain socket and writing to a regular file. Note that we have not run this setup recently with BIND, but did so with CoreDNS in the process of reproducing issue #34.

[timp87]

does use a buffered output when writing to a file

It would be nice to have some control over it.
Like disable buffering, flush buffer every n seconds, handle a signal to flush buffer or something like this.
Right now even buffer flush on exit is missing (tried 0.4.0) which is really disappointing