1 Internet Engineering Task Force (IETF) J. Dickinson
2 Request for Comments: 8618 J. Hague
3 Category: Standards Track S. Dickinson
4 ISSN: 2070-1721 Sinodun IT
5 T. Manderson
6 ICANN
7 J. Bond
8 Wikimedia Foundation, Inc.
9 September 2019
10
11
12 Compacted-DNS (C-DNS): A Format for DNS Packet Capture
13
14 Abstract
15
16 This document describes a data representation for collections of DNS
17 messages. The format is designed for efficient storage and
18 transmission of large packet captures of DNS traffic; it attempts to
19 minimize the size of such packet capture files but retain the full
20 DNS message contents along with the most useful transport metadata.
21 It is intended to assist with the development of DNS traffic-
22 monitoring applications.
23
24 Status of This Memo
25
26 This is an Internet Standards Track document.
27
28 This document is a product of the Internet Engineering Task Force
29 (IETF). It represents the consensus of the IETF community. It has
30 received public review and has been approved for publication by the
31 Internet Engineering Steering Group (IESG). Further information on
32 Internet Standards is available in Section 2 of RFC 7841.
33
34 Information about the current status of this document, any errata,
35 and how to provide feedback on it may be obtained at
36 https://www.rfc-editor.org/info/rfc8618.
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52 Dickinson, et al. Standards Track [Page 1]
53 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
54
55
56 Copyright Notice
57
58 Copyright (c) 2019 IETF Trust and the persons identified as the
59 document authors. All rights reserved.
60
61 This document is subject to BCP 78 and the IETF Trust's Legal
62 Provisions Relating to IETF Documents
63 (https://trustee.ietf.org/license-info) in effect on the date of
64 publication of this document. Please review these documents
65 carefully, as they describe your rights and restrictions with respect
66 to this document. Code Components extracted from this document must
67 include Simplified BSD License text as described in Section 4.e of
68 the Trust Legal Provisions and are provided without warranty as
69 described in the Simplified BSD License.
70
71 Table of Contents
72
73 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4
74 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5
75 3. Data Collection Use Cases . . . . . . . . . . . . . . . . . . 5
76 4. Design Considerations . . . . . . . . . . . . . . . . . . . . 8
77 5. Choice of CBOR . . . . . . . . . . . . . . . . . . . . . . . 10
78 6. C-DNS Format Conceptual Overview . . . . . . . . . . . . . . 10
79 6.1. Block Parameters . . . . . . . . . . . . . . . . . . . . 14
80 6.2. Storage Parameters . . . . . . . . . . . . . . . . . . . 14
81 6.2.1. Optional Data Items . . . . . . . . . . . . . . . . . 15
82 6.2.2. Optional RRs and OPCODEs . . . . . . . . . . . . . . 16
83 6.2.3. Storage Flags . . . . . . . . . . . . . . . . . . . . 17
84 6.2.4. IP Address Storage . . . . . . . . . . . . . . . . . 17
85 7. C-DNS Format Detailed Description . . . . . . . . . . . . . . 18
86 7.1. Map Quantities and Indexes . . . . . . . . . . . . . . . 18
87 7.2. Tabular Representation . . . . . . . . . . . . . . . . . 18
88 7.3. "File" . . . . . . . . . . . . . . . . . . . . . . . . . 19
89 7.3.1. "FilePreamble" . . . . . . . . . . . . . . . . . . . 20
90 7.3.1.1. "BlockParameters" . . . . . . . . . . . . . . . . 20
91 7.3.1.1.1. "StorageParameters" . . . . . . . . . . . . . 21
92 7.3.1.1.1.1. "StorageHints" . . . . . . . . . . . . . 22
93 7.3.1.1.2. "CollectionParameters" . . . . . . . . . . . 24
94 7.3.2. "Block" . . . . . . . . . . . . . . . . . . . . . . . 25
95 7.3.2.1. "BlockPreamble" . . . . . . . . . . . . . . . . . 26
96 7.3.2.2. "BlockStatistics" . . . . . . . . . . . . . . . . 27
97 7.3.2.3. "BlockTables" . . . . . . . . . . . . . . . . . . 28
98 7.3.2.3.1. "ClassType" . . . . . . . . . . . . . . . . . 29
99 7.3.2.3.2. "QueryResponseSignature" . . . . . . . . . . 30
100 7.3.2.3.3. "Question" . . . . . . . . . . . . . . . . . 33
101 7.3.2.3.4. "RR" . . . . . . . . . . . . . . . . . . . . 34
102 7.3.2.3.5. "MalformedMessageData" . . . . . . . . . . . 34
103
104
105
106
107 Dickinson, et al. Standards Track [Page 2]
108 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
109
110
111 7.3.2.4. "QueryResponse" . . . . . . . . . . . . . . . . . 35
112 7.3.2.4.1. "ResponseProcessingData" . . . . . . . . . . 36
113 7.3.2.4.2. "QueryResponseExtended" . . . . . . . . . . . 37
114 7.3.2.5. "AddressEventCount" . . . . . . . . . . . . . . . 38
115 7.3.2.6. "MalformedMessage" . . . . . . . . . . . . . . . 39
116 8. Versioning . . . . . . . . . . . . . . . . . . . . . . . . . 39
117 9. C-DNS to PCAP . . . . . . . . . . . . . . . . . . . . . . . . 40
118 9.1. Name Compression . . . . . . . . . . . . . . . . . . . . 42
119 10. Data Collection . . . . . . . . . . . . . . . . . . . . . . . 42
120 10.1. Matching Algorithm . . . . . . . . . . . . . . . . . . . 43
121 10.2. Message Identifiers . . . . . . . . . . . . . . . . . . 45
122 10.2.1. Primary ID (Required) . . . . . . . . . . . . . . . 45
123 10.2.2. Secondary ID (Optional) . . . . . . . . . . . . . . 46
124 10.3. Algorithm Parameters . . . . . . . . . . . . . . . . . . 46
125 10.4. Algorithm Requirements . . . . . . . . . . . . . . . . . 46
126 10.5. Algorithm Limitations . . . . . . . . . . . . . . . . . 47
127 10.6. Workspace . . . . . . . . . . . . . . . . . . . . . . . 47
128 10.7. Output . . . . . . . . . . . . . . . . . . . . . . . . . 47
129 10.8. Post-Processing . . . . . . . . . . . . . . . . . . . . 47
130 11. Implementation Guidance . . . . . . . . . . . . . . . . . . . 47
131 11.1. Optional Data . . . . . . . . . . . . . . . . . . . . . 48
132 11.2. Trailing Bytes . . . . . . . . . . . . . . . . . . . . . 48
133 11.3. Limiting Collection of RDATA . . . . . . . . . . . . . . 49
134 11.4. Timestamps . . . . . . . . . . . . . . . . . . . . . . . 49
135 12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 49
136 12.1. Transport Types . . . . . . . . . . . . . . . . . . . . 49
137 12.2. Data Storage Flags . . . . . . . . . . . . . . . . . . . 50
138 12.3. Response-Processing Flags . . . . . . . . . . . . . . . 51
139 12.4. AddressEvent Types . . . . . . . . . . . . . . . . . . . 51
140 13. Security Considerations . . . . . . . . . . . . . . . . . . . 52
141 14. Privacy Considerations . . . . . . . . . . . . . . . . . . . 52
142 15. References . . . . . . . . . . . . . . . . . . . . . . . . . 53
143 15.1. Normative References . . . . . . . . . . . . . . . . . . 53
144 15.2. Informative References . . . . . . . . . . . . . . . . . 55
145 Appendix A. CDDL . . . . . . . . . . . . . . . . . . . . . . . . 58
146 Appendix B. DNS Name Compression Example . . . . . . . . . . . . 69
147 B.1. NSD Compression Algorithm . . . . . . . . . . . . . . . . 70
148 B.2. Knot Authoritative Compression Algorithm . . . . . . . . 70
149 B.3. Observed Differences . . . . . . . . . . . . . . . . . . 71
150 Appendix C. Comparison of Binary Formats . . . . . . . . . . . . 71
151 C.1. Comparison with Full PCAP Files . . . . . . . . . . . . . 74
152 C.2. Simple versus Block Coding . . . . . . . . . . . . . . . 74
153 C.3. Binary versus Text Formats . . . . . . . . . . . . . . . 75
154 C.4. Performance . . . . . . . . . . . . . . . . . . . . . . . 75
155 C.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 75
156 C.6. Block Size Choice . . . . . . . . . . . . . . . . . . . . 76
157
158
159
160
161
162 Dickinson, et al. Standards Track [Page 3]
163 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
164
165
166 Appendix D. Data Fields for Traffic Regeneration . . . . . . . . 77
167 D.1. Recommended Fields for Traffic Regeneration . . . . . . . 77
168 D.2. Issues with Small Data Captures . . . . . . . . . . . . . 77
169 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 78
170 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 79
171
172 1. Introduction
173
174 There has long been a need for server operators to collect DNS
175 Queries and Responses on authoritative and recursive name servers for
176 monitoring and analysis. This data is used in a number of ways,
177 including traffic monitoring, analyzing network attacks, and "day in
178 the life" (DITL) [ditl] analysis.
179
180 A wide variety of tools already exist that facilitate the collection
181 of DNS traffic data, such as the DNS Statistics Collector (DSC)
182 [dsc], packetq [packetq], dnscap [dnscap], and dnstap [dnstap].
183 However, there is no standard exchange format for large DNS packet
184 captures. The PCAP ("packet capture") [pcap] format or the PCAP Next
185 Generation (PCAP-NG) [pcapng] format is typically used in practice
186 for packet captures, but these file formats can contain a great deal
187 of additional information that is not directly pertinent to DNS
188 traffic analysis and thus unnecessarily increases the capture file
189 size. Additionally, these tools and formats typically have no filter
190 mechanism to selectively record only certain fields at capture time,
191 requiring post-processing for anonymization or pseudonymization of
192 data to protect user privacy.
193
194 There has also been work on using text-based formats to describe DNS
195 packets (for example, see [dnsxml] and [RFC8427]), but this work is
196 largely aimed at producing convenient representations of single
197 messages.
198
199 Many DNS operators may receive hundreds of thousands of Queries per
200 second on a single name server instance, so a mechanism to minimize
201 the storage and transmission size (and therefore upload overhead) of
202 the data collected is highly desirable.
203
204 The format described in this document, C-DNS (Compacted-DNS), focuses
205 on the problem of capturing and storing large packet capture files of
206 DNS traffic with the following goals in mind:
207
208 o Minimize the file size for storage and transmission.
209
210 o Minimize the overhead of producing the packet capture file and the
211 cost of any further (general-purpose) compression of the file.
212
213
214
215
216
217 Dickinson, et al. Standards Track [Page 4]
218 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
219
220
221 This document contains:
222
223 o A discussion of some common use cases in which DNS data is
224 collected; see Section 3.
225
226 o A discussion of the major design considerations in developing an
227 efficient data representation for collections of DNS messages; see
228 Section 4.
229
230 o A description of why the Concise Binary Object Representation
231 (CBOR) [RFC7049] was chosen for this format; see Section 5.
232
233 o A conceptual overview of the C-DNS format; see Section 6.
234
235 o The definition of the C-DNS format for the collection of DNS
236 messages; see Section 7.
237
238 o Notes on converting C-DNS data to PCAP format; see Section 9.
239
240 o Some high-level implementation considerations for applications
241 designed to produce C-DNS; see Section 10.
242
243 2. Terminology
244
245 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
246 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
247 "OPTIONAL" in this document are to be interpreted as described in
248 BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
249 capitals, as shown here.
250
251 "Packet" refers to an individual IPv4 or IPv6 packet. Typically,
252 packets are UDP datagrams, but such packets may also be part of a TCP
253 data stream. "Message", unless otherwise qualified, refers to a DNS
254 payload extracted from a UDP datagram or a TCP data stream.
255
256 The parts of DNS messages are named as they are in [RFC1035].
257 Specifically, the DNS message has five sections: Header, Question,
258 Answer, Authority, and Additional.
259
260 3. Data Collection Use Cases
261
262 From a purely server operator perspective, collecting full packet
263 captures of all packets going into or out of a name server provides
264 the most comprehensive picture of network activity. However, there
265 are several design choices or other limitations that are common to
266 many DNS installations and operators.
267
268
269
270
271
272 Dickinson, et al. Standards Track [Page 5]
273 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
274
275
276 o DNS servers are hosted in a variety of situations:
277
278 * Self-hosted servers
279
280 * Third-party hosting (including multiple third parties)
281
282 * Third-party hardware (including multiple third parties)
283
284 o Data is collected under different conditions:
285
286 * On well-provisioned servers running in a steady state
287
288 * On heavily loaded servers
289
290 * On virtualized servers
291
292 * On servers that are under DoS attack
293
294 * On servers that are unwitting intermediaries in DoS attacks
295
296 o Traffic can be collected via a variety of mechanisms:
297
298 * Within the name server implementation itself
299
300 * On the same hardware as the name server itself
301
302 * Using a network tap on an adjacent host to listen to DNS
303 traffic
304
305 * Using port mirroring to listen from another host
306
307 o The capabilities of data collection (and upload) networks vary:
308
309 * Out-of-band networks with the same capacity as the in-band
310 network
311
312 * Out-of-band networks with less capacity than the in-band
313 network
314
315 * Everything being on the in-band network
316
317 Thus, there is a wide range of use cases, from very limited data
318 collection environments (third-party hardware, servers that are under
319 attack, packet capture on the name server itself and no out-of-band
320 network) to "limitless" environments (self-hosted, well-provisioned
321 servers, using a network tap or port mirroring with out-of-band
322 networks with the same capacity as the in-band network). In the
323
324
325
326
327 Dickinson, et al. Standards Track [Page 6]
328 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
329
330
331 former case, it is infeasible to reliably collect full packet
332 captures, especially if the server is under attack. In the latter
333 case, collection of full packet captures may be reasonable.
334
335 As a result of these restrictions, the C-DNS data format is designed
336 with the most limited use case in mind, such that:
337
338 o Data collection will occur on the same hardware as the name server
339 itself
340
341 o Collected data will be stored on the same hardware as the name
342 server itself, at least temporarily
343
344 o Collected data being returned to some central analysis system will
345 use the same network interface as the DNS Queries and Responses
346
347 o There can be multiple third-party servers involved
348
349 Because of these considerations, a major factor in the design of the
350 format is minimal storage size of the capture files.
351
352 Another significant consideration for any application that records
353 DNS traffic is that the running of the name server software and the
354 transmission of DNS Queries and Responses are the most important jobs
355 of a name server; capturing data is not. Any data collection system
356 co-located with the name server needs to be intelligent enough to
357 carefully manage its CPU, disk, memory, and network utilization.
358 This leads to designing a format that requires a relatively low
359 overhead to produce and minimizes the requirement for further
360 potentially costly compression.
361
362 However, it is also essential that interoperability with less
363 restricted infrastructure is maintained. In particular, it is highly
364 desirable that the collection format should facilitate the
365 re-creation of common formats (such as PCAP) that are as close to the
366 original as is realistic, given the restrictions above.
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382 Dickinson, et al. Standards Track [Page 7]
383 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
384
385
386 4. Design Considerations
387
388 This section presents some of the major design considerations used in
389 the development of the C-DNS format.
390
391 1. The basic unit of data is a combined DNS Query and the associated
392 Response (a "Query/Response (Q/R) data item"). The same
393 structure will be used for unmatched Queries and Responses.
394 Queries without Responses will be captured omitting the Response
395 data. Responses without Queries will be captured omitting the
396 Query data (but using the Question section from the Response, if
397 present, as an identifying QNAME).
398
399 * Rationale: A Query and the associated Response represent the
400 basic level of a client's interaction with the server. Also,
401 combining the Query and Response into one item often reduces
402 storage requirements due to commonality in the data of the two
403 messages.
404
405 In the context of generating a C-DNS file, it is assumed that
406 only those DNS payloads that can be parsed to produce a
407 well-formed DNS message are stored in the structured Query/
408 Response data items of the C-DNS format and that all other
409 messages will (optionally) be recorded as separate malformed
410 messages. Parsing a well-formed message means, at a minimum, the
411 following:
412
413 * The packet has a well-formed 12-byte DNS Header with a
414 recognized OPCODE.
415
416 * The section counts are consistent with the section contents.
417
418 * All of the Resource Records (RRs) can be fully parsed.
419
420 2. All top-level fields in each Query/Response data item will be
421 optional.
422
423 * Rationale: Different operators will have different
424 requirements for data to be available for analysis. Operators
425 with minimal requirements should not have to pay the cost of
426 recording full data, though this will limit the ability to
427 perform certain kinds of data analysis and also to reconstruct
428 packet captures. For example, omitting the RRs from a
429 Response will reduce the C-DNS file size; in principle,
430 Responses can be synthesized if there is enough context.
431 Operators may have different policies for collecting user data
432 and can choose to omit or anonymize certain fields at capture
433 time, e.g., client address.
434
435
436
437 Dickinson, et al. Standards Track [Page 8]
438 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
439
440
441 3. Multiple Query/Response data items will be collected into blocks
442 in the format. Common data in a block will be abstracted and
443 referenced from individual Query/Response data items by indexing.
444 The maximum number of Query/Response data items in a block will
445 be configurable.
446
447 * Rationale: This blocking and indexing action provides a
448 significant reduction in the volume of file data generated.
449 Although this introduces complexity, it provides compression
450 of the data that makes use of knowledge of the DNS message
451 structure.
452
453 * It is anticipated that the files produced can be subject to
454 further compression using general-purpose compression tools.
455 Measurements show that blocking significantly reduces the CPU
456 required to perform such strong compression. See
457 Appendix C.2.
458
459 * Examples of commonality between DNS messages are that in most
460 cases the QUESTION RR is the same in the Query and Response
461 and that there is a finite set of Query "signatures" (based on
462 a subset of attributes). For many authoritative servers,
463 there is very likely to be a finite set of Responses that are
464 generated, of which a large number are NXDOMAIN.
465
466 4. Traffic metadata can optionally be included in each block.
467 Specifically, counts of some types of non-DNS packets (e.g.,
468 ICMP, TCP resets) sent to the server may be of interest.
469
470 5. The wire-format content of malformed DNS messages may optionally
471 be recorded.
472
473 * Rationale: Any structured capture format that does not capture
474 the DNS payload byte for byte will be limited to some extent
475 in that it cannot represent malformed DNS messages. Only
476 those messages that can be fully parsed and transformed into
477 the structured format can be fully represented. Note,
478 however, that this can result in rather misleading statistics.
479 For example, a malformed Query that cannot be represented in
480 the C-DNS format will lead to the (well-formed) DNS Response
481 with error code FORMERR appearing as "unmatched". Therefore,
482 it can greatly aid downstream analysis to have the wire format
483 of the malformed DNS messages available directly in the
484 C-DNS file.
485
486
487
488
489
490
491
492 Dickinson, et al. Standards Track [Page 9]
493 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
494
495
496 5. Choice of CBOR
497
498 This document presents a detailed format description for C-DNS. The
499 format uses CBOR [RFC7049].
500
501 The choice of CBOR was made taking a number of factors into account.
502
503 o CBOR is a binary representation and thus is economical in storage
504 space.
505
506 o Other binary representations were investigated, and whilst all had
507 attractive features, none had a significant advantage over CBOR.
508 See Appendix C for some discussion of this.
509
510 o CBOR is an IETF specification and is familiar to IETF
511 participants. It is based on the now-common ideas of lists and
512 objects and thus requires very little familiarization for those in
513 the wider industry.
514
515 o CBOR is a simple format and can easily be implemented from scratch
516 if necessary. Formats that are more complex require library
517 support, which may present problems on unusual platforms.
518
519 o CBOR can also be easily converted to text formats such as JSON
520 [RFC8259] for debugging and other human inspection requirements.
521
522 o CBOR data schemas can be described using the Concise Data
523 Definition Language (CDDL) [RFC8610].
524
525 6. C-DNS Format Conceptual Overview
526
527 The following figures show purely schematic representations of the
528 C-DNS format to convey the high-level structure of the C-DNS format.
529 Section 7 provides a detailed discussion of the CBOR representation
530 and individual elements.
531
532 Figure 1 shows the C-DNS format at the top level, including the file
533 header and data blocks. The Query/Response data items, Address/Event
534 Count data items, and Malformed Message data items link to various
535 Block Tables.
536
537
538
539
540
541
542
543
544
545
546
547 Dickinson, et al. Standards Track [Page 10]
548 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
549
550
551 +-------+
552 + C-DNS |
553 +-------+--------------------------+
554 | File Type Identifier |
555 +----------------------------------+
556 | File Preamble |
557 | +--------------------------------+
558 | | Format Version |
559 | +--------------------------------+
560 | | Block Parameters |
561 +-+--------------------------------+
562 | Block |
563 | +--------------------------------+
564 | | Block Preamble |
565 | +--------------------------------+
566 | | Block Statistics |
567 | +--------------------------------+
568 | | Block Tables |
569 | +--------------------------------+
570 | | Query/Response data items |
571 | +--------------------------------+
572 | | Address/Event Count data items |
573 | +--------------------------------+
574 | | Malformed Message data items |
575 +-+--------------------------------+
576 | Block |
577 | +--------------------------------+
578 | | Block Preamble |
579 | +--------------------------------+
580 | | Block Statistics |
581 | +--------------------------------+
582 | | Block Tables |
583 | +--------------------------------+
584 | | Query/Response data items |
585 | +--------------------------------+
586 | | Address/Event Count data items |
587 | +--------------------------------+
588 | | Malformed Message data items |
589 +-+--------------------------------+
590 | Further Blocks... |
591 +----------------------------------+
592
593 Figure 1: The C-DNS Format
594
595 Figure 2 shows some more-detailed relationships within each Block,
596 specifically those between the Query/Response data item and the
597 relevant Block Tables. Some fields have been omitted for clarity.
598
599
600
601
602 Dickinson, et al. Standards Track [Page 11]
603 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
604
605
606 +----------------+
607 | Query/Response |
608 +-------------------------+
609 | Time Offset |
610 +-------------------------+ +------------------+
611 | Client Address |---------+->| IP Address array |
612 +-------------------------+ | +------------------+
613 | Client Port | |
614 +-------------------------+ | +------------------+
615 | Transaction ID | +---)->| Name/RDATA array |<--------+
616 +-------------------------+ | | +------------------+ |
617 | Query Signature |--+ | | |
618 +-------------------------+ | | | +-----------------+ |
619 | Client Hoplimit (q) | +--)---)->| Query Signature | |
620 +-------------------------+ | | +-----------------+-------+ |
621 | Response Delay (r) | | +--| Server Address | |
622 +-------------------------+ | +-------------------------+ |
623 | Query Name |--+--+ | Server Port | |
624 +-------------------------+ | +-------------------------+ |
625 | Query Size (q) | | | Transport Flags | |
626 +-------------------------+ | +-------------------------+ |
627 | Response Size (r) | | | QR Type | |
628 +-------------------------+ | +-------------------------+ |
629 | Response Processing (r) | | | QR Signature Flags | |
630 | +-----------------------+ | +-------------------------+ |
631 | | Bailiwick |--+ | Query OPCODE (q) | |
632 | +-----------------------+ +-------------------------+ |
633 | | Flags | | QR DNS Flags | |
634 +-+-----------------------+ +-------------------------+ |
635 | Extra Query Info (q) | | Query RCODE (q) | |
636 | +-----------------------+ +-------------------------+ |
637 | | Question |--+---+ +--+-Query Class/Type (q) | |
638 | +-----------------------+ | | +-------------------------+ |
639 | | Answer |--+ | | | Query QDCOUNT (q) | |
640 | +-----------------------+ | | | +-------------------------+ |
641 | | Authority |--+ | | | Query ANCOUNT (q) | |
642 | +-----------------------+ | | | +-------------------------+ |
643 | | Additional |--+ | | | Query NSCOUNT (q) | |
644
645
646
647
648
649
650
651
652
653
654
655
656
657 Dickinson, et al. Standards Track [Page 12]
658 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
659
660
661 +-+-----------------------+ | | | +-------------------------+ |
662 | Extra Response Info (r) | |-+ | | | Query ARCOUNT (q) | |
663 | +-----------------------+ | | | | +-------------------------+ |
664 | | Answer |--+ | | | | Query EDNS version (q) | |
665 | +-----------------------+ | | | | +-------------------------+ |
666 | | Authority |--+ | | | | Query EDNS UDP Size (q) | |
667 | +-----------------------+ | | | | +-------------------------+ |
668 | | Additional |--+ | | | | Query OPT RDATA (q) |--+
669 +-+-----------------------+ | | | +-------------------------+ |
670 | | | | Response RCODE (r) | |
671 | | | +-------------------------+ |
672 + -----------------------------+ | +----------+ |
673 | | | |
674 | + -----------------------------+ | |
675 | | +---------------+ +----------+ | |
676 | +->| Question List |->| Question | | |
677 | | array | | array | | |
678 | +---------------+ +----------+--+ | |
679 | | Name |--+-----)--------------------+
680 | +-------------+ | | +------------+
681 | | Class/Type |--)---+-+->| Class/Type |
682 | +-------------+ | | | array |
683 | | | +------------+--+
684 | | | | CLASS |
685 | +---------------+ +----------+ | | +---------------+
686 +--->| RR List array |->| RR array | | | | TYPE |
687 +---------+-----+ +----------+--+ | | +---------------+
688 | Name |--+ |
689 +-------------+ |
690 | Class/Type |------+
691 +-------------+
692
693 Figure 2: The Query/Response Data Item and Subsidiary Tables
694
695 In Figure 2, data items annotated (q) are only present when a
696 Query/Response has a Query, and those annotated (r) are only present
697 when a Query/Response Response is present.
698
699 A C-DNS file begins with a file header containing a File Type
700 Identifier and a File Preamble. The File Preamble contains
701 information on the file Format Version and an array of Block
702 Parameters items (the contents of which include Collection and
703 Storage Parameters used for one or more Blocks).
704
705 The file header is followed by a series of Blocks.
706
707
708
709
710
711
712 Dickinson, et al. Standards Track [Page 13]
713 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
714
715
716 A Block consists of a Block Preamble item, some Block Statistics for
717 the traffic stored within the Block, and then various arrays of
718 common data collectively called the Block Tables. This is then
719 followed by an array of the Query/Response data items detailing the
720 Queries and Responses stored within the Block. The array of
721 Query/Response data items is in turn followed by the Address/Event
722 Count data items (an array of per-client counts of particular IP
723 events) and then Malformed Message data items (an array of malformed
724 messages that are stored in the Block).
725
726 The exact nature of the DNS data will affect what Block size is the
727 best fit; however, sample data for a root server indicated that Block
728 sizes up to 10,000 Query/Response data items give good results. See
729 Appendix C.6 for more details.
730
731 This design exploits data commonality and block-based storage to
732 minimize the C-DNS file size. As a result, C-DNS cannot be streamed
733 below the level of a Block.
734
735 6.1. Block Parameters
736
737 The details of the Block Parameters items are not shown in the
738 diagrams but are discussed here for context.
739
740 An array of Block Parameters items is stored in the File Preamble
741 (with a minimum of one item at index 0); a Block Parameters item
742 consists of a collection of Storage and Collection Parameters that
743 applies to any given Block. An array is used in order to support use
744 cases such as wanting to merge C-DNS files from different sources.
745 The Block Preamble item then contains an optional index for the Block
746 Parameters item that applies for that Block; if not present, the
747 index defaults to 0. Hence, in effect, a global Block Parameters
748 item is defined that can then be overridden per Block.
749
750 6.2. Storage Parameters
751
752 The Block Parameters item includes a Storage Parameters item -- this
753 contains information about the specific data fields stored in the
754 C-DNS file.
755
756 These parameters include:
757
758 o The sub-second timing resolution used by the data.
759
760 o Information (hints) on which optional data are omitted. See
761 Section 6.2.1.
762
763
764
765
766
767 Dickinson, et al. Standards Track [Page 14]
768 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
769
770
771 o Recorded OPCODES [opcodes] and RR TYPEs [rrtypes]. See
772 Section 6.2.2.
773
774 o Flags indicating, for example, whether the data is sampled or
775 anonymized. See Sections 6.2.3 and 14.
776
777 o Client and server IPv4 and IPv6 address prefixes. See
778 Section 6.2.4.
779
780 6.2.1. Optional Data Items
781
782 To enable implementations to store data to their precise requirements
783 in as space-efficient a manner as possible, all fields in the
784 following arrays are optional:
785
786 o Query/Response
787
788 o Query Signature
789
790 o Malformed Messages
791
792 In other words, an implementation can choose to omit any data item
793 that is not required for its use case (whilst observing the
794 restrictions relating to IP address storage described in
795 Section 6.2.4). In addition, implementations may be configured to
796 not record all RRs or to only record messages with certain OPCODES.
797
798 This does, however, mean that a consumer of a C-DNS file faces two
799 problems:
800
801 1. How can it quickly determine if a file definitely does not
802 contain the data items it requires to complete a particular task
803 (e.g., reconstructing DNS traffic or performing a specific piece
804 of data analysis)?
805
806 2. How can it determine whether a data item is not present because
807 it was (1) explicitly not recorded or (2) not available/present?
808
809 For example, capturing C-DNS data from within a name server
810 implementation makes it unlikely that the Client Hoplimit can be
811 recorded. Or, if there is no Query ARCOUNT recorded and no Query OPT
812 RDATA [RFC6891] recorded, is that because no Query contained an OPT
813 RR, or because that data was not stored?
814
815 The Storage Parameters item therefore also contains a Storage Hints
816 item, which specifies which items the encoder of the file omits from
817 the stored data and will therefore never be present. (This approach
818 is taken because a flag that indicated which items were included for
819
820
821
822 Dickinson, et al. Standards Track [Page 15]
823 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
824
825
826 collection would not guarantee that the item was present -- only that
827 it might be.) An implementation decoding that file can then use
828 these flags to quickly determine whether the input data is not rich
829 enough for its needs.
830
831 One scenario where this may be particularly important is the case of
832 regenerating traffic. It is possible to collect such a small set of
833 data items that an implementation decoding the file cannot determine
834 if a given Query/Response data item was generated from just a Query,
835 just a Response, or a Query/Response pair. This makes it impossible
836 to reconstruct DNS traffic even if sensible defaults are provided for
837 the missing data items. This is discussed in more detail in
838 Section 9.
839
840 6.2.2. Optional RRs and OPCODEs
841
842 Also included in the Storage Parameters item are explicit arrays
843 listing the RR TYPEs and the OPCODEs to be recorded. These arrays
844 remove any ambiguity over whether, for example, messages containing
845 particular OPCODEs are not present because (1) certain OPCODEs did
846 not occur or (2) the implementation is not configured to record them.
847
848 In the case of OPCODEs, for a message to be fully parsable, the
849 OPCODE must be known to the collecting implementation. Any message
850 with an OPCODE unknown to the collecting implementation cannot be
851 validated as correctly formed and so must be treated as malformed.
852 Messages with OPCODES known to the recording application but not
853 listed in the Storage Parameters item are discarded by the recording
854 application during C-DNS capture (regardless of whether they are
855 malformed or not).
856
857 In the case of RRs, each record in a message must be fully parsable,
858 including parsing the record RDATA, as otherwise the message cannot
859 be validated as correctly formed. Any RR with an RR TYPE not known
860 to the collecting implementation cannot be validated as correctly
861 formed and so must be treated as malformed.
862
863 Once a message is correctly parsed, an implementation is free to
864 record only a subset of the RRs present.
865
866
867
868
869
870
871
872
873
874
875
876
877 Dickinson, et al. Standards Track [Page 16]
878 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
879
880
881 6.2.3. Storage Flags
882
883 The Storage Parameters item contains flags that can be used to
884 indicate if:
885
886 o the data is anonymized,
887
888 o the data is produced from sample data, or
889
890 o names in the data have been normalized (converted to uniform
891 case).
892
893 The Storage Parameters item also contains optional fields holding
894 details of the sampling method used and the anonymization method
895 used. It is RECOMMENDED that these fields contain URIs [RFC3986]
896 pointing to resources describing the methods used. See Section 14
897 for further discussion of anonymization and normalization.
898
899 6.2.4. IP Address Storage
900
901 The format can store either full IP addresses or just IP prefixes;
902 the Storage Parameters item contains fields to indicate if only IP
903 prefixes were stored.
904
905 If the IP address prefixes are absent, then full addresses are
906 stored. In this case, the IP version can be directly inferred from
907 the stored address length and the fields "qr-transport-flags" in
908 QueryResponseSignature, "ae-transport-flags" in AddressEventCount,
909 and "mm-transport-flags" in MalformedMessageData (which contain the
910 IP version bit) are optional.
911
912 If IP address prefixes are given, only the prefix bits of addresses
913 are stored. In this case, in order to determine the IP version, the
914 fields "qr-transport-flags" in QueryResponseSignature, "ae-transport-
915 flags" in AddressEventCount, and "mm-transport-flags" in
916 MalformedMessageData MUST be present. See Sections 7.3.2.3.2 and
917 7.3.2.3.5.
918
919 As an example of storing only IP prefixes, if a client IPv6 prefix of
920 48 is specified, a client address of 2001:db8:85a3::8a2e:370:7334
921 will be stored as 0x20010db885a3, reducing address storage space
922 requirements. Similarly, if a client IPv4 prefix of 16 is specified,
923 a client address of 192.0.2.1 will be stored as 0xc000 (192.0).
924
925
926
927
928
929
930
931
932 Dickinson, et al. Standards Track [Page 17]
933 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
934
935
936 7. C-DNS Format Detailed Description
937
938 The CDDL definition for the C-DNS format is given in Appendix A.
939
940 7.1. Map Quantities and Indexes
941
942 All map keys are integers with values specified in the CDDL. String
943 keys would significantly bloat the file size.
944
945 All key values specified are positive integers under 24, so their
946 CBOR representation is a single byte. Positive integer values not
947 currently used as keys in a map are reserved for use in future
948 standard extensions.
949
950 Implementations may choose to add additional implementation-specific
951 entries to any map. Negative integer map keys are reserved for these
952 values. Key values from -1 to -24 also have a single-byte CBOR
953 representation, so such implementation-specific extensions are not at
954 any space efficiency disadvantage.
955
956 An item described as an index is the index of the data item in the
957 referenced array. Indexes are 0-based.
958
959 7.2. Tabular Representation
960
961 The following sections present the C-DNS specification in tabular
962 format with a detailed description of each item.
963
964 In all quantities that contain bit flags, bit 0 indicates the least
965 significant bit, i.e., flag "n" in quantity "q" is on if
966 "(q & (1 << n)) != 0".
967
968 For the sake of readability, all type and field names defined in the
969 CDDL definition are shown in double quotes. Type names are by
970 convention camel case (e.g., "BlockTables"), and field names are
971 lowercase with hyphens (e.g., "block-tables").
972
973 For the sake of brevity, the following conventions are used in the
974 tables:
975
976 o The column M marks whether items in a map are mandatory.
977
978 * X - Mandatory items.
979
980 * C - Conditionally mandatory items. Such items are usually
981 optional but may be mandatory in some configurations.
982
983 * If the column is empty, the item is optional.
984
985
986
987 Dickinson, et al. Standards Track [Page 18]
988 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
989
990
991 o The column T gives the CBOR datatype of the item.
992
993 * U - Unsigned integer.
994
995 * I - Signed integer (i.e., either a CBOR unsigned integer or a
996 CBOR negative integer).
997
998 * B - Boolean.
999
1000 * S - Byte string.
1001
1002 * T - Text string.
1003
1004 * M - Map.
1005
1006 * A - Array.
1007
1008 In the case of maps and arrays, more information on the type of each
1009 value, including the CDDL definition name if applicable, is given in
1010 the description.
1011
1012 7.3. "File"
1013
1014 A C-DNS file has an outer structure "File", an array that contains
1015 the following:
1016
1017 +---------------+---+---+-------------------------------------------+
1018 | Field | M | T | Description |
1019 +---------------+---+---+-------------------------------------------+
1020 | file-type-id | X | T | String "C-DNS" identifying the file type. |
1021 | | | | |
1022 | file-preamble | X | M | Version and parameter information for the |
1023 | | | | whole file. Map of type "FilePreamble"; |
1024 | | | | see Section 7.3.1. |
1025 | | | | |
1026 | file-blocks | X | A | Array of items of type "Block"; see |
1027 | | | | Section 7.3.2. The array may be empty if |
1028 | | | | the file contains no data. |
1029 +---------------+---+---+-------------------------------------------+
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042 Dickinson, et al. Standards Track [Page 19]
1043 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1044
1045
1046 7.3.1. "FilePreamble"
1047
1048 Information about data in the file. A map containing the following:
1049
1050 +----------------------+---+---+------------------------------------+
1051 | Field | M | T | Description |
1052 +----------------------+---+---+------------------------------------+
1053 | major-format-version | X | U | Unsigned integer "1". The major |
1054 | | | | version of the format used in the |
1055 | | | | file. See Section 8. |
1056 | | | | |
1057 | minor-format-version | X | U | Unsigned integer "0". The minor |
1058 | | | | version of the format used in the |
1059 | | | | file. See Section 8. |
1060 | | | | |
1061 | private-version | | U | Version indicator available for |
1062 | | | | private use by implementations. |
1063 | | | | |
1064 | block-parameters | X | A | Array of items of type |
1065 | | | | "BlockParameters". See Section |
1066 | | | | 7.3.1.1. The array must contain |
1067 | | | | at least one entry. (The |
1068 | | | | "block-parameters-index" item in |
1069 | | | | each "BlockPreamble" indicates |
1070 | | | | which array entry applies to that |
1071 | | | | "Block".) |
1072 +----------------------+---+---+------------------------------------+
1073
1074 7.3.1.1. "BlockParameters"
1075
1076 Parameters relating to data storage and collection that apply to one
1077 or more items of type "Block". A map containing the following:
1078
1079 +-----------------------+---+---+-----------------------------------+
1080 | Field | M | T | Description |
1081 +-----------------------+---+---+-----------------------------------+
1082 | storage-parameters | X | M | Parameters relating to data |
1083 | | | | storage in a "Block" item. Map |
1084 | | | | of type "StorageParameters"; see |
1085 | | | | Section 7.3.1.1.1. |
1086 | | | | |
1087 | collection-parameters | | M | Parameters relating to collection |
1088 | | | | of the data in a "Block" item. |
1089 | | | | Map of type |
1090 | | | | "CollectionParameters"; see |
1091 | | | | Section 7.3.1.1.2. |
1092 +-----------------------+---+---+-----------------------------------+
1093
1094
1095
1096
1097 Dickinson, et al. Standards Track [Page 20]
1098 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1099
1100
1101 7.3.1.1.1. "StorageParameters"
1102
1103 Parameters relating to how data is stored in the items of type
1104 "Block". A map containing the following:
1105
1106 +------------------+---+---+----------------------------------------+
1107 | Field | M | T | Description |
1108 +------------------+---+---+----------------------------------------+
1109 | ticks-per-second | X | U | Sub-second timing is recorded in |
1110 | | | | ticks. This specifies the number of |
1111 | | | | ticks in a second. |
1112 | | | | |
1113 | max-block-items | X | U | The maximum number of items stored in |
1114 | | | | any of the arrays in a "Block" item |
1115 | | | | (Q/R, Address/Event Count, or |
1116 | | | | Malformed Message data items). An |
1117 | | | | indication to a decoder of the |
1118 | | | | resources needed to process the file. |
1119 | | | | |
1120 | storage-hints | X | M | Collection of hints as to which fields |
1121 | | | | are omitted in the arrays that have |
1122 | | | | optional fields. Map of type |
1123 | | | | "StorageHints". See Section |
1124 | | | | 7.3.1.1.1.1. |
1125 | | | | |
1126 | opcodes | X | A | Array of OPCODES [opcodes] (unsigned |
1127 | | | | integers, each in the range 0 to 15 |
1128 | | | | inclusive) recorded by the collecting |
1129 | | | | implementation. See Section 6.2.2. |
1130 | | | | |
1131 | rr-types | X | A | Array of RR TYPEs [rrtypes] (unsigned |
1132 | | | | integers, each in the range 0 to 65535 |
1133 | | | | inclusive) recorded by the collecting |
1134 | | | | implementation. See Section 6.2.2. |
1135 | | | | |
1136 | storage-flags | | U | Bit flags indicating attributes of |
1137 | | | | stored data. |
1138 | | | | Bit 0. 1 if the data has been |
1139 | | | | anonymized. |
1140 | | | | Bit 1. 1 if the data is sampled data. |
1141 | | | | Bit 2. 1 if the names have been |
1142 | | | | normalized (converted to uniform |
1143 | | | | case). |
1144 | | | | |
1145 | client-address | | U | IPv4 client address prefix length, in |
1146 | -prefix-ipv4 | | | the range 1 to 32 inclusive. If |
1147 | | | | specified, only the address prefix |
1148 | | | | bits are stored. |
1149
1150
1151
1152 Dickinson, et al. Standards Track [Page 21]
1153 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1154
1155
1156 | | | | |
1157 | client-address | | U | IPv6 client address prefix length, in |
1158 | -prefix-ipv6 | | | the range 1 to 128 inclusive. If |
1159 | | | | specified, only the address prefix |
1160 | | | | bits are stored. |
1161 | | | | |
1162 | server-address | | U | IPv4 server address prefix length, in |
1163 | -prefix-ipv4 | | | the range 1 to 32 inclusive. If |
1164 | | | | specified, only the address prefix |
1165 | | | | bits are stored. |
1166 | | | | |
1167 | server-address | | U | IPv6 server address prefix length, in |
1168 | -prefix-ipv6 | | | the range 1 to 128 inclusive. If |
1169 | | | | specified, only the address prefix |
1170 | | | | bits are stored. |
1171 | | | | |
1172 | sampling-method | | T | Information on the sampling method |
1173 | | | | used. See Section 6.2.3. |
1174 | | | | |
1175 | anonymization | | T | Information on the anonymization |
1176 | -method | | | method used. See Section 6.2.3. |
1177 +------------------+---+---+----------------------------------------+
1178
1179 7.3.1.1.1.1. "StorageHints"
1180
1181 An indicator of which fields the collecting implementation omits in
1182 the maps with optional fields. Note that hints have a top-down
1183 precedence. In other words, where a map contains another map, the
1184 hint on the containing map overrides any hints in the contained map
1185 and the contained map is omitted. A map containing the following:
1186
1187 +------------------+---+---+----------------------------------------+
1188 | Field | M | T | Description |
1189 +------------------+---+---+----------------------------------------+
1190 | query-response | X | U | Hints indicating which "QueryResponse" |
1191 | -hints | | | fields are omitted; see Section |
1192 | | | | 7.3.2.4. If a bit is unset, the field |
1193 | | | | is omitted from the capture. |
1194 | | | | Bit 0. time-offset |
1195 | | | | Bit 1. client-address-index |
1196 | | | | Bit 2. client-port |
1197 | | | | Bit 3. transaction-id |
1198 | | | | Bit 4. qr-signature-index |
1199 | | | | Bit 5. client-hoplimit |
1200 | | | | Bit 6. response-delay |
1201 | | | | Bit 7. query-name-index |
1202 | | | | Bit 8. query-size |
1203 | | | | Bit 9. response-size |
1204
1205
1206
1207 Dickinson, et al. Standards Track [Page 22]
1208 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1209
1210
1211 | | | | Bit 10. response-processing-data |
1212 | | | | Bit 11. query-question-sections |
1213 | | | | Bit 12. query-answer-sections |
1214 | | | | Bit 13. query-authority-sections |
1215 | | | | Bit 14. query-additional-sections |
1216 | | | | Bit 15. response-answer-sections |
1217 | | | | Bit 16. response-authority-sections |
1218 | | | | Bit 17. response-additional-sections |
1219 | | | | |
1220 | query-response | X | U | Hints indicating which |
1221 | -signature-hints | | | "QueryResponseSignature" fields are |
1222 | | | | omitted; see Section 7.3.2.3.2. If a |
1223 | | | | bit is unset, the field is omitted |
1224 | | | | from the capture. |
1225 | | | | Bit 0. server-address-index |
1226 | | | | Bit 1. server-port |
1227 | | | | Bit 2. qr-transport-flags |
1228 | | | | Bit 3. qr-type |
1229 | | | | Bit 4. qr-sig-flags |
1230 | | | | Bit 5. query-opcode |
1231 | | | | Bit 6. qr-dns-flags |
1232 | | | | Bit 7. query-rcode |
1233 | | | | Bit 8. query-classtype-index |
1234 | | | | Bit 9. query-qdcount |
1235 | | | | Bit 10. query-ancount |
1236 | | | | Bit 11. query-nscount |
1237 | | | | Bit 12. query-arcount |
1238 | | | | Bit 13. query-edns-version |
1239 | | | | Bit 14. query-udp-size |
1240 | | | | Bit 15. query-opt-rdata-index |
1241 | | | | Bit 16. response-rcode |
1242 | | | | |
1243 | rr-hints | X | U | Hints indicating which optional "RR" |
1244 | | | | fields are omitted; see Section |
1245 | | | | 7.3.2.3.4. If a bit is unset, the |
1246 | | | | field is omitted from the capture. |
1247 | | | | Bit 0. ttl |
1248 | | | | Bit 1. rdata-index |
1249 | other-data-hints | X | U | Hints indicating which other datatypes |
1250 | | | | are omitted. If a bit is unset, the |
1251 | | | | datatype is omitted from the capture. |
1252 | | | | Bit 0. malformed-messages |
1253 | | | | Bit 1. address-event-counts |
1254 +------------------+---+---+----------------------------------------+
1255
1256
1257
1258
1259
1260
1261
1262 Dickinson, et al. Standards Track [Page 23]
1263 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1264
1265
1266 7.3.1.1.2. "CollectionParameters"
1267
1268 Parameters providing information regarding how data in the file was
1269 collected (applicable for some, but not all, collection
1270 environments). The values are informational only and serve as
1271 metadata to downstream analyzers as to the configuration of a
1272 collecting implementation. They can provide context when
1273 interpreting what data is present/absent from the capture but cannot
1274 necessarily be validated against the data captured.
1275
1276 These parameters have no default. If they do not appear, nothing can
1277 be inferred about their value.
1278
1279 A map containing the following items:
1280
1281 +------------------+---+---+----------------------------------------+
1282 | Field | M | T | Description |
1283 +------------------+---+---+----------------------------------------+
1284 | query-timeout | | U | To be matched with a Query, a Response |
1285 | | | | must arrive within this number of |
1286 | | | | milliseconds. |
1287 | | | | |
1288 | skew-timeout | | U | The network stack may report a |
1289 | | | | Response before the corresponding |
1290 | | | | Query. A Response is not considered |
1291 | | | | to be missing a Query until after this |
1292 | | | | many microseconds. |
1293 | | | | |
1294 | snaplen | | U | Collect up to this many bytes per |
1295 | | | | packet. |
1296 | | | | |
1297 | promisc | | B | "true" if promiscuous mode |
1298 | | | | [pcap-options] was enabled on the |
1299 | | | | interface, "false" otherwise. |
1300 | | | | |
1301 | interfaces | | A | Array of identifiers (of type text |
1302 | | | | string) of the interfaces used for |
1303 | | | | collection. |
1304 | | | | |
1305 | server-addresses | | A | Array of server collection IP |
1306 | | | | addresses (of type byte string). |
1307 | | | | Metadata for downstream analyzers; |
1308 | | | | does not affect collection. |
1309 | | | | |
1310
1311
1312
1313
1314
1315
1316
1317 Dickinson, et al. Standards Track [Page 24]
1318 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1319
1320
1321 | vlan-ids | | A | Array of identifiers (of type unsigned |
1322 | | | | integer, each in the range 1 to 4094 |
1323 | | | | inclusive) of VLANs [IEEE802.1Q] |
1324 | | | | selected for collection. VLAN IDs are |
1325 | | | | unique only within an administrative |
1326 | | | | domain. |
1327 | | | | |
1328 | filter | | T | Filter for input, in "tcpdump" |
1329 | | | | [pcap-filter] style. |
1330 | | | | |
1331 | generator-id | | T | Implementation-specific human-readable |
1332 | | | | string identifying the collection |
1333 | | | | method. |
1334 | | | | |
1335 | host-id | | T | String identifying the collecting |
1336 | | | | host. |
1337 +------------------+---+---+----------------------------------------+
1338
1339 7.3.2. "Block"
1340
1341 Container for data with common collection and storage parameters. A
1342 map containing the following:
1343
1344 +--------------------+---+---+--------------------------------------+
1345 | Field | M | T | Description |
1346 +--------------------+---+---+--------------------------------------+
1347 | block-preamble | X | M | Overall information for the "Block" |
1348 | | | | item. Map of type "BlockPreamble"; |
1349 | | | | see Section 7.3.2.1. |
1350 | | | | |
1351 | block-statistics | | M | Statistics about the "Block" item. |
1352 | | | | Map of type "BlockStatistics"; see |
1353 | | | | Section 7.3.2.2. |
1354 | | | | |
1355 | block-tables | | M | The arrays containing data |
1356 | | | | referenced by individual |
1357 | | | | "QueryResponse" or |
1358 | | | | "MalformedMessage" items. Map of |
1359 | | | | type "BlockTables"; see Section |
1360 | | | | 7.3.2.3. |
1361 | | | | |
1362 | query-responses | | A | Details of individual C-DNS Q/R data |
1363 | | | | items. Array of items of type |
1364 | | | | "QueryResponse"; see Section |
1365 | | | | 7.3.2.4. If present, the array must |
1366 | | | | not be empty. |
1367 | | | | |
1368
1369
1370
1371
1372 Dickinson, et al. Standards Track [Page 25]
1373 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1374
1375
1376 | address-event | | A | Per-client counts of ICMP messages |
1377 | -counts | | | and TCP resets. Array of items of |
1378 | | | | type "AddressEventCount"; see |
1379 | | | | Section 7.3.2.5. If present, the |
1380 | | | | array must not be empty. |
1381 | | | | |
1382 | malformed-messages | | A | Details of malformed DNS messages. |
1383 | | | | Array of items of type |
1384 | | | | "MalformedMessage"; see Section |
1385 | | | | 7.3.2.6. If present, the array must |
1386 | | | | not be empty. |
1387 +--------------------+---+---+--------------------------------------+
1388
1389 7.3.2.1. "BlockPreamble"
1390
1391 Overall information for a "Block" item. A map containing the
1392 following:
1393
1394 +------------------+---+---+----------------------------------------+
1395 | Field | M | T | Description |
1396 +------------------+---+---+----------------------------------------+
1397 | earliest-time | C | A | A timestamp (two unsigned integers, of |
1398 | | | | type "Timestamp") for the earliest |
1399 | | | | record in the "Block" item. The first |
1400 | | | | integer is the number of seconds since |
1401 | | | | the POSIX epoch [posix-time] |
1402 | | | | ("time_t"), excluding leap seconds. |
1403 | | | | The second integer is the number of |
1404 | | | | ticks (see Section 7.3.1.1.1) since |
1405 | | | | the start of the second. This field |
1406 | | | | is mandatory unless all block items |
1407 | | | | containing a time offset from the |
1408 | | | | start of the Block also omit that time |
1409 | | | | offset. |
1410 | | | | |
1411 | block-parameters | | U | The index of the item in the |
1412 | -index | | | "block-parameters" array (in the |
1413 | | | | "file-preamble" item) applicable to |
1414 | | | | this block. If not present, index 0 |
1415 | | | | is used. See Section 7.3.1. |
1416 +------------------+---+---+----------------------------------------+
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427 Dickinson, et al. Standards Track [Page 26]
1428 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1429
1430
1431 7.3.2.2. "BlockStatistics"
1432
1433 Basic statistical information about a "Block" item. A map containing
1434 the following:
1435
1436 +---------------------+---+---+-------------------------------------+
1437 | Field | M | T | Description |
1438 +---------------------+---+---+-------------------------------------+
1439 | processed-messages | | U | Total number of well-formed DNS |
1440 | | | | messages processed from the input |
1441 | | | | traffic stream during collection of |
1442 | | | | data in this "Block" item. |
1443 | | | | |
1444 | qr-data-items | | U | Total number of Q/R data items in |
1445 | | | | this "Block" item. |
1446 | | | | |
1447 | unmatched-queries | | U | Number of unmatched Queries in this |
1448 | | | | "Block" item. |
1449 | | | | |
1450 | unmatched-responses | | U | Number of unmatched Responses in |
1451 | | | | this "Block" item. |
1452 | | | | |
1453 | discarded-opcode | | U | Number of DNS messages processed |
1454 | | | | from the input traffic stream |
1455 | | | | during collection of data in this |
1456 | | | | "Block" item but not recorded |
1457 | | | | because their OPCODE is not in the |
1458 | | | | list to be collected. |
1459 | | | | |
1460 | malformed-items | | U | Number of malformed messages |
1461 | | | | processed from the input traffic |
1462 | | | | stream during collection of data in |
1463 | | | | this "Block" item. |
1464 +---------------------+---+---+-------------------------------------+
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482 Dickinson, et al. Standards Track [Page 27]
1483 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1484
1485
1486 7.3.2.3. "BlockTables"
1487
1488 Map of arrays containing data referenced by individual
1489 "QueryResponse" or "MalformedMessage" items in this "Block". Each
1490 element is an array that, if present, must not be empty.
1491
1492 An item in the "qlist" array contains indexes to values in the "qrr"
1493 array. Therefore, if "qlist" is present, "qrr" must also be present.
1494 Similarly, if "rrlist" is present, "rr" must also be present.
1495
1496 The map contains the following items:
1497
1498 +-------------------+---+---+---------------------------------------+
1499 | Field | M | T | Description |
1500 +-------------------+---+---+---------------------------------------+
1501 | ip-address | | A | Array of IP addresses, in network |
1502 | | | | byte order (of type byte string). If |
1503 | | | | client or server address prefixes are |
1504 | | | | set, only the address prefix bits are |
1505 | | | | stored. Each string is therefore up |
1506 | | | | to 4 bytes long for an IPv4 address, |
1507 | | | | or up to 16 bytes long for an IPv6 |
1508 | | | | address. See Section 7.3.1.1.1. |
1509 | | | | |
1510 | classtype | | A | Array of RR CLASS and TYPE |
1511 | | | | information. Type is "ClassType". |
1512 | | | | See Section 7.3.2.3.1. |
1513 | | | | |
1514 | name-rdata | | A | Array where each entry is the |
1515 | | | | contents of a single NAME or RDATA in |
1516 | | | | wire format (of type byte string). |
1517 | | | | Note that NAMEs, and labels within |
1518 | | | | RDATA contents, are full domain names |
1519 | | | | or labels; no name compression (per |
1520 | | | | [RFC1035]) is used on the individual |
1521 | | | | names/labels within the format. |
1522 | | | | |
1523 | qr-sig | | A | Array of Q/R data item signatures. |
1524 | | | | Type is "QueryResponseSignature". |
1525 | | | | See Section 7.3.2.3.2. |
1526 | | | | |
1527 | qlist | | A | Array of type "QuestionList". A |
1528 | | | | "QuestionList" is an array of |
1529 | | | | unsigned integers, indexes to |
1530 | | | | "Question" items in the "qrr" array. |
1531 | | | | |
1532
1533
1534
1535
1536
1537 Dickinson, et al. Standards Track [Page 28]
1538 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1539
1540
1541 | qrr | | A | Array of type "Question". Each entry |
1542 | | | | is the contents of a single Question, |
1543 | | | | where a Question is the second or |
1544 | | | | subsequent Question in a Query. See |
1545 | | | | Section 7.3.2.3.3. |
1546 | | | | |
1547 | rrlist | | A | Array of type "RRList". An "RRList" |
1548 | | | | is an array of unsigned integers, |
1549 | | | | indexes to "RR" items in the "rr" |
1550 | | | | array. |
1551 | | | | |
1552 | rr | | A | Array of type "RR". Each entry is |
1553 | | | | the contents of a single RR. See |
1554 | | | | Section 7.3.2.3.4. |
1555 | | | | |
1556 | malformed-message | | A | Array of the contents of malformed |
1557 | -data | | | messages. Array of type |
1558 | | | | "MalformedMessageData". See Section |
1559 | | | | 7.3.2.3.5. |
1560 +-------------------+---+---+---------------------------------------+
1561
1562 7.3.2.3.1. "ClassType"
1563
1564 RR CLASS and TYPE information. A map containing the following:
1565
1566 +-------+---+---+--------------------------+
1567 | Field | M | T | Description |
1568 +-------+---+---+--------------------------+
1569 | type | X | U | TYPE value [rrtypes]. |
1570 | | | | |
1571 | class | X | U | CLASS value [rrclasses]. |
1572 +-------+---+---+--------------------------+
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592 Dickinson, et al. Standards Track [Page 29]
1593 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1594
1595
1596 7.3.2.3.2. "QueryResponseSignature"
1597
1598 Elements of a Q/R data item that are often common between multiple
1599 individual Q/R data items. A map containing the following:
1600
1601 +--------------------+---+---+--------------------------------------+
1602 | Field | M | T | Description |
1603 +--------------------+---+---+--------------------------------------+
1604 | server-address | | U | The index in the "ip-address" array |
1605 | -index | | | of the server IP address. See |
1606 | | | | Section 7.3.2.3. |
1607 | | | | |
1608 | server-port | | U | The server port. |
1609 | | | | |
1610 | qr-transport-flags | C | U | Bit flags describing the transport |
1611 | | | | used to service the Query. Same |
1612 | | | | definition as "mm-transport-flags" |
1613 | | | | in Section 7.3.2.3.5, with an |
1614 | | | | additional indicator for trailing |
1615 | | | | bytes. See Appendix A. |
1616 | | | | Bit 0. IP version. 0 if IPv4, 1 if |
1617 | | | | IPv6. See Section 6.2.4. |
1618 | | | | Bits 1-4. Transport. 4-bit |
1619 | | | | unsigned value where |
1620 | | | | 0 = UDP [RFC1035] |
1621 | | | | 1 = TCP [RFC1035] |
1622 | | | | 2 = TLS [RFC7858] |
1623 | | | | 3 = DTLS [RFC8094] |
1624 | | | | 4 = HTTPS [RFC8484] |
1625 | | | | 15 = Non-standard transport (see |
1626 | | | | below) |
1627 | | | | Values 5-14 are reserved for future |
1628 | | | | use. |
1629 | | | | Bit 5. 1 if trailing bytes in Query |
1630 | | | | packet. See Section 11.2. |
1631 | | | | |
1632 | qr-type | | U | Type of Query/Response transaction |
1633 | | | | based on the definitions in the |
1634 | | | | dnstap schema [dnstap-schema]. |
1635 | | | | 0 = Stub. A transaction between a |
1636 | | | | stub resolver and a DNS server from |
1637 | | | | the perspective of the stub |
1638 | | | | resolver. |
1639 | | | | 1 = Client. A transaction between a |
1640 | | | | client and a DNS server (a proxy or |
1641 | | | | full recursive resolver) from the |
1642 | | | | perspective of the DNS server. |
1643
1644
1645
1646
1647 Dickinson, et al. Standards Track [Page 30]
1648 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1649
1650
1651 | | | | 2 = Resolver. A transaction between |
1652 | | | | a recursive resolver and an |
1653 | | | | authoritative server from the |
1654 | | | | perspective of the recursive |
1655 | | | | resolver. |
1656 | | | | 3 = Authoritative. A transaction |
1657 | | | | between a recursive resolver and an |
1658 | | | | authoritative server from the |
1659 | | | | perspective of the authoritative |
1660 | | | | server. |
1661 | | | | 4 = Forwarder. A transaction |
1662 | | | | between a downstream forwarder and |
1663 | | | | an upstream DNS server (a recursive |
1664 | | | | resolver) from the perspective of |
1665 | | | | the downstream forwarder. |
1666 | | | | 5 = Tool. A transaction between a |
1667 | | | | DNS software tool and a DNS server, |
1668 | | | | from the perspective of the tool. |
1669 | | | | |
1670 | qr-sig-flags | | U | Bit flags explicitly indicating |
1671 | | | | attributes of the message pair |
1672 | | | | represented by this Q/R data item |
1673 | | | | (not all attributes may be recorded |
1674 | | | | or deducible). |
1675 | | | | Bit 0. 1 if a Query was present. |
1676 | | | | Bit 1. 1 if a Response was present. |
1677 | | | | Bit 2. 1 if a Query was present and |
1678 | | | | it had an OPT RR. |
1679 | | | | Bit 3. 1 if a Response was present |
1680 | | | | and it had an OPT RR. |
1681 | | | | Bit 4. 1 if a Query was present but |
1682 | | | | had no Question. |
1683 | | | | Bit 5. 1 if a Response was present |
1684 | | | | but had no Question (only one |
1685 | | | | query-name-index is stored per Q/R |
1686 | | | | data item). |
1687 | | | | |
1688 | query-opcode | | U | Query OPCODE. |
1689 | | | | |
1690 | qr-dns-flags | | U | Bit flags with values from the Query |
1691 | | | | and Response DNS flags. Flag values |
1692 | | | | are 0 if the Query or Response is |
1693 | | | | not present. |
1694 | | | | Bit 0. Query Checking Disabled |
1695 | | | | (CD). |
1696 | | | | Bit 1. Query Authenticated Data |
1697 | | | | (AD). |
1698 | | | | Bit 2. Query reserved (Z). |
1699
1700
1701
1702 Dickinson, et al. Standards Track [Page 31]
1703 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1704
1705
1706 | | | | Bit 3. Query Recursion Available |
1707 | | | | (RA). |
1708 | | | | Bit 4. Query Recursion Desired |
1709 | | | | (RD). |
1710 | | | | Bit 5. Query TrunCation (TC). |
1711 | | | | Bit 6. Query Authoritative Answer |
1712 | | | | (AA). |
1713 | | | | Bit 7. Query DNSSEC answer OK (DO). |
1714 | | | | Bit 8. Response Checking Disabled |
1715 | | | | (CD). |
1716 | | | | Bit 9. Response Authenticated Data |
1717 | | | | (AD). |
1718 | | | | Bit 10. Response reserved (Z). |
1719 | | | | Bit 11. Response Recursion |
1720 | | | | Available (RA). |
1721 | | | | Bit 12. Response Recursion Desired |
1722 | | | | (RD). |
1723 | | | | Bit 13. Response TrunCation (TC). |
1724 | | | | Bit 14. Response Authoritative |
1725 | | | | Answer (AA). |
1726 | | | | |
1727 | query-rcode | | U | Query RCODE. If the Query contains |
1728 | | | | an OPT RR [RFC6891], this value |
1729 | | | | incorporates any EXTENDED-RCODE |
1730 | | | | value [rcodes]. |
1731 | | | | |
1732 | query-classtype | | U | The index in the "classtype" array |
1733 | -index | | | of the CLASS and TYPE of the first |
1734 | | | | Question. See Section 7.3.2.3. |
1735 | | | | |
1736 | query-qdcount | | U | The QDCOUNT in the Query, or |
1737 | | | | Response if no Query present. |
1738 | | | | |
1739 | query-ancount | | U | Query ANCOUNT. |
1740 | | | | |
1741 | query-nscount | | U | Query NSCOUNT. |
1742 | | | | |
1743 | query-arcount | | U | Query ARCOUNT. |
1744 | | | | |
1745 | query-edns-version | | U | The Query EDNS version. ("EDNS" |
1746 | | | | stands for Extension Mechanisms for |
1747 | | | | DNS.) |
1748 | | | | |
1749 | query-udp-size | | U | The Query EDNS sender's UDP payload |
1750 | | | | size. |
1751 | | | | |
1752
1753
1754
1755
1756
1757 Dickinson, et al. Standards Track [Page 32]
1758 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1759
1760
1761 | query-opt-rdata | | U | The index in the "name-rdata" array |
1762 | -index | | | of the OPT RDATA. See Section |
1763 | | | | 7.3.2.3. |
1764 | | | | |
1765 | response-rcode | | U | Response RCODE. If the Response |
1766 | | | | contains an OPT RR [RFC6891], this |
1767 | | | | value incorporates any EXTENDED- |
1768 | | | | RCODE value [rcodes]. |
1769 +--------------------+---+---+--------------------------------------+
1770
1771 Version 1.0 of C-DNS supports transport values corresponding to DNS
1772 transports defined in IETF Standards Track documents at the time of
1773 writing. There are numerous non-standard methods of sending DNS
1774 messages over various transports using a variety of protocols, but
1775 they are out of scope for this document. With the current
1776 specification, these can be generically stored using value 15
1777 (Non-standard transport), or implementations are free to use the
1778 negative integer map keys to define their own mappings. Such
1779 non-standard transports may also be the subject of a future extension
1780 to the specification.
1781
1782 7.3.2.3.3. "Question"
1783
1784 Details on individual Questions in a Question section. A map
1785 containing the following:
1786
1787 +-----------------+---+---+-----------------------------------------+
1788 | Field | M | T | Description |
1789 +-----------------+---+---+-----------------------------------------+
1790 | name-index | X | U | The index in the "name-rdata" array of |
1791 | | | | the QNAME. See Section 7.3.2.3. |
1792 | | | | |
1793 | classtype-index | X | U | The index in the "classtype" array of |
1794 | | | | the CLASS and TYPE of the Question. |
1795 | | | | See Section 7.3.2.3. |
1796 +-----------------+---+---+-----------------------------------------+
1797
1798
1799
1800
1801
1802
1803
1804
1805
1806
1807
1808
1809
1810
1811
1812 Dickinson, et al. Standards Track [Page 33]
1813 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1814
1815
1816 7.3.2.3.4. "RR"
1817
1818 Details on individual RRs in RR sections. A map containing the
1819 following:
1820
1821 +-----------------+---+---+-----------------------------------------+
1822 | Field | M | T | Description |
1823 +-----------------+---+---+-----------------------------------------+
1824 | name-index | X | U | The index in the "name-rdata" array of |
1825 | | | | the NAME. See Section 7.3.2.3. |
1826 | | | | |
1827 | classtype-index | X | U | The index in the "classtype" array of |
1828 | | | | the CLASS and TYPE of the RR. See |
1829 | | | | Section 7.3.2.3. |
1830 | | | | |
1831 | ttl | | U | The RR Time to Live. |
1832 | | | | |
1833 | rdata-index | | U | The index in the "name-rdata" array of |
1834 | | | | the RR RDATA. See Section 7.3.2.3. |
1835 +-----------------+---+---+-----------------------------------------+
1836
1837 7.3.2.3.5. "MalformedMessageData"
1838
1839 Details on malformed DNS messages stored in this "Block" item. A map
1840 containing the following:
1841
1842 +--------------------+---+---+--------------------------------------+
1843 | Field | M | T | Description |
1844 +--------------------+---+---+--------------------------------------+
1845 | server-address | | U | The index in the "ip-address" array |
1846 | -index | | | of the server IP address. See |
1847 | | | | Section 7.3.2.3. |
1848 | | | | |
1849 | server-port | | U | The server port. |
1850 | | | | |
1851 | mm-transport-flags | C | U | Bit flags describing the transport |
1852 | | | | used to service the Query. See |
1853 | | | | Section 6.2.4. |
1854 | | | | Bits 1-4. Transport. 4-bit |
1855 | | | | unsigned value where |
1856 | | | | 0 = UDP [RFC1035] |
1857 | | | | 1 = TCP [RFC1035] |
1858 | | | | 2 = TLS [RFC7858] |
1859 | | | | 3 = DTLS [RFC8094] |
1860 | | | | 4 = HTTPS [RFC8484] |
1861 | | | | 15 = Non-standard transport |
1862 | | | | Values 5-14 are reserved for future |
1863 | | | | use. |
1864
1865
1866
1867 Dickinson, et al. Standards Track [Page 34]
1868 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1869
1870
1871 | | | | |
1872 | mm-payload | | S | The payload (raw bytes) of the DNS |
1873 | | | | message. |
1874 +--------------------+---+---+--------------------------------------+
1875
1876 7.3.2.4. "QueryResponse"
1877
1878 Details on individual Q/R data items.
1879
1880 Note that there is no requirement that the elements of the
1881 "query-responses" array are presented in strict chronological order.
1882
1883 A map containing the following items:
1884
1885 +----------------------+---+---+------------------------------------+
1886 | Field | M | T | Description |
1887 +----------------------+---+---+------------------------------------+
1888 | time-offset | | U | Q/R timestamp as an offset in |
1889 | | | | ticks (see Section 7.3.1.1.1) from |
1890 | | | | "earliest-time". The timestamp is |
1891 | | | | the timestamp of the Query, or the |
1892 | | | | Response if there is no Query. |
1893 | | | | |
1894 | client-address-index | | U | The index in the "ip-address" |
1895 | | | | array of the client IP address. |
1896 | | | | See Section 7.3.2.3. |
1897 | | | | |
1898 | client-port | | U | The client port. |
1899 | | | | |
1900 | transaction-id | | U | DNS transaction identifier. |
1901 | | | | |
1902 | qr-signature-index | | U | The index in the "qr-sig" array of |
1903 | | | | the "QueryResponseSignature" item. |
1904 | | | | See Section 7.3.2.3. |
1905 | | | | |
1906 | client-hoplimit | | U | The IPv4 TTL or IPv6 Hoplimit from |
1907 | | | | the Query packet. |
1908 | | | | |
1909 | response-delay | | I | The time difference between Query |
1910 | | | | and Response, in ticks. See |
1911 | | | | Section 7.3.1.1.1. Only present |
1912 | | | | if there is a Query and a |
1913 | | | | Response. The delay can be |
1914 | | | | negative if the network |
1915 | | | | stack/capture library returns |
1916 | | | | packets out of order. |
1917 | | | | |
1918
1919
1920
1921
1922 Dickinson, et al. Standards Track [Page 35]
1923 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1924
1925
1926 | query-name-index | | U | The index in the "name-rdata" |
1927 | | | | array of the item containing the |
1928 | | | | QNAME for the first Question. See |
1929 | | | | Section 7.3.2.3. |
1930 | | | | |
1931 | query-size | | U | DNS Query message size (see |
1932 | | | | below). |
1933 | | | | |
1934 | response-size | | U | DNS Response message size (see |
1935 | | | | below). |
1936 | | | | |
1937 | response-processing | | M | Data on Response processing. Map |
1938 | -data | | | of type "ResponseProcessingData". |
1939 | | | | See Section 7.3.2.4.1. |
1940 | | | | |
1941 | query-extended | | M | Extended Query data. Map of type |
1942 | | | | "QueryResponseExtended". See |
1943 | | | | Section 7.3.2.4.2. |
1944 | | | | |
1945 | response-extended | | M | Extended Response data. Map of |
1946 | | | | type "QueryResponseExtended". See |
1947 | | | | Section 7.3.2.4.2. |
1948 +----------------------+---+---+------------------------------------+
1949
1950 The "query-size" and "response-size" fields hold the DNS message
1951 size. For UDP, this is the size of the UDP payload that contained
1952 the DNS message. For TCP, it is the size of the DNS message as
1953 specified in the two-byte message length header. Trailing bytes in
1954 UDP Queries are routinely observed in traffic to authoritative
1955 servers, and this value allows a calculation of how many trailing
1956 bytes were present.
1957
1958 7.3.2.4.1. "ResponseProcessingData"
1959
1960 Information on the server processing that produced the Response. A
1961 map containing the following:
1962
1963 +------------------+---+---+----------------------------------------+
1964 | Field | M | T | Description |
1965 +------------------+---+---+----------------------------------------+
1966 | bailiwick-index | | U | The index in the "name-rdata" array of |
1967 | | | | the owner name for the Response |
1968 | | | | bailiwick. See Section 7.3.2.3. |
1969 | | | | |
1970 | processing-flags | | U | Flags relating to Response processing. |
1971 | | | | Bit 0. 1 if the Response came from |
1972 | | | | cache. |
1973 +------------------+---+---+----------------------------------------+
1974
1975
1976
1977 Dickinson, et al. Standards Track [Page 36]
1978 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
1979
1980
1981 7.3.2.4.2. "QueryResponseExtended"
1982
1983 Extended data on the Q/R data item.
1984
1985 Each item in the map is present only if collection of the relevant
1986 details is configured.
1987
1988 A map containing the following items:
1989
1990 +------------------+---+---+----------------------------------------+
1991 | Field | M | T | Description |
1992 +------------------+---+---+----------------------------------------+
1993 | question-index | | U | The index in the "qlist" array of the |
1994 | | | | entry listing any second and |
1995 | | | | subsequent Questions in the Question |
1996 | | | | section for the Query or Response. |
1997 | | | | See Section 7.3.2.3. |
1998 | | | | |
1999 | answer-index | | U | The index in the "rrlist" array of the |
2000 | | | | entry listing the Answer RR sections |
2001 | | | | for the Query or Response. See |
2002 | | | | Section 7.3.2.3. |
2003 | | | | |
2004 | authority-index | | U | The index in the "rrlist" array of the |
2005 | | | | entry listing the Authority RR |
2006 | | | | sections for the Query or Response. |
2007 | | | | See Section 7.3.2.3. |
2008 | | | | |
2009 | additional-index | | U | The index in the "rrlist" array of the |
2010 | | | | entry listing the Additional RR |
2011 | | | | sections for the Query or Response. |
2012 | | | | See Section 7.3.2.3. Note that Query |
2013 | | | | OPT RR data can optionally be stored |
2014 | | | | in the QuerySignature. |
2015 +------------------+---+---+----------------------------------------+
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
2030
2031
2032 Dickinson, et al. Standards Track [Page 37]
2033 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2034
2035
2036 7.3.2.5. "AddressEventCount"
2037
2038 Counts of various IP-related events relating to traffic with
2039 individual client addresses. A map containing the following:
2040
2041 +--------------------+---+---+--------------------------------------+
2042 | Field | M | T | Description |
2043 +--------------------+---+---+--------------------------------------+
2044 | ae-type | X | U | The type of event. The following |
2045 | | | | event types are currently defined: |
2046 | | | | 0. TCP reset. |
2047 | | | | 1. ICMP time exceeded. |
2048 | | | | 2. ICMP destination unreachable. |
2049 | | | | 3. ICMPv6 time exceeded. |
2050 | | | | 4. ICMPv6 destination unreachable. |
2051 | | | | 5. ICMPv6 packet too big. |
2052 | | | | |
2053 | ae-code | | U | A code relating to the event. For |
2054 | | | | ICMP or ICMPv6 events, this MUST be |
2055 | | | | the ICMP [RFC792] or ICMPv6 |
2056 | | | | [RFC4443] code. For other events, |
2057 | | | | the contents are undefined. |
2058 | | | | |
2059 | ae-transport-flags | C | U | Bit flags describing the transport |
2060 | | | | used to service the event. See |
2061 | | | | Section 6.2.4. |
2062 | | | | Bit 0. IP version. 0 if IPv4, 1 if |
2063 | | | | IPv6. |
2064 | | | | Bits 1-4. Transport. 4-bit |
2065 | | | | unsigned value where |
2066 | | | | 0 = UDP [RFC1035] |
2067 | | | | 1 = TCP [RFC1035] |
2068 | | | | 2 = TLS [RFC7858] |
2069 | | | | 3 = DTLS [RFC8094] |
2070 | | | | 4 = HTTPS [RFC8484] |
2071 | | | | 15 = Non-standard transport |
2072 | | | | Values 5-14 are reserved for future |
2073 | | | | use. |
2074 | | | | |
2075 | ae-address-index | X | U | The index in the "ip-address" array |
2076 | | | | of the client address. See Section |
2077 | | | | 7.3.2.3. |
2078 | | | | |
2079 | ae-count | X | U | The number of occurrences of this |
2080 | | | | event during the Block collection |
2081 | | | | period. |
2082 +--------------------+---+---+--------------------------------------+
2083
2084
2085
2086
2087 Dickinson, et al. Standards Track [Page 38]
2088 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2089
2090
2091 7.3.2.6. "MalformedMessage"
2092
2093 Details on Malformed Message data items. A map containing the
2094 following:
2095
2096 +----------------------+---+---+------------------------------------+
2097 | Field | M | T | Description |
2098 +----------------------+---+---+------------------------------------+
2099 | time-offset | | U | Message timestamp as an offset in |
2100 | | | | ticks (see Section 7.3.1.1.1) from |
2101 | | | | "earliest-time". |
2102 | | | | |
2103 | client-address-index | | U | The index in the "ip-address" |
2104 | | | | array of the client IP address. |
2105 | | | | See Section 7.3.2.3. |
2106 | | | | |
2107 | client-port | | U | The client port. |
2108 | | | | |
2109 | message-data-index | | U | The index in the "malformed- |
2110 | | | | message-data" array of the message |
2111 | | | | data for this message. See |
2112 | | | | Section 7.3.2.3. |
2113 +----------------------+---+---+------------------------------------+
2114
2115 8. Versioning
2116
2117 The C-DNS File Preamble includes a file Format Version; a major and
2118 minor version number are required fields. This document defines
2119 version 1.0 of the C-DNS specification. This section describes the
2120 intended use of these version numbers in future specifications.
2121
2122 It is noted that version 1.0 includes many optional fields;
2123 therefore, consumers of version 1.0 should be inherently robust to
2124 parsing files with variable data content.
2125
2126 Within a major version, a new minor version MUST be a strict superset
2127 of the previous minor version, with no semantic changes to existing
2128 fields. New keys MAY be added to existing maps, and new maps MAY be
2129 added. A consumer capable of reading a particular major.minor
2130 version MUST also be capable of reading all previous minor versions
2131 of the same major version. It SHOULD also be capable of parsing all
2132 subsequent minor versions, ignoring any keys or maps that it does not
2133 recognize.
2134
2135
2136
2137
2138
2139
2140
2141
2142 Dickinson, et al. Standards Track [Page 39]
2143 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2144
2145
2146 A new major version indicates changes to the format that are not
2147 backwards compatible with previous major versions. A consumer
2148 capable of only reading a particular major version (greater than 1)
2149 is neither required nor expected to be capable of reading a previous
2150 major version.
2151
2152 9. C-DNS to PCAP
2153
2154 It is usually possible to reconstruct PCAP files from the C-DNS
2155 format in a lossy fashion. Some of the issues with reconstructing
2156 both the DNS payload and the full packet stream are outlined here.
2157
2158 The reconstruction of well-formed DNS messages depends on two
2159 factors:
2160
2161 1. Whether or not a particular subset of the optional fields were
2162 captured in the C-DNS file, specifically the data fields
2163 necessary to reconstruct a valid IP header and DNS payload for
2164 both Query and Response (see Appendix D.1). Clearly, if not all
2165 these data fields were captured, the reconstruction is likely to
2166 be imperfect even if reasonable defaults are provided for the
2167 reconstruction.
2168
2169 2. Whether or not at least one field was captured that unambiguously
2170 identifies the Query/Response data item as containing just a
2171 Query, just a Response, or a Query/Response pair. Obviously, the
2172 qr-sig-flags defined in Section 7.3.2.3.2 is such a field;
2173 however, this field is optional. For more details, see
2174 Appendix D.2.
2175
2176 It is noted again that simply having hints that indicate that certain
2177 data fields were not omitted does not guarantee that those data
2178 fields were actually captured. Therefore, the ability to reconstruct
2179 PCAP data (in the absence of defaults) can in principle vary for each
2180 record captured in a C-DNS file, and between Blocks that have
2181 differing hints.
2182
2183 Even if all sections of the Response were captured, one cannot
2184 reconstruct the DNS Response payload exactly, due to the fact that
2185 some DNS names in the message on the wire may have been compressed.
2186 Section 9.1 discusses this in more detail.
2187
2188
2189
2190
2191
2192
2193
2194
2195
2196
2197 Dickinson, et al. Standards Track [Page 40]
2198 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2199
2200
2201 Some transport information is not captured in the C-DNS format. For
2202 example, the following aspects of the original packet stream cannot
2203 be reconstructed from the C-DNS format:
2204
2205 o IP fragmentation
2206
2207 o TCP stream information:
2208
2209 * Multiple DNS messages may have been sent in a single TCP
2210 segment
2211
2212 * A DNS payload may have been split across multiple TCP segments
2213
2214 * Multiple DNS messages may have been sent on a single TCP
2215 session
2216
2217 o TLS session information:
2218
2219 * TLS version or cipher suites
2220
2221 * TLS-related features such as TCP Fast Open (TFO) [RFC7413] or
2222 TLS session resumption [RFC5077]
2223
2224 o DNS-over-HTTPS [RFC8484] message details:
2225
2226 * Whether the message used POST or GET
2227
2228 * HTTPS Headers
2229
2230 o Malformed DNS messages if the wire format is not recorded
2231
2232 o Any non-DNS messages that were in the original packet stream,
2233 e.g., ICMP
2234
2235 Simple assumptions can be made on the reconstruction: fragmented and
2236 DNS-over-TCP messages can be reconstructed into single packets, and a
2237 single TCP session can be constructed for each TCP packet.
2238
2239 Additionally, if malformed messages and non-DNS packets are captured
2240 separately, they can be merged with packet captures reconstructed
2241 from C-DNS to produce a more complete packet stream.
2242
2243
2244
2245
2246
2247
2248
2249
2250
2251
2252 Dickinson, et al. Standards Track [Page 41]
2253 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2254
2255
2256 9.1. Name Compression
2257
2258 All the names stored in the C-DNS format are full domain names; no
2259 name compression (per [RFC1035]) is used on the individual names
2260 within the format. Therefore, when reconstructing a packet, name
2261 compression must be used in order to reproduce the on-the-wire
2262 representation of the packet.
2263
2264 Name compression per [RFC1035] works by substituting trailing
2265 sections of a name with a reference back to the occurrence of those
2266 sections earlier in the message. Not all name server software uses
2267 the same algorithm when compressing domain names within the
2268 Responses. Some attempt maximum recompression at the expense of
2269 runtime resources, others use heuristics to balance compression and
2270 speed, and others use different rules for what is a valid compression
2271 target.
2272
2273 This means that Responses to the same Query from different name
2274 server software that match in terms of DNS payload content (header,
2275 counts, RRs with name compression removed) do not necessarily match
2276 byte for byte on the wire.
2277
2278 Therefore, it is not possible to ensure that the DNS Response payload
2279 is reconstructed byte for byte from C-DNS data. However, it can at
2280 least, in principle, be reconstructed to have the correct payload
2281 length (since the original Response length is captured) if there is
2282 enough knowledge of the commonly implemented name compression
2283 algorithms. For example, a simplistic approach would be to try each
2284 algorithm in turn to see if it reproduces the original length,
2285 stopping at the first match. This would not guarantee that the
2286 correct algorithm has been used, as it is possible to match the
2287 length whilst still not matching the on-the-wire bytes; however,
2288 without further information added to the C-DNS data, this is the best
2289 that can be achieved.
2290
2291 Appendix B presents an example of two different compression
2292 algorithms used by well-known name server software.
2293
2294 10. Data Collection
2295
2296 This section describes a non-normative proposed algorithm for the
2297 processing of a captured stream of DNS Queries and Responses and
2298 production of a stream of Q/R data items, matching Queries and
2299 Responses where possible.
2300
2301
2302
2303
2304
2305
2306
2307 Dickinson, et al. Standards Track [Page 42]
2308 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2309
2310
2311 For the purposes of this discussion, it is assumed that the input has
2312 been preprocessed such that:
2313
2314 1. All IP fragmentation reassembly, TCP stream reassembly, and
2315 so on, have already been performed.
2316
2317 2. Each message is associated with transport metadata required to
2318 generate the Primary ID (see Section 10.2.1).
2319
2320 3. Each message has a well-formed DNS Header of 12 bytes, and (if
2321 present) the first Question in the Question section can be parsed
2322 to generate the Secondary ID (see below). As noted earlier, this
2323 requirement can result in a malformed Query being removed in the
2324 preprocessing stage, but the correctly formed Response with RCODE
2325 of FORMERR being present.
2326
2327 DNS messages are processed in the order they are delivered to the
2328 implementation.
2329
2330 It should be noted that packet capture libraries do not necessarily
2331 provide packets in strict chronological order. This can, for
2332 example, arise on multi-core platforms where packets arriving at a
2333 network device are processed by different cores. On systems where
2334 this behavior has been observed, the timestamps associated with each
2335 packet are consistent; Queries always have a timestamp prior to the
2336 Response timestamp. However, the order in which these packets appear
2337 in the packet capture stream is not necessarily strictly
2338 chronological; a Response can appear in the capture stream before the
2339 Query that provoked the Response. For this discussion, this
2340 non-chronological delivery is termed "skew".
2341
2342 In the presence of skew, Response packets can arrive for matching
2343 before the corresponding Query. To avoid generating false instances
2344 of Responses without a matching Query, and Queries without a matching
2345 Response, the matching algorithm must take the possibility of skew
2346 into account.
2347
2348 10.1. Matching Algorithm
2349
2350 A schematic representation of the algorithm for matching Q/R data
2351 items is shown in Figure 3. It takes individual DNS Query or
2352 Response messages as input, and it outputs matched Q/R data items.
2353 The numbers in the figure identify matching operations listed in
2354 Table 1. Specific details of the algorithm -- for example, queues,
2355 timers, and identifiers -- are given in the following sections.
2356
2357
2358
2359
2360
2361
2362 Dickinson, et al. Standards Track [Page 43]
2363 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2364
2365
2366 .----------------------.
2367 | Process next message |<------------------+
2368 `----------------------' |
2369 | |
2370 +------------------------------+ |
2371 | Generate message identifiers | |
2372 +------------------------------+ |
2373 | |
2374 Response | Query |
2375 +--------------< >---------------+ |
2376 | | |
2377 +--------------------+ +--------------------+ |
2378 | Find earliest QR | | Create QR item (2) | |
2379 | item in OFIFO (1) | +--------------------+ |
2380 +--------------------+ | |
2381 | +---------------+ |
2382 Match | No match | Append new QR | |
2383 +--------< >------+ | item to OFIFO | |
2384 | | +---------------+ |
2385 +-----------+ +--------+ | |
2386 | Update QR | | Add to | +-------------------+ |
2387 | item (3) | | RFIFO | | Find earliest QR | |
2388 +-----------+ +--------+ | item in RFIFO (1) | |
2389 | | +-------------------+ |
2390 +-----------------+ | |
2391 | | |
2392 | +----------------+ Match | No match |
2393 | | Remove R |-------< >-----+ |
2394 | | from RFIFO (3) | | |
2395 | +----------------+ | |
2396 | | | |
2397 +--------------+-----------------------+ |
2398 | |
2399 +----------------------------------------------+ |
2400 | Update all timed-out (QT) OFIFO QR items (4) | |
2401 +----------------------------------------------+ |
2402 | |
2403 +--------------------------------+ |
2404 | Remove all timed-out (ST) R | |
2405 | from RFIFO, create QR item (5) | |
2406 +--------------------------------+ |
2407 ____________________|_______________________ |
2408 / / |
2409 / Remove all consecutive done entries from /-------+
2410 / front of OFIFO for further processing /
2411 /____________________________________________/
2412
2413
2414
2415
2416
2417 Dickinson, et al. Standards Track [Page 44]
2418 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2419
2420
2421 OFIFO = output FIFO containing Q/R data items (Section 10.6)
2422 RFIFO = Response FIFO containing unmatched Response items
2423 (Section 10.6)
2424 QT = Query Timeout (Section 10.3)
2425 ST = Skew Timeout (Section 10.3)
2426
2427 Figure 3: Query/Response Matching Algorithm
2428
2429 +-----------+-------------------------------------------+
2430 | Reference | Operation |
2431 +-----------+-------------------------------------------+
2432 | (1) | Find earliest QR item in FIFO where: |
2433 | | * QR.done = false |
2434 | | * QR.Q.PrimaryID == R.PrimaryID |
2435 | | and, if both QR.Q and R have SecondaryID: |
2436 | | * QR.Q.SecondaryID == R.SecondaryID |
2437 | | |
2438 | (2) | Set: |
2439 | | QR.Q := Q |
2440 | | QR.R := nil |
2441 | | QR.done := false |
2442 | | |
2443 | (3) | Set: |
2444 | | QR.R := R |
2445 | | QR.done := true |
2446 | | |
2447 | (4) | Set: |
2448 | | QR.done := true |
2449 | | |
2450 | (5) | Set: |
2451 | | QR.Q := nil |
2452 | | QR.R := R |
2453 | | QR.done := true |
2454 +-----------+-------------------------------------------+
2455
2456 Table 1: Operations Used in the Matching Algorithm
2457
2458 10.2. Message Identifiers
2459
2460 10.2.1. Primary ID (Required)
2461
2462 A Primary ID is constructed for each message. It is composed of the
2463 following data:
2464
2465 1. Source IP Address
2466
2467 2. Destination IP Address
2468
2469
2470
2471
2472 Dickinson, et al. Standards Track [Page 45]
2473 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2474
2475
2476 3. Source Port
2477
2478 4. Destination Port
2479
2480 5. Transport
2481
2482 6. DNS Message ID
2483
2484 10.2.2. Secondary ID (Optional)
2485
2486 If present, the first Question in the Question section is used as a
2487 Secondary ID for each message. Note that there may be well-formed
2488 DNS Queries that have a QDCOUNT of 0, and some Responses may have a
2489 QDCOUNT of 0 (for example, Responses with RCODE=FORMERR or NOTIMP).
2490 In this case, the Secondary ID is not used in matching.
2491
2492 10.3. Algorithm Parameters
2493
2494 1. Query Timeout (QT). A Query arrives with timestamp t1. If no
2495 Response matching that Query has arrived before other input
2496 arrives timestamped later than (t1 + QT), a Q/R data item
2497 containing only a Query is recorded. The QT value is typically
2498 on the order of 5 seconds.
2499
2500 2. Skew Timeout (ST). A Response arrives with timestamp t2. If a
2501 Response has not been matched by a Query before input arrives
2502 timestamped later than (t2 + ST), a Q/R data item containing only
2503 a Response is recorded. The ST value is typically a few
2504 microseconds.
2505
2506 10.4. Algorithm Requirements
2507
2508 The algorithm is designed to handle the following input data:
2509
2510 1. Multiple Queries with the same Primary ID (but different
2511 Secondary ID) arriving before any Responses for these Queries
2512 are seen.
2513
2514 2. Multiple Queries with the same Primary and Secondary ID arriving
2515 before any Responses for these Queries are seen.
2516
2517 3. Queries for which no later Response can be found within the
2518 specified timeout.
2519
2520 4. Responses for which no previous Query can be found within the
2521 specified timeout.
2522
2523
2524
2525
2526
2527 Dickinson, et al. Standards Track [Page 46]
2528 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2529
2530
2531 10.5. Algorithm Limitations
2532
2533 For cases 1 and 2 listed in the above requirements, it is not
2534 possible to unambiguously match Queries with Responses. This
2535 algorithm chooses to match to the earliest Query with the correct
2536 Primary and Secondary ID.
2537
2538 10.6. Workspace
2539
2540 The algorithm employs two FIFO queues:
2541
2542 o OFIFO: an output FIFO containing Q/R data items in chronological
2543 order.
2544
2545 o RFIFO: a FIFO holding Responses without a matching Query in order
2546 of arrival.
2547
2548 10.7. Output
2549
2550 The output is a list of Q/R data items. Both the Query and Response
2551 elements are optional in these items; therefore, Q/R data items have
2552 one of three types of content:
2553
2554 1. A matched pair of Query and Response messages
2555
2556 2. A Query message with no Response
2557
2558 3. A Response message with no Query
2559
2560 The timestamp of a list item is that of the Query for cases 1 and 2
2561 and that of the Response for case 3.
2562
2563 10.8. Post-Processing
2564
2565 When ending a capture, all items in the RFIFO are timed out
2566 immediately, generating Response only entries to the OFIFO. These
2567 and all other remaining entries in the OFIFO should be treated as
2568 timed-out Queries.
2569
2570 11. Implementation Guidance
2571
2572 Whilst this document makes no specific recommendations with respect
2573 to "Canonical CBOR" (see Section 3.9 of [RFC7049]), the following
2574 guidance may be of use to implementers.
2575
2576 Adherence to the first two rules given in Section 3.9 of [RFC7049]
2577 will minimize file sizes.
2578
2579
2580
2581
2582 Dickinson, et al. Standards Track [Page 47]
2583 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2584
2585
2586 Adherence to the last two rules given in Section 3.9 of [RFC7049] for
2587 all maps and arrays would unacceptably constrain implementations --
2588 for example, in the use case of real-time data collection in
2589 constrained environments where outputting Block Tables after Q/R data
2590 items and allowing indefinite-length maps and arrays could reduce
2591 memory requirements.
2592
2593 It is recommended that implementations that have fundamental
2594 restrictions on what data fields they can collect SHOULD always store
2595 hints with the bits unset for those fields, i.e., they unambiguously
2596 indicate that those data fields will be omitted from captured C-DNS.
2597
2598 11.1. Optional Data
2599
2600 When decoding C-DNS data, some of the items required for a particular
2601 function that the consumer wishes to perform may be missing.
2602 Consumers should consider providing configurable default values to be
2603 used in place of the missing values in their output.
2604
2605 11.2. Trailing Bytes
2606
2607 A DNS Query message in a UDP or TCP payload can be followed by some
2608 additional (spurious) bytes, which are not stored in C-DNS.
2609
2610 When DNS traffic is sent over TCP, each message is prefixed with a
2611 two-byte length field, which gives the message length, excluding the
2612 two-byte length field. In this context, trailing bytes can occur in
2613 two circumstances, with different results:
2614
2615 1. The number of bytes consumed by fully parsing the message is less
2616 than the number of bytes given in the length field (i.e., the
2617 length field is incorrect and too large). In this case, the
2618 surplus bytes are considered trailing bytes in a manner analogous
2619 to UDP and recorded as such. If only this case occurs, it is
2620 possible to process a packet containing multiple DNS messages
2621 where one or more have trailing bytes.
2622
2623 2. There are surplus bytes between the end of a well-formed message
2624 and the start of the length field for the next message. In this
2625 case, the first of the surplus bytes will be processed as the
2626 first byte of the next length field, and parsing will proceed
2627 from there, almost certainly leading to the next and any
2628 subsequent messages in the packet being considered malformed.
2629 This will not generate a trailing-bytes record for the processed
2630 well-formed message.
2631
2632
2633
2634
2635
2636
2637 Dickinson, et al. Standards Track [Page 48]
2638 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2639
2640
2641 11.3. Limiting Collection of RDATA
2642
2643 Implementations should consider providing a configurable maximum
2644 RDATA size for captures -- for example, to avoid memory issues when
2645 confronted with large zone transfer records.
2646
2647 11.4. Timestamps
2648
2649 The preamble to each block includes a timestamp of the earliest
2650 record in the Block. As described in Section 7.3.2.1, the timestamp
2651 is an array of two unsigned integers. The first is a POSIX "time_t"
2652 [posix-time]. Consumers of C-DNS should be aware of this, as it
2653 excludes leap seconds and therefore may cause minor anomalies in the
2654 data, e.g., when calculating Query throughput.
2655
The IETF is responsible for the creation and maintenance of the DNS RFCs. The ICANN DNS RFC annotation project provides a forum for collecting community annotations on these RFCs as an aid to understanding for implementers and any interested parties. The annotations displayed here are not the result of the IETF consensus process.
This RFC is included in the DNS RFCs annotation project whose home page is here.
2656 12. IANA Considerations
2657
2658 IANA has created a registry "C-DNS DNS Capture Format" containing the
2659 subregistries defined in Sections 12.1 to 12.4 inclusive.
2660
2661 In all cases, new entries may be added to the subregistries by Expert
2662 Review as defined in [RFC8126]. Experts are expected to exercise
2663 their own expert judgment and should consider the following general
2664 guidelines in addition to any provided guidelines that are particular
2665 to a subregistry.
2666
2667 o There should be a real and compelling use for any new value.
2668
2669 o Values assigned should be carefully chosen to minimize storage
2670 requirements for common cases.
2671
2672 12.1. Transport Types
2673
2674 IANA has created a registry "C-DNS Transports" of C-DNS transport
2675 type identifiers. The primary purpose of this registry is to provide
2676 unique identifiers for all transports used for DNS Queries.
2677
2678 The following note is included in this registry: "In version 1.0 of
2679 C-DNS [RFC8618], there is a field to identify the type of DNS
2680 transport. This field is 4 bits in size."
2681
2682
2683
2684
2685
2686
2687
2688
2689
2690
2691
2692 Dickinson, et al. Standards Track [Page 49]
2693 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2694
2695
2696 The initial contents of the registry are as follows. See
2697 Sections 7.3.2.3.2, 7.3.2.3.5, and 7.3.2.5 of this document:
2698
2699 +------------+------------------------+-----------+
2700 | Identifier | Name | Reference |
2701 +------------+------------------------+-----------+
2702 | 0 | UDP | RFC 8618 |
2703 | 1 | TCP | RFC 8618 |
2704 | 2 | TLS | RFC 8618 |
2705 | 3 | DTLS | RFC 8618 |
2706 | 4 | HTTPS | RFC 8618 |
2707 | 5-14 | Unassigned | |
2708 | 15 | Non-standard transport | RFC 8618 |
2709 +------------+------------------------+-----------+
2710
2711 Expert reviewers should take the following point into consideration:
2712 Is the requested DNS transport described by a Standards Track RFC?
2713
2714 12.2. Data Storage Flags
2715
2716 IANA has created a registry "C-DNS Storage Flags" of C-DNS data
2717 storage flags. The primary purpose of this registry is to provide
2718 indicators giving hints on processing of the data stored.
2719
2720 The following note is included in this registry: "In version 1.0 of
2721 C-DNS [RFC8618], there is a field describing attributes of the data
2722 recorded. The field is a CBOR [RFC7049] unsigned integer holding bit
2723 flags."
2724
2725 The initial contents of the registry are as follows. See
2726 Section 7.3.1.1.1 of this document:
2727
2728 +------+------------------+-----------------------------+-----------+
2729 | Bit | Name | Description | Reference |
2730 +------+------------------+-----------------------------+-----------+
2731 | 0 | anonymized-data | The data has been | RFC 8618 |
2732 | | | anonymized. | |
2733 | | | | |
2734 | 1 | sampled-data | The data is sampled data. | RFC 8618 |
2735 | | | | |
2736 | 2 | normalized-names | Names in the data have been | RFC 8618 |
2737 | | | normalized. | |
2738 | | | | |
2739 | 3-63 | Unassigned | | |
2740 +------+------------------+-----------------------------+-----------+
2741
2742
2743
2744
2745
2746
2747 Dickinson, et al. Standards Track [Page 50]
2748 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2749
2750
2751 12.3. Response-Processing Flags
2752
2753 IANA has created a registry "C-DNS Response Flags" of C-DNS response-
2754 processing flags. The primary purpose of this registry is to provide
2755 indicators giving hints on the generation of a particular Response.
2756
2757 The following note is included in this registry: "In version 1.0 of
2758 C-DNS [RFC8618], there is a field describing attributes of the
2759 Responses recorded. The field is a CBOR [RFC7049] unsigned integer
2760 holding bit flags."
2761
2762 The initial contents of the registry are as follows. See
2763 Section 7.3.2.4.1 of this document:
2764
2765 +------+------------+-------------------------------+-----------+
2766 | Bit | Name | Description | Reference |
2767 +------+------------+-------------------------------+-----------+
2768 | 0 | from-cache | The Response came from cache. | RFC 8618 |
2769 | 1-63 | Unassigned | | |
2770 +------+------------+-------------------------------+-----------+
2771
2772 12.4. AddressEvent Types
2773
2774 IANA has created a registry "C-DNS Address Event Types" of C-DNS
2775 AddressEvent types. The primary purpose of this registry is to
2776 provide unique identifiers of different types of C-DNS address events
2777 and so specify the contents of the optional companion field "ae-code"
2778 for each type.
2779
2780 The following note is included in this registry: "In version 1.0 of
2781 C-DNS [RFC8618], there is a field identifying types of the events
2782 related to client addresses. This field is a CBOR [RFC7049] unsigned
2783 integer. There is a related optional field "ae-code", which, if
2784 present, holds an additional CBOR unsigned integer giving additional
2785 information specific to the event type."
2786
2787
2788
2789
2790
2791
2792
2793
2794
2795
2796
2797
2798
2799
2800
2801
2802 Dickinson, et al. Standards Track [Page 51]
2803 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2804
2805
2806 The initial contents of the registry are as follows. See
2807 Section 7.3.2.5 of this document:
2808
2809 +------------------------+---------------+--------------+-----------+
2810 | Identifier | Event Type | ae-code | Reference |
2811 | | | Contents | |
2812 +------------------------+---------------+--------------+-----------+
2813 | 0 | TCP reset | None | RFC 8618 |
2814 | | | | |
2815 | 1 | ICMP time | ICMP code | RFC 8618 |
2816 | | exceeded | [icmpcodes] | |
2817 | | | | |
2818 | 2 | ICMP | ICMP code | RFC 8618 |
2819 | | destination | [icmpcodes] | |
2820 | | unreachable | | |
2821 | | | | |
2822 | 3 | ICMPv6 time | ICMPv6 code | RFC 8618 |
2823 | | exceeded | [icmp6codes] | |
2824 | | | | |
2825 | 4 | ICMPv6 | ICMPv6 code | RFC 8618 |
2826 | | destination | [icmp6codes] | |
2827 | | unreachable | | |
2828 | | | | |
2829 | 5 | ICMPv6 packet | ICMPv6 code | RFC 8618 |
2830 | | too big | [icmp6codes] | |
2831 | | | | |
2832 | 6-18446744073709551615 | Unassigned | | |
2833 +------------------------+---------------+--------------+-----------+
2834
2835 Expert reviewers should take the following point into consideration:
2836 "ae-code" contents must be defined for a type or, if not appropriate,
2837 specified as "None". A specification of "None" requires less storage
2838 and is therefore preferred.
2839
2840 13. Security Considerations
2841
2842 Any control interface MUST perform authentication and encryption.
2843
2844 Any data upload MUST be authenticated and encrypted.
2845
2846 14. Privacy Considerations
2847
2848 Storage of DNS traffic by operators in PCAP and other formats is a
2849 long-standing and widespread practice. Section 2.5 of
2850 [DNS-Priv-Cons] provides an analysis of the risks to Internet users
2851 regarding the storage of DNS traffic data in servers (recursive
2852 resolvers, authoritative servers, and rogue servers).
2853
2854
2855
2856
2857 Dickinson, et al. Standards Track [Page 52]
2858 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2859
2860
2861 Section 5.2 of [DNS-Priv-Svc] describes mitigations for those risks
2862 for data stored on recursive resolvers (but that could by extension
2863 apply to authoritative servers). These include data-handling
2864 practices and methods for data minimization, IP address
2865 pseudonymization, and anonymization. Appendix C of [DNS-Priv-Svc]
2866 presents an analysis of seven published anonymization processes. In
2867 addition, the ICANN Root Server System Advisory Committee (RSSAC)
2868 have recently published [RSSAC04] ("Recommendations on Anonymization
2869 Processes for Source IP Addresses Submitted for Future Analysis").
2870
2871 The above analyses consider full data capture (e.g., using PCAP) as a
2872 baseline for privacy considerations; therefore, this format
2873 specification introduces no new user privacy issues beyond those of
2874 full data capture (which are quite severe). It does provide
2875 mechanisms to selectively record only certain fields at the time of
2876 data capture, to improve user privacy and to explicitly indicate that
2877 data is sampled, anonymized, or both. It also provides flags to
2878 indicate if data normalization has been performed; data normalization
2879 increases user privacy by reducing the potential for fingerprinting
2880 individuals. However, a trade-off is the potential reduction of the
2881 capacity to identify attack traffic via Query name signatures.
2882 Operators should carefully consider their operational requirements
2883 and privacy policies and SHOULD capture at the source the minimum
2884 user data required to meet their needs.
2885
2886 15. References
2887
2888 15.1. Normative References
2889
2890 [pcap-filter]
2891 tcpdump.org, "Manpage of PCAP-FILTER", November 2017,
2892 <https://www.tcpdump.org/manpages/pcap-filter.7.html>.
2893
2894 [pcap-options]
2895 tcpdump.org, "Manpage of PCAP", July 2018,
2896 <https://www.tcpdump.org/manpages/pcap.3pcap.html>.
2897
2898 [posix-time]
2899 The Open Group, "IEEE Standard for Information
2900 Technology--Portable Operating System Interface (POSIX(R))
2901 Base Specifications, Issue 7", IEEE Standard 1003.1-2017,
2902 Section 4.16, DOI 10.1109/IEEESTD.2018.8277153.
2903
2904 [RFC792] Postel, J., "Internet Control Message Protocol", STD 5,
2905 RFC 792, DOI 10.17487/RFC0792, September 1981,
2906 <https://www.rfc-editor.org/info/rfc792>.
2907
2908
2909
2910
2911
2912 Dickinson, et al. Standards Track [Page 53]
2913 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2914
2915
2916 [RFC1035] Mockapetris, P., "Domain names - implementation and
2917 specification", STD 13, RFC 1035, DOI 10.17487/RFC1035,
2918 November 1987, <https://www.rfc-editor.org/info/rfc1035>.
2919
2920 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
2921 Requirement Levels", BCP 14, RFC 2119,
2922 DOI 10.17487/RFC2119, March 1997,
2923 <https://www.rfc-editor.org/info/rfc2119>.
2924
2925 [RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform
2926 Resource Identifier (URI): Generic Syntax", STD 66,
2927 RFC 3986, DOI 10.17487/RFC3986, January 2005,
2928 <https://www.rfc-editor.org/info/rfc3986>.
2929
2930 [RFC4443] Conta, A., Deering, S., and M. Gupta, Ed., "Internet
2931 Control Message Protocol (ICMPv6) for the Internet
2932 Protocol Version 6 (IPv6) Specification", STD 89,
2933 RFC 4443, DOI 10.17487/RFC4443, March 2006,
2934 <https://www.rfc-editor.org/info/rfc4443>.
2935
2936 [RFC6891] Damas, J., Graff, M., and P. Vixie, "Extension Mechanisms
2937 for DNS (EDNS(0))", STD 75, RFC 6891,
2938 DOI 10.17487/RFC6891, April 2013,
2939 <https://www.rfc-editor.org/info/rfc6891>.
2940
2941 [RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object
2942 Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049,
2943 October 2013, <https://www.rfc-editor.org/info/rfc7049>.
2944
2945 [RFC7858] Hu, Z., Zhu, L., Heidemann, J., Mankin, A., Wessels, D.,
2946 and P. Hoffman, "Specification for DNS over Transport
2947 Layer Security (TLS)", RFC 7858, DOI 10.17487/RFC7858,
2948 May 2016, <https://www.rfc-editor.org/info/rfc7858>.
2949
2950 [RFC8094] Reddy, T., Wing, D., and P. Patil, "DNS over Datagram
2951 Transport Layer Security (DTLS)", RFC 8094,
2952 DOI 10.17487/RFC8094, February 2017,
2953 <https://www.rfc-editor.org/info/rfc8094>.
2954
2955 [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for
2956 Writing an IANA Considerations Section in RFCs", BCP 26,
2957 RFC 8126, DOI 10.17487/RFC8126, June 2017,
2958 <https://www.rfc-editor.org/info/rfc8126>.
2959
2960 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in
2961 RFC 2119 Key Words", BCP 14, RFC 8174,
2962 DOI 10.17487/RFC8174, May 2017,
2963 <https://www.rfc-editor.org/info/rfc8174>.
2964
2965
2966
2967 Dickinson, et al. Standards Track [Page 54]
2968 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
2969
2970
2971 [RFC8484] Hoffman, P. and P. McManus, "DNS Queries over HTTPS
2972 (DoH)", RFC 8484, DOI 10.17487/RFC8484, October 2018,
2973 <https://www.rfc-editor.org/info/rfc8484>.
2974
2975 [RFC8610] Birkholz, H., Vigano, C., and C. Bormann, "Concise Data
2976 Definition Language (CDDL): A Notational Convention to
2977 Express Concise Binary Object Representation (CBOR) and
2978 JSON Data Structures", RFC 8610, DOI 10.17487/RFC8610,
2979 June 2019, <https://www.rfc-editor.org/info/rfc8610>.
2980
2981 15.2. Informative References
2982
2983 [Avro] The Apache Software Foundation, "Apache Avro(TM)", 2019,
2984 <https://avro.apache.org/>.
2985
2986 [ditl] DNS-OARC, "DITL", 2018,
2987 <https://www.dns-oarc.net/oarc/data/ditl>.
2988
2989 [DNS-Priv-Cons]
2990 Bortzmeyer, S. and S. Dickinson, "DNS Privacy
2991 Considerations", Work in Progress,
2992 draft-ietf-dprive-rfc7626-bis-00, July 2019.
2993
2994 [DNS-Priv-Svc]
2995 Dickinson, S., Overeinder, B., van Rijswijk-Deij, R., and
2996 A. Mankin, "Recommendations for DNS Privacy Service
2997 Operators", Work in Progress, draft-ietf-dprive-bcp-op-03,
2998 July 2019.
2999
3000 [dnscap] DNS-OARC, "DNSCAP", 2018,
3001 <https://www.dns-oarc.net/tools/dnscap>.
3002
3003 [dnstap] "dnstap", 2016, <https://dnstap.info/>.
3004
3005 [dnstap-schema]
3006 "dnstap schema", commit d860ec1, November 2016,
3007 <https://github.com/dnstap/dnstap.pb/blob/master/
3008 dnstap.proto>.
3009
3010 [dnsxml] Daley, J., Ed., Morris, S., and J. Dickinson, "dnsxml - A
3011 standard XML representation of DNS data", Work in
3012 Progress, draft-daley-dnsxml-00, July 2013.
3013
3014 [dsc] Wessels, D. and J. Lundstrom, "DSC", 2016,
3015 <https://www.dns-oarc.net/tools/dsc>.
3016
3017 [gzip] "gzip", <https://www.gzip.org/>.
3018
3019
3020
3021
3022 Dickinson, et al. Standards Track [Page 55]
3023 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3024
3025
3026 [icmp6codes]
3027 IANA, "ICMPv6 "Code" Fields",
3028 <https://www.iana.org/assignments/icmpv6-parameters/>.
3029
3030 [icmpcodes]
3031 IANA, "Code Fields",
3032 <https://www.iana.org/assignments/icmp-parameters/>.
3033
3034 [IEEE802.1Q]
3035 IEEE, "IEEE Standard for Local and Metropolitan Area
3036 Networks--Bridges and Bridged Networks", IEEE
3037 Standard 802.1Q.
3038
3039 [Knot] "Knot DNS", <https://www.knot-dns.cz/>.
3040
3041 [lz4] "LZ4", <https://lz4.github.io/lz4/>.
3042
3043 [mmark] Gieben, M., "mmark", commit de69698, May 2019,
3044 <https://github.com/mmarkdown/mmark>.
3045
3046 [NSD] NLnet Labs, "NSD", 2019,
3047 <https://www.nlnetlabs.nl/projects/nsd/about/>.
3048
3049 [opcodes] IANA, "DNS OpCodes",
3050 <https://www.iana.org/assignments/dns-parameters/>.
3051
3052 [packetq] .SE - The Internet Infrastructure Foundation, "PacketQ",
3053 commit c9b2e89, February 2019,
3054 <https://github.com/DNS-OARC/PacketQ>.
3055
3056 [pcap] "PCAP", 2019, <https://www.tcpdump.org/>.
3057
3058 [pcapng] "pcapng: PCAP next generation file format specification",
3059 commit 3c35b6a, March 2019,
3060 <https://github.com/pcapng/pcapng>.
3061
3062 [Protocol-Buffers]
3063 Google LLC, "Protocol Buffers",
3064 <https://developers.google.com/protocol-buffers/>.
3065
3066 [rcodes] IANA, "DNS RCODEs",
3067 <https://www.iana.org/assignments/dns-parameters/>.
3068
3069 [RFC5077] Salowey, J., Zhou, H., Eronen, P., and H. Tschofenig,
3070 "Transport Layer Security (TLS) Session Resumption without
3071 Server-Side State", RFC 5077, DOI 10.17487/RFC5077,
3072 January 2008, <https://www.rfc-editor.org/info/rfc5077>.
3073
3074
3075
3076
3077 Dickinson, et al. Standards Track [Page 56]
3078 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3079
3080
3081 [RFC7413] Cheng, Y., Chu, J., Radhakrishnan, S., and A. Jain, "TCP
3082 Fast Open", RFC 7413, DOI 10.17487/RFC7413, December 2014,
3083 <https://www.rfc-editor.org/info/rfc7413>.
3084
3085 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
3086 Interchange Format", STD 90, RFC 8259,
3087 DOI 10.17487/RFC8259, December 2017,
3088 <https://www.rfc-editor.org/info/rfc8259>.
3089
3090 [RFC8427] Hoffman, P., "Representing DNS Messages in JSON",
3091 RFC 8427, DOI 10.17487/RFC8427, July 2018,
3092 <https://www.rfc-editor.org/info/rfc8427>.
3093
3094 [rrclasses]
3095 IANA, "DNS CLASSes",
3096 <https://www.iana.org/assignments/dns-parameters/>.
3097
3098 [rrtypes] IANA, "Resource Record (RR) TYPEs",
3099 <https://www.iana.org/assignments/dns-parameters/>.
3100
3101 [RSSAC04] ICANN, "Recommendations on Anonymization Processes for
3102 Source IP Addresses Submitted for Future Analysis",
3103 August 2018, <https://www.icann.org/en/system/files/files/
3104 rssac-040-07aug18-en.pdf>.
3105
3106 [snappy] "snappy", <https://google.github.io/snappy/>.
3107
3108 [snzip] "Snzip, a compression/decompression tool based on snappy",
3109 commit 809c6f2, October 2018,
3110 <https://github.com/kubo/snzip>.
3111
3112 [xz] "XZ Utils", <https://tukaani.org/xz/>.
3113
3114 [zstd] "Zstandard - Real-time data compression algorithm",
3115 <https://facebook.github.io/zstd/>.
3116
3117
3118
3119
3120
3121
3122
3123
3124
3125
3126
3127
3128
3129
3130
3131
3132 Dickinson, et al. Standards Track [Page 57]
3133 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3134
3135
3136 Appendix A. CDDL
3137
3138 This appendix gives a CDDL [RFC8610] specification for C-DNS.
3139
3140 CDDL does not permit a range of allowed values to be specified for a
3141 bitfield. Where necessary, those values are given as a CDDL group,
3142 but the group definition is commented out to prevent CDDL tooling
3143 from warning that the group is unused.
3144
3145 ; CDDL specification of the file format for C-DNS,
3146 ; which describes a collection of DNS messages and
3147 ; traffic metadata.
3148
3149 ;
3150 ; The overall structure of a file.
3151 ;
3152 File = [
3153 file-type-id : "C-DNS",
3154 file-preamble : FilePreamble,
3155 file-blocks : [* Block],
3156 ]
3157
3158 ;
3159 ; The File Preamble.
3160 ;
3161 FilePreamble = {
3162 major-format-version => 1,
3163 minor-format-version => 0,
3164 ? private-version => uint,
3165 block-parameters => [+ BlockParameters],
3166 }
3167 major-format-version = 0
3168 minor-format-version = 1
3169 private-version = 2
3170 block-parameters = 3
3171
3172 BlockParameters = {
3173 storage-parameters => StorageParameters,
3174 ? collection-parameters => CollectionParameters,
3175 }
3176 storage-parameters = 0
3177 collection-parameters = 1
3178
3179 IPv6PrefixLength = 1..128
3180 IPv4PrefixLength = 1..32
3181 OpcodeRange = 0..15
3182 RRTypeRange = 0..65535
3183
3184
3185
3186
3187 Dickinson, et al. Standards Track [Page 58]
3188 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3189
3190
3191 StorageParameters = {
3192 ticks-per-second => uint,
3193 max-block-items => uint,
3194 storage-hints => StorageHints,
3195 opcodes => [+ OpcodeRange],
3196 rr-types => [+ RRTypeRange],
3197 ? storage-flags => StorageFlags,
3198 ? client-address-prefix-ipv4 => IPv4PrefixLength,
3199 ? client-address-prefix-ipv6 => IPv6PrefixLength,
3200 ? server-address-prefix-ipv4 => IPv4PrefixLength,
3201 ? server-address-prefix-ipv6 => IPv6PrefixLength,
3202 ? sampling-method => tstr,
3203 ? anonymization-method => tstr,
3204 }
3205 ticks-per-second = 0
3206 max-block-items = 1
3207 storage-hints = 2
3208 opcodes = 3
3209 rr-types = 4
3210 storage-flags = 5
3211 client-address-prefix-ipv4 = 6
3212 client-address-prefix-ipv6 = 7
3213 server-address-prefix-ipv4 = 8
3214 server-address-prefix-ipv6 = 9
3215 sampling-method = 10
3216 anonymization-method = 11
3217
3218 ; A hint indicates whether the collection method will always omit
3219 ; the item from the file.
3220 StorageHints = {
3221 query-response-hints => QueryResponseHints,
3222 query-response-signature-hints =>
3223 QueryResponseSignatureHints,
3224 rr-hints => RRHints,
3225 other-data-hints => OtherDataHints,
3226 }
3227 query-response-hints = 0
3228 query-response-signature-hints = 1
3229 rr-hints = 2
3230 other-data-hints = 3
3231
3232 QueryResponseHintValues = &(
3233 time-offset : 0,
3234 client-address-index : 1,
3235 client-port : 2,
3236 transaction-id : 3,
3237 qr-signature-index : 4,
3238 client-hoplimit : 5,
3239
3240
3241
3242 Dickinson, et al. Standards Track [Page 59]
3243 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3244
3245
3246 response-delay : 6,
3247 query-name-index : 7,
3248 query-size : 8,
3249 response-size : 9,
3250 response-processing-data : 10,
3251 query-question-sections : 11, ; Second & subsequent
3252 ; Questions
3253 query-answer-sections : 12,
3254 query-authority-sections : 13,
3255 query-additional-sections : 14,
3256 response-answer-sections : 15,
3257 response-authority-sections : 16,
3258 response-additional-sections : 17,
3259 )
3260 QueryResponseHints = uint .bits QueryResponseHintValues
3261
3262 QueryResponseSignatureHintValues = &(
3263 server-address-index : 0,
3264 server-port : 1,
3265 qr-transport-flags : 2,
3266 qr-type : 3,
3267 qr-sig-flags : 4,
3268 query-opcode : 5,
3269 qr-dns-flags : 6,
3270 query-rcode : 7,
3271 query-classtype-index : 8,
3272 query-qdcount : 9,
3273 query-ancount : 10,
3274 query-nscount : 11,
3275 query-arcount : 12,
3276 query-edns-version : 13,
3277 query-udp-size : 14,
3278 query-opt-rdata-index : 15,
3279 response-rcode : 16,
3280 )
3281 QueryResponseSignatureHints =
3282 uint .bits QueryResponseSignatureHintValues
3283
3284 RRHintValues = &(
3285 ttl : 0,
3286 rdata-index : 1,
3287 )
3288 RRHints = uint .bits RRHintValues
3289
3290 OtherDataHintValues = &(
3291 malformed-messages : 0,
3292 address-event-counts : 1,
3293 )
3294
3295
3296
3297 Dickinson, et al. Standards Track [Page 60]
3298 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3299
3300
3301 OtherDataHints = uint .bits OtherDataHintValues
3302
3303 StorageFlagValues = &(
3304 anonymized-data : 0,
3305 sampled-data : 1,
3306 normalized-names : 2,
3307 )
3308 StorageFlags = uint .bits StorageFlagValues
3309
3310 ; Metadata about data collection
3311 VLANIdRange = 1..4094
3312
3313 CollectionParameters = {
3314 ? query-timeout => uint, ; Milliseconds
3315 ? skew-timeout => uint, ; Microseconds
3316 ? snaplen => uint,
3317 ? promisc => bool,
3318 ? interfaces => [+ tstr],
3319 ? server-addresses => [+ IPAddress],
3320 ? vlan-ids => [+ VLANIdRange],
3321 ? filter => tstr,
3322 ? generator-id => tstr,
3323 ? host-id => tstr,
3324 }
3325 query-timeout = 0
3326 skew-timeout = 1
3327 snaplen = 2
3328 promisc = 3
3329 interfaces = 4
3330 server-addresses = 5
3331 vlan-ids = 6
3332 filter = 7
3333 generator-id = 8
3334 host-id = 9
3335
3336 ;
3337 ; Data in the file is stored in Blocks.
3338 ;
3339 Block = {
3340 block-preamble => BlockPreamble,
3341 ? block-statistics => BlockStatistics, ; Much of this
3342 ; could be derived
3343 ? block-tables => BlockTables,
3344 ? query-responses => [+ QueryResponse],
3345 ? address-event-counts => [+ AddressEventCount],
3346 ? malformed-messages => [+ MalformedMessage],
3347 }
3348
3349
3350
3351
3352 Dickinson, et al. Standards Track [Page 61]
3353 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3354
3355
3356 block-preamble = 0
3357 block-statistics = 1
3358 block-tables = 2
3359 query-responses = 3
3360 address-event-counts = 4
3361 malformed-messages = 5
3362
3363 ;
3364 ; The (mandatory) preamble to a Block.
3365 ;
3366 BlockPreamble = {
3367 ? earliest-time => Timestamp,
3368 ? block-parameters-index => uint .default 0,
3369 }
3370 earliest-time = 0
3371 block-parameters-index = 1
3372
3373 ; Ticks are sub-second intervals. The number of ticks in a second is
3374 ; file/block metadata. Signed and unsigned tick types are defined.
3375 ticks = int
3376 uticks = uint
3377
3378 Timestamp = [
3379 timestamp-secs : uint, ; POSIX time
3380 timestamp-ticks : uticks,
3381 ]
3382
3383 ;
3384 ; Statistics about the Block contents.
3385 ;
3386 BlockStatistics = {
3387 ? processed-messages => uint,
3388 ? qr-data-items => uint,
3389 ? unmatched-queries => uint,
3390 ? unmatched-responses => uint,
3391 ? discarded-opcode => uint,
3392 ? malformed-items => uint,
3393 }
3394 processed-messages = 0
3395 qr-data-items = 1
3396 unmatched-queries = 2
3397 unmatched-responses = 3
3398 discarded-opcode = 4
3399 malformed-items = 5
3400
3401
3402
3403
3404
3405
3406
3407 Dickinson, et al. Standards Track [Page 62]
3408 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3409
3410
3411 ;
3412 ; Tables of common data referenced from records in a Block.
3413 ;
3414 BlockTables = {
3415 ? ip-address => [+ IPAddress],
3416 ? classtype => [+ ClassType],
3417 ? name-rdata => [+ bstr], ; Holds both names
3418 ; and RDATA
3419 ? qr-sig => [+ QueryResponseSignature],
3420 ? QuestionTables,
3421 ? RRTables,
3422 ? malformed-message-data => [+ MalformedMessageData],
3423 }
3424 ip-address = 0
3425 classtype = 1
3426 name-rdata = 2
3427 qr-sig = 3
3428 qlist = 4
3429 qrr = 5
3430 rrlist = 6
3431 rr = 7
3432 malformed-message-data = 8
3433
3434 IPv4Address = bstr .size (0..4)
3435 IPv6Address = bstr .size (0..16)
3436 IPAddress = IPv4Address / IPv6Address
3437
3438 ClassType = {
3439 type => uint,
3440 class => uint,
3441 }
3442 type = 0
3443 class = 1
3444
3445 QueryResponseSignature = {
3446 ? server-address-index => uint,
3447 ? server-port => uint,
3448 ? qr-transport-flags => QueryResponseTransportFlags,
3449 ? qr-type => QueryResponseType,
3450 ? qr-sig-flags => QueryResponseFlags,
3451 ? query-opcode => uint,
3452 ? qr-dns-flags => DNSFlags,
3453 ? query-rcode => uint,
3454 ? query-classtype-index => uint,
3455 ? query-qdcount => uint,
3456 ? query-ancount => uint,
3457 ? query-nscount => uint,
3458 ? query-arcount => uint,
3459
3460
3461
3462 Dickinson, et al. Standards Track [Page 63]
3463 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3464
3465
3466 ? query-edns-version => uint,
3467 ? query-udp-size => uint,
3468 ? query-opt-rdata-index => uint,
3469 ? response-rcode => uint,
3470 }
3471 server-address-index = 0
3472 server-port = 1
3473 qr-transport-flags = 2
3474 qr-type = 3
3475 qr-sig-flags = 4
3476 query-opcode = 5
3477 qr-dns-flags = 6
3478 query-rcode = 7
3479 query-classtype-index = 8
3480 query-qdcount = 9
3481 query-ancount = 10
3482 query-nscount = 11
3483 query-arcount = 12
3484 query-edns-version = 13
3485 query-udp-size = 14
3486 query-opt-rdata-index = 15
3487 response-rcode = 16
3488
3489 ; Transport gives the values that may appear in bits 1..4 of
3490 ; TransportFlags. There is currently no way to express this in
3491 ; CDDL, so Transport is unused. To avoid confusion when used
3492 ; with CDDL tools, it is commented out.
3493 ;
3494 ; Transport = &(
3495 ; udp : 0,
3496 ; tcp : 1,
3497 ; tls : 2,
3498 ; dtls : 3,
3499 ; https : 4,
3500 ; non-standard : 15,
3501 ; )
3502
3503 TransportFlagValues = &(
3504 ip-version : 0, ; 0=IPv4, 1=IPv6
3505 ) / (1..4)
3506 TransportFlags = uint .bits TransportFlagValues
3507
3508 QueryResponseTransportFlagValues = &(
3509 query-trailingdata : 5,
3510 ) / TransportFlagValues
3511 QueryResponseTransportFlags =
3512 uint .bits QueryResponseTransportFlagValues
3513
3514
3515
3516
3517 Dickinson, et al. Standards Track [Page 64]
3518 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3519
3520
3521 QueryResponseType = &(
3522 stub : 0,
3523 client : 1,
3524 resolver : 2,
3525 auth : 3,
3526 forwarder : 4,
3527 tool : 5,
3528 )
3529
3530 QueryResponseFlagValues = &(
3531 has-query : 0,
3532 has-response : 1,
3533 query-has-opt : 2,
3534 response-has-opt : 3,
3535 query-has-no-question : 4,
3536 response-has-no-question: 5,
3537 )
3538 QueryResponseFlags = uint .bits QueryResponseFlagValues
3539
3540 DNSFlagValues = &(
3541 query-cd : 0,
3542 query-ad : 1,
3543 query-z : 2,
3544 query-ra : 3,
3545 query-rd : 4,
3546 query-tc : 5,
3547 query-aa : 6,
3548 query-do : 7,
3549 response-cd: 8,
3550 response-ad: 9,
3551 response-z : 10,
3552 response-ra: 11,
3553 response-rd: 12,
3554 response-tc: 13,
3555 response-aa: 14,
3556 )
3557 DNSFlags = uint .bits DNSFlagValues
3558
3559
3560
3561
3562
3563
3564
3565
3566
3567
3568
3569
3570
3571
3572 Dickinson, et al. Standards Track [Page 65]
3573 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3574
3575
3576 QuestionTables = (
3577 qlist => [+ QuestionList],
3578 qrr => [+ Question]
3579 )
3580
3581 QuestionList = [+ uint] ; Index of Question
3582
3583 Question = { ; Second and subsequent Questions
3584 name-index => uint, ; Index to a name in the
3585 ; name-rdata table
3586 classtype-index => uint,
3587 }
3588 name-index = 0
3589 classtype-index = 1
3590
3591 RRTables = (
3592 rrlist => [+ RRList],
3593 rr => [+ RR]
3594 )
3595
3596 RRList = [+ uint] ; Index of RR
3597
3598 RR = {
3599 name-index => uint, ; Index to a name in the
3600 ; name-rdata table
3601 classtype-index => uint,
3602 ? ttl => uint,
3603 ? rdata-index => uint, ; Index to RDATA in the
3604 ; name-rdata table
3605 }
3606 ; Other map key values already defined above.
3607 ttl = 2
3608 rdata-index = 3
3609
3610 MalformedMessageData = {
3611 ? server-address-index => uint,
3612 ? server-port => uint,
3613 ? mm-transport-flags => TransportFlags,
3614 ? mm-payload => bstr,
3615 }
3616 ; Other map key values already defined above.
3617 mm-transport-flags = 2
3618 mm-payload = 3
3619
3620
3621
3622
3623
3624
3625
3626
3627 Dickinson, et al. Standards Track [Page 66]
3628 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3629
3630
3631 ;
3632 ; A single Query/Response data item.
3633 ;
3634 QueryResponse = {
3635 ? time-offset => uticks, ; Time offset from
3636 ; start of Block
3637 ? client-address-index => uint,
3638 ? client-port => uint,
3639 ? transaction-id => uint,
3640 ? qr-signature-index => uint,
3641 ? client-hoplimit => uint,
3642 ? response-delay => ticks,
3643 ? query-name-index => uint,
3644 ? query-size => uint, ; DNS size of Query
3645 ? response-size => uint, ; DNS size of Response
3646 ? response-processing-data => ResponseProcessingData,
3647 ? query-extended => QueryResponseExtended,
3648 ? response-extended => QueryResponseExtended,
3649 }
3650 time-offset = 0
3651 client-address-index = 1
3652 client-port = 2
3653 transaction-id = 3
3654 qr-signature-index = 4
3655 client-hoplimit = 5
3656 response-delay = 6
3657 query-name-index = 7
3658 query-size = 8
3659 response-size = 9
3660 response-processing-data = 10
3661 query-extended = 11
3662 response-extended = 12
3663
3664 ResponseProcessingData = {
3665 ? bailiwick-index => uint,
3666 ? processing-flags => ResponseProcessingFlags,
3667 }
3668 bailiwick-index = 0
3669 processing-flags = 1
3670
3671 ResponseProcessingFlagValues = &(
3672 from-cache : 0,
3673 )
3674 ResponseProcessingFlags = uint .bits ResponseProcessingFlagValues
3675
3676
3677
3678
3679
3680
3681
3682 Dickinson, et al. Standards Track [Page 67]
3683 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3684
3685
3686 QueryResponseExtended = {
3687 ? question-index => uint, ; Index of QuestionList
3688 ? answer-index => uint, ; Index of RRList
3689 ? authority-index => uint,
3690 ? additional-index => uint,
3691 }
3692 question-index = 0
3693 answer-index = 1
3694 authority-index = 2
3695 additional-index = 3
3696
3697 ;
3698 ; Address event data.
3699 ;
3700 AddressEventCount = {
3701 ae-type => &AddressEventType,
3702 ? ae-code => uint,
3703 ae-address-index => uint,
3704 ? ae-transport-flags => TransportFlags,
3705 ae-count => uint,
3706 }
3707 ae-type = 0
3708 ae-code = 1
3709 ae-address-index = 2
3710 ae-transport-flags = 3
3711 ae-count = 4
3712
3713 AddressEventType = (
3714 tcp-reset : 0,
3715 icmp-time-exceeded : 1,
3716 icmp-dest-unreachable : 2,
3717 icmpv6-time-exceeded : 3,
3718 icmpv6-dest-unreachable: 4,
3719 icmpv6-packet-too-big : 5,
3720 )
3721
3722 ;
3723 ; Malformed messages.
3724 ;
3725 MalformedMessage = {
3726 ? time-offset => uticks, ; Time offset from
3727 ; start of Block
3728 ? client-address-index => uint,
3729 ? client-port => uint,
3730 ? message-data-index => uint,
3731 }
3732 ; Other map key values already defined above.
3733 message-data-index = 3
3734
3735
3736
3737 Dickinson, et al. Standards Track [Page 68]
3738 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3739
3740
3741 Appendix B. DNS Name Compression Example
3742
3743 The basic algorithm, which follows the guidance in [RFC1035], is
3744 simply to collect each name, and the offset in the packet at which it
3745 starts, during packet construction. As each name is added, it is
3746 offered to each of the collected names in order of collection,
3747 starting from the first name. If (1) labels at the end of the name
3748 can be replaced with a reference back to part (or all) of the earlier
3749 name and (2) the uncompressed part of the name is shorter than any
3750 compression already found, the earlier name is noted as the
3751 compression target for the name.
3752
3753 The following tables illustrate the step-by-step process of adding
3754 names and performing name compression. In an example packet, the
3755 first name added is foo.example, which cannot be compressed.
3756
3757 +---+-------------+--------------+--------------------+
3758 | N | Name | Uncompressed | Compression Target |
3759 +---+-------------+--------------+--------------------+
3760 | 1 | foo.example | foo.example | None |
3761 +---+-------------+--------------+--------------------+
3762
3763 The next name added is bar.example. This is matched against
3764 foo.example. The example part of this can be used as a compression
3765 target, with the remaining uncompressed part of the name being bar.
3766
3767 +---+-------------+--------------+-----------------------+
3768 | N | Name | Uncompressed | Compression Target |
3769 +---+-------------+--------------+-----------------------+
3770 | 1 | foo.example | foo.example | None |
3771 | 2 | bar.example | bar | 1 + offset to example |
3772 +---+-------------+--------------+-----------------------+
3773
3774 The third name added is www.bar.example. This is first matched
3775 against foo.example, and as before this is recorded as a compression
3776 target, with the remaining uncompressed part of the name being
3777 www.bar. It is then matched against the second name, which again can
3778 be a compression target. Because the remaining uncompressed part of
3779 the name is www, this is an improved compression, and so it is
3780 adopted.
3781
3782 +---+-----------------+--------------+-----------------------+
3783 | N | Name | Uncompressed | Compression Target |
3784 +---+-----------------+--------------+-----------------------+
3785 | 1 | foo.example | foo.example | None |
3786 | 2 | bar.example | bar | 1 + offset to example |
3787 | 3 | www.bar.example | www | 2 |
3788 +---+-----------------+--------------+-----------------------+
3789
3790
3791
3792 Dickinson, et al. Standards Track [Page 69]
3793 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3794
3795
3796 As an optimization, if a name is already perfectly compressed (in
3797 other words, the uncompressed part of the name is empty), then no
3798 further names will be considered for compression.
3799
3800 B.1. NSD Compression Algorithm
3801
3802 Using the above basic algorithm, the packet lengths of Responses
3803 generated by the Name Server Daemon (NSD) [NSD] can be matched almost
3804 exactly. At the time of writing, a tiny number (<.01%) of the
3805 reconstructed packets had incorrect lengths.
3806
3807 B.2. Knot Authoritative Compression Algorithm
3808
3809 The Knot Authoritative name server [Knot] uses different compression
3810 behavior, which is the result of internal optimization designed to
3811 balance runtime speed with compression size gains. In brief, and
3812 omitting complications, Knot Authoritative will only consider the
3813 QNAME and names in the immediately preceding RR section in an RRSET
3814 as compression targets.
3815
3816 A set of smart heuristics as described below can be implemented to
3817 mimic this, and while not perfect, it produces output nearly, but not
3818 quite, as good a match as with NSD. The heuristics are as follows:
3819
3820 1. A match is only perfect if the name is completely compressed AND
3821 the TYPE of the section in which the name occurs matches the TYPE
3822 of the name used as the compression target.
3823
3824 2. If the name occurs in RDATA:
3825
3826 * If the compression target name is in a Query, then only the
3827 first RR in an RRSET can use that name as a compression
3828 target.
3829
3830 * The compression target name MUST be in RDATA.
3831
3832 * The name section TYPE must match the compression target name
3833 section TYPE.
3834
3835 * The compression target name MUST be in the immediately
3836 preceding RR in the RRSET.
3837
3838 Using this algorithm, less than 0.1% of the reconstructed packets had
3839 incorrect lengths.
3840
3841
3842
3843
3844
3845
3846
3847 Dickinson, et al. Standards Track [Page 70]
3848 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3849
3850
3851 B.3. Observed Differences
3852
3853 In sample traffic collected on a root name server, around 2-4% of
3854 Responses generated by Knot had different packet lengths than those
3855 produced by NSD.
3856
3857 Appendix C. Comparison of Binary Formats
3858
3859 Several binary serialization formats were considered. For
3860 completeness, they were also compared to JSON.
3861
3862 o Apache Avro [Avro]. Data is stored according to a predefined
3863 schema. The schema itself is always included in the data file.
3864 Data can therefore be stored untagged, for a smaller serialization
3865 size, and be written and read by an Avro library.
3866
3867 * At the time of writing, Avro libraries are available for C,
3868 C++, C#, Java, Python, Ruby, and PHP. Optionally, tools are
3869 available for C++, Java, and C# to generate code for encoding
3870 and decoding.
3871
3872 o Google Protocol Buffers [Protocol-Buffers]. Data is stored
3873 according to a predefined schema. The schema is used by a
3874 generator to generate code for encoding and decoding the data.
3875 Data can therefore be stored untagged, for a smaller serialization
3876 size. The schema is not stored with the data, so unlike Avro, it
3877 cannot be read with a generic library.
3878
3879 * Code must be generated for a particular data schema to read and
3880 write data using that schema. At the time of writing, the
3881 Google code generator can currently generate code for encoding
3882 and decoding a schema for C++, Go, Java, Python, Ruby, C#,
3883 Objective-C, JavaScript, and PHP.
3884
3885 o CBOR [RFC7049]. This serialization format is comparable to JSON
3886 but with a binary representation. It does not use a predefined
3887 schema, so data is always stored tagged. However, CBOR data
3888 schemas can be described using CDDL [RFC8610], and tools exist to
3889 verify that data files conform to the schema.
3890
3891 * CBOR is a simple format and is simple to implement. At the
3892 time of writing, the CBOR website lists implementations for 16
3893 languages.
3894
3895
3896
3897
3898
3899
3900
3901
3902 Dickinson, et al. Standards Track [Page 71]
3903 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3904
3905
3906 Avro and Protocol Buffers both allow storage of untagged data, but
3907 because they rely on the data schema for this, their implementation
3908 is considerably more complex than CBOR. Using Avro or Protocol
3909 Buffers in an unsupported environment would require notably greater
3910 development effort compared to CBOR.
3911
3912 A test program was written that reads input from a PCAP file and
3913 writes output using one of two basic structures: either a simple
3914 structure, where each Query/Response pair is represented in a single
3915 record entry, or the C-DNS block structure.
3916
3917 The resulting output files were then compressed using a variety of
3918 common general-purpose lossless compression tools to explore the
3919 compressibility of the formats. The compression tools employed were:
3920
3921 o snzip [snzip]. A command-line compression tool based on the
3922 Google Snappy library [snappy].
3923
3924 o lz4 [lz4]. The command-line compression tool from the reference C
3925 LZ4 implementation.
3926
3927 o gzip [gzip]. The ubiquitous GNU zip tool.
3928
3929 o zstd [zstd]. Compression using the Zstandard algorithm.
3930
3931 o xz [xz]. A popular compression tool noted for high compression.
3932
3933 In all cases, the compression tools were run using their default
3934 settings.
3935
3936 Note that this document does not mandate the use of compression, nor
3937 any particular compression scheme, but it anticipates that in
3938 practice output data will be subject to general-purpose compression,
3939 and so this should be taken into consideration.
3940
3941 "test.pcap", a 662 MB capture of sample data from a root instance,
3942 was used for the comparison. The following table shows the formatted
3943 size and size after compression (abbreviated to Comp. in the table
3944 headers), together with the task Resident Set Size (RSS) and the user
3945 time taken by the compression. File sizes are in MB, RSS is in kB,
3946 and user time is in seconds.
3947
3948
3949
3950
3951
3952
3953
3954
3955
3956
3957 Dickinson, et al. Standards Track [Page 72]
3958 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
3959
3960
3961 +-------------+-----------+-------+------------+-------+-----------+
3962 | Format | File Size | Comp. | Comp. Size | RSS | User Time |
3963 +-------------+-----------+-------+------------+-------+-----------+
3964 | PCAP | 661.87 | snzip | 212.48 | 2696 | 1.26 |
3965 | | | lz4 | 181.58 | 6336 | 1.35 |
3966 | | | gzip | 153.46 | 1428 | 18.20 |
3967 | | | zstd | 87.07 | 3544 | 4.27 |
3968 | | | xz | 49.09 | 97416 | 160.79 |
3969 | | | | | | |
3970 | JSON simple | 4113.92 | snzip | 603.78 | 2656 | 5.72 |
3971 | | | lz4 | 386.42 | 5636 | 5.25 |
3972 | | | gzip | 271.11 | 1492 | 73.00 |
3973 | | | zstd | 133.43 | 3284 | 8.68 |
3974 | | | xz | 51.98 | 97412 | 600.74 |
3975 | | | | | | |
3976 | Avro simple | 640.45 | snzip | 148.98 | 2656 | 0.90 |
3977 | | | lz4 | 111.92 | 5828 | 0.99 |
3978 | | | gzip | 103.07 | 1540 | 11.52 |
3979 | | | zstd | 49.08 | 3524 | 2.50 |
3980 | | | xz | 22.87 | 97308 | 90.34 |
3981 | | | | | | |
3982 | CBOR simple | 764.82 | snzip | 164.57 | 2664 | 1.11 |
3983 | | | lz4 | 120.98 | 5892 | 1.13 |
3984 | | | gzip | 110.61 | 1428 | 12.88 |
3985 | | | zstd | 54.14 | 3224 | 2.77 |
3986 | | | xz | 23.43 | 97276 | 111.48 |
3987 | | | | | | |
3988 | PBuf simple | 749.51 | snzip | 167.16 | 2660 | 1.08 |
3989 | | | lz4 | 123.09 | 5824 | 1.14 |
3990 | | | gzip | 112.05 | 1424 | 12.75 |
3991 | | | zstd | 53.39 | 3388 | 2.76 |
3992 | | | xz | 23.99 | 97348 | 106.47 |
3993 | | | | | | |
3994 | JSON block | 519.77 | snzip | 106.12 | 2812 | 0.93 |
3995 | | | lz4 | 104.34 | 6080 | 0.97 |
3996 | | | gzip | 57.97 | 1604 | 12.70 |
3997 | | | zstd | 61.51 | 3396 | 3.45 |
3998 | | | xz | 27.67 | 97524 | 169.10 |
3999 | | | | | | |
4000 | Avro block | 60.45 | snzip | 48.38 | 2688 | 0.20 |
4001 | | | lz4 | 48.78 | 8540 | 0.22 |
4002 | | | gzip | 39.62 | 1576 | 2.92 |
4003 | | | zstd | 29.63 | 3612 | 1.25 |
4004 | | | xz | 18.28 | 97564 | 25.81 |
4005 | | | | | | |
4006
4007
4008
4009
4010
4011
4012 Dickinson, et al. Standards Track [Page 73]
4013 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
4014
4015
4016 | CBOR block | 75.25 | snzip | 53.27 | 2684 | 0.24 |
4017 | | | lz4 | 51.88 | 8008 | 0.28 |
4018 | | | gzip | 41.17 | 1548 | 4.36 |
4019 | | | zstd | 30.61 | 3476 | 1.48 |
4020 | | | xz | 18.15 | 97556 | 38.78 |
4021 | | | | | | |
4022 | PBuf block | 67.98 | snzip | 51.10 | 2636 | 0.24 |
4023 | | | lz4 | 52.39 | 8304 | 0.24 |
4024 | | | gzip | 40.19 | 1520 | 3.63 |
4025 | | | zstd | 31.61 | 3576 | 1.40 |
4026 | | | xz | 17.94 | 97440 | 33.99 |
4027 +-------------+-----------+-------+------------+-------+-----------+
4028
4029 The above results are discussed in the following sections.
4030
4031 C.1. Comparison with Full PCAP Files
4032
4033 An important first consideration is whether moving away from PCAP
4034 offers significant benefits.
4035
4036 The simple binary formats are typically larger than PCAP, even though
4037 they omit some information such as Ethernet Media Access Control
4038 (MAC) addresses. But not only do they require less CPU to compress
4039 than PCAP, the resulting compressed files are smaller than compressed
4040 PCAP.
4041
4042 C.2. Simple versus Block Coding
4043
4044 The intention of the block coding is to perform data deduplication on
4045 Query/Response records within the block. The simple and block
4046 formats shown above store exactly the same information for each
4047 Query/Response record. This information is parsed from the DNS
4048 traffic in the input PCAP file, and in all cases each field has an
4049 identifier and the field data is typed.
4050
4051 The data deduplication on the block formats show an order-of-
4052 magnitude reduction in the size of the format file size against the
4053 simple formats. As would be expected, the compression tools are able
4054 to find and exploit a lot of this duplication, but as the
4055 deduplication process uses knowledge of DNS traffic, it is able to
4056 retain a size advantage. This advantage reduces as stronger
4057 compression is applied, as again would be expected, but even with the
4058 strongest compression applied the block-formatted data remains around
4059 75% of the size of the simple format and its compression requires
4060 roughly a third of the CPU time.
4061
4062
4063
4064
4065
4066
4067 Dickinson, et al. Standards Track [Page 74]
4068 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
4069
4070
4071 C.3. Binary versus Text Formats
4072
4073 Text data formats offer many advantages over binary formats,
4074 particularly in the areas of ad hoc data inspection and extraction.
4075 It was therefore felt worthwhile to carry out a direct comparison,
4076 implementing JSON versions of the simple and block formats.
4077
4078 Concentrating on JSON block format, the format files produced are a
4079 significant fraction of an order of magnitude larger than binary
4080 formats. The impact on file size after compression is as might be
4081 expected from that starting point; the stronger compression produces
4082 files that are 150% of the size of similarly compressed binary format
4083 and require over 4x more CPU to compress.
4084
4085 C.4. Performance
4086
4087 Concentrating again on the block formats, all three produce format
4088 files that are close to an order of magnitude smaller than the
4089 original "test.pcap" file. CBOR produces the largest files and Avro
4090 the smallest, 20% smaller than CBOR.
4091
4092 However, once compression is taken into account, the size difference
4093 narrows. At medium compression (with gzip), the size difference is
4094 4%. Using strong compression (with xz), the difference reduces to
4095 2%, with Avro the largest and Protocol Buffers the smallest, although
4096 CBOR and Protocol Buffers require slightly more compression CPU.
4097
4098 The measurements presented above do not include data on the CPU
4099 required to generate the format files. Measurements indicate that
4100 writing Avro requires 10% more CPU than CBOR or Protocol Buffers. It
4101 appears, therefore, that Avro's advantage in compression CPU usage is
4102 probably offset by a larger CPU requirement in writing Avro.
4103
4104 C.5. Conclusions
4105
4106 The above assessments lead us to the choice of a binary format file
4107 using blocking.
4108
4109 As noted previously, this document anticipates that output data will
4110 be subject to compression. There is no compelling case for one
4111 particular binary serialization format in terms of either final file
4112 size or machine resources consumed, so the choice must be largely
4113 based on other factors. CBOR was therefore chosen as the binary
4114 serialization format for the reasons listed in Section 5.
4115
4116
4117
4118
4119
4120
4121
4122 Dickinson, et al. Standards Track [Page 75]
4123 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
4124
4125
4126 C.6. Block Size Choice
4127
4128 Given the choice of a CBOR format using blocking, the question arises
4129 of what an appropriate default value for the maximum number of
4130 Query/Response pairs in a block should be. This has two components:
4131
4132 1. What is the impact on performance of using different block sizes
4133 in the format file?
4134
4135 2. What is the impact on the size of the format file before and
4136 after compression?
4137
4138 The following table addresses the performance question, showing the
4139 impact on the performance of a C++ program converting "test.pcap"
4140 to C-DNS. File sizes are in MB, RSS is in kB, and user time is
4141 in seconds.
4142
4143 +------------+-----------+--------+-----------+
4144 | Block Size | File Size | RSS | User Time |
4145 +------------+-----------+--------+-----------+
4146 | 1,000 | 133.46 | 612.27 | 15.25 |
4147 | 5,000 | 89.85 | 676.82 | 14.99 |
4148 | 10,000 | 76.87 | 752.40 | 14.53 |
4149 | 20,000 | 67.86 | 750.75 | 14.49 |
4150 | 40,000 | 61.88 | 736.30 | 14.29 |
4151 | 80,000 | 58.08 | 694.16 | 14.28 |
4152 | 160,000 | 55.94 | 733.84 | 14.44 |
4153 | 320,000 | 54.41 | 799.20 | 13.97 |
4154 +------------+-----------+--------+-----------+
4155
4156 Therefore, increasing block size tends to increase maximum RSS a
4157 little, with no significant effect (if anything, a small reduction)
4158 on CPU consumption.
4159
4160
4161
4162
4163
4164
4165
4166
4167
4168
4169
4170
4171
4172
4173
4174
4175
4176
4177 Dickinson, et al. Standards Track [Page 76]
4178 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
4179
4180
4181 The following table demonstrates the effect of increasing block size
4182 on output file size for different compressions.
4183
4184 +------------+--------+-------+-------+-------+-------+-------+
4185 | Block Size | None | snzip | lz4 | gzip | zstd | xz |
4186 +------------+--------+-------+-------+-------+-------+-------+
4187 | 1,000 | 133.46 | 90.52 | 90.03 | 74.65 | 44.78 | 25.63 |
4188 | 5,000 | 89.85 | 59.69 | 59.43 | 46.99 | 37.33 | 22.34 |
4189 | 10,000 | 76.87 | 50.39 | 50.28 | 38.94 | 33.62 | 21.09 |
4190 | 20,000 | 67.86 | 43.91 | 43.90 | 33.24 | 32.62 | 20.16 |
4191 | 40,000 | 61.88 | 39.63 | 39.69 | 29.44 | 28.72 | 19.52 |
4192 | 80,000 | 58.08 | 36.93 | 37.01 | 27.05 | 26.25 | 19.00 |
4193 | 160,000 | 55.94 | 35.10 | 35.06 | 25.44 | 24.56 | 19.63 |
4194 | 320,000 | 54.41 | 33.87 | 33.74 | 24.36 | 23.44 | 18.66 |
4195 +------------+--------+-------+-------+-------+-------+-------+
4196
4197 There is obviously scope for tuning the default block size to the
4198 compression being employed, traffic characteristics, frequency of
4199 output file rollover, etc. Using a strong compression scheme, block
4200 sizes over 10,000 Query/Response pairs would seem to offer limited
4201 improvements.
4202
4203 Appendix D. Data Fields for Traffic Regeneration
4204
4205 D.1. Recommended Fields for Traffic Regeneration
4206
4207 This section specifies the data fields that would need to be captured
4208 in order to perform the fullest PCAP traffic reconstruction for
4209 well-formed DNS messages that is possible with C-DNS.
4210
4211 o All data fields in the QueryResponse type except response-
4212 processing-data.
4213
4214 o All data fields in the QueryResponseSignature type except qr-type.
4215
4216 o All data fields in the RR TYPE.
4217
4218 D.2. Issues with Small Data Captures
4219
4220 At the other extreme, an interesting corner case arises when opting
4221 to perform captures with a smaller data set than that recommended
4222 above. The following list specifies a subset of the above data
4223 fields; if only these data fields are captured, then even a minimal
4224 traffic reconstruction is problematic because there is not enough
4225 information to determine if the Query/Response data item contained
4226 just a Query, just a Response, or a Query/Response pair.
4227
4228
4229
4230
4231
4232 Dickinson, et al. Standards Track [Page 77]
4233 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
4234
4235
4236 o The following data fields from the QueryResponse type:
4237
4238 * time-offset
4239
4240 * client-address-index
4241
4242 * client-port
4243
4244 * transaction-id
4245
4246 * query-name-index
4247
4248 o The following data fields from the QueryResponseSignature type:
4249
4250 * server-address-index
4251
4252 * server-port
4253
4254 * qr-transport-flags
4255
4256 * query-classtype-index
4257
4258 In this case, simply also capturing the qr-sig-flags will provide
4259 enough information to perform a minimal traffic reconstruction
4260 (assuming that suitable defaults for the remaining fields are
4261 provided). Additionally, capturing response-delay, query-opcode, and
4262 response-rcode will avoid having to rely on potentially misleading
4263 defaults for these values and should result in a PCAP that represents
4264 the basics of the real traffic flow.
4265
4266 Acknowledgements
4267
4268 The authors wish to thank CZ.NIC -- in particular, Tomas Gavenciak --
4269 for many useful discussions on binary formats, compression, and
4270 packet matching. Thanks also to Jan Vcelak and Wouter Wijngaards for
4271 discussions on name compression, and Paul Hoffman for a detailed
4272 review of this document and the C-DNS CDDL.
4273
4274 Thanks also to Robert Edmonds, Jerry Lundstrom, Richard Gibson,
4275 Stephane Bortzmeyer, and many other members of DNSOP for review.
4276
4277 Also, thanks to Miek Gieben for [mmark].
4278
4279
4280
4281
4282
4283
4284
4285
4286
4287 Dickinson, et al. Standards Track [Page 78]
4288 RFC 8618 C-DNS: A Format for DNS Packet Capture September 2019
4289
4290
4291 Authors' Addresses
4292
4293 John Dickinson
4294 Sinodun IT
4295 Magdalen Centre
4296 Oxford Science Park
4297 Oxford OX4 4GA
4298 United Kingdom
4299 Email: jad@sinodun.com
4300
4301
4302 Jim Hague
4303 Sinodun IT
4304 Magdalen Centre
4305 Oxford Science Park
4306 Oxford OX4 4GA
4307 United Kingdom
4308 Email: jim@sinodun.com
4309
4310
4311 Sara Dickinson
4312 Sinodun IT
4313 Magdalen Centre
4314 Oxford Science Park
4315 Oxford OX4 4GA
4316 United Kingdom
4317 Email: sara@sinodun.com
4318
4319
4320 Terry Manderson
4321 ICANN
4322 12025 Waterfront Drive
4323 Suite 300
4324 Los Angeles, CA 90094-2536
4325 United States of America
4326 Email: terry.manderson@icann.org
4327
4328
4329 John Bond
4330 Wikimedia Foundation, Inc.
4331 1 Montgomery Street
4332 Suite 1600
4333 San Francisco, CA 94104
4334 United States of America
4335 Email: ietf-wikimedia@johnbond.org
4336
4337
4338
4339
4340
4341
4342 Dickinson, et al. Standards Track [Page 79]
4343
The IANA registry is at https://www.iana.org/assignments/c-dns/c-dns.xhtml