1 Internet Engineering Task Force (IETF) M. Loffredo
2 Request for Comments: 8977 M. Martinelli
3 Category: Standards Track IIT-CNR/Registro.it
4 ISSN: 2070-1721 S. Hollenbeck
5 Verisign Labs
6 January 2021
7
8
9 Registration Data Access Protocol (RDAP) Query Parameters for Result
10 Sorting and Paging
11
12 Abstract
13
14 The Registration Data Access Protocol (RDAP) does not include core
15 functionality for clients to provide sorting and paging parameters
16 for control of large result sets. This omission can lead to
17 unpredictable server processing of queries and client processing of
18 responses. This unpredictability can be greatly reduced if clients
19 can provide servers with their preferences for managing large
20 responses. This document describes RDAP query extensions that allow
21 clients to specify their preferences for sorting and paging result
22 sets.
23
24 Status of This Memo
25
26 This is an Internet Standards Track document.
27
28 This document is a product of the Internet Engineering Task Force
29 (IETF). It represents the consensus of the IETF community. It has
30 received public review and has been approved for publication by the
31 Internet Engineering Steering Group (IESG). Further information on
32 Internet Standards is available in Section 2 of RFC 7841.
33
34 Information about the current status of this document, any errata,
35 and how to provide feedback on it may be obtained at
36 https://www.rfc-editor.org/info/rfc8977.
37
38 Copyright Notice
39
40 Copyright (c) 2021 IETF Trust and the persons identified as the
41 document authors. All rights reserved.
42
43 This document is subject to BCP 78 and the IETF Trust's Legal
44 Provisions Relating to IETF Documents
45 (https://trustee.ietf.org/license-info) in effect on the date of
46 publication of this document. Please review these documents
47 carefully, as they describe your rights and restrictions with respect
48 to this document. Code Components extracted from this document must
49 include Simplified BSD License text as described in Section 4.e of
50 the Trust Legal Provisions and are provided without warranty as
51 described in the Simplified BSD License.
52
53 Table of Contents
54
55 1. Introduction
56 1.1. Conventions Used in This Document
57 2. RDAP Query Parameter Specification
58 2.1. Sorting and Paging Metadata
59 2.1.1. RDAP Conformance
60 2.2. "count" Parameter
61 2.3. "sort" Parameter
62 2.3.1. Sorting Properties Declaration
63 2.3.2. Representing Sorting Links
64 2.4. "cursor" Parameter
65 2.4.1. Representing Paging Links
66 3. Negative Answers
67 4. Implementation Considerations
68 5. IANA Considerations
69 6. Security Considerations
70 7. References
71 7.1. Normative References
72 7.2. Informative References
73 Appendix A. JSONPath Operators
74 Appendix B. Approaches to Result Pagination
75 B.1. Specific Issues Raised by RDAP
76 Appendix C. Implementation Notes
77 C.1. Sorting
78 C.2. Counting
79 C.3. Paging
80 Acknowledgements
81 Authors' Addresses
82
83 1. Introduction
84
85 The availability of functionality for result sorting and paging
86 provides benefits to both clients and servers in the implementation
87 of RESTful services [REST]. These benefits include:
88
89 * reducing the server response bandwidth requirements
90
91 * improving server response time
92
93 * improving query precision and, consequently, obtaining more
94 relevant results
95
96 * decreasing server query processing load
97
98 * reducing client response processing time
99
100 Approaches to implementing features for result sorting and paging can
101 be grouped into two main categories:
102
103 1. Sorting and paging are implemented through the introduction of
104 additional parameters in the query string (e.g., the Open Data
105 Protocol (ODATA) [ODATA-PART1]).
106
107 2. Information related to the number of results and the specific
108 portion of the result set to be returned, in addition to a set of
109 ready-made links for the result set scrolling, are inserted in
110 the HTTP header of the request/response [RFC7231].
111
112 However, there are some drawbacks associated with the use of the HTTP
113 header. First, the header properties cannot be set directly from a
114 web browser. Moreover, in an HTTP session, the information on the
115 status (i.e., the session identifier) is usually inserted in the
116 header or a cookie, while the information on the resource
117 identification or the search type is included in the query string.
118 Finally, providing custom information through HTTP headers assumes
119 the client has prior knowledge of the server implementation, which is
120 widely considered a Representational State Transfer (REST) design
121 anti-pattern. As a result, this document describes a specification
122 based on the use of query parameters.
123
124 Currently, RDAP [RFC7482] defines two query types:
125
126 lookup: the server returns only one object
127
128 search: the server returns a collection of objects
129
130 While the lookup query does not raise issues regarding response size
131 management, the search query can potentially generate a large result
132 set that is often truncated according to server limits. Besides, it
133 is not possible to obtain the total number of objects found that
134 might be returned in a search query response [RFC7483]. Lastly,
135 there is no way to specify sort criteria to return the most relevant
136 objects at the beginning of the result set. Therefore, the client
137 might traverse the whole result set to find the relevant objects or,
138 due to truncation, might not find them at all.
139
140 The specification described in this document extends RDAP query
141 capabilities to enable result sorting and paging by adding new query
142 parameters that can be applied to RDAP search path segments. The
143 service is implemented using the Hypertext Transfer Protocol (HTTP)
144 [RFC7230] and the conventions described in [RFC7480].
145
146 The implementation of the new parameters is technically feasible, as
147 operators for counting, sorting, and paging rows are currently
148 supported by the major relational database management systems.
149
150 1.1. Conventions Used in This Document
151
152 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
153 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
154 "OPTIONAL" in this document are to be interpreted as described in
155 BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
156 capitals, as shown here.
157
158 2. RDAP Query Parameter Specification
159
160 The new query parameters are OPTIONAL extensions of path segments
161 defined in [RFC7482]. They are as follows:
162
163 "count": a boolean value that allows a client to request the return
164 of the total number of objects found
165
166 "sort": a string value that allows a client to request a specific
167 sort order for the result set
168
169 "cursor": a string value representing a pointer to a specific fixed-
170 size portion of the result set
171
172 Augmented Backus-Naur Form (ABNF) [RFC5234] is used in the following
173 sections to describe the formal syntax of these new parameters.
174
175 2.1. Sorting and Paging Metadata
176
177 According to most advanced principles in REST design, collectively
178 known as HATEOAS (Hypermedia as the Engine of Application State)
179 [HATEOAS], a client entering a REST application through an initial
180 URI should use server-provided links to dynamically discover
181 available actions and access the resources it needs. In this way,
182 the client is neither required to have prior knowledge of the service
183 nor, consequently, to hard code the URIs of different resources.
184 This allows the server to make URI changes as the API evolves without
185 breaking clients. Definitively, a REST service should be as self-
186 descriptive as possible.
187
188 Therefore, servers implementing the query parameters described in
189 this specification SHOULD provide additional information in their
190 responses about both the available sorting criteria and possible
191 pagination. Such information is collected in two OPTIONAL response
192 elements named "sorting_metadata" and "paging_metadata".
193
194 The "sorting_metadata" element contains the following properties:
195
196 "currentSort": "String" (OPTIONAL)
197 Either the value of the "sort" parameter as specified in the query
198 string or the sort applied by default, if any.
199
200 "availableSorts": "AvailableSort[]" (OPTIONAL)
201 An array of objects, with each element describing an available
202 sort criterion. The AvailableSort object includes the following
203 members:
204
205 "property": "String" (REQUIRED)
206 The name that can be used by the client to request the sort
207 criterion.
208
209 "default": "Boolean" (REQUIRED)
210 Indicator of whether the sort criterion is applied by default.
211 An RDAP server MUST define only one default sorting property
212 for each object class.
213
214 "jsonPath": "String" (OPTIONAL)
215 The JSONPath expression of the RDAP field corresponding to the
216 property.
217
218 "links": "Link[]" (OPTIONAL)
219 An array of links as described in [RFC8288] containing the
220 query string that applies the sort criterion.
221
222 At least one of the "currentSort" and "availableSorts" properties
223 MUST be present.
224
225 The "paging_metadata" element contains the following fields:
226
227 "totalCount": "Numeric" (OPTIONAL)
228 A numeric value representing the total number of objects found.
229 It MUST be provided if and only if the query string contains the
230 "count" parameter.
231
232 "pageSize": "Numeric" (OPTIONAL)
233 A numeric value representing the number of objects that should
234 have been returned in the current page. It MUST be provided if
235 and only if the total number of objects exceeds the page size.
236 This property is redundant for RDAP clients because the page size
237 can be derived from the length of the search results array, but it
238 can be helpful if the end user interacts with the server through a
239 web browser.
240
241 "pageNumber": "Numeric" (OPTIONAL)
242 A numeric value representing the number of the current page in the
243 result set. It MUST be provided if and only if the total number
244 of objects found exceeds the page size.
245
246 "links": "Link[]" (OPTIONAL)
247 An array of links as described in [RFC8288] containing the
248 reference to the next page. In this specification, only forward
249 pagination is described because it is all that is necessary to
250 traverse the result set.
251
252 2.1.1. RDAP Conformance
253
254 Servers returning the "paging_metadata" element in their response
255 MUST include the string literal "paging" in the rdapConformance
256 array. Servers returning the "sorting_metadata" element MUST include
257 the string literal "sorting".
258
259 2.2. "count" Parameter
260
261 Currently, RDAP does not allow a client to determine the total number
262 of results in a query response when the result set is truncated.
263 This is inefficient because the user cannot determine if the result
264 set is complete.
265
266 The "count" parameter provides additional functionality that allows a
267 client to request information from the server that specifies the
268 total number of objects matching the search pattern.
269
270 The following is an example of an RDAP query including the "count"
271 parameter:
272
273 https://example.com/rdap/domains?name=example*.com&count=true
274
275 The ABNF syntax is the following:
276
277 count = "count=" ( trueValue / falseValue )
278 trueValue = ("true" / "yes" / "1")
279 falseValue = ("false" / "no" / "0")
280
281 A trueValue means that the server MUST provide the total number of
282 objects in the "totalCount" field of the "paging_metadata" element
283 (Figure 1). A falseValue means that the server MUST NOT provide this
284 number.
285
286 {
287 "rdapConformance": [
288 "rdap_level_0",
289 "paging"
290 ],
291 ...
292 "paging_metadata": {
293 "totalCount": 43
294 },
295 "domainSearchResults": [
296 ...
297 ]
298 }
299
300 Figure 1: Example of RDAP Response with "paging_metadata" Element
301 Containing the "totalCount" Field
302
303 2.3. "sort" Parameter
304
305 RDAP does not provide any capability to specify the result set sort
306 criteria. A server could implement a default sorting scheme
307 according to the object class, but this feature is not mandatory and
308 might not meet user requirements. Sorting can be addressed by the
309 client, but this solution is rather inefficient. Sorting features
310 provided by the RDAP server could help avoid truncation of relevant
311 results.
312
313 The "sort" parameter allows the client to ask the server to sort the
314 results according to the values of one or more properties and
315 according to the sort direction of each property. The ABNF syntax is
316 the following:
317
318 sort = "sort=" sortItem *( "," sortItem )
319 sortItem = property-ref [":" ( "a" / "d" ) ]
320 property-ref = ALPHA *( ALPHA / DIGIT / "_" )
321
322 "a" means that an ascending sort MUST be applied; "d" means that a
323 descending sort MUST be applied. If the sort direction is absent, an
324 ascending sort MUST be applied.
325
326 The following are examples of RDAP queries that include the "sort"
327 parameter:
328
329 https://example.com/rdap/domains?name=example*.com&sort=name
330
331 https://example.com/rdap/
332 domains?name=example*.com&sort=registrationDate:d
333
334 https://example.com/rdap/
335 domains?name=example*.com&sort=lockedDate,name
336
337 Except for sorting IP addresses and values denoting dates and times,
338 servers MUST implement sorting according to the JSON value type of
339 the RDAP field the sorting property refers to. That is, JSON strings
340 MUST be sorted lexicographically, and JSON numbers MUST be sorted
341 numerically. Values denoting dates and times MUST be sorted in
342 chronological order. If IP addresses are represented as JSON
343 strings, they MUST be sorted based on their numeric conversion.
344
345 The conversion of an IPv4 address to a number is possible since each
346 dotted format IPv4 address is a representation of a number written in
347 a 256-based manner; for example, 192.168.0.1 means 1*256^0 + 0*256^1
348 + 168*256^2 + 192*256^3 = 3232235521. Similarly, an IPv6 address can
349 be converted into a number by applying the base 65536. Therefore,
350 the numerical representation of the IPv6 address
351 2001:0db8:85a3:0:0:8a2e:0370:7334 is
352 42540766452641154071740215577757643572. Built-in functions and
353 libraries for converting IP addresses into numbers are available in
354 most known programming languages and relational database management
355 systems.
356
357 If the "sort" parameter presents an allowed sorting property, it MUST
358 be provided in the "currentSort" field of the "sorting_metadata"
359 element.
360
361 2.3.1. Sorting Properties Declaration
362
363 In the "sort" parameter ABNF syntax, the element named "property-ref"
364 represents a reference to a property of an RDAP object. Such a
365 reference could be expressed by using a JSONPath expression (named
366 "jsonpath" in the following).
367
368 JSONPath is a syntax, originally based on the XML XPath notation
369 [W3C.CR-xpath-31-20170321], which represents a path to select an
370 element (or a set of elements) in a JSON document [RFC8259]. For
371 example, the jsonpath to select the value of the ASCII name inside an
372 RDAP domain lookup response is "$.ldhName", where $ identifies the
373 root of the document object model (DOM). Another way to select a
374 value inside a JSON document is the JSON Pointer [RFC6901].
375
376 While JSONPath and JSON Pointer are both commonly adopted notations
377 to select any value inside JSON data, neither is particularly concise
378 and easy to use (e.g., "$.domainSearchResults[*].events[?(@.eventActi
379 on='registration')].eventDate" is the jsonpath of the registration
380 date in an RDAP domain search response).
381
382 Therefore, this specification defines the "property-ref" element in
383 terms of names identifying RDAP properties. However, not all the
384 RDAP properties are suitable to be used in sort criteria. These
385 properties include:
386
387 * properties providing service information (e.g., links, notices,
388 and remarks)
389
390 * multivalued properties (e.g., status, roles, and variants)
391
392 * properties representing relationships to other objects (e.g.,
393 entities)
394
395 On the contrary, properties expressed as values of other properties
396 (e.g., registration date) could be used in such a context.
397
398 A list of properties an RDAP server MAY implement is defined. The
399 properties are divided into two groups: object-common properties and
400 object-specific properties.
401
402 * Object-common properties. Object-common properties are derived
403 from merging the "eventAction" and the "eventDate" properties.
404 The following values of the "sort" parameter are defined:
405
406 - registrationDate
407
408 - reregistrationDate
409
410 - lastChangedDate
411
412 - expirationDate
413
414 - deletionDate
415
416 - reinstantiationDate
417
418 - transferDate
419
420 - lockedDate
421
422 - unlockedDate
423
424 * Object-specific properties. Note that some of these properties
425 are also defined as query path segments. These properties
426 include:
427
428 - Domain: name
429
430 - Nameserver: name, ipv4, ipv6
431
432 - Entity: fn, handle, org, email, voice, country, cc, city
433
434 The correspondence between these sorting properties and the RDAP
435 object classes is shown in Table 1. Some of the sorting properties
436 defined for the RDAP entity class are related to jCard elements
437 [RFC7095], but because jCard is the JSON format for vCard, the
438 corresponding definitions are included in the vCard specification
439 [RFC6350].
440
441 An RDAP server MUST NOT use the defined sorting properties with a
442 meaning other than that described in Table 1.
443
444 +============+============+=================+======+=======+======+
445 | Object | Sorting | RDAP property | RFC | RFC | RFC |
446 | class | property | | 7483 | 6350 | 8605 |
447 +============+============+=================+======+=======+======+
448 | Searchable | Common | eventAction | 4.5 | | |
449 | objects | properties | values suffixed | | | |
450 | | | by "Date" | | | |
451 +------------+------------+-----------------+------+-------+------+
452 | Domain | name | unicodeName/ | 5.3 | | |
453 | | | ldhName | | | |
454 +------------+------------+-----------------+------+-------+------+
455 | Nameserver | name | unicodeName/ | 5.2 | | |
456 | | | ldhName | | | |
457 +------------+------------+-----------------+------+-------+------+
458 | | ipv4 | v4 ipAddress | 5.2 | | |
459 +------------+------------+-----------------+------+-------+------+
460 | | ipv6 | v6 ipAddress | 5.2 | | |
461 +------------+------------+-----------------+------+-------+------+
462 | Entity | handle | handle | 5.1 | | |
463 +------------+------------+-----------------+------+-------+------+
464 | | fn | jCard fn | 5.1 | 6.2.1 | |
465 +------------+------------+-----------------+------+-------+------+
466 | | org | jCard org | 5.1 | 6.6.4 | |
467 +------------+------------+-----------------+------+-------+------+
468 | | voice | jCard tel with | 5.1 | 6.4.1 | |
469 | | | type="voice" | | | |
470 +------------+------------+-----------------+------+-------+------+
471 | | email | jCard email | 5.1 | 6.4.2 | |
472 +------------+------------+-----------------+------+-------+------+
473 | | country | country name in | 5.1 | 6.3.1 | |
474 | | | jCard adr | | | |
475 +------------+------------+-----------------+------+-------+------+
476 | | cc | country code in | 5.1 | | 3.1 |
477 | | | jCard adr | | | |
478 +------------+------------+-----------------+------+-------+------+
479 | | city | locality in | 5.1 | 6.3.1 | |
480 | | | jCard adr | | | |
481 +------------+------------+-----------------+------+-------+------+
482
483 Table 1: Definitions of Sorting Properties
484
485 Regarding the definitions in Table 1, some further considerations are
486 needed to disambiguate some cases:
487
488 * Since the response to a search on either domains or nameservers
489 might include both A-labels and U-labels [RFC5890] in general, a
490 consistent sorting policy MUST treat the unicodeName and ldhName
491 as two representations of the same value. The unicodeName value
492 MUST be used while sorting if it is present; when the unicodeName
493 is unavailable, the value of the ldhName MUST be used instead.
494
495 * The jCard "sort-as" parameter MUST be ignored for the sorting
496 capability described in this document.
497
498 * Even if a nameserver can have multiple IPv4 and IPv6 addresses,
499 the most common configuration includes one address for each IP
500 version. Therefore, this specification makes the assumption that
501 nameservers have a single IPv4 and/or IPv6 value. When more than
502 one address per IP version is presented, sorting MUST be applied
503 to the first value.
504
505 * Multiple events with a given action on an object might be
506 returned. If this occurs, sorting MUST be applied to the most
507 recent event.
508
509 * Except for handle values, all the sorting properties defined for
510 entity objects can be multivalued according to the definition of
511 vCard as given in [RFC6350]. When more than one value is
512 presented, sorting MUST be applied to the preferred value
513 identified by the parameter pref="1". If the "pref" parameter is
514 missing, sorting MUST be applied to the first value.
515
516 The "jsonPath" field in the "sorting_metadata" element is used to
517 clarify the RDAP response field the sorting property refers to. The
518 mapping between the sorting properties and the jsonpaths of the RDAP
519 response fields is shown below. The JSONPath operators used herein
520 are described in Appendix A.
521
522 * Searchable objects
523
524 registrationDate
525 $.domainSearchResults[*].events[?(@.eventAction=="registration"
526 )].eventDate
527
528 reregistrationDate
529 $.domainSearchResults[*].events[?(@.eventAction=="reregistratio
530 n")].eventDate
531
532 lastChangedDate
533 $.domainSearchResults[*].events[?(@.eventAction=="last
534 changed")].eventDate
535
536 expirationDate
537 $.domainSearchResults[*].events[?(@.eventAction=="expiration")]
538 .eventDate
539
540 deletionDate
541 $.domainSearchResults[*].events[?(@.eventAction=="deletion")].e
542 ventDate
543
544 reinstantiationDate
545 $.domainSearchResults[*].events[?(@.eventAction=="reinstantiati
546 on")].eventDate
547
548 transferDate
549 $.domainSearchResults[*].events[?(@.eventAction=="transfer")].e
550 ventDate
551
552 lockedDate
553 $.domainSearchResults[*].events[?(@.eventAction=="locked")].eve
554 ntDate
555
556 unlockedDate
557 $.domainSearchResults[*].events[?(@.eventAction=="unlocked")].e
558 ventDate
559
560 * Domain
561
562 name
563 $.domainSearchResults[*].[unicodeName,ldhName]
564
565 * Nameserver
566
567 name
568 $.nameserverSearchResults[*].[unicodeName,ldhName]
569
570 ipv4
571 $.nameserverSearchResults[*].ipAddresses.v4[0]
572
573 ipv6
574 $.nameserverSearchResults[*].ipAddresses.v6[0]
575
576 * Entity
577
578 handle
579 $.entitySearchResults[*].handle
580
581 fn
582 $.entitySearchResults[*].vcardArray[1][?(@[0]=="fn")][3]
583
584 org
585 $.entitySearchResults[*].vcardArray[1][?(@[0]=="org")][3]
586
587 voice
588 $.entitySearchResults[*].vcardArray[1][?(@[0]=="tel" &&
589 @[1].type=="voice")][3]
590
591 email
592 $.entitySearchResults[*].vcardArray[1][?(@[0]=="email")][3]
593
594 country
595 $.entitySearchResults[*].vcardArray[1][?(@[0]=="adr")][3][6]
596
597 cc
598 $.entitySearchResults[*].vcardArray[1][?(@[0]=="adr")][1].cc
599
600 city
601 $.entitySearchResults[*].vcardArray[1][?(@[0]=="adr")][3][3]
602
603 Additional notes on the provided jsonpaths:
604
605 * Those related to the event dates are defined only for the "domain"
606 object. To obtain the equivalent jsonpaths for "entity" and
607 "nameserver", the path segment "domainSearchResults" must be
608 replaced with "entitySearchResults" and "nameserverSearchResults",
609 respectively.
610
611 * Those related to jCard elements are specified without taking into
612 account the "pref" parameter. Servers that sort those values
613 identified by the "pref" parameter SHOULD update a jsonpath by
614 adding an appropriate filter. For example, if the email values
615 identified by pref="1" are considered for sorting, the jsonpath of
616 the "email" sorting property should be
617 $.entitySearchResults[*].vcardArray[1][?(@[0]=="email" &&
618 @[1].pref=="1")][3].
619
620 2.3.2. Representing Sorting Links
621
622 An RDAP server MAY use the "links" array of the "sorting_metadata"
623 element to provide ready-made references [RFC8288] to the available
624 sort criteria (Figure 2). Each link represents a reference to an
625 alternate view of the results.
626
627 The "value", "rel", and "href" JSON values MUST be specified. All
628 other JSON values are OPTIONAL.
629
630 {
631 "rdapConformance": [
632 "rdap_level_0",
633 "sorting"
634 ],
635 ...
636 "sorting_metadata": {
637 "currentSort": "name",
638 "availableSorts": [
639 {
640 "property": "registrationDate",
641 "jsonPath": "$.domainSearchResults[*]
642 .events[?(@.eventAction==\"registration\")].eventDate",
643 "default": false,
644 "links": [
645 {
646 "value": "https://example.com/rdap/domains?name=example*.com
647 &sort=name",
648 "rel": "alternate",
649 "href": "https://example.com/rdap/domains?name=example*.com
650 &sort=registrationDate",
651 "title": "Result Ascending Sort Link",
652 "type": "application/rdap+json"
653 },
654 {
655 "value": "https://example.com/rdap/domains?name=example*.com
656 &sort=name",
657 "rel": "alternate",
658 "href": "https://example.com/rdap/domains?name=example*.com
659 &sort=registrationDate:d",
660 "title": "Result Descending Sort Link",
661 "type": "application/rdap+json"
662 }
663 ]
664 },
665 ...
666 ]
667 },
668 "domainSearchResults": [
669 ...
670 ]
671 }
672
673 Figure 2: Example of a "sorting_metadata" Instance to Implement
674 Result Sorting
675
676 2.4. "cursor" Parameter
677
678 The "cursor" parameter defined in this specification can be used to
679 encode information about any pagination method. For example, in the
680 case of a simple implementation of the "cursor" parameter to
681 represent offset pagination information, the "cursor" value
682 "b2Zmc2V0PTEwMCxsaW1pdD01MA==" is the base64 encoding of
683 "offset=100,limit=50". Likewise, in a simple implementation to
684 represent keyset pagination information, the "cursor" value
685 "ZXhhbXBsZS1OLmNvbQ==" represents the base64 encoding of
686 "key=example-N.com" whereby the key value identifies the last row of
687 the current page.
688
689 Note that this specification uses a base64 encoding for cursor
690 obfuscation just for example. RDAP servers are NOT RECOMMENDED to
691 obfuscate the "cursor" value through a mere base64 encoding.
692
693 This solution lets RDAP providers implement a pagination method
694 according to their needs, a user's access level, and the submitted
695 query. Besides, servers can change the method over time without
696 announcing anything to clients. The considerations that have led to
697 this solution are described in more detail in Appendix B.
698
699 The ABNF syntax of the "cursor" parameter is the following:
700
701 cursor = "cursor=" 1*( ALPHA / DIGIT / "/" / "=" / "-" / "_" )
702
703 The following is an example of an RDAP query including the "cursor"
704 parameter:
705
706 https://example.com/rdap/domains?name=example*.com
707 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M=
708
709 2.4.1. Representing Paging Links
710
711 An RDAP server SHOULD use the "links" array of the "paging_metadata"
712 element to provide a ready-made reference [RFC8288] to the next page
713 of the result set (Figure 3). Examples of additional "rel" values a
714 server MAY implement are "first", "last", and "prev".
715
716 {
717 "rdapConformance": [
718 "rdap_level_0",
719 "paging"
720 ],
721 ...
722 "notices": [
723 {
724 "title": "Search query limits",
725 "type": "result set truncated due to excessive load",
726 "description": [
727 "search results for domains are limited to 50"
728 ]
729 }
730 ],
731 "paging_metadata": {
732 "totalCount": 73,
733 "pageSize": 50,
734 "pageNumber": 1,
735 "links": [
736 {
737 "value": "https://example.com/rdap/domains?name=example*.com",
738 "rel": "next",
739 "href": "https://example.com/rdap/domains?name=example*.com
740 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M=",
741 "title": "Result Pagination Link",
742 "type": "application/rdap+json"
743 }
744 ]
745 },
746 "domainSearchResults": [
747 ...
748 ]
749 }
750
751 Figure 3: Example of a "paging_metadata" Instance to Implement Cursor
752 Pagination
753
754 3. Negative Answers
755
756 The constraints for the values of parameters are defined by their
757 ABNF syntax. Therefore, each request that includes an invalid value
758 for a parameter SHOULD produce an HTTP 400 (Bad Request) response
759 code. The same response SHOULD be returned in the following cases:
760
761 * if sorting by either single or multiple properties, the client
762 provides an unsupported value for the "sort" parameter, as well as
763 a value related to an object property not included in the response
764
765 * if the client submits an invalid value for the "cursor" parameter
766
767 Optionally, the response MAY include additional information regarding
768 either the supported sorting properties or the correct "cursor" value
769 in the HTTP entity body (Figure 4).
770
771 {
772 "errorCode": 400,
773 "title": "Domain sorting property 'unknown' is not valid",
774 "description": [
775 "Supported domain sorting properties are:"
776 "'aproperty', 'anotherproperty'"
777 ]
778
779 }
780
781 Figure 4: Example of RDAP Error Response Due to an Invalid Domain
782 Sorting Property Included in the Request
783
784 4. Implementation Considerations
785
786 Implementation of the new parameters is technically feasible, as
787 operators for counting, sorting, and paging are currently supported
788 by the major relational database management systems. Similar
789 operators are completely or partially supported by the most well-
790 known NoSQL databases (e.g., MongoDB, CouchDB, HBase, Cassandra,
791 Hadoop, etc.). Additional implementation notes are included in
792 Appendix C.
793
794 5. IANA Considerations
795
796 IANA has registered the following values in the "RDAP Extensions"
797 registry:
798
799 Extension identifier: paging
800 Registry operator: Any
801 Published specification: RFC 8977
802 Contact: IETF <iesg@ietf.org>
803 Intended usage: This extension describes a best practice for result
804 set paging.
805
806 Extension identifier: sorting
807 Registry operator: Any
808 Published specification: RFC 8977
809 Contact: IETF <iesg@ietf.org>
810 Intended usage: This extension describes a best practice for result
811 set sorting.
812
813 6. Security Considerations
814
815 Security services for the operations specified in this document are
816 described in [RFC7481].
817
818 A search query typically requires more server resources (such as
819 memory, CPU cycles, and network bandwidth) when compared to a lookup
820 query. This increases the risk of server resource exhaustion and
821 subsequent denial of service. This risk can be mitigated by either
822 restricting search functionality or limiting the rate of search
823 requests. Servers can also reduce their load by truncating the
824 results in a response. However, this last security policy can result
825 in a higher inefficiency or risk due to acting on incomplete
826 information if the RDAP server does not provide any functionality to
827 return the truncated results.
828
829 The new parameters presented in this document provide RDAP operators
830 with a way to implement a server that reduces inefficiency risks.
831 The "count" parameter gives the client the ability to evaluate the
832 completeness of a response. The "sort" parameter allows the client
833 to obtain the most relevant information at the beginning of the
834 result set. This can reduce the number of unnecessary search
835 requests. Finally, the "cursor" parameter enables the user to scroll
836 the result set by submitting a sequence of sustainable queries within
837 server-acceptable limits.
838
839 7. References
840
841 7.1. Normative References
842
843 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
844 Requirement Levels", BCP 14, RFC 2119,
845 DOI 10.17487/RFC2119, March 1997,
846 <https://www.rfc-editor.org/info/rfc2119>.
847
848 [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax
849 Specifications: ABNF", STD 68, RFC 5234,
850 DOI 10.17487/RFC5234, January 2008,
851 <https://www.rfc-editor.org/info/rfc5234>.
852
853 [RFC5890] Klensin, J., "Internationalized Domain Names for
854 Applications (IDNA): Definitions and Document Framework",
855 RFC 5890, DOI 10.17487/RFC5890, August 2010,
856 <https://www.rfc-editor.org/info/rfc5890>.
857
858 [RFC6350] Perreault, S., "vCard Format Specification", RFC 6350,
859 DOI 10.17487/RFC6350, August 2011,
860 <https://www.rfc-editor.org/info/rfc6350>.
861
862 [RFC7095] Kewisch, P., "jCard: The JSON Format for vCard", RFC 7095,
863 DOI 10.17487/RFC7095, January 2014,
864 <https://www.rfc-editor.org/info/rfc7095>.
865
866 [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer
867 Protocol (HTTP/1.1): Message Syntax and Routing",
868 RFC 7230, DOI 10.17487/RFC7230, June 2014,
869 <https://www.rfc-editor.org/info/rfc7230>.
870
871 [RFC7231] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer
872 Protocol (HTTP/1.1): Semantics and Content", RFC 7231,
873 DOI 10.17487/RFC7231, June 2014,
874 <https://www.rfc-editor.org/info/rfc7231>.
875
876 [RFC7480] Newton, A., Ellacott, B., and N. Kong, "HTTP Usage in the
877 Registration Data Access Protocol (RDAP)", RFC 7480,
878 DOI 10.17487/RFC7480, March 2015,
879 <https://www.rfc-editor.org/info/rfc7480>.
880
881 [RFC7481] Hollenbeck, S. and N. Kong, "Security Services for the
882 Registration Data Access Protocol (RDAP)", RFC 7481,
883 DOI 10.17487/RFC7481, March 2015,
884 <https://www.rfc-editor.org/info/rfc7481>.
885
886 [RFC7482] Newton, A. and S. Hollenbeck, "Registration Data Access
887 Protocol (RDAP) Query Format", RFC 7482,
888 DOI 10.17487/RFC7482, March 2015,
889 <https://www.rfc-editor.org/info/rfc7482>.
890
891 [RFC7483] Newton, A. and S. Hollenbeck, "JSON Responses for the
892 Registration Data Access Protocol (RDAP)", RFC 7483,
893 DOI 10.17487/RFC7483, March 2015,
894 <https://www.rfc-editor.org/info/rfc7483>.
895
896 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
897 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
898 May 2017, <https://www.rfc-editor.org/info/rfc8174>.
899
900 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
901 Interchange Format", STD 90, RFC 8259,
902 DOI 10.17487/RFC8259, December 2017,
903 <https://www.rfc-editor.org/info/rfc8259>.
904
905 [RFC8288] Nottingham, M., "Web Linking", RFC 8288,
906 DOI 10.17487/RFC8288, October 2017,
907 <https://www.rfc-editor.org/info/rfc8288>.
908
909 [RFC8605] Hollenbeck, S. and R. Carney, "vCard Format Extensions:
910 ICANN Extensions for the Registration Data Access Protocol
911 (RDAP)", RFC 8605, DOI 10.17487/RFC8605, May 2019,
912 <https://www.rfc-editor.org/info/rfc8605>.
913
914 7.2. Informative References
915
916 [CURSOR] Nimesh, R., "Paginating Real-Time Data with Cursor Based
917 Pagination", July 2014, <https://www.sitepoint.com/
918 paginating-real-time-data-cursor-based-pagination/>.
919
920 [CURSOR-API1]
921 Facebook, "Facebook for Developers -- Using the Graph
922 API", <https://developers.facebook.com/docs/graph-api/
923 using-graph-api>.
924
925 [CURSOR-API2]
926 Twitter, "Twitter Ads API",
927 <https://developer.twitter.com/en/docs/twitter-ads-api>.
928
929 [GOESSNER-JSON-PATH]
930 Goessner, S., "JSONPath - XPath for JSON", February 2007,
931 <https://goessner.net/articles/JsonPath/>.
932
933 [HATEOAS] Jedrzejewski, B., "HATEOAS - a simple explanation",
934 February 2018, <https://www.e4developer.com/2018/02/16/
935 hateoas-simple-explanation/>.
936
937 [JSONPATH-COMPARISON]
938 "JSONPath Comparison",
939 <https://cburgmer.github.io/json-path-comparison/>.
940
941 [JSONPATH-WG]
942 IETF, "JSON Path (jsonpath)",
943 <https://datatracker.ietf.org/wg/jsonpath/about/>.
944
945 [ODATA-PART1]
946 Pizzo, M., Handl, R., and M. Zurmuehl, "OData Version 4.0.
947 Part 1: Protocol Plus Errata 03", June 2016,
948 <https://docs.oasis-
949 open.org/odata/odata/v4.0/errata03/os/complete/part1-
950 protocol/odata-v4.0-errata03-os-part1-protocol-
951 complete.pdf>.
952
953 [REST] Fielding, R., "Architectural Styles and the Design of
954 Network-based Software Architectures", 2000,
955 <https://www.ics.uci.edu/~fielding/pubs/dissertation/
956 fielding_dissertation.pdf>.
957
958 [RFC6901] Bryan, P., Ed., Zyp, K., and M. Nottingham, Ed.,
959 "JavaScript Object Notation (JSON) Pointer", RFC 6901,
960 DOI 10.17487/RFC6901, April 2013,
961 <https://www.rfc-editor.org/info/rfc6901>.
962
963 [SEEK] EverSQL, "Faster Pagination in Mysql - Why Order By With
964 Limit and Offset is Slow?", July 2017,
965 <https://www.eversql.com/faster-pagination-in-mysql-why-
966 order-by-with-limit-and-offset-is-slow/>.
967
968 [W3C.CR-xpath-31-20170321]
969 Robie, J., Dyck, M., and J. Spiegel, "XML Path Language
970 (XPath) 3.1", World Wide Web Consortium Recommendation
971 REC-xpath-31-20170321, March 2017,
972 <https://www.w3.org/TR/2017/REC-xpath-31-20170321/>.
973
974 Appendix A. JSONPath Operators
975
976 The jsonpaths used in this document are provided according to the
977 Goessner proposal [GOESSNER-JSON-PATH].
978
979 Such specification requires that implementations support a set of
980 "basic operators". These operators are used to access the elements
981 of a JSON structure like objects and arrays, as well as their
982 subelements (object members and array items, respectively). No
983 operations are defined for retrieving parent or sibling elements of a
984 given element. The root element is always referred to as $
985 regardless of it being an object or array.
986
987 Additionally, the specification permits implementations to support
988 arbitrary script expressions. These can be used to index into an
989 object or array, or to filter elements from an array. While script
990 expression behavior is implementation-defined, most implementations
991 support the basic relational and logical operators as well as both
992 object member and array item access, sufficiently similar for the
993 purpose of this document. Commonly supported operators/functions
994 divided into "top-level operators" and "filter operators" are
995 documented in Tables 2 and 3, respectively.
996
997 For more information on implementation interoperability issues, see
998 [JSONPATH-COMPARISON]. At the time of writing, work is beginning on
999 a standardization effort too (see [JSONPATH-WG]).
1000
1001 +===================+=========================================+
1002 | Operator | Description |
1003 +===================+=========================================+
1004 | $ | Root element |
1005 +-------------------+-----------------------------------------+
1006 | .<name> | Object member access (dot-notation) |
1007 +-------------------+-----------------------------------------+
1008 | ['<name>'] | Object member access (bracket-notation) |
1009 +-------------------+-----------------------------------------+
1010 | [<number>] | Array item access |
1011 +-------------------+-----------------------------------------+
1012 | * | All elements within the specified scope |
1013 +-------------------+-----------------------------------------+
1014 | [?(<expression>)] | Filter expression |
1015 +-------------------+-----------------------------------------+
1016
1017 Table 2: JSONPath Top-Level Operators
1018
1019 +====================+========================================+
1020 | Operator | Description |
1021 +====================+========================================+
1022 | @ | Current element being processed |
1023 +--------------------+----------------------------------------+
1024 | .<name> | Object member access |
1025 +--------------------+----------------------------------------+
1026 | .[<name1>,<name2>] | Union of object members |
1027 +--------------------+----------------------------------------+
1028 | [<number>] | Array item access |
1029 +--------------------+----------------------------------------+
1030 | == | Left is equal to right |
1031 +--------------------+----------------------------------------+
1032 | != | Left is not equal to right |
1033 +--------------------+----------------------------------------+
1034 | < | Left is less than right |
1035 +--------------------+----------------------------------------+
1036 | <= | Left is less than or equal to right |
1037 +--------------------+----------------------------------------+
1038 | > | Left is greater than right |
1039 +--------------------+----------------------------------------+
1040 | >= | Left is greater than or equal to right |
1041 +--------------------+----------------------------------------+
1042 | && | Logical conjunction |
1043 +--------------------+----------------------------------------+
1044 | || | Logical disjunction |
1045 +--------------------+----------------------------------------+
1046
1047 Table 3: JSONPath Filter Operators
1048
1049 Appendix B. Approaches to Result Pagination
1050
1051 An RDAP query could return a response with hundreds, even thousands,
1052 of objects, especially when partial matching is used. For this
1053 reason, the "cursor" parameter addressing result pagination is
1054 defined to make responses easier to handle.
1055
1056 Presently, the most popular methods to implement pagination in a REST
1057 API include offset pagination and keyset pagination. Neither
1058 pagination method requires the server to handle the result set in a
1059 storage area across multiple requests since a new result set is
1060 generated each time a request is submitted. Therefore, they are
1061 preferred to any other method requiring the management of a REST
1062 session.
1063
1064 Using limit and offset operators represents the traditionally used
1065 method to implement result pagination. Both of them can be used
1066 individually:
1067
1068 "limit=N": means that the server returns the first N objects of the
1069 result set
1070
1071 "offset=N": means that the server skips the first N objects and
1072 returns objects starting from position N+1
1073
1074 When limit and offset are used together, they provide the ability to
1075 identify a specific portion of the result set. For example, the pair
1076 "offset=100,limit=50" returns the first 50 objects starting from
1077 position 101 of the result set.
1078
1079 Though easy to implement, offset pagination also includes drawbacks:
1080
1081 * When offset has a very high value, scrolling the result set could
1082 take some time.
1083
1084 * It always requires fetching all rows before dropping as many rows
1085 as specified by offset.
1086
1087 * It may return inconsistent pages when data are frequently updated
1088 (i.e., real-time data).
1089
1090 Keyset pagination [SEEK] adds a query condition that enables the
1091 selection of the only data not yet returned. This method has been
1092 taken as the basis for the implementation of a "cursor" parameter
1093 [CURSOR] by some REST API providers [CURSOR-API1] [CURSOR-API2]. The
1094 cursor is a URL-safe string opaque to the client and representing a
1095 logical pointer to the first result of the next page.
1096
1097 Nevertheless, even keyset pagination can be troublesome:
1098
1099 * It needs at least one key field.
1100
1101 * It does not allow sorting simply by any field because the sorting
1102 criterion must contain a key.
1103
1104 * It works best with full composite values supported by database
1105 management systems (i.e., [x,y]>[a,b]); emulation is possible but
1106 inelegant and less efficient.
1107
1108 * It does not allow direct navigation to arbitrary pages because the
1109 result set must be scrolled in sequential order starting from the
1110 initial page.
1111
1112 * Implementing bidirectional navigation is tedious because all
1113 comparison and sort operations have to be reversed.
1114
1115 B.1. Specific Issues Raised by RDAP
1116
1117 Some additional considerations can be made in the RDAP context:
1118
1119 * An RDAP object is a conceptual aggregation of information
1120 generally collected from more than one data structure (e.g.,
1121 table), and this makes it even harder to implement keyset
1122 pagination, a task that is already quite difficult. For example,
1123 the entity object can include information from different data
1124 structures (registrars, registrants, contacts, resellers), each
1125 one with its key field mapping the RDAP entity handle.
1126
1127 * Depending on the number of page results as well as the number and
1128 the complexity of the properties of each RDAP object in the
1129 response, the time required by offset pagination to skip the
1130 previous pages could be much faster than the processing time
1131 needed to build the current page. In fact, RDAP objects are
1132 usually formed by information belonging to multiple data
1133 structures and containing multivalued properties (i.e., arrays);
1134 therefore, data selection might be a time-consuming process. This
1135 situation occurs even though the selection is supported by
1136 indexes.
1137
1138 * Depending on the access levels defined by each RDAP operator, the
1139 increase in complexity and the decrease in flexibility of keyset
1140 pagination in comparison to offset pagination could be considered
1141 impractical.
1142
1143 Ultimately, both pagination methods have benefits and drawbacks.
1144
1145 Appendix C. Implementation Notes
1146
1147 This section contains an overview of the main choices made during the
1148 implementation of the capabilities defined in this document in the
1149 RDAP public test server of Registro.it at the Institute of
1150 Informatics and Telematics of the National Research Council (IIT-
1151 CNR). The content of this section can represent guidance for
1152 implementers who plan to provide RDAP users with those capabilities.
1153 The RDAP public test server can be accessed at
1154 <https://rdap.pubtest.nic.it/>. Further documentation about the
1155 server features is available at <https://rdap.pubtest.nic.it/doc/
1156 README.html>.
1157
1158 C.1. Sorting
1159
1160 If no sort criterion is specified in the query string, the results
1161 are sorted by a default property: "name" for domains and nameservers,
1162 and "handle" for entities. The server supports multiple property
1163 sorting but the "sorting_metadata" object includes only the links to
1164 alternative result set views sorted by a single property just to show
1165 the list of sorting properties allowed for each searchable object.
1166 The server supports all the object-specific sorting properties
1167 described in the specification except for nameserver sorting based on
1168 unicodeName, that is, the "name" sorting property is mapped onto the
1169 "ldhName" response field. Regarding the object-common properties,
1170 sorting by registrationDate, expirationDate, lastChangedDate, and
1171 transferDate is supported.
1172
1173 C.2. Counting
1174
1175 The counting operation is implemented through a separate query. Some
1176 relational database management systems support custom operators to
1177 get the total count together with the rows, but the resulting query
1178 can be considerably more expensive than that performed without the
1179 total count. Therefore, as "totalCount" is an optional response
1180 information, always fetching the total number of rows has been
1181 considered an inefficient solution. Furthermore, to avoid the
1182 processing of unnecessary queries, when the "count" parameter is
1183 included in the submitted query, it is not also repeated in the query
1184 strings of the "links" array provided in both "paging_metadata" and
1185 "sorting_metadata" objects.
1186
1187 C.3. Paging
1188
1189 The server implements the cursor pagination through the keyset
1190 pagination when sorting by a unique property is requested or the
1191 default sort is applied. Otherwise, it implements the cursor
1192 pagination through the offset pagination. As most relational
1193 database management systems don't support the comparison of full
1194 composite values natively, the implementation of full keyset
1195 pagination seem to be troublesome so, at least initially, a selective
1196 applicability of keyset pagination is advisable. Moreover, the
1197 "cursor" value encodes not only information about pagination but also
1198 about the search pattern and the other query parameters in order to
1199 check the consistency of the entire query string. If the "cursor"
1200 value is inconsistent with the rest of the query string, the server
1201 returns an error response.
1202
1203 Acknowledgements
1204
1205 The authors would like to acknowledge Brian Mountford, Tom Harrison,
1206 Karl Heinz Wolf, Jasdip Singh, Erik Kline, Éric Vyncke, Benjamin
1207 Kaduk, and Roman Danyliw for their contributions to the development
1208 of this document.
1209
1210 Authors' Addresses
1211
1212 Mario Loffredo
1213 IIT-CNR/Registro.it
1214 Via Moruzzi,1
1215 56124 Pisa
1216 Italy
1217
1218 Email: mario.loffredo@iit.cnr.it
1219 URI: https://www.iit.cnr.it
1220
1221
1222 Maurizio Martinelli
1223 IIT-CNR/Registro.it
1224 Via Moruzzi,1
1225 56124 Pisa
1226 Italy
1227
1228 Email: maurizio.martinelli@iit.cnr.it
1229 URI: https://www.iit.cnr.it
1230
1231
1232 Scott Hollenbeck
1233 Verisign Labs
1234 12061 Bluemont Way
1235 Reston, VA 20190
1236 United States of America
1237
1238 Email: shollenbeck@verisign.com
1239 URI: https://www.verisignlabs.com/
1240
The IETF is responsible for the creation and maintenance of the DNS RFCs. The ICANN DNS RFC annotation project provides a forum for collecting community annotations on these RFCs as an aid to understanding for implementers and any interested parties. The annotations displayed here are not the result of the IETF consensus process.
This RFC is included in the DNS RFCs annotation project whose home page is here.