Fix #575 - Add extended set validation

docs/admins/configuration.rst

@@ -17,6 +17,7 @@ a key ``roa_source`` under a key ``rpki``.
 .. contents::
    :backlinks: none
    :local:
+   :depth: 2
 
 Example configuration file
 --------------------------
@@ -294,21 +295,21 @@ Email
   |br| `parent of newly created object(s).`
 
 
-Authentication
-~~~~~~~~~~~~~~
+Authentication and validation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 * ``auth.override_password``: a salted MD5 hash of the override password,
   which can be used to override any
   authorisation requirements for authoritative databases.
   |br| **Default**: not defined, no override password will be accepted.
   |br| **Change takes effect**: upon the next update attempt.
-* ``auth.authenticate_related_mntners``: whether to check for
-  :ref:`related object maintainers <auth-related-mntners>` when processing
-  updates.
-  |br| **Default**: true, check enabled
-  |br| **Change takes effect**: upon the next update attempt.
 * ``auth.gnupg_keyring``: the full path to the gnupg keyring.
   |br| **Default**: not defined, but required.
   |br| **Change takes effect**: after full IRRd restart.
+* ``auth.authenticate_parents_route_creation``: whether to check for
+  :ref:`related object maintainers <auth-related-mntners-route>` when users create
+  new `route(6)` objects.
+  |br| **Default**: true, check enabled
+  |br| **Change takes effect**: upon the next update attempt.
 
 .. danger::
 
@@ -317,6 +318,77 @@ Authentication
     changes. Therefore, the keyring referred to by ``auth.gnupg_keyring`` can
     not be simply reset, or PGP authentications may fail.
 
+.. _conf-auth-set-creation:
+
+auth.set_creation
+"""""""""""""""""
+The ``auth.set_creation`` setting configures the requirements when creating new
+RPSL set objects. These are `as-set`, `filter-set`, `peering-set`, `route-set`
+and `rtr-set`. It is highly customisable, but therefore also more complex
+than most other settings.
+
+There are two underlying settings:
+
+* ``prefix_required`` configures whether an ASN prefix is required in the name
+  of a set object. When enabled ``AS-EXAMPLE`` is invalid, while
+  ``AS65537:AS-EXAMPLE`` or ``AS65537:AS-EXAMPLE:AS-CUSTOMERS``
+  are valid.
+* ``autnum_authentication`` controls whether the user also needs to pass
+  authentication for the `aut-num` corresponding to the AS number used as the set
+  name prefix. For example, if the set name is ``AS65537:AS-EXAMPLE:AS-CUSTOMERS``,
+  this setting may require the creation to also pass authentication for the
+  `aut-num` AS65537.
+  The options are ``disabled``, ``opportunistic`` or ``required``.
+  When disabled, this check is skipped. For opportunistic, the check is used, but
+  passes if the aut-num does not exist. For required, the check is used and fails
+  if the aut-num does not exist.
+
+Note that even when ``autnum_authentication`` is set to ``required``,
+if at the same time ``prefix_required`` is set to false, a set can be created
+without a prefix or with one, per ``prefix_required``.
+But if it has a prefix, there *must* be a corresponding
+aut-num object for which authentication *must* pass, per ``autnum_authentication``.
+
+You can configure one common for all set classes under the key ``COMMON``,
+and/or specific settings for specific classes using the class name as key.
+An example::
+
+    irrd:
+      auth:
+          set_creation:
+              COMMON:
+                  prefix_required: true
+                  autnum_authentication: required
+              as-set:
+                  prefix_required: true
+                  autnum_authentication: opportunistic
+              rtr-set:
+                  prefix_required: false
+                  autnum_authentication: disabled
+
+This example means:
+
+* New `as-set` objects must include an ASN prefix in their name
+  and the user must pass authentication for the corresponding `aut-num` object,
+  if it exists. If the `aut-num` does not exist, the check passes.
+* New ``rtr-set`` objects are not required to include an ASN prefix in their
+  name, but this is permitted. The user never has to pass authentication for
+  the corresponding `aut-num` object, regardless of whether it exists.
+* All other new set objects must include an ASN prefix in their name, an `aut-num`
+  corresponding that AS number must exist, and the user must pass authentication
+  for that `aut-num` object.
+
+All checks are only applied when users create new set objects in authoritative
+databases. Authoritative updates to existing objects, deletions, or objects from
+mirrors are never affected. When looking for corresponding `aut-num` objects,
+IRRd only looks in the same IRR source.
+
+**Default**: ``prefix_required`` is enabled, ``autnum_authentication``
+set to ``opportunistic`` for all sets. Note that settings under the
+``COMMON`` key override these IRRd defaults, and settings under set-specific
+keys in turn override settings under the ``COMMON`` key.
+|br| **Change takes effect**: upon the next update attempt.
+
 
 Access lists
 ~~~~~~~~~~~~
@@ -752,13 +824,6 @@ Compatibility
   See the :doc:`4.2.0 release notes </releases/4.2.0>` for details.
   |br| **Default**: ``false``, operating normally.
   |br| **Change takes effect**: after SIGHUP, for all subsequent queries.
-* ``compatibility.permit_non_hierarchical_as_set_name``: by default,
-  `as-set` objects created in authoritative databases are required to have a
-  hierarchical name, like ``AS65540:AS-CUSTOMERS``. For example,
-  ``AS-CUSTOMERS`` would not be allowed. If this setting is set to ``true``,
-  this name requirement does not apply, and ``AS-CUSTOMERS`` is permitted.
-  |br| **Default**: ``false``, hierarchical name required.
-  |br| **Change takes effect**: after SIGHUP, for all subsequent updates.
 * ``compatibility.ipv4_only_route_set_members``: if set to ``true``, ``!i``
   queries will not return IPv6 prefixes. This option can be used for limited
   compatibility with IRRd version 2. Enabling this setting may have a

docs/releases/4.1.0.rst

@@ -96,7 +96,7 @@ Other changes
   for further details.
 * When users create `route(6)` objects in authoritative databases, IRRd
   also verifies authorisation from
-  :ref:`related object maintainers <auth-related-mntners>`. This behaviour
+  :ref:`related object maintainers <auth-related-mntners-route>`. This behaviour
   is enabled by default, but can be disabled with the
   ``auth.authenticate_related_mntners`` setting.
 * The ``!j`` command has changed, and now is exclusively used to check

docs/releases/4.3.0.rst

@@ -0,0 +1,40 @@
+============================
+Release notes for IRRd 4.3.0
+============================
+
+Changes to related object authentication and settings
+-----------------------------------------------------
+In version 4.2, IRRd required newly created authoritative `as-set` objects
+to have a hierarchical name where the first element is an AS number.
+In 4.3, this feature has been significantly expanded.
+
+For all RPSL set objects, IRRd can now be configured to require:
+
+* Including an ASN prefix in the name of the set, e.g. ``AS65537:AS-EXAMPLE``
+  being valid, but ``AS-EXAMPLE`` being invalid.
+* Passing authentication for the corresponding `aut-num`, e.g. AS65537 in the
+  example, skipping this check if the `aut-num` does not exist.
+* Passing authentication for the corresponding `aut-num`, e.g. AS65537 in the
+  example, failing this check if the `aut-num` does not exist.
+
+The first two options, requiring a prefix with opportunistic `aut-num` authentication,
+is now the default for all set objects.
+You can :ref:`change the configuration <conf-auth-set-creation>` for specific
+RPSL set objects, or set your own common configuration that applies to all sets.
+
+The ``compatibility.permit_non_hierarchical_as_set_name`` setting has been
+removed, as it is now covered by the ``prefix_required`` setting.
+
+The ``auth.authenticate_related_mntners`` setting has been renamed to 
+``auth.authenticate_parents_route_creation``, as this setting exclusively
+relates to :ref:`authentication for route(6) objects <auth-related-mntners-route>`
+and needs to be distinct from the configuration for RPSL set objects.
+
+If you were using ``auth.authenticate_related_mntners`` or 
+``compatibility.permit_non_hierarchical_as_set_name``, you need to update
+your configuration.
+
+All checks are only applied when users create new set objects in authoritative
+databases. Authoritative updates to existing objects, deletions, or objects from
+mirrors are never affected. When looking for related objects,
+IRRd only looks in the same IRR source.

docs/spelling_wordlist.txt

@@ -1,5 +1,7 @@
 ASes
+aut
 CPython
+customisable
 Escribano
 IPtables
 IPv
@@ -46,6 +48,7 @@ nginx
 noreply
 nrtm
 ntt
+num
 ov
 pidfile
 postgres

docs/users/database-changes.rst

@@ -269,11 +269,13 @@ When you create a new `mntner`, a submission must pass authorisation for
 one of the auth methods of the new mntner. You can submit other objects
 that depend on the new `mntner` in the same submission.
 
-.. _auth-related-mntners:
+.. _auth-related-mntners-route:
 
+Related maintainers in route objects
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 When you create new `route(6)` objects, authentication also needs to pass
 for the parent object. IRRd searches for the parent object in the following
-order, only considering the first match:
+order, only considering the first match, only looking in the same IRR source:
 
 * An `inet(6)num` that is an exact match to the new `route(6)`.
 * The smallest `inet(6)num` that is a less specific of the new `route(6)`.
@@ -282,7 +284,32 @@ order, only considering the first match:
 If no objects match, there is no parent object, and there are no extra
 authentication requirements.
 This behaviour can be disabled by setting
-``auth.authenticate_related_mntners`` to false.
+``auth.authenticate_parents_route_creation`` to false.
+These requirements do not apply when you change or delete existing objects.
+
+.. _auth-related-mntners-set:
+
+Related maintainers in set objects
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+When you create new set objects, you may need to pass authentication for the
+parent `aut-num` object.
+RPSL set objects are `as-set`, `filter-set`, `peering-set`, `route-set` and
+`rtr-set`.
+
+The details of this behaviour and the strictness of the checks are
+:ref:`configured by the IRR operator <conf-auth-set-creation>`. This may
+include a requirement to:
+
+* Include an ASN prefix in the name of your set, e.g. ``AS65537:AS-EXAMPLE``
+  being valid, but ``AS-EXAMPLE`` being invalid.
+* Pass authentication for the corresponding `aut-num`, e.g. AS65537 in the
+  example, skipping this check if the `aut-num` does not exist.
+* Pass authentication for the corresponding `aut-num`, e.g. AS65537 in the
+  example, failing this check if the `aut-num` does not exist.
+
+These requirements do not apply when you change or delete existing objects.
+When looking for corresponding `aut-num` objects,
+IRRd only looks in the same IRR source.
 
 Object templates
 ----------------
@@ -323,12 +350,12 @@ This template shows:
 * The `member-of` attribute is a look-up key, meaning it can be used with
   ``-i`` queries.
 * The `member-of` attribute references to the `route-set` class. It is a
-  weak references, meaning the referred `route-set` does not have to exist,
+  weak reference, meaning the referred `route-set` does not have to exist,
   but is required to meet the syntax of a `route-set` name. The attribute
   is also optional, so it can be left out entirely.
 * The `admin-c` and `tech-c` attributes reference a `role` or `person`.
   This means they may refer to either object class, but must be a
-  reference to a valid, existing `person` or `role. This `person` or
+  reference to a valid, existing `person` or `role`. This `person` or
   `role` can be created as part of the same submission.