Fix #618 - Add alisases for source definitions (#815)

.circleci/config.yml

@@ -107,10 +107,10 @@ commands:
 
       - restore_cache:
           keys:
-            - v3-dependencies-{{ .Environment.CIRCLE_JOB }}-{{ checksum
+            - v4-dependencies-{{ .Environment.CIRCLE_JOB }}-{{ checksum
               "poetry.lock" }}
             # fallback to using the latest cache if no exact match is found
-            - v3-dependencies-{{ .Environment.CIRCLE_JOB }}
+            - v4-dependencies-{{ .Environment.CIRCLE_JOB }}
 
       - run:
           name: install python dependencies
@@ -122,7 +122,7 @@ commands:
       - save_cache:
           paths:
             - /mnt/ramdisk/.venv
-          key: v3-dependencies-{{ .Environment.CIRCLE_JOB }}-{{ checksum
+          key: v4-dependencies-{{ .Environment.CIRCLE_JOB }}-{{ checksum
             "poetry.lock" }}
 
   wait_for_postgres:

docs/admins/configuration.rst

@@ -108,6 +108,10 @@ This sample shows most configuration options
             - MIRROR-SECOND
             - MIRROR-FIRST
             - RPKI
+        source_aliases:
+            EXAMPLE:
+                - RPKI
+                - AUTHDATABASE
 
         sources:
             AUTHDATABASE:
@@ -594,6 +598,17 @@ Sources
   |br| **Default**: not defined. All sources are enabled, but results are not
   ordered by source.
   |br| **Change takes effect**: after SIGHUP, for all subsequent queries.
+* ``source_aliases``: a set of source names that are aliases to real sources.
+  Under ``source_aliases.{name}``, list the sources that should be included
+  in that alias. The alias can then be used as any other source name,
+  including listing in ``sources_default``. Ordering is preserved.
+  Aliases can not be nested. Alias names have the same requirements
+  as source names,
+  If ``rpki.roa_source`` is defined, this may also
+  include ``RPKI``, which contains pseudo-IRR objects generated from ROAs.
+  |br| **Default**: no aliases defined.
+  ordered by source.
+  |br| **Change takes effect**: after SIGHUP, for all subsequent queries.
 * ``sources.{name}``: settings for a particular source. The name must be
   all-uppercase, start with a letter, and end with a letter or digit. Valid
   characters are letters, digits and dashes. The minimum length is two

docs/releases/4.4.0.rst

@@ -75,6 +75,16 @@ consequences. See the :ref:`protected names documentation <auth-protected-names>
 for details. This only affects authoritative databases.
 
 
+Source aliases
+--------------
+Operators can now configure source aliases. A source alias can be used
+in queries, and translates to a specific set of regular sources
+configured in the same IRRD instance. These are configured under the
+``source_aliases`` setting. Returned objects are not modified - the
+alias relates only to the query. The database status under the ``!J`` whois
+and the ``databaseStatus`` GraphQL queries is extended with alias
+information.
+
 Upgrading to IRRd 4.4.0 from 4.3.x
 ----------------------------------
 TODO

docs/users/queries/graphql.rst

@@ -158,18 +158,21 @@ It returns an array of ``DatabaseStatus`` GraphQL objects, which are also
 defined in the schema::
 
     type DatabaseStatus {
-      source: String!
-      authoritative: Boolean!
-      objectClassFilter: [String!]
-      rpkiRovFilter: Boolean!
-      scopefilterEnabled: Boolean!
-      localJournalKept: Boolean!
-      serialOldestJournal: Int
-      serialNewestJournal: Int
-      serialLastExport: Int
-      serialNewestMirror: Int
-      lastUpdate: String
-      synchronisedSerials: Boolean!
+        source_type: String!
+        source: String!
+        authoritative: Boolean!
+        objectClassFilter: [String!]
+        rpkiRovFilter: Boolean!
+        scopefilterEnabled: Boolean!
+        routePreference: Int
+        localJournalKept: Boolean!
+        serialOldestJournal: Int
+        serialNewestJournal: Int
+        serialLastExport: Int
+        serialNewestMirror: Int
+        lastUpdate: String
+        synchronisedSerials: Boolean!
+        aliased_sources: [String!]
     }
 
 These are all the fields that can be queried, and their return types.
@@ -178,6 +181,7 @@ An example query that returns all current fields for all sources::
 
     query {
       databaseStatus {
+        source_type
         source
         authoritative
         objectClassFilter
@@ -190,6 +194,7 @@ An example query that returns all current fields for all sources::
         serialNewestMirror
         lastUpdate
         synchronisedSerials
+        aliased_sources
       }
     }
 
@@ -199,6 +204,7 @@ Which might return::
       "data": {
         "databaseStatus": [
           {
+            "source_type": "regular",
             "source": "NTTCOM",
             "authoritative": false,
             "objectClassFilter": null,
@@ -210,7 +216,8 @@ Which might return::
             "serialLastExport": null,
             "serialNewestMirror": 1228527,
             "lastUpdate": "2020-09-26T15:22:13.977916+00:00",
-            "synchronisedSerials": false
+            "synchronisedSerials": false,
+            "aliased_sources": null
           }
         ]
       },
@@ -244,7 +251,9 @@ Or a set of sources::
     will accept this as well. This works for all array types, i.e. those
     defined with ``[...]``.
 
-The fields have the following meaning:
+Each entry has a ``source_type`` which is either ``regular`` or ``alias``.
+Alias sources have an ``aliased_sources`` field listing the sources for which they
+are an alias. Other sources have the the following fields for each valid source:
 
 * ``source``: the name of the source
 * ``authoritative``: true if this source is authoritative in this IRRd

docs/users/queries/whois.rst

@@ -89,8 +89,10 @@ IRRd style queries
 * ``!J`` returns status information for each source. This can be used to check
   the mirroring status, which databases are authoritative, whether certain
   object classes are excluded, and various other settings.
-  The query syntax is identical to ``!j``, the output is JSON data, with the
-  following keys for each valid source:
+  The query syntax is identical to ``!j``, the output is JSON data.
+  Each entry has a ``source_type`` which is either ``regular`` or ``alias``.
+  Alias sources have an ``aliased_sources`` key listing the sources for which they
+  are an alias. Other sources have the the following keys for each valid source:
 
   * ``authoritative``: true if this source is authoritative in this IRRd
     instance, i.e. whether local changes are allowed. False if the source

irrd/conf/__init__.py

@@ -95,10 +95,15 @@ def __init__(self, user_config_path: Optional[str] = None, commit=True):
         Load the default config and load and check the user provided config.
         If a logfile was specified, direct logs there.
         """
-        from .known_keys import KNOWN_CONFIG_KEYS, KNOWN_SOURCES_KEYS
+        from .known_keys import (
+            KNOWN_CONFIG_KEYS,
+            KNOWN_FLEXIBLE_KEYS,
+            KNOWN_SOURCES_KEYS,
+        )
 
         self.known_config_keys = KNOWN_CONFIG_KEYS
         self.known_sources_keys = KNOWN_SOURCES_KEYS
+        self.known_flexible_keys = KNOWN_FLEXIBLE_KEYS
         self.user_config_path = user_config_path if user_config_path else CONFIG_PATH_DEFAULT
         default_config_path = str(Path(__file__).resolve().parents[0] / "default_config.yaml")
         with open(default_config_path) as default_config:
@@ -151,7 +156,7 @@ def get_setting_live(self, setting_name: str, default: Optional[Any] = None) ->
             components = setting_name.split(".")
             if len(components) == 3 and components[2] not in self.known_sources_keys:
                 raise ValueError(f"Unknown setting {setting_name}")
-        elif not setting_name.startswith("access_lists"):
+        elif not any([setting_name.startswith(k) for k in self.known_flexible_keys]):
             if self.known_config_keys.get(setting_name) is None:
                 raise ValueError(f"Unknown setting {setting_name}")
 
@@ -240,7 +245,7 @@ def _validate_subconfig(key, value):
                     _validate_subconfig(subkey, value2)
 
         for key, value in config.items():
-            if key in ["sources", "access_lists"]:
+            if key in ["sources"] + self.known_flexible_keys:
                 continue
             if self.known_config_keys.get(key) is None:
                 errors.append(f"Unknown setting key: {key}")
@@ -432,6 +437,20 @@ def _validate_subconfig(key, value):
                     "Read documentation carefully."
                 )
 
+        for alias_name, aliased_sources in config.get("source_aliases", {}).items():
+            if not SOURCE_NAME_RE.match(alias_name):
+                errors.append(f"Invalid source alias name: {alias_name}")
+            if alias_name in known_sources:
+                errors.append(
+                    f"Source alias name {alias_name} conflicts with an already configured real source."
+                )
+            for aliased_source in aliased_sources:
+                if aliased_source not in known_sources:
+                    errors.append(
+                        f"Source alias {alias_name} contains reference to unknown source {aliased_source}."
+                    )
+        known_sources.update(config.get("source_aliases", {}).keys())
+
         unknown_default_sources = set(config.get("sources_default", [])).difference(known_sources)
         if unknown_default_sources:
             errors.append(

irrd/conf/known_keys.py

@@ -3,8 +3,7 @@
 from irrd.rpsl.rpsl_objects import OBJECT_CLASS_MAPPING, RPSLSet
 from irrd.vendor.dotted.collection import DottedDict
 
-# Note that sources are checked separately,
-# and 'access_lists' is always permitted
+# Note that sources are checked separately
 KNOWN_CONFIG_KEYS = DottedDict(
     {
         "database_url": {},
@@ -82,6 +81,8 @@
     }
 )
 
+KNOWN_FLEXIBLE_KEYS = ["access_lists", "source_aliases"]
+
 KNOWN_SOURCES_KEYS = {
     "authoritative",
     "keep_journal",

irrd/conf/test_conf.py

@@ -95,7 +95,7 @@ def test_load_valid_reload_valid_config(self, monkeypatch, save_yaml_config, tmp
                         "bcrypt-pw": "legacy",
                     },
                 },
-                "sources_default": ["TESTDB2", "TESTDB"],
+                "sources_default": ["TESTDB2", "TESTDB", "SOURCE-ALIAS"],
                 "sources": {
                     "TESTDB": {
                         "authoritative": True,
@@ -117,6 +117,9 @@ def test_load_valid_reload_valid_config(self, monkeypatch, save_yaml_config, tmp
                     # RPKI source permitted, rpki.roa_source not set
                     "RPKI": {},
                 },
+                "source_aliases": {
+                    "SOURCE-ALIAS": ["TESTDB", "TESTDB2"],
+                },
                 "log": {"level": "DEBUG", "logfile_path": logfile},
             }
         }
@@ -127,7 +130,7 @@ def test_load_valid_reload_valid_config(self, monkeypatch, save_yaml_config, tmp
         save_yaml_config(config, run_init=False)
 
         # Unchanged, no reload performed
-        assert list(get_setting("sources_default")) == ["TESTDB2", "TESTDB"]
+        assert list(get_setting("sources_default")) == ["TESTDB2", "TESTDB", "SOURCE-ALIAS"]
 
         os.kill(os.getpid(), signal.SIGHUP)
         assert list(get_setting("sources_default")) == ["TESTDB2"]
@@ -320,6 +323,11 @@ def test_load_invalid_config(self, save_yaml_config, tmpdir):
                     "lowercase": {},
                     "invalid char": {},
                 },
+                "source_aliases": {
+                    "SOURCE-ALIAS": ["TESTDB-NOTEXIST"],
+                    "TESTDB2": ["TESTDB"],
+                    "invalid name": ["TESTDB"],
+                },
                 "log": {
                     "level": "INVALID",
                     "logging_config_path": "path",
@@ -401,6 +409,11 @@ def test_load_invalid_config(self, save_yaml_config, tmpdir):
         assert "Invalid source name: invalid char" in str(ce.value)
         assert "but rpki.notify_invalid_enabled is not set" in str(ce.value)
         assert "Setting sources_default contains unknown sources: DOESNOTEXIST-DB" in str(ce.value)
+        assert "Source alias SOURCE-ALIAS contains reference to unknown source TESTDB-NOTEXIST" in str(
+            ce.value
+        )
+        assert "Source alias name TESTDB2 conflicts with an already configured real source" in str(ce.value)
+        assert "Invalid source alias name: invalid name" in str(ce.value)
         assert "Invalid log.level: INVALID" in str(ce.value)
         assert "Setting log.logging_config_path can not be combined" in str(ce.value)
         assert "Unknown setting key: unknown_setting" in str(ce.value)

irrd/server/graphql/resolvers.py

@@ -6,7 +6,6 @@
 from graphql import GraphQLError, GraphQLResolveInfo
 from IPy import IP
 
-from irrd.conf import RPKI_IRR_PSEUDO_SOURCE, get_setting
 from irrd.routepref.status import RoutePreferenceStatus
 from irrd.rpki.status import RPKIStatus
 from irrd.rpsl.rpsl_objects import OBJECT_CLASS_MAPPING, lookup_field_names
@@ -15,7 +14,7 @@
 from irrd.storage.queries import RPSLDatabaseJournalQuery, RPSLDatabaseQuery
 from irrd.utils.text import remove_auth_hashes, snake_to_camel_case
 
-from ..query_resolver import QueryResolver
+from ..query_resolver import QueryResolver, QuerySourceManager
 from .schema_generator import SchemaGenerator
 
 """
@@ -94,15 +93,11 @@ def resolve_rpsl_objects(_, info: GraphQLResolveInfo, **kwargs):
     else:
         query.route_preference_status([RoutePreferenceStatus.visible])
 
-    all_valid_sources = set(get_setting("sources", {}).keys())
-    if get_setting("rpki.roa_source"):
-        all_valid_sources.add(RPKI_IRR_PSEUDO_SOURCE)
-    sources_default = set(get_setting("sources_default", []))
+    source_manager = QuerySourceManager()
 
     if "sources" in kwargs:
-        query.sources(kwargs["sources"])
-    elif sources_default and sources_default != all_valid_sources:
-        query.sources(list(sources_default))
+        source_manager.set_query_sources(kwargs["sources"])
+    query.sources(source_manager.sources_resolved)
 
     # All other parameters are generic lookup fields, like `members`
     for attr, value in kwargs.items():

irrd/server/graphql/schema_generator.py

@@ -65,6 +65,7 @@ def __init__(self):
             }
 
             type DatabaseStatus {
+                source_type: String!
                 source: String!
                 authoritative: Boolean!
                 objectClassFilter: [String!]
@@ -78,6 +79,7 @@ def __init__(self):
                 serialNewestMirror: Int
                 lastUpdate: String
                 synchronisedSerials: Boolean!
+                aliased_sources: [String!]
             }
 
             type RPSLJournalEntry {

irrd/server/graphql/tests/test_resolvers.py

@@ -105,6 +105,8 @@ def prepare_resolver(monkeypatch):
 
 class TestGraphQLResolvers:
     def test_resolve_rpsl_objects(self, prepare_resolver, config_override):
+        config_override({"sources": {"TEST1": {}}})
+
         info, mock_database_query, mock_query_resolver = prepare_resolver
 
         with pytest.raises(ValueError):
@@ -159,7 +161,7 @@ def test_resolve_rpsl_objects(self, prepare_resolver, config_override):
         assert info.context["sql_trace"]
 
         mock_database_query.reset_mock()
-        config_override({"sources_default": ["TEST1"]})
+        config_override({"sources_default": ["TEST1"], "sources": {"TEST1": {}}})
         result = list(
             resolvers.resolve_rpsl_objects(
                 None,

irrd/server/graphql/tests/test_schema_generator.py

@@ -41,6 +41,7 @@ def test_schema_generator():
             }
 
             type DatabaseStatus {
+                source_type: String!
                 source: String!
                 authoritative: Boolean!
                 objectClassFilter: [String!]
@@ -54,6 +55,7 @@ def test_schema_generator():
                 serialNewestMirror: Int
                 lastUpdate: String
                 synchronisedSerials: Boolean!
+                aliased_sources: [String!]
             }
 
             type RPSLJournalEntry {