Enable auto-generation of Lotman policies

This gives caches the ability to inform Lotman of known namespaces, which
Lotman can use to automatically create a (currently toothless) purge policy.

A lot of the work here involved figuring out how to specify named lot policies
through our yaml config. Examples of what I came up with can be found in
docs/parameters.yaml. Outside of that, some changes involved figuring out how
to merge local config with potentially-discovered config.

Until Pelican actually uses the new purge code with the Lotman purge plugin,
none of this really does much other than set up a database. For now, this lets
us set up most of the lot config in a cache through Pelican, and then the last
bits for testing purge stuff need to be configured manually. Until then, the
actual purge ordering portion of a policy isn't plumbed into anything -- eventually
it will modify XRootD config directly.

Finally, I still haven't worked out completely how to manage expiration/deletion of
lots. I'll save that for another PR, potentially in the next release cycle.

config/resources/defaults.yaml

@@ -67,6 +67,16 @@ Cache:
   LowWatermark: 90
   HighWaterMark: 95
   BlocksToPrefetch: 0
+Lotman:
+  EnabledPolicy: "fairshare"
+  DefaultLotExpirationLifetime: "2016h"
+  DefaultLotDeletionLifetime: "4032h"
+  PolicyDefinitions:
+    - PolicyName: "fairshare"
+      DivideUnallocated: true
+      PurgeOrder: ["del", "exp", "opp", "ded"]
+      DiscoverPrefixes: true
+      MergeLocalWithDiscovered: false
 LocalCache:
   HighWaterMarkPercentage: 95
   LowWaterMarkPercentage: 85

docs/parameters.yaml

@@ -2692,34 +2692,126 @@ type: bool
 default: false
 components: ["cache"]
 ---
-name: Lotman.Lots
-description: |+
-  Declarative configuration for LotMan.  This is a list of objects, each of which describes a "lot". Every lot
-  can be defined with the following:
-
-  - `LotName`: REQUIRED. The name of the lot.  This is used to identify the lot in the LotMan database.
-  - `Owner`: REQUIRED. A string identifying the owner of the lot's data (as opposed to someone who can modify the lot itself).
-    The Owner field should generally be set to the issue for the lot's namespace path. For example, if the lot
-    tracks namespace `/foo/bar`, the owner might be set to `https://registry.com/api/v1.0/registry/foo/bar`.
-  - `Paths`: OPTIONAL. A list of path objects, each of which describes a path that should be managed by the lot.
-      - `Path`: REQUIRED. The path to be managed by the lot.
-      - `Recursive`: REQUIRED. A boolean indicating whether the path should be managed recursively. If true, the lot will
-        manage all files and directories under the specified path.
-  - `ManagementPolicyAttrs`: REQUIRED. The lot's management policy attributes object. This contains information about resources the lot should
-    be allocated, and how it should be managed.
-      - `DedicatedGB`: REQUIRED. The amount of storage, in GB, that should be dedicated to the lot. This means the lot can assume it
-        always has access to this quantity.
-      - `OpportunisticGB`: REQUIRED. The amount of opportunistic storage, in GB, the lot should have access to, when storage is available.
-      - `MaxNumObjects`: REQUIRED. The maximum number of objects a lot is allowed to store.
-      - `CreationTime`: REQUIRED. A unix timestamp indicating when the lot should begin being considered valid. Times in the future indicate
-        the lot should not be considered valid until that time.
-      - `ExpirationTime`: REQUIRED. A unix timestamp indicating when the lot expires. Lots may continue to function after expiration, but lot
-        data owners should recognize the storage is at-will and may be pre-empted at any time.
-      - `DeletionTime`: REQUIRED. A unix timestamp indicating when the lot and its associated data should be deleted.
-
-  Note that example configurations can be found in lotman/resources/lots-config.yaml
+name: Lotman.PolicyDefinitions
+description: |+
+  A list of named Lotman purge policy definitions that may be enabled by the cache administrator through setting the `Lotman.EnabledPolicy`
+  configuration. Each policy definition is an object with the following fields:
+  - `PolicyName`: The name of the policy. This is used to identify the policy in the `Lotman.EnabledPolicy` configuration.
+  - `PurgeOrder`: An ordered list of strings indicating the order in which lots should be purged. The strings should be one of the following:
+    - `del`: Purge lots that have passed their deletion time.
+    - `exp`: Purge lots that have passed their expiration time.
+    - `opp`: Purge lots that have passed their opportunistic storage quota.
+    - `ded`: Purge lots that have passed their dedicated storage quota.
+  - `DiscoverPrefixes`: A boolean indicating whether Lotman should automatically discover prefixes from the Director. If true, Lotman will
+    attempt to create lots for all discovered federation prefixes. Locally-defined lots will take precedence over discovered lots if the two have
+    the same name.
+  - `MergeLocalWithDiscovered`: A boolean indicating whether Lotman should merge locally-defined lot configurations with discovered namespaces.
+    Most Lot configuration fields will take precedence from local configuration, but the `Paths` and `Parents` fields are additive.
+  - `DivideUnallocated`: A boolean indicating whether Lotman should attempt to make intelligent decisions regarding management policy attributes
+    for lots that have not provided explicit values. These decisions are based on the cache's total storage capacity and the number of lots
+    that have been explicitly configured, and are intended to maximize potential cache utilization. This should be set to "true" in most cases.
+  - `Lots`: A list of lot objects, each of which describes a "lot". Every lot can be defined with the following:
+    - `LotName`: REQUIRED. The name of the lot.  This is used to identify the lot in the LotMan database.
+    - `Owner`: REQUIRED. A string identifying the owner of the lot's data (as opposed to someone who can modify the lot itself).
+      The Owner field should generally be set to the issue for the lot's namespace path. For example, if the lot
+      tracks namespace `/foo/bar`, the owner might be set to `https://registry.com/api/v1.0/registry/foo/bar`.
+    - `Paths`: OPTIONAL. A list of path objects, each of which describes a path that should be managed by the lot.
+        - `Path`: REQUIRED. The path to be managed by the lot.
+        - `Recursive`: REQUIRED. A boolean indicating whether the path should be managed recursively. If true, the lot will
+          manage all files and directories under the specified path.
+    - `ManagementPolicyAttrs`: REQUIRED. The lot's management policy attributes object. This contains information about resources the lot should
+      be allocated, and how it should be managed.
+        - `DedicatedGB`: REQUIRED. The amount of storage, in GB, that should be dedicated to the lot. This means the lot can assume it
+          always has access to this quantity.
+        - `OpportunisticGB`: REQUIRED. The amount of opportunistic storage, in GB, the lot should have access to, when storage is available.
+        - `MaxNumObjects`: REQUIRED. The maximum number of objects a lot is allowed to store.
+        - `CreationTime`: REQUIRED. A unix timestamp indicating when the lot should begin being considered valid. Times in the future indicate
+          the lot should not be considered valid until that time.
+        - `ExpirationTime`: REQUIRED. A unix timestamp indicating when the lot expires. Lots may continue to function after expiration, but lot
+          data owners should recognize the storage is at-will and may be pre-empted at any time.
+        - `DeletionTime`: REQUIRED. A unix timestamp indicating when the lot and its associated data should be deleted.
+
+  For example, Lotman could be configured with the "my-policy" policy with the following:
+  ```yaml
+  Lotman:
+    EnabledPolicy: "my-policy"
+    PolicyDefinitions:
+    - PolicyName: "my-policy"
+      DivideUnallocated: true
+      PurgeOrder: ["del", "exp", "opp", "ded"]
+      DiscoverPrefixes: true
+      MergeLocalWithDiscovered: true
+      Lots:
+      - LotName: "/foo/bar"
+        Owner: "https://registry.com/api/v1.0/registry/foo/bar"
+        Paths:
+          Path: "/foo/bar"
+          Recursive: true
+        ManagementPolicyAttrs:
+          DedicatedGB: 100
+          OpportunisticGB: 100
+          MaxNumObjects: 1000
+          CreationTime: 1614556800
+          ExpirationTime: 1614556800
+          DeletionTime: 1614556800
+      - LotName ... <additional lots>
+  ```
+
+  Additional example configurations can be found in lotman/resources/lots-config.yaml
   For more information about LotMan configuration, see:
   [https://github.com/pelicanplatform/lotman](https://github.com/pelicanplatform/lotman)
 type: object
 default: none
 components: ["cache"]
+---
+name: Lotman.EnabledPolicy
+description: |+
+  The name of the policy to use with Lotman's purge logic. Policy names are defined in the Lotman.PolicyDefinitions list object.
+  If unset, the "fairshare" policy is used, which evenly divides the cache's space amongst all top-level namespaces discoverable
+  through the Director and purges data according in order of lots past deletion, lots past expiration, lots past opportunistic
+  storage, and lots past dedicated storage. The "fairshare" policy is defined as follows:
+  ```yaml
+  Lotman:
+    EnabledPolicy: "fairshare"
+    DefaultLotExpirationLifetime: "2016h"
+    DefaultLotDeletionLifetime: "4032h"
+    PolicyDefinitions:
+    - PolicyName: "fairshare"
+      DivideUnallocated: true
+      PurgeOrder: ["del", "exp", "opp", "ded"]
+      DiscoverPrefixes: true
+      MergeLocalWithDiscovered: false
+  ```
+type: string
+default: "fairshare"
+components: ["cache"]
+---
+name: Lotman.DefaultLotExpirationLifetime
+description: |+
+  The default expiration lifetime for lots that have not provided an explicit expiration time. Valid time units are:
+    - ns for nanoseconds
+    - us (or µs) for microseconds
+    - ms for milliseconds
+    - s for seconds
+    - m for minutes
+    - h for hours
+
+  This value is fed to Lotman as a unix timestamp in microseconds, adjusted from the current time.
+type: duration
+default: 2016h
+components: ["cache"]
+---
+name: Lotman.DefaultLotDeletionLifetime
+description: |+
+  The default deletion lifetime for lots that have not provided an explicit deletion time. Valid time units are:
+    - ns for nanoseconds
+    - us (or µs) for microseconds
+    - ms for milliseconds
+    - s for seconds
+    - m for minutes
+    - h for hours
+
+  This value is fed to Lotman as a unix timestamp in microseconds, adjusted from the current time.
+type: duration
+default: 4032h
+components: ["cache"]

launchers/cache_serve.go

@@ -74,8 +74,16 @@ func CacheServe(ctx context.Context, engine *gin.Engine, egrp *errgroup.Group, m
 			log.Debugln("Registering Lotman API")
 			lotman.RegisterLotman(ctx, engine.Group("/"))
 		}
-		// Bind the c library funcs to Go
-		if success := lotman.InitLotman(); !success {
+
+		// Until https://github.com/PelicanPlatform/lotman/issues/24 is closed, we can only really logic over
+		// top-level prefixes because enumerating all object "directories" under a given federation prefix is
+		// infeasible, but is currently the only way to nest namespaces in Lotman such that a sub namespace
+		// can be assosciated with a top-level prefix.
+		// To that end, we need to filter out any nested namespaces from the cache server's namespace ads.
+		uniqueTopPrefixes := server_utils.FilterTopLevelPrefixes(cacheServer.GetNamespaceAds())
+
+		// Bind the c library funcs to Go, instantiate lots, set up the Lotman database, etc
+		if success := lotman.InitLotman(uniqueTopPrefixes); !success {
 			return nil, errors.New("Failed to initialize lotman")
 		}
 	}

lotman/lotman.go

@@ -29,13 +29,15 @@ import (
 
 	"github.com/gin-gonic/gin"
 	log "github.com/sirupsen/logrus"
+
+	"github.com/pelicanplatform/pelican/server_structs"
 )
 
 func RegisterLotman(ctx context.Context, router *gin.RouterGroup) {
 	log.Warningln("LotMan is not supported on this platform. Skipping...")
 }
 
-func InitLotman() bool {
+func InitLotman(adsFromFed []server_structs.NamespaceAdV2) bool {
 	log.Warningln("LotMan is not supported on this platform. Skipping...")
 	return false
 }