Garbage collect caches that aren't needed any more

It's possible that due to pipelines disappearing, or being updated,
some caches will no longer be needed. If these are shut down, the
number of caches will only grow, which constitutes a leak of resources
(though not necessarily a serious one, since it will max out at
`clusters x types`).

To be able to shut down caches that are no longer needed, we need to
be able to do a few things:

 1. detect when they aren't needed
 2. stop them running when not needed
 3. stop them when the controller is shutting down

To do the first, I index the cache keys used by each pipeline. The
garbage collector regularly checks to see if each cache has entries in
the index; and if not, it's not used by any pipeline and can be shut
down.

To keep track of caches to consider for collection, the GC uses a
rate-limiting work queue. When the cache is created, it's put on the
queue; and each time it's considered and is still needed, it's
requeued with a longer retry, up to about eight minutes. This avoids
the question of finding an appropriate event to hook into, with the
downside of being a bit eventual.

The second and third things can be arranged by deriving contexts from
the manager's context. I have introduced `runner` (in runner.go) which
can be Start()ed by the manager and thus gain access to its context,
and which can then construct a context for each cache. Each cache gets
its own cancel func that can be used to shut it down, but will also be
shut down by the manager when it's shutting down itself.

Signed-off-by: Michael Bridgen <[email protected]>

config/testdata/crds/fake_application.yaml

@@ -0,0 +1,134 @@
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: fakes.test.weave.works
+spec:
+  conversion:
+    strategy: None
+  group: test.weave.works
+  names:
+    kind: Fake
+    listKind: FakeList
+    plural: fakes
+    singular: fake
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: Fake is a fake type, for using in tests
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          status:
+            description: FakeStatus defines the observed state of a Fake
+            properties:
+              lastAppliedRevision:
+                description: The last successfully applied revision. Equals the Revision
+                  of the applied Artifact from the referenced Source.
+                type: string
+
+              conditions:
+                description: Conditions holds the conditions for the Cluster.
+                items:
+                  description: "Condition contains details for one aspect of the current
+                    state of this API Resource. --- This struct is intended for direct
+                    use as an array at the field path .status.conditions.  For example,
+                    type FooStatus struct{ // Represents the observations of a foo's
+                    current state. // Known .status.conditions.type are: \"Available\",
+                    \"Progressing\", and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge
+                    // +listType=map // +listMapKey=type Conditions []metav1.Condition
+                    `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\"
+                    protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields }"
+                  properties:
+                    lastTransitionTime:
+                      description: lastTransitionTime is the last time the condition
+                        transitioned from one status to another. This should be when
+                        the underlying condition changed.  If that is not known, then
+                        using the time when the API field changed is acceptable.
+                      format: date-time
+                      type: string
+                    message:
+                      description: message is a human readable message indicating
+                        details about the transition. This may be an empty string.
+                      maxLength: 32768
+                      type: string
+                    observedGeneration:
+                      description: observedGeneration represents the .metadata.generation
+                        that the condition was set based upon. For instance, if .metadata.generation
+                        is currently 12, but the .status.conditions[x].observedGeneration
+                        is 9, the condition is out of date with respect to the current
+                        state of the instance.
+                      format: int64
+                      minimum: 0
+                      type: integer
+                    reason:
+                      description: reason contains a programmatic identifier indicating
+                        the reason for the condition's last transition. Producers
+                        of specific condition types may define expected values and
+                        meanings for this field, and whether the values are considered
+                        a guaranteed API. The value should be a CamelCase string.
+                        This field may not be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    status:
+                      description: status of the condition, one of True, False, Unknown.
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: type of condition in CamelCase or in foo.example.com/CamelCase.
+                        --- Many .condition.type values are consistent across resources
+                        like Available, but because arbitrary conditions can be useful
+                        (see .node.status.conditions), the ability to deconflict is
+                        important. The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                  required:
+                  - lastTransitionTime
+                  - message
+                  - reason
+                  - status
+                  - type
+                  type: object
+                type: array
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: GitopsCluster
+    listKind: GitopsClusterList
+    plural: gitopsclusters
+    singular: gitopscluster
+  conditions:
+  - lastTransitionTime: "2022-08-05T17:42:36Z"
+    message: no conflicts found
+    reason: NoConflicts
+    status: "True"
+    type: NamesAccepted
+  - lastTransitionTime: "2022-08-05T17:42:38Z"
+    message: the initial names have been accepted
+    reason: InitialNamesAccepted
+    status: "True"
+    type: Established
+  storedVersions:
+  - v1alpha1