VIP 27: Move some VCL variables to the catflap facility

Synopsis

VCL currently has two "lookup modifiers" with the variables req.hash_always_miss and req.hash_ignore_busy. The goal behind this VIP is to evaluate the relevance of turning them into catflap consumers.

Why?

The VCF facility is currently undocumented on purpose and only has test coverage via vmod-debug. Turning those two variables into catflap functions would remove special cases from the core lookup code and help work out actual needs when it comes to tweaking cache lookups and how to compose tweaks. Shipping catflaps in tree might also encourage third-party catflaps once we reach a stabilization point.

How?

There is no clear "how" currently. The existing 4.0 and 4.1 VCL syntax will need to maintain the existing variables, but we could imagine shipping a vmod-lookup or something similar to offer built-in VCFs.

In order to compose VCFs, we need to update the current API so that more than one can be registered. We might consider a registration similar to beresp.filters:

sub vcl_recv {
    if (client.ip ~ varnish_neighbors) {
        set req.lookup = "ignore_busy";
    } else if (req.method == "REFRESH") {
        set req.method = "GET";
        set req.lookup = "always_miss";
    } else {
        # chaining example
        set req.lookup = "always_miss ignore_busy";
    }
}

PR #2858 which preceded the current implementation would have addressed these requirements by the option for catflap chaining in the catflap code itself. So with that implementation, VCL could look like

sub vcl_recv {
    # catflap chaining
    lookup.always_miss();
    lookup.ignore_busy();
}

with each registering the previous callback and priv pointer, calling them unless it decided that it needed to provide the final result.

Note: there may be other aspects to consider which were not thought of when the catflap code was proposed.

A new lookup modifier can be considered too: some kind of hash_ignore_bgfetch that ignores a busy object only when an eligible stale object is also available.

sub vcl_recv {
    if (req.http.host ~ "very slow backend") {
        set req.lookup = "ignore_bgfetch";
    }
}